Power centralized control center data verification and backup method and device

Abstract

The invention provides a power centralized control center data verification and backup method and device, and the method comprises the steps: obtaining hardware resource information corresponding to an operating system, and constructing a thread resource pool; the method comprises the following steps: connecting data sources of an electric power centralized control center, classifying and rating the data sources, and carrying out real-time demand analysis by adopting a parameterized mixed time algorithm to obtain a data task queue with a priority mark; monitoring the load state of an operating system and the length of a task queue, performing resource evaluation by applying an online distributed queue length estimation algorithm, dynamically creating a working thread and allocating resources; based on a thread resource allocation scheme, executing parallel data verification by adopting a sub-exponential algorithm on an independent set; and evaluating a backup demand according to the data verification result set, determining a backup strategy, and executing parallel data backup. According to the method, the characteristics of a domestic operating system are fully utilized, efficient verification and backup of the data of the electric power centralized control center are achieved, and the system resource utilization rate and the data safety are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of power system automation, and in particular to a data verification and backup method and device for a power centralized control center. Background Art

[0002] Data verification and backup technology for power control centers is a crucial component of power system automation and information security, crucial for the safe and stable operation of power systems and data security. As power systems continue to expand and become more intelligent, the amount of data processed by power control centers is growing exponentially, placing higher demands on data verification and backup efficiency.

[0003] Currently, power control centers generally use scheduled tasks or single-threaded processing modes for data verification and backup. For example, they employ full backup strategies within fixed time windows or data integrity verification methods based on simple checksums. These technologies perform well when processing small amounts of data, but are less efficient when dealing with massive amounts of power data.

[0004] An existing optimization technique uses a priority-based task scheduling mechanism to allocate processing resources based on data importance and combines it with an incremental backup strategy to reduce redundant data transmission. This technique uses a single scheduler to manage all verification and backup tasks, incorporating the concepts of task priorities and resource quotas into the scheduling logic. However, it fails to fully utilize the characteristics of modern multi-core processor architectures and is not sufficiently compatible with domestic operating systems.

[0005] However, this technology has obvious shortcomings: first, a single scheduler can easily become a performance bottleneck, especially under high load conditions; second, the resource allocation strategy is too static and cannot be dynamically adjusted according to the real-time load of the system; third, it fails to fully utilize the unique resource management interface provided by domestic operating systems, resulting in poor performance on domestic platforms; finally, the lack of an effective distributed queue length estimation mechanism makes it difficult to achieve globally optimal task allocation. Summary of the Invention

[0006] The purpose of the present invention is to provide a data verification and backup method and device for a power centralized control center, aiming to solve one or more of the technical problems in the prior art, such as a single scheduler becoming a performance bottleneck, resource allocation strategies being too static, failure to fully utilize the characteristics of domestic operating systems, and lack of an effective distributed queue length estimation mechanism.

[0007] To achieve the above objectives, the present invention provides a data verification and backup method for a power centralized control center, comprising the following steps: Obtaining hardware resource information corresponding to the operating system, calculating effective resource parameters based on the hardware resource information, building a thread resource pool, and obtaining a system resource capability assessment report; Connect to the data source of the power centralized control center, classify the data type and importance of the data source, use a parameterized hybrid time algorithm to perform real-time demand analysis, and obtain a data task queue with priority tags; Monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to perform resource evaluation, dynamically create worker threads based on the load threshold of the operating system and the backlog level of the task queue, and allocate CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; Based on the thread resource allocation scheme, the verification task is assigned to the working thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are summarized to obtain a data verification result set; Evaluate backup requirements based on the data verification result set, determine backup strategies, execute parallel data backup, and obtain data backup execution status and results.

[0008] The present invention also provides a data verification and backup device for a power centralized control center, comprising: A system resource evaluation module is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, build a thread resource pool, and obtain a system resource capacity evaluation report; A data prioritization module, connected to the system resource assessment module, is used to connect to the data source of the power centralized control center, classify the data type and importance of the data source, use a parameterized hybrid time algorithm to perform real-time demand analysis, and obtain a data task queue with priority marking; a thread resource allocation module, connected to the data prioritization module, for monitoring the load status of the operating system and the length of the task queue, applying an online distributed queue length estimation algorithm to perform resource evaluation, dynamically creating worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocating CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; a data verification module, connected to the thread resource allocation module, configured to allocate verification tasks to working threads based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on independent sets, and aggregate verification results of each thread to obtain a data verification result set; The data backup module is connected to the data verification module and is used to evaluate the backup requirements according to the data verification result set, determine the backup strategy, perform parallel data backup, and obtain the data backup execution status and results.

[0009] The beneficial effects of the present invention are: On the one hand, by performing parallel data verification through a sub-exponential algorithm on independent sets, combined with priority-tagged task queues and dynamic thread resource allocation, the verification efficiency and backup accuracy of the power control center data are significantly improved. On the other hand, the online distributed queue length estimation algorithm is applied to dynamically evaluate system resources, achieving optimal allocation of CPU time slices, memory quotas, and I / O bandwidth, effectively solving the resource competition problem in multi-threaded parallel processing. Therefore, the present invention fully utilizes the characteristics of domestic operating systems to achieve efficient verification and backup of power control center data, improving system resource utilization and data security. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0011] Figure 1 Flowchart of the data verification and backup method for a power centralized control center provided by an embodiment of the present invention; Figure 2 A flowchart of data classification and task prioritization provided by an embodiment of the present invention; Figure 3 This is a structural diagram of the data verification and backup device for the power centralized control center provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0012] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0013] like Figure 1 As shown, an embodiment of the present invention provides a data verification and backup method for a power centralized control center, comprising the following steps: Step S1: Obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, build a thread resource pool, and obtain a system resource capability assessment report; During the implementation of the present invention, the system initialization phase first requires acquiring and evaluating available hardware resources. When booting based on a domestic operating system, hardware resource information is obtained by calling underlying APIs, including the GetSystemInfo() function and the sysconf() function, as well as by reading relevant files in the / proc virtual file system. Through these APIs, key hardware parameters such as the number of processor cores, the main frequency of each core, the total physical memory capacity, available memory space, disk I / O bandwidth, and network interface bandwidth can be accurately obtained. After obtaining these raw hardware resource parameters, the actual amount of resources available for data verification and backup tasks can be calculated based on the resource usage of other processes in the current operating environment. For example, by analyzing indicators such as the current CPU average load, memory utilization, and I / O wait time, available CPU time slices, allocatable memory space, and available I / O bandwidth can be determined. Based on these effective resource parameters, a thread pool manager is pre-created, the minimum and maximum thread count thresholds for the thread pool are set, and inter-thread communication mechanisms and synchronization primitives such as mutexes, condition variables, and semaphores are initialized. Finally, the resource evaluation algorithm is used to calculate the maximum number of verification and backup tasks that can be processed concurrently under the current hardware conditions, and a system resource capacity evaluation report is generated to provide a decision-making basis for subsequent task scheduling and resource allocation.

[0014] Step S2: Connecting to the data source of the power centralized control center, classifying the data type and importance rating of the data source, using a parameterized hybrid time algorithm to perform real-time demand analysis, and obtaining a data task queue with priority marking; After obtaining the system resource capacity assessment report, the system needs to connect to various data sources within the power control center and classify and rank these data. First, connect to various data sources, such as the SCADA real-time database, historical database, configuration information repository, and waveform recording file system, using configured database connection parameters, API interfaces, or file system paths. For each data source, metadata such as data structure description, update frequency, and access patterns is obtained. Subsequently, data is categorized across multiple dimensions based on pre-set classification rules, including data type (real-time, historical, configuration data), data structure (structured, semi-structured, unstructured), update frequency (high, medium, low), and storage location. After data classification, the importance of each data type is ranked using the sub-exponential algorithm on independent sets. This algorithm first constructs a data dependency graph, analyzes the citation and reference relationships between data points, and calculates graph-theoretic metrics such as in-degree, out-degree, and centrality for each data node. Key data nodes are then identified. The ranking process also considers factors such as the business impact of the data, regulatory requirements, data recovery difficulty, and historical query frequency. A parameterized hybrid time algorithm is also used to analyze the timeliness requirements of various data types. Based on three core parameters: time window, data decay rate, and business time sensitivity, this algorithm establishes a timeliness model for each data type and calculates the maximum allowable delay. Finally, a weighted algorithm combines data importance and timeliness requirements to derive a comprehensive priority index for each data task. Based on this index, a multi-level task queue is constructed, outputting a priority-tagged data task queue.

[0015] The various data sources in the power control center include, but are not limited to, SCADA real-time databases, historical databases, configuration information repositories, and waveform recording file systems. First, the configuration database interface is called to obtain a list of all registered data sources in the control center, along with metadata such as connection parameters, data structure descriptions, access permissions, and update frequency. These data sources are classified into multiple dimensions based on preset classification rules: data type is divided into real-time data streams, static configuration data, periodically collected data, and event-triggered data; data structure is divided into structured data, semi-structured data, and unstructured data; data update frequency is divided into high-frequency update data, medium-frequency update data, and low-frequency update data; and data storage location is divided into primary system storage, backup system storage, and remote site storage. For newly discovered unclassified data sources, machine learning algorithms can be applied for feature analysis, attempting to classify them into existing categories or create new ones.

[0016] The parameterized hybrid time algorithm combines deterministic and stochastic time analysis to more accurately describe the timeliness characteristics of power system data. Based on three core parameters: time window, data decay rate, and service time sensitivity, the algorithm generates a timeliness model through hybrid analysis. The mathematical model constructed by the algorithm can be expressed as: E(t) = α·W(t) + β·D(t) + γ·S(t) Where E(t) is the timeliness score of the data at time t, W(t) is the time window function, D(t) is the data decay function, S(t) is the service sensitivity function, α, β, and γ are weight coefficients, and α+β+γ=1.

[0017] Different timeliness evaluation criteria can be selected for different types of data: for SCADA real-time data, strict time window constraints (usually seconds) are adopted; for metering data, corresponding time windows (minutes to hours) are set according to their collection cycle; for configuration data, attention is paid to the synchronization timeliness requirements after data changes; for historical data and statistical data, the main consideration is the time constraints of their business query needs.

[0018] Step S3: monitoring the load status and task queue length of the operating system, applying an online distributed queue length estimation algorithm to perform resource evaluation, dynamically creating worker threads based on the load threshold of the operating system and the backlog level of the task queue, and allocating CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; After obtaining a priority-tagged data task queue, thread resource allocation needs to be dynamically adjusted based on the current operating system load and task queue status. By periodically calling the operating system monitoring API, key performance indicators such as CPU utilization, memory usage, and I / O wait time are obtained to calculate the current system load level. Simultaneously, an online distributed queue length estimation algorithm is applied to analyze the current length and backlog of task queues at each priority level. Based on historical data and current trends, it predicts future task growth trends. This algorithm accurately estimates queue lengths for each node in a distributed environment, avoiding the global synchronization overhead required by traditional methods. Based on the system load and task queue length estimates, the expected task wait time and remaining system resources are calculated. The system then determines whether and how many new worker threads should be created based on a pre-set thread creation policy. Thread creation is triggered when a high-priority task queue backlog is detected or when task wait times exceed a threshold. When the system load approaches a limit or the task queue length decreases significantly, the number of active threads may be reduced. Verification and backup threads are dynamically created by calling thread management interfaces provided by domestic operating systems, such as pthread_create() or specific thread pool management APIs. Threads of different priorities are allocated different CPU time slices, memory quotas, and I / O bandwidth through resource control mechanisms provided by the operating system, such as control groups, process / thread priority settings (nice value adjustment), and CPU affinity settings. High-priority threads receive more resource quotas and higher execution priority, ensuring that critical tasks can be processed promptly. Finally, an optimized thread resource allocation plan is output, including information such as each thread's ID, priority, resource quota, and task allocation.

[0019] The online distributed queue length estimation algorithm accurately estimates the current length and growth trend of each priority task queue through local observation and statistical modeling without the need for global synchronization. The algorithm is based on random sampling and Markov modeling, and its core steps include: 1. Randomly sample the local queue length at each node; 2. Build a time series model of queue length; 3. Apply Bayesian inference to estimate the global queue state; 4. Combine historical data to predict future cohort growth trends.

[0020] The complexity of the algorithm is O(k·log n), where k is the number of sampling points and n is the queue length, which is significantly better than the O(n) method that requires global synchronization.

[0021] This algorithm is used to estimate the backlog of task queues at each priority level and predict task growth trends over a period of time (e.g., the next 30 minutes to 2 hours). Business model factors, such as peak and trough periods in the power system and scheduled maintenance periods, are taken into account to adjust the prediction model and ensure rational resource allocation.

[0022] Step S4: Based on the thread resource allocation scheme, the verification task is assigned to the working thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are summarized to obtain a data verification result set; After obtaining the optimized thread resource allocation plan, parallel data verification begins. First, verification tasks are assigned to corresponding worker threads based on task priority and thread resource availability, ensuring that high-priority data is processed by high-priority threads. A task-thread mapping table can be maintained, recording the worker thread to which each verification task is assigned, as well as information such as the task's status, start time, and expected completion time. For each data verification task, an appropriate verification algorithm is selected based on the data type and importance. For example, a lightweight CRC32 algorithm can be used for real-time SCADA data, high-strength hashing algorithms such as SHA-256 can be used for critical configuration data, and block hashing and tree hashing structures can be used for large historical data files. When processing large data sets, a sub-exponential algorithm on independent sets is applied for optimization. This algorithm first analyzes the logical structure and dependencies of the data, decomposing the large dataset into multiple logical subsets. It then identifies independent sets within these subsets—sets of data blocks that have no dependencies and can be processed in parallel. Based on these identified independent sets, verification tasks are reorganized, splitting the original large verification task into multiple smaller subtasks that can be executed in parallel without interfering with each other. Streaming and block processing techniques can also be used to control the amount of data loaded into memory each time and reduce memory requirements. Each worker thread executes the assigned verification task in parallel and writes the verification results to a shared result pool. To avoid excessive memory pressure, an intermediate result caching and persistence mechanism can be used. When the intermediate verification results reach a certain size, they will be written to temporary storage. After all subtasks are completed, result aggregation processing is initiated, using a hierarchical merging strategy: first merge within the thread group, then merge across thread groups, and finally the main thread performs the final merge. The merging process is not just a simple result splicing, but also includes consistency checks to ensure that the data reference relationships across independent sets are correct. Ultimately, a data verification result set is generated, which contains information such as overall verification statistics, detailed verification records, problem lists, and performance indicators.

[0023] During verification task allocation, the thread resource allocation plan is first analyzed to determine the number of threads available for data verification, each thread's priority, and resource quota. A "priority matching" strategy is implemented to assign high-priority tasks to high-priority threads, ensuring that critical data verification tasks receive sufficient computing resources and execution priority. Furthermore, the system considers the current load of each thread to prevent overloading some threads while leaving others idle. Load balance is maintained across threads by calculating metrics such as the length of each thread's current task queue and the estimated remaining time for the task being processed. The system also considers the physical storage location of the data and the CPU core location of the processing thread to optimize data locality and reduce data transmission latency.

[0024] Step S5: Evaluate the backup requirements based on the data verification result set, determine the backup strategy, perform parallel data backup, and obtain the data backup execution status and results.

[0025] After obtaining the data verification results, the intelligent backup strategy execution phase begins. First, the data verification results are analyzed for information such as data integrity, last backup time, and data change rate. Combined with the data's importance level, the backup priority and backup method for each data set are assessed. Data identified as problematic during verification, data that has not been backed up for a long time, or critical data with a high change rate can be given higher backup priority. Based on the backup needs assessment results, a parameterized hybrid time algorithm is applied to calculate the optimal backup window. This algorithm considers the timeliness of the data, system load fluctuations, and storage resource availability to determine the optimal backup execution time for different data types. Furthermore, a full or incremental backup strategy is determined based on data change. For data with a high change rate, a full backup may be chosen; for data with a low change rate but large data volume, incremental backup is preferred to reduce network transmission and storage overhead. Backup parameters such as the compression level, encryption requirements, and storage location are also determined to generate a detailed backup strategy execution plan. During the backup execution phase, backup tasks are assigned to the appropriate backup threads based on the backup strategy execution plan and thread resource allocation scheme. Establish a backup progress monitoring mechanism to track the execution status of each backup task in real time, including the amount of data processed, the current backup rate, and the estimated completion time. Each backup thread performs data compression, encryption, and transmission operations in parallel to back up the data to the designated storage location. During the backup process, detect and handle various abnormal situations, such as data transmission interruption, insufficient storage space, or data verification failure. For recoverable abnormalities, automatically start the retry process. After the backup is completed, perform a verification check on the backup data to ensure the integrity and recoverability of the backup data, and record metadata information such as the backup completion time, backup content summary, storage location, and verification value. Finally, output the data backup execution status and result report to provide a basis for subsequent data recovery and auditing.

[0026] The following embodiments of the present invention illustrate specific implementations of the above steps.

[0027] In step S1, hardware resource information corresponding to the operating system is obtained, and effective resource parameters are evaluated based on the hardware resource information, including: Step S1.1: Obtain the number of processor cores, memory capacity, and I / O bandwidth information through the system API to obtain the original hardware resource parameter set; Step S1.2: Based on the original hardware resource parameter set, read the current process occupancy data, calculate the allocatable CPU time slice, available memory space and I / O bandwidth, and obtain the effective resource parameter set; Step S1.3: Based on the effective resource parameter set, set the minimum and maximum thresholds of the thread pool, initialize the thread communication mechanism, and obtain the initial state of the thread resource pool; Step S1.4: Calculate the number of concurrently processable verification and backup tasks based on the initial state of the thread resource pool and the effective resource parameter set, and output a system resource capability assessment report.

[0028] In step S1.1, hardware resource information is obtained by calling the API provided by the domestic operating system. Specifically, the GetSystemInfo() function or the sysconf() function is used to obtain CPU-related information, such as the number of processor cores, the main frequency of each core, and the cache size. The total physical memory capacity and currently available memory are obtained through the memory management API. Disk I / O and network interface bandwidth information is obtained by reading files such as / proc / diskstats and / proc / net / dev in the / proc virtual file system. The presence and performance parameters of specific hardware accelerators (such as encryption chips and compression cards) are detected. These raw hardware resource parameters are integrated into a structured parameter set, providing basic data for subsequent resource evaluation.

[0029] In step S1.2, based on the original hardware resource parameter set obtained and combined with the current system operating status, the actual amount of resources that can be allocated to the data verification and backup tasks is calculated. By reading files such as / proc / stat and / proc / meminfo, indicators such as the current CPU usage, memory occupancy, and I / O wait time are obtained, and the resource usage of other processes in the system is analyzed. Considering that the power control center usually also runs critical business software such as SCADA systems and historical databases, a certain proportion of resources can be reserved for these core businesses to ensure that data verification and backup tasks do not affect normal business operations. After calculation, the percentage of allocable CPU time slices, the amount of available memory space, and the available I / O bandwidth are obtained to form an effective resource parameter set.

[0030] In step S1.3, the thread resource pool is configured and initialized based on the valid resource parameter set. The minimum and maximum thread count thresholds for the thread pool are set based on the number of available CPU cores and the load. The minimum number of threads ensures that the system always maintains a certain level of parallel processing capability, while the maximum number of threads prevents the creation of too many threads, which would result in excessive thread switching overhead. Inter-thread communication mechanisms can also be initialized, including synchronization primitives such as mutexes, conditional variables, semaphores, and message queues, to provide the infrastructure for subsequent parallel task processing. In addition, parameters such as thread stack size, thread priority policy, and thread affinity can be configured to optimize thread operation efficiency. After completing these configurations, a thread pool manager is created, which is responsible for thread creation, destruction, and status monitoring, and outputs the initial state of the thread resource pool.

[0031] In step S1.4, the system's task processing capability is evaluated based on the initial state of the thread resource pool and the set of valid resource parameters. By analyzing historical task execution data, a relationship model between resource consumption, task type, and data volume is established to predict the resource requirements of different task types. Based on the available resources, the maximum number of concurrently processable verification tasks and backup tasks, as well as the expected task execution efficiency, are calculated under the current conditions. Resource bottlenecks can also be assessed to identify key resources (such as CPU, memory, or I / O bandwidth) that may limit system performance. Finally, a detailed system resource capacity assessment report is generated, including a hardware configuration summary, available resource statistics, concurrent processing capacity assessment, and potential bottleneck analysis, providing a decision-making basis for subsequent task scheduling and resource allocation.

[0032] like Figure 2 As shown, in step S2, the data source is classified into data types and rated in importance, including: Step S2.1: Receive data from the power control center data source, obtain data structure description and update frequency information, classify the data type, and obtain data source classification results; Step S2.2: Based on the data source classification results, obtain data access frequency and data dependency information, calculate the data importance index, and obtain a data importance rating result; Step S2.3: Based on the data importance rating results, a directed dependency graph is constructed, and criticality index calculation is performed to obtain the data node criticality; Step S2.4: Based on the criticality of the data nodes, a sub-exponential algorithm on independent sets is applied to identify key data nodes, and a data classification and rating report is output.

[0033] In step S2.1, various data sources within the power control center are connected and metadata information is obtained. First, connections are established to various data sources, including the SCADA real-time database, historical database, configuration information repository, and waveform recording file system, using connection parameters stored in configuration files or databases. For each data source, metadata is obtained, including data structure descriptions (such as database table structures, XML schema definitions, and file format specifications), data update frequency, data volume, and access interfaces. Parallel processing interfaces provided by domestic operating systems are utilized to simultaneously process metadata analysis tasks for multiple data sources, improving classification efficiency. Data is categorized across multiple dimensions according to pre-set classification rules: data type is categorized into real-time data streams (such as SCADA real-time measurement point data), static configuration data (such as protection device settings), periodically collected data (such as metering data), and event-triggered data (such as alarm logs); data structure is categorized into structured, semi-structured, and unstructured data; data update frequency is categorized into high-frequency, medium-frequency, and low-frequency data; and data storage location is categorized into primary system storage, backup system storage, and remote site storage. For newly discovered unclassified data sources, machine learning algorithms can be applied to analyze their characteristics, attempting to classify them into existing categories or create new ones. Ultimately, a data source classification result is generated, including each data source's category identifier, connection information, data volume estimation, and access methods.

[0034] In step S2.2, based on the data source classification results, data importance is further analyzed. Access frequency statistics are obtained for each type of data, including query counts, modification frequency, and export frequency. This information reflects the extent of data usage in business operations. Data dependencies are analyzed to identify which data is referenced or dependent on other data, thereby constructing a data dependency network. Data importance can also be determined by considering various factors: business impact coefficient, which assesses the impact of data loss or errors on power business operations; regulatory compliance requirements, which determine whether the data type is required to be preserved by regulations; data recovery difficulty, which assesses the difficulty of retrieving or reconstructing data if lost; historical query frequency, which analyzes the historical frequency of queries or use of this type of data; and data link criticality, which determines whether the data is located on the critical path within the entire data flow. By weighting these factors, a importance index between 0 and 100 is calculated for each data type. Based on this index value, the data is classified into four levels: critical (80-100), important (60-79), regular (40-59), and general (0-39). The rating results are output in the form of a data importance rating results table, which includes information such as data source ID, data name, importance index, grade classification and rating basis.

[0035] In step S2.3, based on the data importance rating results, a directed graph model of data dependencies is constructed and criticality metrics are calculated. In this directed graph, nodes represent data entities, and edges represent dependencies or references between data. Edges are directed from the dependent to the dependent. For each node in the graph, various centrality metrics are calculated: degree centrality, which measures the number of direct connections a node has; closeness centrality, which measures the average distance from a node to all other nodes; betweenness centrality, which measures the frequency with which a node serves as the shortest path between pairs of other nodes; and eigenvector centrality, which considers the importance of node connections rather than just the number of connections. Global graph properties, such as the clustering coefficient, connectivity, and diameter, are also calculated to understand the overall structure of the data dependency network. By combining these metrics, a criticality score is calculated for each data node, identifying key data nodes that are at the core of the data dependency network.

[0036] In step S2.4, based on the criticality of the data nodes, a sub-exponential algorithm on independent sets is applied to identify the most critical set of data nodes. An independent set is a group of non-adjacent nodes in a graph. Sub-exponential algorithms, with a time complexity between polynomial and exponential, are particularly well-suited for finding specific patterns in large-scale graphs. First, the maximum-weighted independent set is found in the data dependency graph—that is, the set of nodes with the highest total criticality without violating the independence constraint (no direct connections between nodes). This problem is NP-hard and can be solved using approximate algorithms with sub-exponential time complexity, such as branch-and-bound, local search, or genetic algorithms. By identifying these critical data nodes, data validation and backup can be prioritized, maximizing data security within limited resources. Finally, a detailed data classification and rating report is generated, including data classification results, importance ratings, criticality analysis, and key node identification, providing a scientific basis for subsequent task prioritization.

[0037] In step S3, an online distributed queue length estimation algorithm is applied to perform resource evaluation, including: Step S3.1: Periodically obtain CPU usage, memory occupancy, and I / O waiting time, calculate the current system load level, and obtain a load status report of the operating system; Step S3.2: Based on the load status report of the operating system, analyze the backlog of tasks of each priority level, predict the task growth trend, and obtain the task queue length estimation result; Step S3.3: Based on the task queue length estimation result, the task waiting time and the system resource margin are calculated, the thread creation strategy is determined, and the thread creation decision result is obtained; Step S3.4: Based on the thread creation decision result, the thread management interface is called to create a working thread, resource quotas are allocated, and an optimized thread resource allocation plan is output.

[0038] In step S3.1, system resource usage is periodically called through the operating system monitoring API to assess the current load level. At preset intervals (typically a few to tens of seconds), system-provided performance monitoring interfaces are called, such as / proc / stat to obtain CPU usage, / proc / meminfo to obtain memory usage, and / proc / diskstats and / proc / net / dev to obtain I / O device usage status. Several key performance indicators are calculated: CPU usage, including the percentage of user-mode, system-mode, I / O wait, and idle time; memory utilization, including physical memory usage, swap space usage, and paging frequency; I / O wait time, including disk read and write latency and network transmission latency; and the system load average, reflecting the overall system load level. Resource usage of each thread in the thread pool is also monitored, including CPU time, memory allocation, and the number of I / O operations. Based on this real-time monitoring data, the current system load level is calculated and compared with preset load thresholds (such as 70% CPU usage and 80% memory usage) to assess whether the system is under high, normal, or low load. This information is integrated into an operating system load status report to provide a basis for subsequent resource allocation decisions.

[0039] In step S3.2, based on the operating system load status report and the current task queue status, an online distributed queue length estimation algorithm is applied to analyze the task backlog and predict future trends. In a distributed environment, accurately obtaining the global task queue length typically requires expensive synchronization operations. However, the online distributed queue length estimation algorithm employed in the present invention can accurately estimate the current length and growth trend of each priority task queue through local observation and statistical models, without the need for global synchronization. First, local observation data is collected for each priority queue, including the enqueue rate, dequeue rate, and current queue length. Then, time series analysis methods, such as exponential smoothing, ARIMA models, or machine learning algorithms, are applied to predict the task growth trend over a future period (e.g., the next 30 minutes to two hours) based on historical data. The prediction model can also be adjusted to account for business model factors, such as peak and off-peak periods in the power system and scheduled maintenance periods. This method accurately estimates the current backlog and future growth trend of each priority task queue, outputting a task queue length estimation result containing metrics such as the current queue length, predicted growth rate, and queue saturation.

[0040] In step S3.3, based on the estimated task queue length and system load, key performance indicators are calculated and a thread creation strategy is determined. First, the expected waiting time for tasks of each priority level is calculated. This metric directly impacts the timeliness of data processing. Furthermore, the system's resource headroom is assessed, representing the ratio of the remaining available capacity of each resource (CPU, memory, and I / O) to its total capacity. Based on the pre-set thread management strategy, a balance is sought between task waiting time and system resource headroom. When the expected waiting time of a high-priority task exceeds a threshold (e.g., a critical task should not wait longer than 5 seconds) and sufficient system resource headroom is available, a new worker thread is created. When the task queue is empty or system resources are nearing saturation, the number of active threads is reduced. The overhead of thread creation and destruction is also considered to avoid frequent changes in the number of threads. Based on this analysis, a thread creation decision is generated, including the number of threads to be created or destroyed, thread priority settings, and resource quota allocation recommendations.

[0041] In step S3.4, based on the thread creation decision, the operating system's thread management interface is called to dynamically adjust the number of worker threads and allocate resources. New worker threads are created using the pthread_create() function or specific thread pool management APIs, or the thread count is reduced by sending termination signals to existing threads. For newly created threads, their priorities, CPU affinity, and resource limits are set. Resource control mechanisms provided by domestic operating systems, such as control groups (cgroups), process / thread priority settings (using nice values ​​or real-time priorities), and CPU affinity settings (binding threads to specific CPU cores), are used to allocate different resource quotas to threads of different priorities. High-priority threads typically receive larger CPU time slices, larger memory quotas, and higher I / O bandwidth priority. Load balancing among worker threads ensures an even distribution of tasks, preventing some threads from being overloaded while others remain idle. Finally, an optimized thread resource allocation plan is output, containing detailed information such as each thread's ID, priority, resource quota, CPU affinity, and current task allocation, providing a foundation for subsequent parallel data verification and backup tasks.

[0042] In step S4, a sub-exponential algorithm on an independent set is used to perform parallel data verification, including: Step S4.1: Analyze the logical structure and dependencies of the data, decompose the data set into multiple logical subsets, construct a data dependency graph based on the reference relationships and update sequences between the data, and obtain a data decomposition plan; Step S4.2: Based on the data decomposition scheme, identify a set of data blocks that have no dependencies on each other, determine an independent set partitioning strategy, and obtain an independent set partitioning result; Step S4.3: Based on the independent set division result, perform verification task reorganization on each independent set, set parallel processing parameters, and obtain a subtask division plan; Step S4.4: Based on the subtask division scheme, a stream processing mechanism is established to control the data loading granularity and obtain a verification execution plan; Step S4.5: According to the verification execution plan, call the verification algorithm to perform parallel data verification, record the verification process data, and output the independent set verification results.

[0043] In step S4.1, the logical structure and dependencies of the data to be verified are analyzed to decompose the large dataset into smaller logical subsets. By analyzing data schema definitions, database table structures, or file format specifications, the internal structure and organization of the data are understood. For structured data (such as relational databases), foreign key relationships, trigger dependencies, and view dependencies between tables are identified. For semi-structured data (such as XML or JSON files), nested and reference relationships between elements are analyzed. For unstructured data (such as oscillographic files), the organization of internal data blocks is identified based on the file format specifications. The update sequence of data can also be considered to determine which data requires a specific order for processing to ensure consistency. Based on this analysis, a data dependency graph is constructed, where nodes represent data entities (such as database tables, configuration files, or data blocks) and edges represent dependencies, with edges pointing from the dependent to the dependent. A graph partitioning algorithm, such as the minimum cut set algorithm or spectral clustering algorithm, is applied to decompose the dataset into multiple logical subsets, minimizing the dependencies between subsets. Finally, a data decomposition plan is generated, which includes the subset partitioning results, the dependencies between subsets, and a recommended processing order.

[0044] In step S4.2, based on the data decomposition scheme, independent data sets that can be processed in parallel are further identified. In graph theory, an independent set refers to a group of non-adjacent nodes in a graph, meaning there are no direct edges connecting them. In a data dependency graph, an independent set represents a group of data entities that have no dependencies and can be processed in parallel without causing consistency issues. Finding the largest independent set is an NP-hard problem. Heuristic or approximate algorithms, such as greedy algorithms, local search, or genetic algorithms, can be used to find an optimal independent set partition within an acceptable timeframe. The physical storage location of the data can also be considered, with data stored on the same physical device being assigned to the same independent set to reduce data transmission overhead. Different independent set partitioning strategies are employed for different types of power data: for large historical databases, independent sets can be partitioned by time (year, month, day) or spatial (region, substation, device); for configuration databases, independent sets can be partitioned by functional module or configuration object type; and for large waveform file collections, independent sets can be partitioned by device type or time window. The independent set partitioning results are output, including a list of members, size estimates, and processing priority recommendations for each independent set.

[0045] In step S4.3, based on the independent set partitioning results, the verification tasks within each independent set are reorganized and parameterized. First, the data characteristics of each independent set are evaluated, including data size, data type complexity, and verification algorithm requirements. Then, the verification tasks within each independent set are further decomposed into multiple smaller subtasks. These subtasks serve as the basic units actually assigned to worker threads. The subtask granularity setting requires a balance between parallelism and scheduling overhead: too large a granularity limits parallel processing capabilities, while too small a granularity increases task scheduling overhead. The subtask granularity is adaptively adjusted based on data characteristics and system resource availability, typically keeping the subtask execution time between a few seconds and tens of seconds. For each subtask, parallel processing parameters are set, including the number of threads, memory allocation limits, I / O buffer size, and verification algorithm selection. Subtasks can also be prioritized to ensure that verification tasks for important data are executed first. Finally, a subtask partitioning scheme is generated, containing each subtask's data range, resource requirements, execution parameters, and priority settings.

[0046] In step S4.4, based on the subtask partitioning scheme, a streaming processing mechanism is established to control the granularity of data loading and generate a detailed verification execution plan. For large datasets, loading all data into memory for processing at once is generally not feasible. Therefore, a streaming processing approach is required to load and process data in batches. By employing a data pipeline mechanism that includes data reading, preprocessing, verification calculation, and result output, each stage can be executed in parallel, improving processing efficiency. The amount of data loaded in each batch (data block size) is determined based on available memory and data characteristics, typically ranging from a few MB to several hundred MB. For extremely large datasets, disk-based external sorting and verification algorithms can be used to further reduce memory requirements. Data prefetching can also be implemented, preloading the next data block while processing the current data block, reducing I / O wait time. For data that requires multiple accesses, a caching mechanism is used to avoid repeated reads. Based on these optimizations, a detailed verification execution plan is generated, including information such as the data loading strategy, processing flow, resource allocation, and exception handling mechanisms.

[0047] In step S4.5, according to the verification execution plan, the corresponding verification algorithm is invoked to perform parallel data verification and record verification process data. First, subtasks are assigned to worker threads based on the task-thread mapping relationship. Each worker thread performs verification operations on the assigned data subset according to the specified verification algorithm and parameters. Different verification algorithms are selected based on the data type and importance. For real-time SCADA data, lightweight CRC32 or Adler-32 algorithms are typically used; for critical configuration data, SHA-256 or higher-strength hash algorithms are used; for large historical data files, block hashing and tree hashing structures are used; for structured databases, in addition to file-level verification, record-level and field-level consistency checks are also performed. During execution, threads dynamically adjust processing strategies based on data conditions. For example, more fine-grained verification may be performed on data areas where anomalies are detected. Threads also regularly report execution progress and resource usage to facilitate system monitoring and adjustment. A robust exception handling mechanism can capture and classify various exceptions, such as data access errors, verification algorithm execution anomalies, or insufficient resources. For recoverable exceptions, a retry process is automatically initiated. For non-recoverable serious exceptions, detailed information is recorded and notified to the upper-level management module. Each thread writes the verification results to a shared result pool, including information such as the checksum / hash value, processing time, and exception history. Finally, the independent set verification results are output to provide input for subsequent result aggregation.

[0048] In step S4, the verification results of each thread are summarized, including: Step S4.6: Monitor the verification progress of each working thread, collect intermediate verification results, establish a result buffer, and obtain a thread verification status report; Step S4.7: Based on the thread verification status report, perform a merge of the thread group results, verify data consistency, and obtain a merge result within the group; Step S4.8: Based on the intra-group merge result, perform a cross-thread group merge, check the reference relationship integrity, and obtain a global merge result; Step S4.9: Based on the global merge result, generate verification statistics and a problem list, evaluate verification efficiency, and obtain a performance analysis report; Step S4.10: Based on the performance analysis report, perform hierarchical storage of the results, establish a verification problem index, and output a data verification result set.

[0049] In step S4.6, the verification progress of each worker thread is monitored in real time, and intermediate verification results are collected. Through inter-thread communication mechanisms such as shared memory, message queues, or conditional variables, the execution status of each thread is regularly obtained, including information such as the amount of data processed, current processing rate, estimated completion time, and resource usage. Threads are checked for abnormal conditions, such as prolonged unresponsiveness, abnormal resource usage, or frequent errors. Intermediate verification results generated by each thread are stored in the result buffer, a temporary in-memory storage area for verification results that have not yet been merged. To avoid excessive memory pressure, when the buffer reaches a preset threshold, some results are written to a temporary file or database for subsequent processing. The efficient inter-process communication (IPC) mechanism and memory-mapped file technology provided by domestic operating systems are used to optimize the transmission and storage efficiency of intermediate results. Based on this collected information, a thread verification status report is generated, containing detailed information such as each thread's ID, current task, progress percentage, processing rate, resource usage, and exception records.

[0050] In step S4.7, based on the thread verification status report, the results within the thread group are merged. Worker threads are organized into multiple thread groups, each responsible for processing an associated data subset. Within each thread group, a check is first performed to determine whether all threads have completed their assigned tasks or whether the preset merge trigger condition (e.g., more than 80% of tasks have been completed) has been met. Then, all intermediate results for the thread group are read from the result buffer, and an intra-group merge is performed. The merge process is more than simply concatenating results; it also includes data consistency verification to ensure that data references within the group are correct, data versions are consistent, and any possible boundary conflicts are resolved. For any issues discovered during the verification process, detailed information is recorded, including the issue type, location, severity, and possible cause. Statistical metrics for the intra-group verification are calculated, such as verification pass rate, issue density, and processing efficiency. After the intra-group merge is complete, an intra-group merge result is generated, containing information such as the merged verification data, consistency verification results, and statistical metrics.

[0051] In step S4.8, based on the intra-group merge results of each thread group, a cross-thread group merge is performed to form a globally consistent verification result. First, the merge order between thread groups is determined, usually based on data dependencies, to ensure that the dependent data is merged first. Then, cross-group merges are performed step by step to integrate multiple intra-group merge results into a larger result set. During the cross-group merge process, the reference integrity of the cross-group data is checked to ensure that the reference relationships between data processed by different groups are correct. For example, check whether foreign key references are valid, whether the dependencies between configuration items are satisfied, and whether the continuity of time series data is maintained. It can also resolve cross-group data conflicts, such as the handling strategy when different groups produce different verification results for the same data item. For any cross-group issues discovered, detailed information is recorded and the scope of their impact is assessed. Finally, a global merge result is generated, which includes information such as the verification results of all data, the cross-group reference relationship verification results, and the global consistency assessment.

[0052] In step S4.9, based on the global merge results, detailed verification statistics and a problem list are generated, and verification efficiency is evaluated. Several statistical metrics are calculated: the total amount of data processed, including the number of records, files, or data size; the verification pass rate (the percentage of data items that passed verification); the problem distribution (the number of problems by problem type, severity, and data category); and the verification coverage (the proportion of data items actually verified out of the total number that should have been verified). A detailed problem list is also generated, containing detailed information for each discovered problem: problem ID, problem type (such as data corruption, inconsistency, or missing data), problem location (such as database table name and record ID, file path and offset), severity (typically categorized as fatal, severe, warning, and prompt), discovery time, and possible cause analysis. Verification efficiency is evaluated using multiple performance metrics: total processing time, average processing rate (data processed per second), resource utilization (average CPU, memory, and I / O usage), parallel efficiency (the ratio of actual speedup to theoretical speedup), and bottleneck analysis (identifying the main factors limiting performance). Based on this analysis, a performance analysis report is generated, containing detailed statistics, performance metrics, and optimization recommendations.

[0053] In step S4.10, based on the performance analysis report, the results are stored hierarchically, a verification issue index is established, and a data verification result set is ultimately output. This tiered storage strategy stores verification results at different levels based on data importance and access frequency: key summary information is stored in memory for immediate access; detailed verification results are stored in local fast storage (such as SSDs) for efficient querying; and historical verification data and raw verification logs are stored in large-capacity storage devices for long-term archiving. Verification results are compressed and encrypted according to pre-set rules to reduce storage space usage and protect sensitive information. For discovered data issues, a multidimensional index is established to support fast querying and filtering by issue type, severity, data category, and time range. Depending on the configuration, the corresponding processing flow is automatically triggered: critical and severe issues may generate an alert to notify the administrator; issues that can be automatically fixed may initiate the remediation process; and issues requiring manual intervention may generate a work order and assign it to the relevant personnel. Finally, a complete data verification result set is output, including overall verification statistics, detailed verification records, a list of issues, performance analysis, and follow-up recommendations, providing key input for the subsequent execution of the intelligent backup strategy.

[0054] In step S5, backup requirements are evaluated and a backup strategy is determined, including: Step S5.1: Analyze the data verification result set, obtain data integrity status and change information, calculate the data change rate, and obtain the data status assessment result; Step S5.2: Based on the data status evaluation result, obtain the last backup time and storage space information, evaluate the backup priority, and obtain a backup priority list; Step S5.3: Based on the backup priority list, a parameterized hybrid time algorithm is applied to calculate the backup time window, a backup method is selected, and an initial backup plan is obtained; Step S5.4: Based on the initial backup plan, evaluate the storage resource status, optimize the compression parameter settings, and obtain a storage strategy plan; Step S5.5: Generate a backup task description based on the storage strategy solution, allocate storage locations, and output a backup strategy execution plan.

[0055] In step S5.1, the data verification result set is analyzed to assess the data integrity status and changes. First, the integrity status information for each data type is extracted from the verification result set, including data integrity, corruption or inconsistency issues, and the severity of the issues. For data with identified issues, the impact of the issues on data availability is assessed to determine whether urgent backup or recovery is necessary. Data change information is analyzed, comparing the current data state with the data state at the time of the last verification or backup to identify which data has changed and the extent of the changes. The data change rate, the proportion of changed data items to the total data items, is calculated. This metric is crucial for selecting a full or incremental backup strategy. Different change detection methods are used for different data types: for structured data, record modification timestamps or version numbers can be compared; for file system data, file modification times and sizes can be compared; and for content-sensitive data, data hash values ​​or checksums can be compared. Change patterns and trends are analyzed, such as whether changes are concentrated in specific areas or distributed throughout the dataset, and whether the change frequency is stable or fluctuating. Based on this analysis, a data status assessment result is generated, containing information such as the integrity status, change rate, change pattern, and impact assessment for each data type.

[0056] In step S5.2, the backup priority of each data set is evaluated based on the data status assessment results, combined with historical backup information and storage resource status. First, information such as the last backup time, backup type (full or incremental), and backup results for each data type is obtained from the backup metadata repository. The backup interval for each data type is calculated—the difference between the current time and the last successful backup time—and compared with the recommended backup cycle for that data type. The storage system status is queried to obtain information such as available storage space, storage device performance, and backup media type to assess the adequacy of storage resources. Based on this information, a multi-factor scoring model is used to calculate a backup priority score for each data type, taking into account the following factors: data importance level (data importance rating results from step S2); data integrity status (data integrity status from step S5.1), with problematic data receiving a higher priority; data change rate (data with a high change rate generally requires more frequent backups); the ratio of the backup interval to the recommended cycle (data exceeding the recommended cycle receives a higher priority); and storage resource status (lower-importance data may be downgraded when storage resources are limited). According to the priority score, all data sets are sorted and a backup priority list is generated, which contains information such as the ID, name, priority score, recommended backup type, and recommended backup time of each data set.

[0057] In step S5.3, based on the backup priority list, a parameterized hybrid time algorithm is applied to calculate the optimal backup time window and select an appropriate backup method. The parameterized hybrid time algorithm combines deterministic and stochastic time analysis to more accurately describe the backup timing characteristics of power system data. The algorithm relies on three core parameters: a time window parameter, which defines the time period during which backups can be performed; a data decay parameter, which describes the temporal variation of data value; and a system load parameter, which reflects the load variations in different time periods. First, historical system load data is analyzed to identify periods with low load, which are suitable for resource-intensive backup tasks. Then, data usage patterns are considered to avoid performing backups during periods of intensive data access or updates. Time-varying characteristics of network bandwidth and storage system performance can also be considered to select a time window that optimizes resource conditions. The choice of backup method is primarily based on the data change rate and the type of the last backup. For data with a high change rate (e.g., exceeding 30%), a full backup is typically selected. For data with a low change rate but large data volume, an incremental backup is preferred. For data that has not undergone a full backup for a long time, even if the change rate is low, a full backup may be selected to prevent recovery complexity caused by long incremental chains. We can also consider data consistency requirements. For related datasets with strong consistency requirements, we may choose to back them up simultaneously to ensure consistency. Based on these analyses, we determine the backup time window and backup method for each dataset and generate an initial backup plan.

[0058] In step S5.4, based on the initial backup plan, storage resource availability is further evaluated, compression parameter settings are optimized, and a detailed storage strategy is formulated. First, the current state of the backup storage system is analyzed, including the available space, I / O performance, reliability level, and cost characteristics of each storage device. The storage requirements for each backup task are estimated, taking into account factors such as the original data size, expected compression ratio, and backup type (full or incremental). Then, based on the data's importance and access frequency, an appropriate storage tier is selected for each backup task. Critical data may be stored on high-performance, high-reliability storage devices; historical data may be stored on large-capacity, but lower-performance storage devices; and rarely accessed archived data may be stored on offline or nearline storage media. The data compression strategy can also be optimized, selecting an appropriate compression algorithm and level based on the data type, importance, and system resource availability. For situations with abundant CPU resources but limited storage space, a high-compression-ratio, but CPU-intensive algorithm may be selected; for situations with limited CPU resources, an algorithm with lower compression efficiency but low CPU consumption may be selected; and for critical data requiring rapid recovery, a lower compression level or even no compression may be selected. Data security requirements can also be considered, with encryption parameters configured for sensitive data and an appropriate encryption algorithm and key management scheme selected. Based on these analyses, a storage policy plan is generated, which contains detailed information such as storage device allocation, compression parameter settings, encryption configuration, and expected storage space requirements for each backup task.

[0059] In step S5.5, a detailed backup task description is generated based on the storage policy scheme, a specific storage location is assigned, and the final backup policy execution plan is output. First, a unique task ID and description information is created for each backup task, including data source information, backup type, expected execution time, and expected completion time. Then, based on the storage policy scheme, a specific storage location is assigned to each backup task, including storage device identification, directory path, or object storage bucket name. Detailed backup parameter configuration is generated, including data source connection parameters, backup scope definition, filter conditions, compression and encryption settings, verification options, etc. For incremental backup tasks, the location and identification of the baseline backup are specified to ensure that incremental backups can be applied correctly. Resource limit parameters for backup tasks are set, such as maximum CPU usage, memory limit, and I / O bandwidth limit, to prevent backup tasks from affecting the normal operation of the system. For the backup of related data, dependencies and execution order between tasks are set to ensure data consistency. The monitoring and reporting parameters of the backup task are configured, defining the indicators to be recorded and the types of reports to be generated. Finally, a complete backup strategy execution plan is output, which includes detailed descriptions of all backup tasks, execution parameters, resource limitations, dependencies, monitoring configurations, and other information, providing detailed guidance for subsequent parallel data backup execution.

[0060] In step S5, parallel data backup is performed, including: Step S5.6: Based on the backup strategy execution plan, establish a backup progress monitoring mechanism, initialize the backup thread pool, and obtain the backup environment configuration; Step S5.7: Based on the backup environment configuration, assign backup tasks to working threads, set resource usage limits, and obtain a task allocation plan; Step S5.8: Based on the task allocation scheme, perform data compression and transmission operations, monitor the backup rate, and obtain real-time backup status; Step S5.9: Based on the real-time backup status, detect backup anomalies such as data transmission interruption, insufficient storage space, and data verification failure, execute failed tasks and retry, and obtain an anomaly handling log; Step S5.10: Verify the integrity of the backup data according to the exception handling log, record the backup metadata, and output the data backup execution status and results.

[0061] In step S5.6, a backup progress monitoring mechanism is established based on the backup policy execution plan, and the backup thread pool is initialized. A backup task monitor is first created to track the execution status and progress of all backup tasks. The monitor sets multiple monitoring points and regularly collects task execution data, including the amount of data processed, the current processing rate, the estimated remaining time, and resource usage. An alarm mechanism is established to trigger alarms of corresponding levels when task execution is abnormal or progress deviates significantly from expectations. Simultaneously, the backup thread pool is initialized, and the initial number of threads and resource quota are determined based on the resource allocation recommendations in the backup policy execution plan and the current system load. The backup thread pool is configured with a thread creation policy, a task queue management policy, and an exception handling policy to ensure efficient and stable operation. Shared resources required for backup, such as the buffer pool, connection pool, and temporary storage area, are initialized, and resource allocation and recycling policies are set. The backup data stream processing pipeline is configured, including processing stages such as data reading, filtering, conversion, compression, encryption, and writing, and appropriate parameters are set for each stage. Finally, a backup environment configuration is generated, which includes information such as monitoring mechanism settings, thread pool configuration, resource initialization status, and processing pipeline parameters.

[0062] In step S5.7, based on the backup environment configuration, backup tasks are assigned to worker threads and resource usage limits are set. The order of task execution is first determined based on the backup task's priority and dependencies. For tasks without dependencies, higher-priority tasks are prioritized; for tasks with dependencies, dependent tasks are ensured to execute first. Task scheduling algorithms, such as priority queue scheduling or fair share scheduling, are used to assign backup tasks to worker threads in the backup thread pool. The allocation process considers load balancing to prevent overloading of some threads while idling others. Resource usage limits are set for each backup task, including CPU usage limits, memory allocation limits, and I / O bandwidth limits, to ensure that backup tasks do not excessively consume system resources and impact other services. Higher-priority tasks are allocated more resource quotas, while lower-priority tasks are subject to stricter resource limits. Considering the principle of data locality, tasks accessing the same or nearby storage devices are assigned to the same or similar threads whenever possible to reduce data transmission overhead. Large backup tasks are split into multiple subtasks and assigned to multiple threads for parallel processing to improve execution efficiency. Finally, a task allocation plan is generated, which contains detailed information such as thread allocation, execution order, resource constraints, and subtask division (if any) for each backup task.

[0063] In step S5.8, based on the task allocation scheme, data compression and transfer operations are performed, and the backup rate is monitored in real time. Each backup thread begins executing data backup operations according to its assigned task. For full backup tasks, the thread first establishes a connection with the data source and reads all target data. For incremental backup tasks, the thread identifies data that has changed since the last backup and backs up only the modified data. Following the configured processing pipeline, the thread performs a series of processing on the data: first, data filtering is performed to exclude data items that do not need to be backed up; then, data conversion is performed to convert the data into a format suitable for backup; compression algorithms are applied to reduce the data size; and, if encryption is configured, the data is encrypted. Finally, the processed data is written to the target storage location. During execution, the backup rate and resource usage, including data read rate, processing rate, transfer rate, and write rate, are monitored in real time. Processing parameters such as buffer size, batch size, and parallelism are dynamically adjusted to optimize backup performance. The backup progress percentage and estimated completion time are regularly calculated to update task status information. For tasks that take longer than expected, resource allocation or priority is adjusted to ensure timely completion of critical tasks. This monitoring data is consolidated into a real-time backup status, including information such as the progress, rate, resource usage, and estimated completion time of each task.

[0064] In step S5.9, various backup anomalies are detected and handled based on the real-time backup status. A comprehensive anomaly detection mechanism is implemented, capable of identifying various potential issues that may arise during the backup process: data source connection anomalies, such as connection loss, authentication failure, or insufficient permissions; data read anomalies, such as file non-existence, data corruption, or format errors; processing anomalies, such as compression failure, encryption errors, or insufficient memory; transmission anomalies, such as network outages, timeouts, or insufficient bandwidth; storage anomalies, such as insufficient storage space, write failures, or device failures; and consistency anomalies, such as inconsistencies caused by data modification during the backup process. For detected anomalies, appropriate handling strategies are implemented based on the type and severity of the anomaly. For temporary anomalies (such as network fluctuations), the operation is automatically retried, typically with a maximum number of retries and a backoff strategy. For resource-related anomalies (such as insufficient storage space), attempts are made to release resources or find alternatives. For unrecoverable, serious anomalies, the current task is terminated, detailed error information is logged, and the administrator is notified. A task recovery mechanism can also be implemented to resume interrupted backup tasks from the point where they were interrupted, avoiding the waste of resources caused by restarting. All exceptions and handling operations are recorded in the exception handling log, including the exception time, type, detailed description, scope of impact, handling operations and results.

[0065] In step S5.10, the backup data integrity is verified based on the exception handling log, backup metadata is recorded, and the final backup execution status and results are output. For completed backup tasks, backup verification is performed to ensure the integrity and recoverability of the backup data. The verification process includes: checksum verification, which calculates the checksum of the backup data and compares it with the checksum of the original data; structure verification, which ensures the structural integrity of the backup data, such as database table structures and file directory structures; sample content verification, which randomly selects a portion of data items and compares the backup content with the original content; and recovery testing, which performs a simulated recovery operation on a portion of the backup data to verify the feasibility of the recovery process. For backups that pass verification, detailed backup metadata is recorded, including: backup identification information, such as the backup ID, name, and description; time information, such as the start time, completion time, and duration; content summary, such as the backup data type, scope, and quantity; storage information, such as the storage location, file name, and size; verification information, such as the checksum, digital signature, and verification results; and association information, such as the baseline backup ID (for incremental backups) and dependencies. This metadata is stored in the backup directory, and the backup metadata database is simultaneously updated for easy query and management. Backups that fail verification are marked as "failed," and the cause of the failure and the scope of impact are recorded. Finally, a complete data backup execution status and results report is generated, including information on the execution status of all backup tasks, success rates, exception statistics, performance indicators, and recommended actions. This report serves as a summary of the backup operation and provides a basis for subsequent backup strategy optimization and troubleshooting.

[0066] like Figure 3 As shown, an embodiment of the present invention further provides a data verification and backup device for a power centralized control center, comprising: The system resource evaluation module 10 is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, build a thread resource pool, and obtain a system resource capability evaluation report; The data prioritization module 20 is connected to the system resource assessment module 10 and is used to connect to the data source of the power centralized control center, classify the data type and importance of the data source, use a parameterized hybrid time algorithm to perform real-time demand analysis, and obtain a data task queue with priority marking; The thread resource allocation module 30 is connected to the data prioritization module 20 and is used to monitor the load status of the operating system and the length of the task queue, apply an online distributed queue length estimation algorithm to perform resource evaluation, dynamically create worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocate CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; The data verification module 40 is connected to the thread resource allocation module 30 and is used to allocate verification tasks to working threads based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on independent sets, and summarize the verification results of each thread to obtain a data verification result set; The data backup module 50 is connected to the data verification module 40 and is used to evaluate the backup requirements according to the data verification result set, determine the backup strategy, perform parallel data backup, and obtain the data backup execution status and results.

[0067] The above description is only a preferred embodiment of the present invention and does not limit the patent scope of the present invention. All equivalent structural transformations made by using the contents of the present invention description and drawings under the inventive concept of the present invention, or direct / indirect application in other related technical fields are included in the patent protection scope of the present invention.

Claims

1. A data verification and backup method for a power centralized control center, characterized in that: The following steps are involved: Obtaining hardware resource information corresponding to the operating system, calculating effective resource parameters based on the hardware resource information, building a thread resource pool, and obtaining a system resource capability assessment report; Connect to the data source of the power centralized control center, classify the data type and importance of the data source, use a parameterized hybrid time algorithm to perform real-time demand analysis, and obtain a data task queue with priority tags; Monitor the load status and task queue length of the operating system, apply an online distributed queue length estimation algorithm to perform resource evaluation, dynamically create worker threads based on the load threshold of the operating system and the backlog level of the task queue, and allocate CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; Based on the thread resource allocation scheme, the verification task is assigned to the working thread, and the sub-exponential algorithm on the independent set is used to perform parallel data verification. The verification results of each thread are summarized to obtain a data verification result set; Evaluate backup requirements based on the data verification result set, determine backup strategies, execute parallel data backup, and obtain data backup execution status and results.

2. The method according to claim 1, characterized in that Obtaining hardware resource information corresponding to the operating system and evaluating effective resource parameters based on the hardware resource information include: Obtain the number of processor cores, memory capacity, and I / O bandwidth information through the system API to obtain the original hardware resource parameter set; Based on the original hardware resource parameter set, read the current process occupancy data, calculate the allocatable CPU time slice, available memory space and I / O bandwidth, and obtain the effective resource parameter set; Based on the effective resource parameter set, the minimum threshold and maximum threshold of the thread pool are set, the thread communication mechanism is initialized, and the initial state of the thread resource pool is obtained; According to the initial state of the thread resource pool and the effective resource parameter set, the number of verification and backup tasks that can be processed concurrently is calculated, and a system resource capacity evaluation report is output.

3. The method according to claim 2, characterized in that Categorize the data types and importance of the data sources, including: Receive data from the power control center data source, obtain data structure description and update frequency information, classify the data type, and obtain data source classification results; Based on the data source classification results, data access frequency and data dependency information are obtained, a data importance index is calculated, and a data importance rating result is obtained; Based on the data importance rating results, a directed dependency graph is constructed, and criticality index calculation is performed to obtain the criticality of the data nodes; Based on the criticality of the data nodes, a sub-exponential algorithm on independent sets is applied to identify key data nodes, and a data classification and rating report is output.

4. The method according to claim 3, characterized in that Use parameterized hybrid time algorithm to perform real-time demand analysis, including: Obtain data type and business requirement information, determine the data valid time window, and obtain the time window parameter set; Based on the time window parameter set, a data timeliness model is established, the data decay rate is calculated, and the decay curve parameters are obtained; Based on the attenuation curve parameters, the tolerance of the service to data delay is evaluated, the maximum allowable delay time is calculated, and a timeliness evaluation result is obtained; According to the timeliness evaluation result and the data importance rating result, a comprehensive priority index is calculated, a multi-level task queue is constructed, and a data task queue with priority marking is output.

5. The method according to claim 4, characterized in that Apply online distributed queue length estimation algorithms for resource assessment, including: Periodically obtain CPU usage, memory occupancy, and I / O wait time, calculate the current system load level, and obtain a load status report of the operating system; Based on the load status report of the operating system, analyzing the backlog of tasks of each priority level, predicting the task growth trend, and obtaining a task queue length estimation result; Based on the task queue length estimation result, the task waiting time and system resource margin are calculated, the thread creation strategy is determined, and the thread creation decision result is obtained; According to the thread creation decision result, the thread management interface is called to create a working thread, resource quota is allocated, and an optimized thread resource allocation plan is output.

6. The method according to claim 5, characterized in that Parallel data validation is performed using a sub-exponential algorithm on independent sets, including: Analyze the logical structure and dependencies of the data, decompose the data set into multiple logical subsets, build a data dependency graph based on the reference relationships and update sequence between the data, and obtain a data decomposition plan; Based on the data decomposition scheme, a set of data blocks having no dependency on each other is identified, an independent set partitioning strategy is determined, and an independent set partitioning result is obtained; Based on the independent set division result, reorganize the verification tasks for each independent set, set parallel processing parameters, and obtain a subtask division scheme; Based on the subtask division scheme, a stream processing mechanism is established to control the data loading granularity and obtain a verification execution plan; According to the verification execution plan, the verification algorithm is called to perform parallel data verification, the verification process data is recorded, and the independent set verification results are output.

7. The method according to claim 6, characterized in that Summarize the verification results of each thread, including: Monitor the verification progress of each working thread, collect intermediate verification results, establish a result buffer, and obtain thread verification status reports; Based on the thread verification status report, perform a merge of the thread group results, verify data consistency, and obtain a merge result within the group; Based on the intra-group merge result, perform cross-thread group result merge, check the reference relationship integrity, and obtain a global merge result; Based on the global merging results, generate verification statistics and a problem list, evaluate verification efficiency, and obtain a performance analysis report; According to the performance analysis report, the execution results are stored in a hierarchical manner, a verification problem index is established, and a data verification result set is output.

8. The method according to claim 1, characterized in that Evaluate backup needs and determine backup strategies, including: Analyze the data verification result set, obtain data integrity status and change information, calculate the data change rate, and obtain data status assessment results; Based on the data status evaluation result, obtaining the last backup time and storage space information, evaluating the backup priority, and obtaining a backup priority list; Based on the backup priority list, a parameterized hybrid time algorithm is applied to calculate the backup time window, a backup method is selected, and an initial backup plan is obtained; Based on the initial backup plan, evaluate the storage resource status, optimize the compression parameter settings, and obtain a storage strategy plan; According to the storage strategy solution, a backup task description is generated, a storage location is allocated, and a backup strategy execution plan is output.

9. The method according to claim 8, characterized in that Perform parallel data backups, including: Based on the backup strategy execution plan, a backup progress monitoring mechanism is established, the backup thread pool is initialized, and the backup environment configuration is obtained; Based on the backup environment configuration, assign backup tasks to working threads, set resource usage limits, and obtain a task allocation plan; Based on the task allocation scheme, perform data compression and transmission operations, monitor the backup rate, and obtain real-time backup status; Based on the real-time backup status, detect backup anomalies such as data transmission interruption, insufficient storage space, data verification failure, etc., execute failed tasks and retry, and obtain an exception handling log; According to the exception handling log, the backup data integrity is verified, the backup metadata is recorded, and the data backup execution status and results are output.

10. A data verification and backup device for a power centralized control center, characterized in that: include: A system resource evaluation module is used to obtain hardware resource information corresponding to the operating system, calculate effective resource parameters based on the hardware resource information, build a thread resource pool, and obtain a system resource capacity evaluation report; A data prioritization module, connected to the system resource assessment module, is used to connect to the data source of the power centralized control center, classify the data type and importance of the data source, use a parameterized hybrid time algorithm to perform real-time demand analysis, and obtain a data task queue with priority marking; a thread resource allocation module, connected to the data prioritization module, for monitoring the load status of the operating system and the length of the task queue, applying an online distributed queue length estimation algorithm to perform resource evaluation, dynamically creating worker threads based on the load threshold of the operating system and the backlog of the task queue, and allocating CPU time slices, memory quotas, and I / O bandwidth to threads of different priorities to obtain an optimized thread resource allocation plan; a data verification module, connected to the thread resource allocation module, configured to allocate verification tasks to working threads based on the thread resource allocation scheme, perform parallel data verification using a sub-exponential algorithm on independent sets, and aggregate verification results of each thread to obtain a data verification result set; The data backup module is connected to the data verification module and is used to evaluate the backup requirements according to the data verification result set, determine the backup strategy, perform parallel data backup, and obtain the data backup execution status and results.

Images