Power system operation data synchronization method, system and device and storage medium

Abstract

The invention provides an electric power system operation data synchronization method, system and device and a storage medium, and the method comprises the steps: carrying out the priority marking of all kinds of received to-be-synchronized operation data, adding the data into a corresponding data class message queue, and when the number of corresponding data class queues and the number of data class to-be-stored data are zero, carrying out the synchronization of the data class to-be-synchronized operation data; reading a corresponding number of to-be-synchronized data from the data class message queue according to the size of the corresponding data class fragment to construct a data class synchronization fragment, adding the data class synchronization fragment into a task sorting queue, and sequentially reading the data class synchronization fragment from the task sorting queue according to the maximum task number of the storage queue to add into a to-be-executed storage queue, and when the to-be-executed warehousing queue is scheduled, sequentially reading the data class synchronous fragments in the to-be-executed warehousing queue, and inserting the data class synchronous fragments into the target table in the preset storage database in batches. According to the method, the task priority is set based on the data synchronization timeliness, and a data fragmentation scheduling mechanism is combined, so that all the power system operation data can meet the corresponding data synchronization timeliness requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of power system monitoring, and in particular to a method, system, device and storage medium for synchronizing power system operation data. Background Art

[0002] The operation of the power system will have a direct impact on social production and life. In order to reliably monitor the status of the power system, it is necessary to synchronize system operation data such as telemetry and telesignaling to the storage database in real time, and cooperate with the ledger data of each power equipment for real-time analysis of the system operation status.

[0003] The existing methods for synchronizing power system operation data are as follows: Figure 1 As shown in the figure, the telemetry data is usually inserted into the storage database using the batch processing method of JDBC after parsing the telemetry file in CIME format read through the SFTP service, and the telemetry data is directly connected to the telemetering data source to insert the acquired telemetering data into the storage database using the batch processing method of JDBC. In the synchronization process, the synchronization tasks of telemetry data and telemetering data, as well as the synchronization operations of various types of telemetry data and telemetering data are executed concurrently. Based on this, the storage database used to synchronize the running data is shared, and the database resources are limited. When other database operations including telemetry data insertion are executed, the telemetering data storage will inevitably be blocked; however, in actual applications, the telemetering data has a low frequency (once every 10 seconds, about 100 items each time) and the state analysis triggers the data source. The timeliness of the telemetering data storage is required to be high (no slower than 5 seconds), but the telemetering data type is mostly and the data volume is large (about 600,000 items per minute). Each storage takes a long time, which seriously affects the timeliness of the telemetering data storage, and thus affects the smooth execution of the subsequent real-time analysis task of the system operation status. Therefore, there is an urgent need to provide a synchronization method that ensures that all power system operation data stored in the database can meet the corresponding data timeliness requirements when storage database resources are limited. Summary of the invention

[0004] The purpose of the present invention is to provide a method for synchronizing power system operation data. By introducing a message queue cluster to temporarily store the operation data to be synchronized, and setting the task priority based on the data synchronization timeliness, combined with a data sharding scheduling mechanism that dynamically controls the synchronization task order, it can ensure that all power system operation data can meet the corresponding data synchronization timeliness application requirements when the storage database resources are limited.

[0005] In order to achieve the above objectives, it is necessary to provide a power system operation data synchronization method, system, computer equipment and storage medium in response to the above technical problems.

[0006] In a first aspect, an embodiment of the present invention provides a method for synchronizing operation data of a power system, the method comprising the following steps:

[0007] In response to receiving various types of data to be synchronized, the various types of data to be synchronized are marked with priorities and added to the corresponding data class message queues, and the corresponding data class queue numbers and data class waiting to be stored numbers are obtained; the various types of data to be synchronized include telesignaling data and telemetry data;

[0008] Determine whether the data class queue number and data class waiting to be stored corresponding to each type of data to be synchronized are both zero. If so, obtain the corresponding data class shard size, read the corresponding number of data to be synchronized from the corresponding data class message queue, build the corresponding data class synchronization shard and add it to the task sorting queue. Otherwise, continue to monitor the corresponding data class queue number and data class waiting to be stored.

[0009] Obtain the maximum number of tasks in the warehousing queue, and according to the maximum number of tasks in the warehousing queue, sequentially read the corresponding number of data class synchronization shards from the task sorting queue and add them to the warehousing queue to be executed;

[0010] In response to the scheduling of the to-be-executed warehousing queue, the data class synchronization fragments in the to-be-executed warehousing queue are sequentially read, and each data class synchronization fragment is batch-inserted into a corresponding target table in a preset storage database.

[0011] Furthermore, the step of marking the priority of various types of data to be synchronized includes:

[0012] The data types of various types of to-be-synchronized running data are obtained respectively, and the corresponding to-be-synchronized running data are marked with priorities according to the task synchronization timeliness requirements corresponding to the data types.

[0013] Furthermore, the step of reading a corresponding number of to-be-synchronized data from a corresponding data class message queue, constructing a corresponding data class synchronization shard and adding it to a task sorting queue includes:

[0014] According to the size of the data class shard, a corresponding number of data to be synchronized is sequentially read from the corresponding data class message queue and filled into the corresponding data class synchronization shard;

[0015] According to the data priority in the data class synchronization shard, the data class synchronization shard is marked with a priority, and the marked data class synchronization shard is added to the task sorting queue;

[0016] According to the preset sorting principle, all data class synchronization fragments in the task sorting queue are reordered; the preset sorting principle includes sorting different priorities from high to low and sorting the same priority according to the queue entry time.

[0017] Furthermore, the step of sequentially reading a corresponding number of to-be-synchronized data from a corresponding data class message queue and filling the corresponding data class synchronization shard according to the size of the data class shard comprises:

[0018] When the amount of data to be synchronized in the data class message queue is smaller than the size of the data class shard, a maximum waiting time timer is started, and when the maximum waiting time timer times out, reading of the data class message queue is stopped.

[0019] Furthermore, the method further comprises:

[0020] In response to the completion of batch insertion of each data class synchronization shard, the corresponding data class shard size and the maximum number of tasks in the storage queue are updated.

[0021] Furthermore, in response to the completion of batch insertion of each data class synchronization shard, the step of updating the corresponding data class shard size and the maximum number of tasks in the storage queue includes:

[0022] When the batch insertion of each data class synchronization shard is completed, the corresponding data class synchronization data volume and data synchronization duration are obtained, and the data class synchronization data volume and the data class synchronization duration are divided to obtain the corresponding data class insertion number per second;

[0023] According to the rounded value of the number of insertions per second of the data class, the corresponding data class shard size is updated;

[0024] According to the number of insertions per second of the data class and the amount of data in all data class synchronization shards in the task sorting queue, the current maximum number of shards allowed to be processed is obtained, and according to the current maximum number of shards allowed to be processed, the maximum number of tasks in the warehousing queue is updated.

[0025] Furthermore, the step of obtaining the maximum number of shards currently allowed to be processed according to the number of insertions per second of the data class and the data volume of all synchronized shards of the data class in the task sorting queue includes:

[0026] Sequentially obtain the sum of the sorted data volume of each data class synchronization shard and all previous data class synchronization shards in the task sorting queue;

[0027] The maximum sum of sorted data volumes that is less than the number of insertions per second of the data class is obtained, and the number of synchronized shards of the data class corresponding to the maximum sum of sorted data volumes is used as the maximum number of shards currently allowed to be processed.

[0028] In a second aspect, an embodiment of the present invention provides a power system operation data synchronization system, the system comprising:

[0029] A data receiving module is used to respond to the reception of various types of data to be synchronized, mark the priority of various types of data to be synchronized, add them to the corresponding data class message queue, and obtain the corresponding data class queue number and data class waiting to be stored number; the various types of data to be synchronized include telesignaling data and telemetry data;

[0030] The task sorting module is used to determine whether the data class queue number and data class waiting to be stored corresponding to each type of data to be synchronized are both zero. If so, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, and the corresponding data class synchronization shard is constructed and added to the task sorting queue. Otherwise, the corresponding data class queue number and data class waiting to be stored are continuously monitored.

[0031] The task scheduling module is used to obtain the maximum number of tasks in the storage queue, and according to the maximum number of tasks in the storage queue, sequentially read the corresponding number of data class synchronization shards from the task sorting queue and add them to the storage queue to be executed;

[0032] The synchronous warehousing module is used to respond to the scheduling of the waiting-to-be-executed warehousing queue, sequentially read the data class synchronization fragments in the waiting-to-be-executed warehousing queue, and batch-insert each data class synchronization fragment into the corresponding target table in the preset storage database.

[0033] In a third aspect, an embodiment of the present invention further provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0034] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0035] The above-mentioned application provides a method, system, computer equipment and storage medium for synchronizing operation data of an electric power system. The method realizes that in response to the reception of various types of operation data to be synchronized, various types of operation data to be synchronized are marked with priorities and added to the corresponding data class message queues, the corresponding data class queue numbers and data class waiting to be stored are obtained, and the data class queue numbers and data class waiting to be stored corresponding to various types of operation data to be synchronized are respectively judged whether they are both zero. If so, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, the corresponding data class synchronization shard is constructed and added to the task sorting queue. Otherwise, the corresponding data class queue number and the data class waiting to be stored are continuously monitored, and the maximum number of tasks in the storage queue is obtained. According to the maximum number of tasks in the storage queue, the corresponding number of data class synchronization shards are sequentially read from the task sorting queue and added to the storage queue to be executed. In response to the scheduling of the storage queue to be executed, the data class synchronization shards in the storage queue to be executed are sequentially read, and each data class synchronization shard is inserted into the corresponding target table in the preset storage database in batches. Technical solution. Compared with the existing technology, the power system operation data synchronization method introduces a message queue cluster to temporarily store the operation data to be synchronized, and on the basis of setting the task priority based on the data synchronization timeliness, combined with the data sharding scheduling mechanism that dynamically controls the synchronization task order, can ensure that all power system operation data can meet the corresponding data synchronization timeliness application requirements when the storage database resources are limited. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the existing method for synchronizing the operation data of the power system;

[0037] Figure 2 It is a schematic diagram of a process framework for synchronizing power system operation data in an embodiment of the present invention;

[0038] Figure 3 is a flow chart of a method for synchronizing power system operation data in an embodiment of the present invention;

[0039] Figure 4 is another flow chart of the method for synchronizing power system operation data according to an embodiment of the present invention;

[0040] Figure 5 is another detailed process architecture diagram of power system operation data synchronization in an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the structure of a power system operation data synchronization system according to an embodiment of the present invention;

[0042] Figure 7 is another structural schematic diagram of the power system operation data synchronization system according to an embodiment of the present invention;

[0043] Figure 8 It is a diagram of the internal structure of a computer device in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical scheme and beneficial effects of the present application clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. Obviously, the embodiments described below are part of the embodiments of the present invention and are only used to illustrate the present invention, but are not used to limit the scope of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] The power system operation data synchronization method provided by the present invention can be understood as being based on the synchronous and concurrent execution of various types of telemetry data and telesignaling data in the existing power system operation data, and the storage of telemetry data with many types and large data volume takes a long time, which seriously affects the timeliness of the storage of telesignaling data, and thus affects the smooth execution of subsequent real-time analysis tasks of the system operation status. Figure 2 The method shown introduces a message queue cluster to temporarily store the operation data to be synchronized, and on the basis of setting the task priority based on the data synchronization timeliness, combines the data sharding scheduling mechanism that dynamically controls the synchronization task order to realize the efficient batch synchronization of the power system operation data. The following embodiments will explain the power system operation data synchronization method of the present invention in detail.

[0046] In one embodiment, Figure 3 As shown, a method for synchronizing power system operation data is provided, comprising the following steps:

[0047] S11. In response to receiving various types of operation data to be synchronized, various types of operation data to be synchronized are marked with priorities and added to corresponding data class message queues, and the corresponding data class queue numbers and data class waiting to be stored numbers are obtained; wherein, the operation data to be synchronized can be understood as power system operation data that needs to be synchronized to a preset storage database for subsequent analysis, including various telesignaling data, various telemetry data and other related data that need to be collected, wherein various telesignaling data refers to the switching quantity of telecontrol communication data, such as the opening / closing state of a circuit breaker or disconnector, the action / reset of a protection signal, the activation / exit of an AGC / AVC function, etc., including sub-type data such as displacement, over-limit and SOE (Sequence Of Event), and various telemetry data refers to various physical quantities in a remote system that are monitored by remotely transmitting data, such as voltage, current and frequency, etc., including sub-type data such as circuit breakers, switches, transformer windings, current mutual inductance and voltage mutual inductance.

[0048] In order to ensure that data with high timeliness requirements can be quickly synchronized, this embodiment preferably pre-sets corresponding priorities for various types of data to be synchronized according to timeliness requirements. For example, telematics data with high timeliness requirements use a higher priority, and telemetry data with low timeliness requirements use a relatively lower priority. For other types of data to be synchronized, the priority can be set between the telematics data priority and the telemetry data priority, or a priority lower than the telemetry data priority according to the actual application scenario. Specifically, the step of marking the priority of various types of data to be synchronized includes:

[0049] The data types of various types of data to be synchronized are obtained respectively, and the corresponding data to be synchronized are marked with priorities according to the task synchronization timeliness requirements corresponding to the data types; wherein the data type can be understood as the major category to which the data to be synchronized belongs, such as telemetry data category and telesignaling data category, and the priority marking of various types of data to be synchronized can be understood as setting priorities according to the major category to which the specific data to be synchronized belongs, for example, each sub-type data in the telesignaling data uniformly adopts one priority level because they belong to the telesignaling data, and each sub-type data in the telemetry data uniformly adopts one priority level because they belong to the remote-side data, etc., which will not be repeated here.

[0050] The data class message queue in this embodiment can be understood as a message queue corresponding to each sub-type of data. For example, each telemetry sub-type data corresponds to a data class message queue and each telesignal sub-type data corresponds to a data class message queue. It can be deployed and implemented based on the Kafka message queue. The specific implementation method can refer to the relevant existing technology and will not be described in detail here. It should be noted that in actual applications, each data class message queue is maintained and scheduled by a data filling thread.

[0051] The above-mentioned data class queue number can be understood as the number of synchronization sharding tasks with the same subtype as the data to be synchronized in the task sorting queue for sharding all types of data to be synchronized, and the corresponding data class waiting to be stored can be understood as the number of synchronization sharding tasks with the same subtype as the data to be synchronized in the waiting to be executed storage queue for warehousing the synchronization shards that can currently execute storage operations. Considering the characteristics that in actual data synchronization, data of the same subtype needs to be synchronized to the unified target table of the same storage database, and the same target table cannot perform multiple different data insertion operations at the same time, in order to ensure that the resource utilization efficiency and data synchronization efficiency are improved as much as possible under the condition of limited database resources, this embodiment preferably senses and triggers the synchronization queue of various subtypes of data to be synchronized by real-time monitoring of the queuing status of each subtype of data synchronization tasks in the task sorting queue and the storage queue to be executed.

[0052] S12. Determine whether the number of data class queues and the number of data class to be stored corresponding to each type of data to be synchronized are both zero. If so, obtain the corresponding data class shard size, read the corresponding number of data to be synchronized from the corresponding data class message queue, construct the corresponding data class synchronization shards and add them to the task sorting queue. Otherwise, continue to monitor the corresponding number of data class queues and the number of data class to be stored; that is, when a data filling thread receives a certain type of data to be synchronized and adds it to the corresponding data class message queue after marking its priority, it is necessary to determine whether the number of data class queues and the number of data class to be stored corresponding to the subtype of the data to be synchronized are both zero (whether there are still data synchronization tasks of each subtype waiting to be processed in the task sorting queue and the queue to be executed). If there are no tasks of the same type to be synchronized, it is determined that the data filling thread can batch package the data from the corresponding data class message queue and insert it into the task sorting queue. If there are still tasks of the same type to be synchronized, it is determined that the current filling thread needs to suspend data packaging processing until it is subsequently monitored that the number of data class queues and the number of data class to be stored are both zero and then perform packaging processing. It should be noted that the task sorting queue, as mentioned above, is a queue in which the synchronization shards shared by all subtypes of data to be synchronized are queued and wait for scheduling to be inserted into the queue for execution. In actual applications, the scheduling can be maintained by the corresponding task sorting thread, that is, the task sorting thread reads the corresponding number of synchronization shards from the task sorting thread according to the actual application requirements and database operation performance, and adds them to the queue for execution to wait for the execution of the warehousing operation.

[0053] The data class shard size in this embodiment can be understood as the data size of a single shard obtained by sharding the corresponding subtype of data to be synchronized in order to ensure that data with high timeliness requirements can be dynamically inserted into synchronization tasks to improve the synchronization efficiency of high-priority data as much as possible, taking into account the fact that in actual applications, some types of data with low timeliness requirements but large data volumes will occupy database resources for too long when performing synchronization operations at one time, thereby causing other data with high timeliness requirements to have a large synchronization delay. That is, the large amount of data to be synchronized is sharded according to the corresponding data class shard size, and each piece of data is packaged into a data synchronization subtask, that is, a large synchronization task is split into countless small synchronization tasks to leave idle time periods for other higher priority tasks, thereby saving data synchronization time. Specifically, the steps of reading a corresponding number of data to be synchronized from the corresponding data class message queue, constructing corresponding data class synchronization shards and adding them to the task sorting queue include:

[0054] According to the data class shard size, a corresponding number of data to be synchronized is sequentially read from the corresponding data class message queue and filled into the corresponding data class synchronization shard; wherein, the data class synchronization shard can be understood as a data packaging list with a data volume less than or equal to the data class shard size; for example, a data packaging list (initially empty) is established according to the data class shard size, a preset number of items (for example, 100 items) are pulled from the corresponding data class message queue and filled into the data packaging list, and the process of pulling data and filling it into the data packaging list is continuously repeated in a loop until the data packaging list is filled with the length of the data class shard size to obtain a data class synchronization shard, and then the subsequent steps are executed.

[0055] At the same time, in order to prevent the risk that a certain subtype will not be able to fill up the corresponding data packaging list for a long time due to too little data, thereby occupying system processing resources and increasing the data synchronization time, this embodiment preferably sets the maximum waiting time allowed for each type of data packaging process to be synchronized; specifically, according to the size of the data class shard, the steps of sequentially reading the corresponding number of data to be synchronized from the corresponding data class message queue and filling it into the corresponding data class synchronization shard include:

[0056] When the amount of data to be synchronized in the data class message queue is less than the size of the data class fragment, the maximum waiting time timer is started, and when the maximum waiting time timer times out, the reading of the data class message queue is stopped; wherein, the maximum waiting time allowed by the maximum waiting time timer can be determined according to actual application requirements, and the maximum waiting time for each type of data to be synchronized can be the same or different, and is not specifically limited here. In actual applications, when the amount of data to be synchronized in a data class message queue being scheduled for processing is less than the corresponding data class fragment size, it is necessary to start the corresponding maximum waiting time timer, and when the maximum waiting time timer times out, stop reading the data class message queue, that is, if the data packaging list is not fully filled after the maximum waiting time is exceeded, it is also necessary to stop the current packaging operation to obtain a data class synchronization fragment as a subsequent data synchronization subtask.

[0057] According to the data priority in the data class synchronization shard, the data class synchronization shard is prioritized and the labeled data class synchronization shard is added to the task sorting queue; wherein the data priority of the data class synchronization shard can be understood as the priority corresponding to the major data category to which the data to be synchronized and run stored in the shard belongs, that is, the priority corresponding to the data class synchronization shard is determined according to the priority of the data in the synchronization shard, so that the task sorting queue can sort and update according to the priority of the data class synchronization shard.

[0058] According to the preset sorting principle, all data class synchronization fragments in the task sorting queue are reordered; wherein, the preset sorting principle can be understood as the principle of dynamically executing task queuing of the task sorting queue that is preferably set in order to ensure that high-priority data class synchronization fragments can perform synchronous warehousing operations first (dynamic insertion of other tasks is allowed between tasks), while also ensuring that data with the same priority can still maintain the original timing and be smoothly warehousing, specifically including sorting different priorities from high to low and sorting the same priority according to the time of entering the queue; that is, each time a new data class synchronization fragment is inserted into the task sorting queue, a reordering action needs to be performed so that the high priority is placed at the front of the queue as much as possible, and is subsequently read and inserted into the queue to be executed. In practical applications, assuming that the shard size of telemetry data is 2000, the length of the current queue to be executed is 1, and only one task can be transferred from the task sorting queue to the queue to be executed; however, there are currently 20,000 data of circuit breakers (low priority) and 40,000 data of switches (low priority), and the queue to be executed is already executing the first synchronous shard task of the circuit breaker, and the current task sorting queue contains the second synchronous shard (2001-4000) of the circuit breaker and the first synchronous shard (1-2000) of the switch. If a telemetering change task of 100 data is entered at this time, it was originally the last one in the task sorting queue, but after being re-sorted by the task sorting queue, it becomes the first in the task sorting queue because of its high priority, and can be moved to the queue to be executed first later.

[0059] S13. Obtain the maximum number of tasks in the warehousing queue, and according to the maximum number of tasks in the warehousing queue, sequentially read the corresponding number of data class synchronization fragments from the task sorting queue and add them to the waiting warehousing queue; wherein, the maximum number of tasks in the warehousing queue can be understood as the maximum number of fragment synchronization tasks allowed to be stored in the waiting warehousing queue determined according to the database processing performance; correspondingly, the waiting warehousing queue can be understood as a queue that is maintained and scheduled by the warehousing thread and stores executable batch warehousing operation synchronization tasks. In actual applications, the warehousing thread will obtain a certain number of data class synchronization fragments from the above-mentioned task sorting queue according to the actual maximum number of tasks in the warehousing queue, and add them to the waiting warehousing queue, waiting for each data class synchronization fragment to be synchronously stored in the corresponding target table.

[0060] S14. In response to the scheduling of the to-be-executed warehousing queue, sequentially read the data class synchronization shards in the to-be-executed warehousing queue, and insert each data class synchronization shard into the corresponding target table in the preset storage database in batches; wherein, the operation of batch inserting the data class synchronization shards in the to-be-executed warehousing queue into the corresponding target table in the preset storage database can be understood as determining the corresponding target table to be synchronously inserted according to the data subtype of the data class synchronization shard, and then batch writing all the data in the data class synchronization shard into the corresponding target table. The specific batch insertion operation can be implemented with reference to the relevant existing technology and will not be repeated here.

[0061] It should be noted that, in the above embodiments, each thread configuration and queue scheduling method only needs to meet the requirements of temporarily storing the data to be synchronized by introducing a message queue cluster, setting the task priority based on the data synchronization time limit, and combining the data sharding scheduling mechanism that dynamically controls the order of synchronization tasks to ensure that various types of data to be synchronized can be synchronized efficiently on demand. The specific settings can be implemented by referring to the relevant existing technologies and will not be described in detail here.

[0062] The present embodiment provides a technical solution for responding to the reception of various types of operation data to be synchronized, marking the priority of various types of operation data to be synchronized and adding them to the corresponding data class message queue, and when it is determined that the corresponding data class queue number and the data class to be entered into the warehouse are both zero, obtaining the corresponding data class shard size, reading the corresponding number of data to be synchronized from the corresponding data class message queue, constructing the corresponding data class synchronization shards and adding them to the task sorting queue, sequentially reading the corresponding number of data class synchronization shards from the task sorting queue and adding them to the to-be-executed warehouse queue according to the maximum number of tasks in the warehouse queue, and sequentially reading the data class synchronization shards in the to-be-executed warehouse queue in response to the scheduling of the to-be-executed warehouse queue and batch-inserting them into the corresponding target table in the preset storage database. The technical solution introduces a message queue cluster to temporarily store the operation data to be synchronized, and on the basis of setting the task priority based on the data synchronization timeliness, combines the data shard scheduling mechanism that dynamically controls the synchronization task order, so as to ensure that all power system operation data can meet the corresponding data synchronization timeliness application requirements when the storage database resources are limited.

[0063] In addition, considering that during the actual operation of the storage database, there may be other operations such as data statistics or data queries at different times that will occupy database resources and thus affect the performance of data synchronization insertion, that is, the amount of data that can be inserted into the storage database at different times may be different. In order to ensure that the storage database is not overloaded and maximize the efficiency of database resource utilization, this embodiment preferably adaptively adjusts the batch synchronization data volume (data class shard size) of each type of data to be synchronized and the maximum amount of inserted data supported by the storage database (maximum number of tasks in the storage queue) according to the actual insertion performance when the storage database executes batch insertion of synchronization shards of different data types at different time periods. Specifically, Figure 4 As shown, the method also includes:

[0064] S15. In response to the completion of batch insertion of each data class synchronization shard, update the corresponding data class shard size and the maximum number of tasks in the storage queue; wherein, the update of the data class shard size and the maximum number of tasks in the storage queue can be understood as re-determining the batch synchronization data volume of the corresponding type of data in the subsequent synchronization process and the maximum insertion data volume of the storage queue to be executed according to the actual storage performance of inserting different types of data synchronization shards in the storage queue to be executed in the storage thread; specifically, the step of updating the corresponding data class shard size and the maximum number of tasks in the storage queue in response to the completion of batch insertion of each data class synchronization shard includes:

[0065] When the batch insertion of each data class synchronization shard is completed, the corresponding data class synchronization data volume and data synchronization duration are obtained, and the data class synchronization data volume and the data class synchronization duration are divided to obtain the corresponding data class insertion number per second; wherein, the data class synchronization data volume can be understood as the number of data in the current data class synchronization shard; the corresponding data synchronization duration can be understood as the total time consumed by the data class synchronization shard to insert into the storage database based on the timer (that is, data synchronization duration = the end time of the shard data insertion into the database - the start time of the shard data insertion into the database), and the specific method of using the timer to monitor the data synchronization duration can be implemented by referring to the relevant prior art. The number of data class insertions per second in this embodiment can be understood as an indicator that reflects the data insertion capability of the database at the corresponding moment, specifically expressed as: data class insertion number per second = data class synchronization data volume / data synchronization duration.

[0066] According to the rounded value of the number of insertions per second of the data class, the corresponding data class shard size is updated; the specific data class shard size update value is expressed as: updated data class shard size = rounded [data class insertions per second].

[0067] According to the number of insertions per second of the data class and the amount of data of all synchronized shards of the data class in the task sorting queue, the current maximum number of shards allowed to be processed is obtained, and according to the current maximum number of shards allowed to be processed, the maximum number of tasks of the warehousing queue is updated; wherein, the current maximum number of shards allowed to be processed can be understood as the maximum number of synchronized shards allowed to be added to the subsequent warehousing queue to be executed that can be determined based on the data insertion performance of the current storage database; specifically, the step of obtaining the current maximum number of shards allowed to be processed according to the number of insertions per second of the data class and the amount of data of all synchronized shards of the data class in the task sorting queue includes:

[0068] Sequentially obtain the sum of the sorted data volume of each data class synchronization shard and all previous data class synchronization shards in the task sorting queue;

[0069] The maximum sum of sorted data volumes that is less than the number of insertions per second of the data class is obtained, and the number of synchronized shards of the data class corresponding to the maximum sum of sorted data volumes is used as the maximum number of shards currently allowed to be processed.

[0070] In order to understand the above process of obtaining the maximum number of slices currently allowed to be processed, the telemetry data and telesignal data are used as examples for explanation below; if the number of data class insertions per second obtained through the above steps is 3400 / second, and there are 3 tasks in the task sorting queue, the synchronization slice of the high-priority telesignal change (100 data), the second synchronization slice of the low-priority circuit breaker (2000 data) and the first synchronization slice of the low-priority knife switch (2000 data); then after statistical analysis, it is found that 200+2000<3400<200+2000+2000, then it is considered that the current maximum number of slices allowed to be processed corresponding to the subsequent queue to be executed can only be two synchronization slices, that is, the synchronization slice of the telesignal change and the second synchronization slice of the circuit breaker can be moved from the task sorting queue to the queue to be executed during the next scheduling of the entry thread, while the first synchronization slice of the knife switch needs to wait for the next scheduling.

[0071] The present application embodiment Figure 5In response to the reception of various types of data to be synchronized, the various types of data to be synchronized are marked with priorities and added to the corresponding data class message queues, and when it is determined that the corresponding data class queue number and the data class to be entered into the warehouse are both zero, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, the corresponding data class synchronization shards are constructed and added to the task sorting queue, and according to the maximum number of tasks in the entry queue, the corresponding number of data class synchronization shards are sequentially read from the task sorting queue and added to the entry queue to be executed, and in response to the scheduling of the entry queue to be executed, the data class synchronization shards in the entry queue to be executed are sequentially read and batch-inserted into the corresponding target table in the preset storage database, and in response to the completion of the batch insertion of each data class synchronization shard, the corresponding data class shard size and the maximum number of tasks in the entry queue are updated. This solution effectively solves the existing Various types of telemetry data and telesignaling data in the power system operation data are executed synchronously and concurrently, and the storage of telemetry data with many types and large data volume takes a long time, which seriously affects the timeliness of the storage of telesignaling data, and thus affects the smooth execution of subsequent real-time analysis tasks of the system operation status. By introducing a message queue cluster to temporarily store the operation data to be synchronized, and setting the task priority based on the data synchronization timeliness, combined with the data sharding scheduling mechanism that dynamically controls the order of synchronization tasks and the batch synchronization data volume adaptive adjustment mechanism based on database data insertion performance monitoring, it can not only ensure that all power system operation data can meet the corresponding data synchronization timeliness when the storage database resources are limited, but also effectively balance the storage database load and resource utilization efficiency, thereby providing technical support for the efficient and reliable synchronization of power system operation data.

[0072] It should be noted that although the steps in the above flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders.

[0073] In one embodiment, Figure 6 As shown, a power system operation data synchronization system is provided, the system comprising:

[0074] The data receiving module 1 is used for responding to the reception of various types of data to be synchronized, marking the priority of various types of data to be synchronized and adding them to the corresponding data class message queue, and obtaining the corresponding data class queue number and data class waiting to be stored number; the various types of data to be synchronized include telesignaling data and telemetry data;

[0075] Task sorting module 2 is used to determine whether the data class queue number and data class waiting to be stored corresponding to each type of data to be synchronized are both zero. If so, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, and the corresponding data class synchronization shard is constructed and added to the task sorting queue. Otherwise, the corresponding data class queue number and data class waiting to be stored are continuously monitored;

[0076] Task scheduling module 3, used to obtain the maximum number of tasks in the warehousing queue, and according to the maximum number of tasks in the warehousing queue, sequentially read the corresponding number of data class synchronization shards from the task sorting queue and add them to the warehousing queue to be executed;

[0077] The synchronous warehousing module 4 is used to respond to the scheduling of the waiting-to-be-executed warehousing queue, sequentially read the data class synchronization fragments in the waiting-to-be-executed warehousing queue, and batch-insert each data class synchronization fragment into the corresponding target table in the preset storage database.

[0078] In one embodiment, Figure 7 As shown, a power system operation data synchronization system is provided, and the system also includes:

[0079] The performance monitoring module 5 is used to update the corresponding data class shard size and the maximum number of tasks in the storage queue in response to the completion of batch insertion of each data class synchronization shard.

[0080] For the specific limitations of the power system operation data synchronization system, please refer to the limitations of the power system operation data synchronization method above, and the corresponding technical effects can also be obtained equivalently, which will not be repeated here. Each module in the above-mentioned power system operation data synchronization system can be implemented in whole or in part through software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0081] Figure 8 FIG. 1 shows an internal structure diagram of a computer device in an embodiment, and the computer device may specifically be a terminal or a server. Figure 8As shown, the computer device includes a processor, a memory, a network interface, a display, a camera and an input device connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for synchronizing power system operation data is implemented. The display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the computer device housing, or an external keyboard, touchpad or mouse, etc.

[0082] It can be understood by those skilled in the art that Figure 8 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computing device may include more or fewer components than those shown in the figure, or combine certain components, or have the same component arrangement.

[0083] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the steps of the above method are implemented when the processor executes the computer program.

[0084] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0085] In summary, an embodiment of the present invention provides a method, system, device and storage medium for synchronizing operation data of an electric power system. The method for synchronizing operation data of an electric power system realizes that in response to the reception of various types of operation data to be synchronized, various types of operation data to be synchronized are marked with priorities and added to the corresponding data class message queues, and the corresponding data class queue numbers and data class to-be-stored numbers are obtained. It is determined whether the data class queue numbers and data class to-be-stored numbers corresponding to various types of operation data to be synchronized are both zero. If so, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, the corresponding data class synchronization shard is constructed and added to the task sorting queue. Otherwise, the corresponding data class queue numbers and data class to-be-stored numbers are continued to be monitored. , obtain the maximum number of tasks in the storage queue, and according to the maximum number of tasks in the storage queue, sequentially read the corresponding number of data class synchronization slices from the task sorting queue to add to the storage queue to be executed, and in response to the scheduling of the storage queue to be executed, sequentially read the data class synchronization slices in the storage queue to be executed, and batch insert each data class synchronization slice into the corresponding target table in the preset storage database. The technical solution introduces a message queue cluster to temporarily store the operation data to be synchronized, and on the basis of setting the task priority based on the data synchronization timeliness, combines the data slice scheduling mechanism that dynamically controls the synchronization task order, so as to ensure that all power system operation data can meet the corresponding data synchronization timeliness application requirements when the storage database resources are limited.

[0086] Each embodiment in this specification is described in a progressive manner, and the same or similar parts of each embodiment can be directly referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. It should be noted that the technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, all possible combinations of the technical features in the above-mentioned embodiments are not described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0087] The above-mentioned embodiments only express several preferred implementation modes of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for ordinary technicians in the technical field, several improvements and substitutions can be made without departing from the technical principles of the present invention, and these improvements and substitutions should also be regarded as the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be based on the protection scope of the claims.

Claims

1. A method for synchronizing power system operation data, characterized in that: The method comprises the following steps: In response to receiving various types of data to be synchronized, the various types of data to be synchronized are marked with priorities and added to the corresponding data class message queues, and the corresponding data class queue numbers and data class waiting to be stored numbers are obtained; the various types of data to be synchronized include telesignaling data and telemetry data; Determine whether the data class queue number and data class waiting to be stored corresponding to each type of data to be synchronized are both zero. If so, obtain the corresponding data class shard size, read the corresponding number of data to be synchronized from the corresponding data class message queue, build the corresponding data class synchronization shard and add it to the task sorting queue. Otherwise, continue to monitor the corresponding data class queue number and data class waiting to be stored. Obtain the maximum number of tasks in the warehousing queue, and according to the maximum number of tasks in the warehousing queue, sequentially read the corresponding number of data class synchronization shards from the task sorting queue and add them to the warehousing queue to be executed; In response to the scheduling of the to-be-executed warehousing queue, the data class synchronization fragments in the to-be-executed warehousing queue are sequentially read, and each data class synchronization fragment is batch-inserted into a corresponding target table in a preset storage database.

2. The method for synchronizing power system operation data according to claim 1, characterized in that: The step of marking the priority of various types of data to be synchronized includes: The data types of various types of to-be-synchronized running data are obtained respectively, and the corresponding to-be-synchronized running data are marked with priorities according to the task synchronization timeliness requirements corresponding to the data types.

3. The method for synchronizing power system operation data according to claim 1, characterized in that: The steps of reading a corresponding number of to-be-synchronized data from a corresponding data class message queue, constructing corresponding data class synchronization fragments and adding them to a task sorting queue include: According to the size of the data class shard, sequentially read the corresponding number of data to be synchronized from the corresponding data class message queue and fill it into the corresponding data class synchronization shard; According to the data priority in the data class synchronization shard, the data class synchronization shard is marked with a priority, and the marked data class synchronization shard is added to the task sorting queue; According to the preset sorting principle, all data class synchronization fragments in the task sorting queue are reordered; the preset sorting principle includes sorting different priorities from high to low and sorting the same priority according to the queue entry time.

4. The method for synchronizing power system operation data according to claim 1, characterized in that: The step of sequentially reading a corresponding number of to-be-synchronized data from a corresponding data class message queue and filling the corresponding data class synchronization shard according to the size of the data class shard comprises: When the amount of data to be synchronized in the data class message queue is smaller than the size of the data class shard, a maximum waiting time timer is started, and when the maximum waiting time timer times out, reading of the data class message queue is stopped.

5. The method for synchronizing power system operation data according to claim 1, characterized in that: The method further comprises: In response to the completion of batch insertion of each data class synchronization shard, the corresponding data class shard size and the maximum number of tasks in the storage queue are updated.

6. The method for synchronizing power system operation data according to claim 5, characterized in that: In response to the completion of batch insertion of each data class synchronization shard, the step of updating the corresponding data class shard size and the maximum number of tasks in the storage queue includes: When the batch insertion of each data class synchronization shard is completed, the corresponding data class synchronization data volume and data synchronization duration are obtained, and the data class synchronization data volume and the data class synchronization duration are divided to obtain the corresponding data class insertion number per second; According to the rounded value of the number of insertions per second of the data class, the corresponding data class shard size is updated; According to the number of insertions per second of the data class and the amount of data in all data class synchronization shards in the task sorting queue, the current maximum number of shards allowed to be processed is obtained, and according to the current maximum number of shards allowed to be processed, the maximum number of tasks in the warehousing queue is updated.

7. The method for synchronizing power system operation data according to claim 6, characterized in that: The step of obtaining the maximum number of shards currently allowed to be processed according to the number of insertions per second of the data class and the amount of data of all synchronized shards of the data class in the task sorting queue comprises: Sequentially obtain the sum of the sorted data volume of each data class synchronization shard and all previous data class synchronization shards in the task sorting queue; The maximum sum of sorted data volumes that is less than the number of insertions per second of the data class is obtained, and the number of synchronized shards of the data class corresponding to the maximum sum of sorted data volumes is used as the maximum number of shards currently allowed to be processed.

8. A power system operation data synchronization system, characterized in that: The system comprises: A data receiving module is used to respond to the reception of various types of data to be synchronized, mark the priority of various types of data to be synchronized, add them to the corresponding data class message queue, and obtain the corresponding data class queue number and data class waiting to be stored number; the various types of data to be synchronized include telesignaling data and telemetry data; The task sorting module is used to determine whether the data class queue number and data class waiting to be stored corresponding to each type of data to be synchronized are both zero. If so, the corresponding data class shard size is obtained, and the corresponding number of data to be synchronized is read from the corresponding data class message queue, and the corresponding data class synchronization shard is constructed and added to the task sorting queue. Otherwise, the corresponding data class queue number and data class waiting to be stored are continuously monitored. The task scheduling module is used to obtain the maximum number of tasks in the storage queue, and according to the maximum number of tasks in the storage queue, sequentially read the corresponding number of data class synchronization shards from the task sorting queue and add them to the storage queue to be executed; The synchronous warehousing module is used to respond to the scheduling of the waiting-to-be-executed warehousing queue, sequentially read the data class synchronization fragments in the waiting-to-be-executed warehousing queue, and batch-insert each data class synchronization fragment into the corresponding target table in the preset storage database.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images