Distribution network dispatching disaster recovery data processing system

Abstract

The invention discloses a data processing system for distribution network dispatching disaster recovery. The data processing system comprises a data acquisition cluster, a data aggregation cluster and a data forwarding cluster, the data acquisition cluster is used for starting a corresponding number of data acquisition micro-services according to the grouping number of the acquisition channels, receiving message data uploaded by the terminal equipment through the data acquisition micro-services, acquiring first data of the power distribution network equipment from the message data, and sending the first data to the data aggregation cluster; the data aggregation cluster is used for starting a corresponding number of data aggregation micro-services according to the number of partitions of the acquisition equipment, aggregating the first data through the data aggregation micro-services, and preprocessing the first data to obtain second data; and the data forwarding cluster is used for integrating all the second data into third data and forwarding the third data to the target receiver. According to the invention, mass data can be processed, and the requirement of mass data processing can be met, so that the data processing efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing system for distribution network scheduling disaster recovery. Background Art

[0002] The functions, modes and forms of distribution network dispatching business are constantly undergoing great changes. A large number of emerging grid-connected entities such as small and medium-sized distributed new energy, adjustable loads and new energy storage will be added to the distribution network, making the data collection and exchange process of the regional distribution network dispatching system in the future complex, the operating load high, and difficult to continuously access massive amounts of data.

[0003] At present, the prefecture-level distribution network dispatching system has built multiple security zones, and uses the main and backup service mode to process different types of data in a partitioned manner. The data collection and processing capabilities are constrained by the resources of a single server and cannot be continuously improved. New technologies such as "cloud, big data, Internet of Things, mobile and intelligence" are constantly being studied and applied by power dispatching automation experts to the field of intelligent power grid dispatching, especially in the network / provincial dispatching system, to achieve integrated monitoring, control and management of a larger range of power grids.

[0004] The existing data processing system for distribution network dispatching and disaster recovery is difficult to meet the processing needs of massive data, resulting in low data processing efficiency. Summary of the invention

[0005] The present invention provides a data processing system for distribution network dispatching disaster recovery, so as to solve the technical problem that the existing data processing system for distribution network dispatching disaster recovery is difficult to meet the processing requirements of massive data, resulting in low data processing efficiency.

[0006] The present invention provides a data processing system for distribution network scheduling disaster recovery, comprising:

[0007] Data collection cluster, data aggregation cluster and data forwarding cluster;

[0008] The data collection cluster is used to start a corresponding number of data collection microservices according to the number of groups of the collection channel, receive the message data uploaded by the terminal device through the data collection microservice, obtain the first data of the distribution network device from the message data, and send the first data to the data aggregation cluster;

[0009] The data aggregation cluster is used to start a corresponding number of data aggregation microservices according to the number of partitions of the acquisition device, aggregate the first data through the data aggregation microservice, pre-process the first data, and obtain the second data;

[0010] The data forwarding cluster is used to integrate all second data into third data and forward the third data to a target receiver.

[0011] Furthermore, the data processing system also includes a channel management cluster, which is used to create a channel polling thread according to the channel parameters of the acquisition channel, and monitor the channel status of the acquisition channel through the channel polling thread, wherein the channel parameters include data transmission frequency and data type.

[0012] Furthermore, the data processing system also includes a message management cluster, which is used to identify the message type of each message data and store each message data according to the message type.

[0013] Furthermore, the data processing system also includes a task scheduling cluster, which is used to deploy the data collection cluster and the data aggregation cluster according to the labels set on the task nodes, and deploy the data forwarding cluster, the channel management cluster and the message management cluster in a one-master-multiple-standby manner.

[0014] Furthermore, the data collection cluster is also used to verify the acquired first data, and generate a missing data alarm when the first data verification fails.

[0015] Furthermore, the preprocessing includes data cleaning, data conversion and data section storage.

[0016] Furthermore, the data forwarding cluster is also used for:

[0017] The third data is forwarded to a target receiver at a preset time interval, and the third data is checked during the forwarding process. When it is found that the third data is lost or corrupted, a retransmission mechanism is automatically triggered.

[0018] Furthermore, the task scheduling cluster is also used to estimate the resource demand for a preset time period based on the maximum likelihood distribution estimation method according to the current system load and historical data, and adjust the number of microservices in each cluster according to the resource demand for the preset time period.

[0019] The data acquisition cluster of the embodiment of the present invention dynamically starts a corresponding number of data acquisition microservices according to the number of groups of the acquisition channel, and the data aggregation cluster starts a corresponding number of data aggregation microservices according to the number of partitions of the acquisition device. It can start corresponding data processing resources according to actual data processing needs, thereby effectively improving resource utilization and ensuring the real-time and accuracy of data processing. It can also process massive data and meet the needs of massive data processing, thereby effectively improving data processing efficiency.

[0020] Furthermore, the task scheduling cluster of an embodiment of the present invention is based on the maximum likelihood distribution estimation method. It estimates the resource requirements for a preset time period according to the current system load and historical data, and adjusts the number of microservices in each cluster according to the resource requirements for the preset time period. Through maximum likelihood estimation, it can predict future resource requirements based on current and historical data, thereby dynamically adjusting resource allocation in different time periods and optimizing resource utilization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a structural diagram of a data processing system for distribution network scheduling disaster recovery provided by an embodiment of the present invention;

[0022] Figure 2 This is a diagram of the microservice operation architecture of the cloud-based front-end system provided by an embodiment of the present invention;

[0023] Figure 3 This is a diagram of the data collection microservice component architecture provided by an embodiment of the present invention;

[0024] Figure 4 This is a diagram of the data aggregation microservice component architecture provided by an embodiment of the present invention;

[0025] Figure 5 This is a diagram of the data forwarding microservice component architecture provided by an embodiment of the present invention;

[0026] Figure 6 This is a diagram of the channel management microservice component architecture provided by an embodiment of the present invention;

[0027] Figure 7 is a channel link state transition diagram provided by an embodiment of the present invention;

[0028] Figure 8 This is a diagram of the message management microservice component architecture provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the feature. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0031] In the description of this application, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in this application can be understood according to specific circumstances.

[0032] The embodiment of the present invention is applicable to a cloud-based front-end system for distribution network scheduling and disaster recovery.

[0033] See also Figure 1 The present invention provides a data processing system for distribution network scheduling disaster recovery, including:

[0034] Data collection cluster, data aggregation cluster and data forwarding cluster;

[0035] In an embodiment of the present invention, the data collection cluster, the data aggregation cluster and the data forwarding cluster all support event tracing, which is a distributed event storage and analysis technology that flexibly schedules the operating status of microservices and key business operations. In the cloud-based front-end microservice collection task cluster, events (Event) are defined as operating events such as the collection microservice status and the collection channel status, which are published to the operating message queue according to the time of state change. The data aggregation service includes an event aggregation component (EventAggregate), which can cache events for nearly 1 hour based on a distributed real-time library, and supports loading channel optimization components in the form of snapshots; the data aggregation service includes an event storage component (EventStore), which stores events in real time in a time series database.

[0036] In the embodiment of the present invention, the cloud-based front-end supports the management of multiple collection domains, and multiple clusters are run collaboratively in each domain to meet the needs of seamless expansion of massive channels.

[0037] Each collection domain can implement a collection subsystem for the dispatch of a local distribution network. The microservices in the collection domain do not interact with the microservices in other collection domains, thus achieving data isolation. Figure 2 , Figure 2The figure shows a microservice operation architecture diagram of a cloud-based front-end system for provincial distribution network dispatching and disaster recovery.

[0038] The data collection cluster is used to start a corresponding number of data collection microservices according to the number of groups of the collection channel, receive the message data uploaded by the terminal device through the data collection microservice, obtain the first data of the distribution network device from the message data, and send the first data to the data aggregation cluster;

[0039] In an embodiment of the present invention, the data collection cluster collects data of the corresponding equipment of the distribution network scheduling disaster recovery through a preset collection channel, and the collection channel is used to transmit the data of the corresponding distribution network equipment. The collection channel includes a duty channel and other channels, wherein the duty channel is the channel currently in use. The embodiment of the present invention can filter customized important data from other channels through preset filtering rules.

[0040] The embodiment of the present invention can also receive message data uploaded by other systems. The data collection cluster can add a processing component of a lightweight Internet of Things protocol on the basis of parsing the power protocol.

[0041] See also Figure 3 , in the embodiment of the present invention, Figure 3 This is an architecture diagram of a data collection microservice component in a data collection cluster.

[0042] The embodiment of the present invention can add "event snapshot" and "pre-parameter cache" to the distributed real-time library to drive the operation of other components. The "acquisition protocol component" in the acquisition group implements the protocol interpretation of the message and the framing and sending of the downlink message. The protocol includes common power protocols and Internet of Things protocols; the "device link component" completes the start and stop control of the communication link of the acquisition device, and realizes the basic function of channel message reception and transmission. The acquisition group initializes the component according to the grouping of the pre-parameters; the message generated data parsed by the acquisition group is published to the running message queue through the raw data publishing component. Among them, the raw data is the unprocessed original data.

[0043] Figure 3 The data collection microservice shown can realize concurrent processing of data flows between components within the service. Including: Southbound to the collection device: the forwarding processing component subscribes to the message to be forwarded, cooperates with the corresponding collection group, and forwards the data; the command issuing component subscribes to the control command to be forwarded, and cooperates with the corresponding collection group to forward the data. Northbound to other collection and processing services: the data publishing component obtains the data processed by the collection group and publishes the data to the running message queue; the message publishing component obtains the message processed by the collection group and publishes it to the running message queue; the control response publishing component obtains the control response information received by the collection group and publishes the data to the running message queue.

[0044] Figure 3The device links shown are managed by channel groups. The newly added link parameters in each group can be dynamically synchronized to the pre-parameter cache component of the related running microservice; the new channel group parameters can also be synchronized to the data cache component in real time. The collection group collaboration module of the data collection microservice regularly accesses the pre-parameter cache to drive the creation, operation and destruction of the collection group.

[0045] Figure 3 The event tracing component of the data acquisition microservice shown in the figure periodically loads the events of the device link-related channels from the distributed real-time library into the "event snapshot", and periodically publishes its own operation events to the operation message queue.

[0046] A data aggregation cluster is used to start a corresponding number of data aggregation microservices according to the number of partitions of the collection device, aggregate the first data through the data aggregation microservice, pre-process the first data, and obtain the second data;

[0047] See also Figure 4 , in the embodiment of the present invention, Figure 4 The figure shows a data aggregation microservice component architecture diagram in a data aggregation cluster. "Event snapshot" and "pre-parameter cache" can be installed from the distributed real-time library to drive the operation of other components. The section service provides access to the acquisition equipment and channel section data interface; the alarm component handles abnormal events in the channel multi-source processing process and generates alarm information; the master-slave coordination component determines whether the aggregation microservice is the master service or the backup service.

[0048] Figure 4 The data section component shown uses the section snapshot of the dynamic pointer to cache the collected section data. Define point as the measurement point, point.id as the globally unique point ID of the measurement point; JObject is a JSON type of data object collected by the measurement point, which can be represented as any row of data records containing point ID, collection millisecond time, collection value, flag, etc.; table is the name of the section data table. The section snapshot adopts a three-layer structure:

[0049] a) Value: Record = shared_ptr <jobject>

[0050] b) Point value table: ValueTable = map <point.id,shared_ptr <record>>

[0051] c) Section snapshot library: SectionSnap=map <table,shared_ptr <valuetable>>

[0052] Figure 4 The data section component shown provides commonly used operation interfaces: put(JObjects) to submit batch records, update(JObject) to update records, get(pointid) to obtain specified records, get_station(stationid) to obtain records of the entire station, batch_get(pointids) to batch obtain records of a specified point ID set, and select(condition) to query records based on conditions.

[0053] Figure 4 The stream processing group shown in the figure completes the processing of various data streams in the business logic in groups, and concurrently subscribes to messages from different topics (Topics) of the running message queue within the converged microservice.

[0054] Figure 4 The stream processing group 1 shown: in the raw data processing stream, the raw data is preferably first obtained from the duty channel and loaded into the raw and cooked data queue; the raw and cooked data conversion component obtains the raw data calculation in the pre-parameter drive queue to generate the cooked data queue; the cooked data publishing component publishes the data in the cooked data queue to the running message queue in a first-in-first-out manner.

[0055] Figure 4 In the stream processing group 2: channel status processing and dead data detection stream shown, when the running status update component is started, it identifies the status of the relevant collection microservices and collection channels in the event snapshot. If the status is abnormal, it waits for 1 second and then restarts. After restarting the component three times in a row, a service abnormality alarm message is published; if it starts normally, the running status event is updated, and the dead data detection component is driven to analyze the cooked data queue. If all cooked data are the same, the dead data alarm message of the collection device is published; the running status event can also drive the data section storage component, subscribe to the published cooked data, and maintain the aggregated data section.

[0056] Figure 4 Flow processing group 3 shown: In the request processing flow, the SCADA data request component subscribes to the resend request signal issued by other microservices, drives the data section resend component after correct analysis, and publishes all the cooked data in the aggregated data section to the running message queue.

[0057] Figure 4 The event tracing component of the data aggregation microservice shown in the figure periodically loads the collection microservice status and collection channel status from the distributed real-time library into the "event snapshot", and periodically publishes its own master and backup status and abnormal status to the running message queue.

[0058] The data forwarding cluster is used to integrate all the second data into third data and forward the third data to a target receiver.

[0059] See also Figure 5 , in the embodiment of the present invention, Figure 5 The figure shows a data forwarding microservice component architecture diagram in a data forwarding cluster. Components such as message subscription, message publishing, section service, alarm component, event tracing, data section, master-slave coordination component and cache are reusable common components.

[0060] Figure 5 The stream processing group shown in the figure completes the processing of various data streams in the business logic in groups, and concurrently subscribes to messages from different topics (Topics) of the running message queue within the forwarding microservice.

[0061] Figure 5 Flow processing group 1 shown: In the whole-station forwarding flow, first access the section service of the corresponding plant station in the data aggregation service cluster according to the plant station index, and load the whole-station section cache; forward invalid data found by the data verification component, and then access the data section of the distributed real-time library to load normal data; finally, the section publishes all data through the whole-station channel forwarding component.

[0062] Figure 5 The stream processing group 2 shown: in the point forwarding flow, the point data identification component obtains the ID set (pointids) of the collected measurement points according to the point information, and subscribes to the data to be forwarded from the running message queue; the point section maintenance component calls the put (JObjects) of the data section component to maintain the forwarding data section; finally, the section publishes all the data through the section channel forwarding component.

[0063] The data acquisition cluster of the embodiment of the present invention dynamically starts a corresponding number of data acquisition microservices according to the number of groups of the acquisition channel, and the data aggregation cluster starts a corresponding number of data aggregation microservices according to the number of partitions of the acquisition device. It can start corresponding data processing resources according to actual data processing needs, thereby effectively improving resource utilization and ensuring the real-time and accuracy of data processing. It can also process massive data and meet the needs of massive data processing, thereby effectively improving data processing efficiency.

[0064] In one embodiment, the data processing system also includes a channel management cluster, which is used to create a channel polling thread according to the channel parameters of the acquisition channel, and monitor the channel status of the acquisition channel through the channel polling thread, wherein the channel parameters include data transmission frequency and data type.

[0065] See also Figure 6 , in the embodiment of the present invention, Figure 6 The figure shows a channel management microservice component architecture diagram in the channel management cluster. Components such as message subscription, message publishing, section service, alarm component, event tracing, statistics cache service and cache are reusable public components.

[0066] Figure 6 The statistical cache service shown in the figure analyzes the channel parameters in the pre-parameter cache, creates measurement points for statistics and analysis on demand, and can store the statistical and analytical result data of each component in the stream processor group (the data structure is consistent with the cross-section snapshot), and store the changed data in the Channel_Stat table of the distributed real-time library. When the channel management microservice is started, the required initialization data is loaded from the Channel_Stat table according to the pre-parameters, which facilitates seamless switching when the primary and standby channel managers switch.

[0067] Figure 6 The master-slave coordination component shown in the figure determines whether the channel management microservice is the master service or the backup service. Normally, the master channel management microservice completes most of the channel management, statistical calculation, and command processing functions. The backup channel manager only needs to passively receive messages pushed by the master channel manager, update its own status and statistical cache services, and keep the cross-section data consistent with the master channel manager. When the backup service switches to the master service, it is a key point to ensure the synchronization of the position of the pre-command message queue of the command parsing component. The design uses the process identifier of the channel management microservice to subscribe to the channel original message statistics queue message. The message queue will return the queue offset position of the latest consumption of the original master service; the new master channel manager consumes data from this position, thereby achieving seamless switching of consumption data.

[0068] Figure 6 The stream processing group shown in the figure completes the processing of various data streams in the business logic in groups, and concurrently subscribes to messages from different topics of the running message queue within the channel management microservice.

[0069] Figure 6 In the flow processing group 1: channel state management flow shown, the channel monitoring component analyzes the data in the event tracing and statistical cache to determine the initial state of the device link channel in the collection microservice. The fault detection component performs path fault / recovery detection based on the current link state and real-time message statistics, and distinguishes between receiving and forwarding paths. The cause of the path fault must be identified, and the dead data and dead data recovery of the path must be identified; the data of the path detection must maintain the statistical cache service, including the last received and sent frame number, initial bit error rate time, real-time bit error rate, initial fault time, initial fault flag, initial recovery time, initial recovery flag, etc.

[0070] Figure 6 In the flow processing group 1: channel state management flow shown in the figure, the path decision component analyzes the path detection results, and realizes the path state transition by collecting the path state change (failure / recovery) of the device and other operation events as the driving events through the embedded state transition diagram. Figure 7 Path determination status diagram. If two or more paths are determined to be normal, that is, multiple paths are running at the same time, the path decision problem will be faced. The path decision component selects the path with the highest dynamic priority as the duty channel of the main path. The dynamic priority is calculated by comprehensively considering various factors such as channel priority, path priority, path fault data quality, and statistical data quality. The duty publishing component maintains the duty channel information in the front cache, synchronously maintains the data in the distributed real-time library, and publishes the duty channel to the running message queue.

[0071] Figure 6 In the flow processing group 2: channel operation statistics flow shown in the figure, the collection group operation statistics provide real-time status statistics of the collection link operation, the dead data alarm statistics component provides real-time device alarm statistics, and the path state transition statistics component counts the real-time path state transition. Typical link state transitions can be seen Figure 7 The acquisition link has flexible four-state conversion functions such as initialization, normal, disabled, and closed. The link statistics provide the state section statistics of all links in the system, such as the number of disabled links and details; the path state conversion statistics are the historical link status of a single acquisition device.

[0072] Figure 6 The flow processing group 3 shown: in the pre-command processing flow, the command parsing and verification component subscribes to the command issuance request command from the running message bus, performs command parsing and security analysis, and verifies permissions; the command issuance component verifies the status of the link to be issued, and when the link is normal, the instructions that can be issued can be issued.

[0073] In one embodiment, the data processing system further includes a message management cluster, which is used to identify a message type of each message data and store each message data according to the message type.

[0074] See also Figure 8 , in the embodiment of the present invention, Figure 8 The figure shows a message management microservice component architecture diagram in the message management cluster. Components such as message subscription, message publishing, event tracing, file storage service, object storage service, and cache are reusable public components.

[0075] Figure 6 The file storage service shown in the figure accesses the file service interface and stores messages in a specific location of the distributed file system. A common method is to place messages in the structure of the operation and maintenance management unit to which the collected devices belong, such as "Guangzhou / Panyu / Xiaoguwei Street / Guangdong Science Center / ". The object storage service accesses the object storage service interface and stores messages in a specific location of the distributed storage system.

[0076] Figure 8 The flow processing group 1 shown in the figure: in the message classification storage flow, the message cache component caches the messages generated by the data collection microservice message publishing component to prevent frequent file writing from causing system IO busy; the message storage component identifies the type of service configuration information summary storage service, determines whether to call the file storage service or the object storage service, and realizes batch storage of messages. When caching and storing messages, the collection protocol library should be called to identify the message type, and the sent and received messages should be saved according to the type requirements of the channel message storage. The message type supports at least the following categories: all messages, telemetry messages, telesignaling messages, SOE messages, power messages, remote control messages, remote adjustment messages, link control messages, etc. The message cleaning component has a complete expired message file cleaning mechanism, and can be configured to back up to other devices before cleaning files. The file index publishing organization publishes the location information of the stored messages to the running message queue.

[0077] Figure 8 The flow processing group 2 shown in the figure: in the CASE generation processing flow, the CASE time tag processing component subscribes to the CASE trigger event generated by SCADA and generates a CASE message containing a time tag. After receiving the CASE generation notification message, the message cutting component records the "cut timestamp" and intercepts the message files within a period of time before and after the timestamp to form a CASE file of the channel message. The CASE file storage service identifies the type of service configuration information summary storage service, determines whether to call the file storage service or the object storage service, and realizes batch storage of CASE messages. The CASE trigger event includes at least: accident change, important telesignal change, telemetry mutation, etc. In order to prevent the conflict between message cutting and file operation of message storage, the file access needs to be locked when the message is cut.

[0078] In one embodiment, the data processing system also includes a ground resource occupancy model, which is compatible with the CIN model and the physical model and can be integrated into a "collection protocol component" of a lightweight Internet of Things protocol. The embodiment of the present invention uses an information model based on the JSON format, and can reasonably divide and classify models of equipment in different fields such as wind power stations, photovoltaic stations, energy storage stations, and charging stations according to primary equipment and secondary functions: the equipment model of primary equipment such as switches, transformers, inverters, etc. should contain information such as operating parameters, operating status, and online monitoring; the secondary function model should contain functional information such as collection measurement, relay protection, and operation analysis; the name of the secondary measurement point should adopt the data object naming of the ICD model of DL / T 860, and clearly correspond to the DO object in the ICD model.

[0079] Taking the photovoltaic station JSON format integrated terminal collection and exchange information model as an example, it is classified according to the station, equipment, measurement point and collection data:

[0080] 1) Station model: PV station operation monitoring model, with typical collection and measurement covering operation, commissioning, shutdown, fault / alarm, power limit and other states, as well as daily, monthly, annual and total power generation statistics.

[0081] 2) Equipment model: contains three parts of information: schema, profile, and services. There are five types of services: parameter, analog, discrete, control, and accumulator. Each service contains several collected data items. The equipment in the PV station covers equipment parameter models such as grid-connected switches, step-up transformers, inverters, combiner boxes, collector lines, and PV panels, as well as equipment connection relationship models contained in the parameters.

[0082] 3) Measurement point model: telemetering, telesignaling, remote control, remote adjustment and accumulated quantity models. The measurement point model needs to be associated with a device model and the acquisition serial number of the point in the device. The measurement points and acquisition serial numbers in a device correspond one to one.

[0083] 4) Collection data: including collection sequence number, collection value, millisecond time, data type and other attributes.

[0084] The terminal device can be configured with station models, equipment models and measurement point models. After the connection relationship is determined with the cloud front-end, the full model and full collected data are automatically uploaded, and the changed model and changed data are uploaded in real time. The full model and data can be configured to be uploaded once a day, which can support the realization of plug-and-play terminals and efficient adaptive updates of medium and low voltage power grid topologies.

[0085] In one embodiment, the data processing system also includes a task scheduling cluster, which is used to deploy data collection clusters and data aggregation clusters according to labels set on task nodes, and to deploy data forwarding clusters, channel management clusters and message management clusters in a one-master-multiple-backup manner.

[0086] In the embodiment of the present invention, the microservice cluster task scheduling in the data processing system provided by the embodiment of the present invention is based on the principle of "load balancing and taking into account the lowest operation risk coefficient", that is, Figure 2 The microservices in each cluster are reasonably scheduled to run on nodes that are adapted to the corresponding "labels". The core of task scheduling is label-based microservice scheduling. Each task node in the cloud front-end needs to be bound to a label. The label has a domain to which it belongs and identifies the role it can assume. A node can be bound to multiple labels and play multiple roles. The label of the node can be changed dynamically during operation; the task (microservice to be scheduled to run) also needs to be bound to a label. A task can be bound to multiple labels to indicate on which types of nodes the task can run.

[0087] 1) Data collection cluster task scheduling strategy:

[0088] In order to achieve partitioned collection, the data collection cluster can be deployed across collection areas and secure access areas. The data collection microservice carries out channel data collection within the group based on channel grouping. The scheduling strategy is as follows: the scheduler comprehensively considers the operating conditions of each front-end node and the collection network conditions, and schedules the microservice to run on available nodes; at the same time, the built-in TCP / UDP server link distributor allocates the new client link request to the TCP server microservice with the least number of existing connections.

[0089] Typically, collection devices have multiple channel sources, and multiple channels of the same device are configured in different collection groups, that is, they are scanned by different data collection microservices. This is to reduce the risk of reduced redundancy of the collection channels. The groups where the channels of the same device are located have mutually exclusive properties. When the task scheduling cluster schedules the microservices of these groups, it tries to deploy them to different collection nodes, which is also to minimize the risk of reduced channel redundancy.

[0090] In the same collection area, the channel grouping parameters configure the collection area to which it belongs. Data collection microservices can only be scheduled and run on the nodes corresponding to the specified area label and cannot cross areas. For example, if 3 collection nodes and 8 data collection microservices are to be deployed in one area, the task scheduler will first deploy "1+|number of collection microservices / number of nodes|" microservices in sequence except for the last node according to the simple load balancing principle, and deploy the remaining microservices on the last node.

[0091] 2) Data aggregation cluster task scheduling strategy:

[0092] In order to achieve unified data aggregation processing, the data aggregation cluster is generally deployed flexibly on demand in the security zone I, and its scheduling strategy is different from that of the data collection cluster. The logical grouping of data aggregation is realized by taking the collection device partition as the processing unit. Each group starts a corresponding aggregation microservice and processes the collection data aggregation in the same partition. The data aggregation microservice itself has a master and standby mode. In addition to considering the "load balancing" of each node operation, it is also necessary to consider the "mutually exclusive" deployment of the master and standby processes of the microservice, that is, the master and standby nodes of the same partition aggregation microservice are arranged to run on different collection nodes as much as possible to reduce the risk of node failure and reduce the redundancy of microservice operation.

[0093] 3) Cluster task scheduling strategies such as data forwarding cluster, channel management cluster, and message management cluster:

[0094] The cloud-based front-end data forwarding, channel management, and message management clusters are oriented to all partitioned data, and their task scheduling methods are different from those of the collection and aggregation clusters. The message storage, forwarder, and scheduling manager are deployed in a master-slave mode. The entire system only runs one master instance and one backup instance, which are uniformly scheduled and run by the task scheduling cluster.

[0095] In one embodiment, the task scheduling cluster is also used to estimate the resource demand for a preset time period based on the maximum likelihood distribution estimation method according to the current system load and historical data, and adjust the number of microservices in each cluster according to the resource demand for the preset time period.

[0096] In an embodiment of the present invention, the task scheduling cluster needs to consider the mutual exclusion constraint of the task under the premise of label constraint. For the cloud-based front-end data acquisition cluster and data aggregation cluster, for high availability considerations, its microservice instances can have multiple copies. At this time, it is required that these instance copies cannot run on the same node to avoid single point failure. When performing task scheduling, it is also necessary to perceive the resource usage of the node and balance the load of the node through an algorithm that combines distribution estimation and game calculation.

[0097] 1) Elastic scaling scheduling of microservice clusters based on maximum likelihood distribution estimation calculation:

[0098] In order to solve the problem of elastic scaling scheduling of multi-mode microservices in front of cloudification, the embodiment of the present invention first adopts a simple maximum likelihood distribution estimation algorithm to analyze various data of running microservices collected by the event tracing component, including component operation event information, microservice operation resource occupancy, etc., estimate the reliability of microservice operation and the carrying capacity of services on nodes, and calculate the highest probability of nodes where services can be deployed.

[0099] The task scheduler needs to find the maximum likelihood function of the unique parameter set θ in the parameter space; the parameter space of cloud front-end includes parameters such as acquisition domain, microservice cluster, and node. The expression of the maximum likelihood distribution estimation algorithm can be written as:

[0100] node_availability(srv)=argmax(L(θ|X))

[0101] Among them, the argmax operation returns the parameter that produces the maximum value of the target likelihood function. The likelihood function L(θ|X) starts from the observation result X, and the parameter of the distribution function is the probability size of θ. In the cloud frontend, X represents the events perceived by the cluster event tracing and the distribution statistics of each event.

[0102] The binomial likelihood function simplifies to the form:

[0103] L(θ|X)=[N,K]p k (1-p) N-K

[0104] In the above formula, probability p is the probability of observation result X occurring at node k when parameter θ is known; N is the number of calculations, and K is the number of times node k is selected. p of node k is the parameter that the task scheduler tries to estimate; it can maximize the node found and schedule the new task to be expanded to run on it; conversely, it can minimize the node found and stop the task to be reduced from running on it.

[0105] 2) Abnormal microservice drift scheduling based on sorting game calculation:

[0106] The task scheduler monitors the operating conditions of the collection nodes and the collection network. Collection node failures and collection network failures will cause microservices to drift between collection nodes. Drift scheduling must also comply with the "load balancing" principle.

[0107] The carrying capacity of the collection node k can be equivalently converted into the sum of the order of currently used resources and the order of traceable events that have occurred. The abnormal microservice drifts to the node with the smallest carrying capacity among the available nodes except the original running node. Processing steps in the data preparation phase of the abnormal microservice y order:

[0108] a) Identify the running node before y as n;

[0109] b) Identify the labels associated with y and determine the set of nodes N that can run y;

[0110] c) Calculate the resource sorting of the node set N and obtain the corresponding sorting queues Q: cpu_queue (CPU sorting), mem_queue (memory sorting), disk_queue (disk sorting), net_queue (network sorting), event_queue (event sorting), and alarm_queue (alarm sorting).

[0111] d)Priority definition: alarm>net>mem>disk>cpu>event.

[0112] The microservice sorting game algorithm studies X nodes, X = {X1, X2, ..., Xm}, m is the total number of nodes; schedules Y abnormal microservices, Y = {Y1, Y2, ..., Yr}, r is the total number of abnormal microservices, y∈Y; the scheduling strategy of Q can be set to "reverse order of comprehensive sorting" as the positive order of the target scheduling nodes, and this strategy will not be changed during the scheduling process.

[0113] The scheduling strategy of the node to be scheduled is denoted as s, and the strategy set is denoted as Sj = {M1, M2, ..., Mm}. After the abnormal microservice is calculated, the scheduling strategy situation S = (s1, s2, ..., sr) ∈ S1*S2... * Sr is formed, which is the mapping of the abnormal microservice set to the target node set. If any microservice cannot change its strategy alone to make its target value less than the original target value while other microservice strategies remain unchanged, the current strategy situation satisfies the Nash equilibrium.

[0114] In one embodiment, the data collection cluster is further used to verify the acquired first data, and generate a missing data alarm when the first data verification fails.

[0115] In the embodiment of the present invention, the first data that can be obtained can be checked to determine whether important data is leaked, wherein the important data can be preset specific data.

[0116] In one embodiment, preprocessing includes data cleaning, data conversion, and data section storage.

[0117] In one embodiment, the data forwarding cluster is further used for:

[0118] The third data is forwarded to the target receiver at a preset time interval, and the third data is checked during the forwarding process. When it is found that the third data is lost or corrupted, a retransmission mechanism is automatically triggered.

[0119] In an embodiment of the present invention, the data processing system further includes an expansion module for elastically expanding nodes, clusters, services and channels, thereby realizing the collection of massive scheduling data in multiple domains and clusters.

[0120] 1) Expand the number of nodes: When adding a new node, mark the node with a business label, and the task scheduling cluster automatically plans the new node into the business cluster according to the label to participate in the scheduling calculation; the task scheduling strategy is to calculate the latest carrying capacity, identify the node with large carrying capacity, and then determine the CPU and memory usage of the microservices on the node, and schedule the microservices with no event alarms in the past hour and low CPU and memory usage to run on the newly added node with the lowest carrying capacity; poll the above process n times:

[0121]

[0122] Schedule n microservices with low resource usage to run on the new node.

[0123] 2) Expand the number of clusters:

[0124] The cloud-based front-end can replicate the domains of clusters including data collection, data aggregation, data forwarding, channel management, and message management in a multi-domain manner, expand them into a new cluster with the same number of nodes, flexibly load new collection parameters, and carry out data collection.

[0125] 3) Expand the number of microservices in the cluster:

[0126] The cloud-based front-end data collection cluster and data aggregation cluster support elastic scaling scheduling of microservice clusters based on maximum likelihood distribution estimation calculations. They can also pre-configure thresholds for the use of resources such as CPU and memory, and drive the scheduling of microservices through threshold crossings or event alarms.

[0127] 4) Expand the number of channel links in the data collection microservice:

[0128] The channel data collection microservice runs in container mode. According to the device link number (channel number) passed in when the container is started, it reads the link parameters, establishes the link connection, and collects the link data. When the microservice fails, the channel it takes over is interrupted and dispatched to other data collection microservices for processing. The device link is managed by channel group. The newly added link parameters in each group can be dynamically synchronized to the corresponding running microservice to identify the required protocol and collect link data.

[0129] The implementation of the embodiments of the present invention has the following beneficial effects:

[0130] The data acquisition cluster of the embodiment of the present invention dynamically starts a corresponding number of data acquisition microservices according to the number of groups of the acquisition channel, and the data aggregation cluster starts a corresponding number of data aggregation microservices according to the number of partitions of the acquisition device. It can start corresponding data processing resources according to actual data processing needs, thereby effectively improving resource utilization and ensuring the real-time and accuracy of data processing. It can also process massive data and meet the needs of massive data processing, thereby effectively improving data processing efficiency.

[0131] Furthermore, the task scheduling cluster of an embodiment of the present invention is based on the maximum likelihood distribution estimation method. It estimates the resource requirements for a preset time period according to the current system load and historical data, and adjusts the number of microservices in each cluster according to the resource requirements for the preset time period. Through maximum likelihood estimation, it can predict future resource requirements based on current and historical data, thereby dynamically adjusting resource allocation in different time periods and optimizing resource utilization efficiency.

[0132] The above specific embodiments further illustrate the purpose, technical solutions and beneficial effects of the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. It is particularly pointed out that for those skilled in the art, any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention should be included in the scope of protection of the present invention.< / valuetable> < / record> < / jobject>

Claims

1. A data processing system for distribution network scheduling disaster recovery, characterized in that: include: Data collection cluster, data aggregation cluster and data forwarding cluster; The data collection cluster is used to start a corresponding number of data collection microservices according to the number of groups of the collection channel, receive the message data uploaded by the terminal device through the data collection microservice, obtain the first data of the distribution network device from the message data, and send the first data to the data aggregation cluster; The data aggregation cluster is used to start a corresponding number of data aggregation microservices according to the number of partitions of the acquisition device, aggregate the first data through the data aggregation microservice, pre-process the first data, and obtain the second data; The data forwarding cluster is used to integrate all second data into third data and forward the third data to a target receiver.

2. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: It also includes a channel management cluster, which is used to create a channel polling thread according to the channel parameters of the acquisition channel, and monitor the channel status of the acquisition channel through the channel polling thread, wherein the channel parameters include data transmission frequency and data type.

3. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: It also includes a message management cluster, which is used to identify the message type of each message data and store each message data according to the message type.

4. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: It also includes a task scheduling cluster, which is used to deploy the data collection cluster and the data aggregation cluster according to the labels set on the task nodes, and to deploy the data forwarding cluster, the channel management cluster and the message management cluster in a one-master-multiple-standby manner.

5. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: The data collection cluster is further used to verify the acquired first data, and generate a missing data alarm when the first data fails the verification.

6. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: The preprocessing includes data cleaning, data conversion and data section storage.

7. The data processing system for distribution network scheduling disaster recovery according to claim 1, characterized in that: The data forwarding cluster is further used for: The third data is forwarded to a target receiver at a preset time interval, and the third data is checked during the forwarding process. When it is found that the third data is lost or corrupted, a retransmission mechanism is automatically triggered.

8. The data processing system for distribution network scheduling disaster recovery according to claim 4, characterized in that: The task scheduling cluster is also used to estimate the resource demand for a preset time period based on the maximum likelihood distribution estimation method according to the current system load and historical data, and adjust the number of microservices in each cluster according to the resource demand for the preset time period.

Images