Centralized control station service instruction storage method and device, storage medium and processor

Abstract

The invention discloses a centralized control station service instruction storage method and device, a storage medium and a processor. In the scheme, a service data packet is received; the service data packet comprises a plurality of pieces of service data; analyzing the service data packet to obtain a plurality of service instructions; sending the plurality of service instructions to a message queue, and if the sending is successful, processing the plurality of service instructions in the message queue; and consuming the processed multiple service instructions according to a preset period, and storing and compressing the consumed multiple service instructions according to columns. Different from the problem of low storage efficiency when the service flow data of the centralized control station is processed in the prior art, the method has obvious advantages.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, storage medium and processor for storing business instructions of a centralized control station. Background Art

[0002] With the rapid development of smart grids, the processing and storage of business flow data of centralized control stations, as the core component of the power system, has become increasingly important. The business flow data of centralized control stations includes but is not limited to telemetry data, telesignaling data, control commands, etc. Business flow data is usually generated on a large scale and at high speed, which places high requirements on storage technology.

[0003] At present, centralized control stations usually use relational databases or distributed file systems to store business traffic data. Relational databases store data in the form of tables. Each table consists of rows and columns. Rows represent records and columns represent fields. Data is stored by defining relationships and constraints between tables. However, the connections between multiple tables are complex, and there is a problem of low storage efficiency when the amount of data is large. Distributed file systems store data in multiple nodes in a dispersed manner. In order to improve fault tolerance and availability, multiple copies of each block are stored on different nodes, but this method reduces storage efficiency.

[0004] The current storage technology has the problem of low storage efficiency when processing business flow data. Therefore, how to improve the storage efficiency of business flow data in the centralized control station is a technical problem that needs to be solved urgently. Summary of the invention

[0005] Based on the above problems, the present application provides a method, device, storage medium and processor for storing business instructions of a centralized control station, aiming to improve the storage efficiency of business flow data of the centralized control station.

[0006] The embodiments of the present application disclose the following technical solutions:

[0007] The first aspect of the present application provides a method for storing service instructions of a centralized control station, the method comprising:

[0008] Receive a service data packet; the service data packet includes a plurality of service data;

[0009] Parsing the service data packet to obtain multiple service instructions;

[0010] Sending the multiple business instructions to a message queue, and if the sending is successful, processing the multiple business instructions in the message queue;

[0011] Consume multiple processed business instructions according to a preset cycle;

[0012] Store and compress multiple business instructions after consumption by column.

[0013] Optionally, the service data packet is parsed to obtain multiple service instructions, including:

[0014] Parsing the service data packet to obtain the protocol type of each service data in the service data packet;

[0015] A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

[0016] Optionally, the multiple service instructions are sent to a message queue, and if the sending is successful, the multiple service instructions are processed in the message queue, including:

[0017] If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion;

[0018] A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

[0019] Optionally, after storing and compressing the consumed multiple business instructions by column, it also includes:

[0020] The stored data is encoded using encoding technology; the encoding technology includes dictionary encoding and bitmap encoding.

[0021] A second aspect of the present application provides a centralized control station service instruction storage device, the device comprising:

[0022] A receiving module, used to receive a service data packet; the service data packet includes a plurality of service data;

[0023] A parsing module, used for parsing the service data packet to obtain a plurality of service instructions;

[0024] A processing module, used for sending a plurality of business instructions to a message queue, and if the sending is successful, processing the business instructions in the message queue;

[0025] A consumption module is used to consume the processed multiple business instructions according to a preset period;

[0026] The storage compression module is used to store and compress multiple business instructions after consumption by column.

[0027] Optionally, the parsing module is specifically used for:

[0028] Parsing the service data packet to obtain the protocol type of each service data in the service data packet;

[0029] A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

[0030] Optionally, the processing module is specifically used for:

[0031] If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion;

[0032] A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

[0033] Optionally, the device further includes: an encoding module;

[0034] The encoding module is used to encode the stored and compressed data using encoding technology; the encoding technology includes dictionary encoding and bitmap encoding.

[0035] The third aspect of the present application provides a computer-readable storage medium, in which a computer program is stored. When the program is executed by a processor, the centralized control station business instruction storage method provided in any implementation of the first aspect is implemented.

[0036] A fourth aspect of the present application provides a processor, which is used to run a computer program, and when the program is run, the centralized control station business instruction storage method provided in any implementation of the first aspect is executed.

[0037] Compared with the prior art, this application has the following beneficial effects:

[0038] The business instruction storage method of the centralized control station provided in the present application receives a business data packet; parses the business data packet to obtain multiple business instructions; sends the multiple business instructions to the message queue, and if the sending is successful, processes the multiple business instructions in the message queue; consumes the processed multiple business instructions according to a preset cycle, and stores and compresses the consumed multiple business instructions by column. The column-based storage method makes the data type of each column the same and often has a high degree of similarity, which is suitable for efficient compression technology (such as dictionary encoding and bitmap encoding), reduces storage space occupancy and improves I / O efficiency, thereby improving the storage efficiency of the business flow data of the centralized control station. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0040] Figure 1 A flowchart of a method for storing business instructions of a centralized control station provided in an embodiment of the present application;

[0041] Figure 2 A flowchart of another centralized control station service instruction storage method provided in an embodiment of the present application;

[0042] Figure 3 A flowchart of a method for processing a service data packet provided in an embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of a centralized control station business instruction storage device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] As described above, the storage technology currently used in centralized control stations suffers from low storage efficiency when processing business traffic data.

[0045] In view of the above problems, the inventors have proposed a method, device, storage medium and processor for storing business instructions of a centralized control station after research, which receive a business data packet; the business data packet contains multiple business data; the business data packet is parsed to obtain multiple business instructions; the multiple business instructions are sent to a message queue, and if the sending is successful, the multiple business instructions are processed in the message queue; the processed multiple business instructions are consumed according to a preset cycle; the multiple business instructions after consumption are stored and compressed by column.

[0046] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] See also Figure 1 , which is a flow chart of a method for storing service instructions of a centralized control station provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0048] S101. Receive a service data packet.

[0049] Wherein, the business data packet contains multiple business data.

[0050] Receiving business data packets is the first step in the processing flow of the centralized control station. This process involves obtaining data packets containing multiple business data from external sources and preparing them for subsequent parsing, processing and storage.

[0051] S102: Parse the service data packet to obtain multiple service instructions.

[0052] The service data packet is parsed to obtain multiple service instructions, including:

[0053] Parsing the service data packet to obtain the protocol type of each service data in the service data packet;

[0054] A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

[0055] Each processing node implements dedicated processing logic for a specific protocol, including but not limited to decoding, verification, conversion, etc., ensuring that each node can correctly understand and process the corresponding type of business data.

[0056] After processing is complete, the processing node returns the parsed business instruction. Ensure that each business instruction is assigned a globally unique identifier. This can be achieved by using UUID or other similar algorithms when generating instructions to avoid duplication or conflict.

[0057] For example, the service data packets are processed using Vector Packet Processing (VPP), which is an efficient data packet processing framework, especially suitable for high-performance network applications. It was originally developed by the FD.io (Fast Data Input / Output) project, aiming to provide a flexible, scalable and high-performance platform to process a large number of data packets.

[0058] S103: Send the multiple service instructions to a message queue. If the sending is successful, process the multiple service instructions in the message queue.

[0059] A message queue is a buffer that stores messages. It can temporarily hold messages until the consumer is ready to process them. Message queues can be persistent to ensure that messages are not lost even in the event of a system crash. Common types of message queues include Apache Kafka, RabbitMQ, and Amazon SQS.

[0060] Sending multiple business instructions to a message queue, and if the sending is successful, processing the multiple business instructions in the message queue, including:

[0061] If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion;

[0062] A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

[0063] Once the business instructions are successfully sent to the message queue, the next step is to perform intermediate processing on these instructions. The main purpose of intermediate processing is to simplify the format and perform necessary format conversion to ensure the consistency and efficiency of subsequent processing.

[0064] In order to ensure the consistency of business instructions after intermediate processing in a distributed system, the Raft consensus algorithm can be used. The Raft algorithm provides a simple and effective way to manage replication logs and ensure state synchronization between multiple nodes.

[0065] In the process of processing multiple business instructions, the use of message queues combined with intermediate processing and consistency algorithms (such as Raft) can not only efficiently process a large number of business instructions, but also ensure data consistency and reliability.

[0066] S104: Consume the processed multiple service instructions according to a preset cycle.

[0067] According to the system's load capacity and business needs, determine an appropriate consumption cycle (for example, every second, every minute). This cycle should be short enough to ensure real-time performance, while also taking into account the system's maximum processing capacity to prevent overload.

[0068] In one practicable implementation, a mechanism is designed to dynamically adjust the consumption period, for example, the consumption interval can be extended during low traffic periods to save resources, and the interval can be shortened during high traffic periods to speed up processing.

[0069] Configure the number of messages each consumer pulls from the message queue each time (i.e., batch size). A reasonable batch size can maximize CPU utilization while reducing network overhead.

[0070] Through the above method, it can be ensured that the processed multiple business instructions are effectively consumed according to the preset cycle, thereby improving the flexibility, reliability and scalability of the system.

[0071] S105: Store and compress the consumed multiple business instructions by column.

[0072] Column storage organizes data by columns, that is, data in the same field is stored together continuously. Column storage queries usually only involve specific columns, not entire rows. In a large database containing millions of rows, if you only need to query the data of a certain column, the column storage database only needs to read the data of that column.

[0073] For example, a column-oriented NoSQL database is used for storage. Column-oriented NoSQL databases combine the characteristics of column storage and NoSQL databases. Common column-oriented NoSQL databases include Apache Cassandra, HBase, and ClickHouse. Column-oriented NoSQL databases provide strong support for processing large-scale data sets and complex queries through their unique storage methods and optimized query engines.

[0074] Taking advantage of the characteristics of column storage, the data in each column is efficiently compressed. Because the data types in the same column are the same and highly similar, it is very suitable to apply compression algorithms (such as LZ4, Snappy) and dictionary encoding technologies, thereby reducing storage space usage and improving I / O efficiency. Column storage usually supports efficient batch write operations, reducing the overhead caused by frequent small-scale writes. At the same time, because the data is organized by column, the write process is also more orderly, which helps to improve the overall write speed.

[0075] The centralized control station service instruction storage method provided in this embodiment adopts a column-based storage method, so that the data type of each column is the same and often has a high degree of similarity, which is suitable for efficient compression technology (such as dictionary coding and bitmap coding), reduces storage space occupancy and improves I / O efficiency, thereby improving the storage efficiency of centralized control station service flow data.

[0076] See also Figure 2 , which is a flow chart of another centralized control station service instruction storage method provided by the embodiment of the present application. Based on the above embodiment, coding technology is added. Figure 2 As shown, the method comprises the following steps:

[0077] S201. Receive a service data packet.

[0078] Wherein, the business data packet contains multiple business data.

[0079] In a smart grid centralized control station environment, business data packets may come from different sensors, control systems and other devices, using a variety of protocols such as Modbus, DNP3, etc.

[0080] S202: Parse the service data packet to obtain multiple service instructions.

[0081] The service data packet is parsed to obtain multiple service instructions, including:

[0082] Parsing the service data packet to obtain the protocol type of each service data in the service data packet;

[0083] A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

[0084] For example, the Vector Packet Processing (VPP) framework is used to efficiently perform batch reception and parallel parsing to extract specific business instructions. Figure 3 As shown, Figure 3 The service data packet processing flow is shown, which is applicable to scenarios using the VPP framework. Packet Vector stores the received service data packets in a vector structure, which can accommodate multiple data packets (packet-1, packet-3, packet-4,..., packet-n) for subsequent batch processing. dpdk-input is the first node to receive data packets, usually through DPDK (Data Plane Development Kit) technology to efficiently capture data packets from the network card. Data packets input from DPDK are first passed to the Ethernet input node (ethernet-input). Basic Ethernet frame parsing is performed here, and the service data in the data packet is distributed to different processing nodes according to the parsed protocol type. If the data packet contains an MPLS (Multiprotocol Label Switching) label, it will be forwarded to the mpls-ethernet-input node for further processing. For IPv6 data packets, they will be sent to the ip6-input node for parsing and processing, and the parsed IPv6 data packets will be sent to the ip6-lookup node for route lookup or next-hop address determination. For IPv4 packets, they are sent to the ip4-input node for parsing and processing, for ARP (Address Resolution Protocol) packets, they are sent to the arp-input node for parsing and processing, and for LLC (Logical Link Control) packets, they are sent to the llc-input node for parsing and processing.

[0085] S203: Send the multiple service instructions to the message queue. If the sending is successful, process the multiple service instructions in the message queue.

[0086] Use the confirmation mechanism provided by the message queue. Do not assume that the command has been successfully sent before receiving the successful confirmation returned by the message queue. Only when the confirmation is received can the batch of commands be considered to have been received by the message queue.

[0087] Sending multiple business instructions to a message queue, and if the sending is successful, processing the multiple business instructions in the message queue, including:

[0088] If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion;

[0089] A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

[0090] Once the business instructions are successfully sent to the message queue, the next step is to perform intermediate processing on these instructions. The main purpose of intermediate processing is to simplify the format and perform necessary format conversion to ensure the consistency and efficiency of subsequent processing.

[0091] S204: Consume the processed multiple service instructions according to a preset cycle.

[0092] The preset period can be set by a timer, a scheduler or an event-driven method.

[0093] Use a timer or scheduler (such as Cron job) to trigger periodic consumption tasks. For some programming languages ​​or frameworks, you can also use the built-in scheduling function directly. Based on the event-driven approach, for example, when the number of messages in the message queue accumulates to a certain amount, the consumption process is automatically triggered. This method can adapt to different traffic patterns more flexibly.

[0094] S205: Store and compress the consumed multiple business instructions by column.

[0095] Choose a database system that supports columnar storage, such as Apache Cassandra, HBase, ClickHouse, etc. Columnar storage can help process a large number of real-time business instructions more efficiently, provide faster data access speed and higher throughput, which is crucial for timely response to dynamic changes in the power system.

[0096] The data types in the same column in column storage are the same and often have high similarity, which makes them very suitable for applying efficient compression algorithms (such as LZ4, Snappy, Zstandard, etc.). Highly similar data can achieve a better compression ratio, thereby reducing disk space usage. It can be seen that storing and compressing multiple business instructions after consumption by column improves storage efficiency.

[0097] S206: Encode the stored and compressed data using encoding technology.

[0098] The encoding techniques include dictionary encoding and bitmap encoding.

[0099] In practical applications, dictionary encoding and bitmap encoding are usually used in combination to take full advantage of their respective advantages. For example, dictionary encoding can be performed first: for some columns, dictionary encoding can be performed first to convert the original values ​​into smaller integer IDs, and then bitmap encoding can be applied to these integer IDs. Further encoding of stored and compressed data using encoding techniques such as dictionary encoding and bitmap encoding can significantly improve data compression ratio, query efficiency, and overall performance.

[0100] The centralized control station service instruction storage method provided in this embodiment stores the multiple service instructions after consumption by column, and further compresses the data using compression technology (such as vector compression), which can significantly reduce storage space and improve I / O efficiency. Encoding the stored and compressed data using encoding technology can significantly improve the compression ratio of the data and further improve the storage efficiency, which is particularly important for applications such as smart grid centralized control stations that need to efficiently process a large number of real-time service instructions.

[0101] Based on the centralized control station business instruction storage method introduced in the above embodiment, the present application also provides a centralized control station business instruction storage device. Figure 4 Figure 1 is a schematic diagram of the structure of the device. Figure 4 As shown, the device comprises:

[0102] The receiving module 401 is used to receive a service data packet; the service data packet includes a plurality of service data;

[0103] The parsing module 402 is used to parse the service data packet to obtain multiple service instructions;

[0104] The processing module 403 is used to send the multiple business instructions to the message queue, and if the sending is successful, the business instructions are processed in the message queue;

[0105] The consumption module 404 is used to consume the processed multiple business instructions according to a preset period;

[0106] The storage compression module 405 is used to store and compress the consumed multiple business instructions in columns.

[0107] Optionally, the parsing module is specifically used for:

[0108] Parsing the service data packet to obtain the protocol type of each service data in the service data packet;

[0109] A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

[0110] Optionally, the processing module is specifically used for:

[0111] If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion;

[0112] A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

[0113] Optionally, the device further includes: an encoding module;

[0114] The encoding module is used to encode the stored and compressed data using encoding technology; the encoding technology includes dictionary encoding and bitmap encoding.

[0115] In addition, an embodiment of the present application also provides a computer-readable storage medium, in which a computer program is stored. When the program is executed by a processor, the centralized control station business instruction storage method described in any of the method embodiments is implemented.

[0116] In addition, an embodiment of the present application further provides a processor, which is used to run a computer program, and when the program is run, the centralized control station business instruction storage method introduced in any implementation manner of the aforementioned method embodiment is executed.

[0117] It should be noted that each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, in which the unit described as a separate component may or may not be physically separated, and the component prompted as a unit may or may not be a physical unit, that is, it may be located in one place, or it may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative work.

[0118] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for storing service instructions of a centralized control station, characterized in that: include: Receive a service data packet; the service data packet includes multiple service data; Parsing the service data packet to obtain multiple service instructions; Sending the multiple business instructions to a message queue, and if the sending is successful, processing the multiple business instructions in the message queue; Consume multiple processed business instructions according to a preset cycle; Store and compress multiple business instructions after consumption by column.

2. The method according to claim 1, characterized in that The service data packet is parsed to obtain multiple service instructions, including: Parsing the service data packet to obtain the protocol type of each service data in the service data packet; A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

3. The method according to claim 1, characterized in that Sending multiple business instructions to a message queue, and if the sending is successful, processing the multiple business instructions in the message queue, including: If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion; A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

4. The method according to claim 1, characterized in that: After storing and compressing the consumed multiple business instructions by column, it also includes: The stored and compressed data is encoded using encoding techniques; the encoding techniques include dictionary encoding and bitmap encoding.

5. A centralized control station service instruction storage device, characterized in that: include: A receiving module, used to receive a service data packet; the service data packet includes a plurality of service data; A parsing module, used for parsing the service data packet to obtain a plurality of service instructions; A processing module, used for sending a plurality of business instructions to a message queue, and if the sending is successful, processing the business instructions in the message queue; A consumption module is used to consume the processed multiple business instructions according to a preset period; The storage compression module is used to store and compress multiple business instructions after consumption by column.

6. The device according to claim 5, characterized in that The analysis module is specifically used for: Parsing the service data packet to obtain the protocol type of each service data in the service data packet; A processing node corresponding to the protocol type of each piece of business data is called to process each piece of business data to obtain a plurality of business instructions; the business instructions carry a unique identifier.

7. The device according to claim 5, characterized in that The processing module is specifically used for: If the sending is successful, then multiple business instructions are intermediately processed in the message queue; the intermediate processing method includes format simplification and format conversion; A consistency algorithm is used to perform consistency processing on multiple business instructions that have undergone intermediate processing.

8. The device according to claim 5, characterized in that The device also includes: an encoding module; The encoding module is used to encode the stored and compressed data using encoding technology; the encoding technology includes dictionary encoding and bitmap encoding.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the program is executed by a processor, the method for storing business instructions of a centralized control station as described in any one of claims 1 to 4 is implemented.

10. A processor, characterized in that: Used to run a computer program, which, when running, executes the centralized control station business instruction storage method according to any one of claims 1 to 4.

Images