Business traffic storage method, device, electronic device and storage medium

Abstract

The present application provides a method, apparatus, electronic device, and storage device for storing service traffic. The method includes: collecting service traffic based on a collection thread at the collection speed of the current collection period, and writing the collected service traffic into a first cache; wherein the collection speed of the current collection period is determined according to the storage speed and the working state of disk I / O; analyzing the service traffic read from the first cache based on an analysis thread to generate service data, and writing the service data into a second cache; reading the service data from the second cache based on a storage thread and writing the service data into a database according to the storage mode of the current storage period; wherein the storage mode of the current storage period is determined according to the collection speed and the working state of disk I / O. This method can improve the storage efficiency of service traffic.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a method, apparatus, electronic device, and storage medium for storing service traffic. Background Art

[0002] The power system has comprehensively adopted network layer devices for channel security protection. In recent years, application layer attacks against the 104 protocol have emerged in an endless stream. For a threat collection and analysis system, it is necessary to collect and store protocol data for subsequent program analysis of traffic and subsequent auditing.

[0003] The power production control area is divided into Security Zone I and Zone II. Security Zone I is also called the real-time control area, with a large amount of service traffic but not a very high frequency; Security Zone II is also called the non-real-time control area, with less service traffic.

[0004] In related technologies, a method of collecting and storing one protocol traffic at a time is used to fixedly store service traffic. Such a storage method has low storage efficiency. Summary of the Invention

[0005] In view of this, the present application provides a method, apparatus, electronic device, and storage medium for storing service traffic, which can improve storage efficiency.

[0006] To solve the above technical problems, the technical solution of the present application is implemented as follows:

[0007] In one embodiment, a method for storing service traffic is provided. The method includes:

[0008] Collect service traffic based on a collection thread according to the collection speed of the current collection period, and write the collected service traffic into a first buffer; wherein, the collection speed of the current collection period is determined according to the storage speed and the working state of disk I / O; the storage speed and the working state of disk I / O are the storage speed and the working state of disk I / O of the latest storage period that can be obtained;

[0009] Based on an analysis thread, read the service traffic in the first buffer for analysis, generate service data, and write it into a second buffer;

[0010] Based on a storage thread, read the service data in the second buffer and write it into a database according to the storage mode of the current storage period; wherein, the storage mode of the current storage period is determined according to the collection speed and the working state of disk I / O; the collection speed is the collection speed of the latest collection period that can be obtained currently; the working state of disk I / O is the working state of disk I / O of the previous storage period.

[0011] Among them, determining the collection speed of the current collection period according to the storage speed and the working state of disk I / O includes:

[0012] Obtain the storage speed of the latest storage cycle and the working status of disk I / O based on the storage thread;

[0013] Determine whether the storage speed is greater than a preset storage speed threshold;

[0014] In response to the storage speed reaching the preset storage speed limit value, notify the acquisition thread to reduce the acquisition speed as the acquisition speed for the current acquisition cycle;

[0015] In response to the storage speed being greater than the preset storage speed threshold, not reaching the preset storage speed limit value, and the working status of disk I / O being in a saturated working state, notify the acquisition thread to reduce the acquisition speed as the acquisition speed for the current acquisition cycle;

[0016] In response to the storage speed not being greater than the preset storage speed threshold and the working status of disk I / O being in a non-saturated working state, notify the acquisition thread to increase the acquisition speed as the acquisition speed for the current acquisition cycle.

[0017] Among them, determining the storage mode of the current storage cycle according to the acquisition speed and the working status of disk I / O includes:

[0018] Obtain the acquisition speed of the latest acquisition cycle and the working status of disk I / O in the previous storage cycle; among them, the acquisition speed is calculated by the acquisition thread at the end of the latest acquisition cycle;

[0019] If the acquisition speed increases and the working status of disk I / O does not reach the saturated working state, adjust the storage mode to a storage mode with a faster storage speed as the storage mode for the current storage cycle;

[0020] If the acquisition speed decreases, adjust the storage mode to a storage mode with a smaller storage speed as the storage mode for the current storage cycle.

[0021] Among them, the method further includes:

[0022] Calculate the storage speed based on the storage thread cycle;

[0023] Compare the calculated storage speed of the current storage cycle with the acquisition speed of the latest acquisition cycle that can be obtained;

[0024] In response to the acquisition speed of the latest acquisition cycle being greater than the storage speed of the current storage cycle, adjust the storage mode to a storage mode with a greater storage speed;

[0025] In response to the acquisition speed of the latest acquisition cycle being less than the storage speed of the current storage cycle, adjust the storage mode to a storage mode with a smaller storage speed.

[0026] Among them, the acquisition speeds include: a first acquisition speed, a second acquisition data, and a third acquisition speed; among them, the first acquisition speed is less than the second acquisition speed; the second acquisition speed is less than the third acquisition speed;

[0027] The storage modes include: a first storage mode, a second storage mode, and a third storage mode; among them, the first storage mode includes a first compression mode and a first packaging mode; the second storage mode includes a second compression mode and a second packaging mode; the third storage mode includes a third compression mode and a third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode.

[0028] Among them,

[0029] During the storage process, when compressing based on the first compression mode, the stored content of the service data is the content of the original APDU frame;

[0030] When compressing based on the second compression mode, the U-frame and S-frame are optimized;

[0031] When compressing based on the third compression mode, the APDU content of the U-frame and S-frame is not stored, and the I-frame is optimized.

[0032] Among them,

[0033] During the storage process, when packing based on the first packing mode, each piece of service data is written into the disk database respectively;

[0034] When packing based on the second packing mode, an ordered linked list with a capacity greater than a preset ordered linked list capacity value is stored in the disk database, and the content of the corresponding ordered linked list is cleared; the ordered linked list stores multiple pieces of service data;

[0035] When packing based on the third packing mode, a hash linked list with a storage capacity greater than a preset hash linked list capacity value is stored in the disk database, and the content of the corresponding hash linked list is cleared; the hash linked list adds service data to different hash linked lists according to the source and destination IPs.

[0036] In another embodiment, a service traffic storage device is provided, and the device includes:

[0037] An acquisition unit, configured to execute the acquisition of service traffic based on an acquisition thread according to the acquisition speed of the current acquisition cycle, and write the acquired service traffic into a first cache; among them, the acquisition speed of the current acquisition cycle is determined according to the storage speed of the storage thread in the previous storage cycle and the working state of the IO;

[0038] An analysis unit, configured to perform analysis based on an analysis thread to read the service traffic in the first cache, generate service data, and write the service data into a second cache;

[0039] A storage unit, configured to perform storage based on a storage thread to read protocol packets in the second cache and write the protocol packets into a database according to a storage mode of a current storage period; wherein, the storage mode of the current storage period is determined according to a collection speed of a previous collection period and a working state of disk I / O.

[0040] In another embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, steps of the service traffic storage method are implemented.

[0041] In another embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, steps of the service traffic storage method are implemented.

[0042] As can be seen from the above technical solutions, in the above embodiments, data collection, analysis, and storage are carried out in cooperation by three threads of collection, analysis, and storage, and the storage mode is dynamically adjusted based on the collection speed and the working state of disk I / O, and the collection speed is dynamically adjusted based on the storage speed and the working state of disk I / O, so that collection, analysis, and storage reach a balanced state according to the actual application situation, thereby improving the efficiency of service traffic storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for description in the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0044] Figure 1 is a schematic diagram of an implementation environment of a service traffic storage method shown according to an exemplary embodiment;

[0045] Figure 2 is a flowchart of a service traffic storage method shown according to a schematic embodiment;

[0046] Figure 3 is a schematic diagram of determining a collection speed of a current collection period shown according to a schematic embodiment;

[0047] Figure 4 is a schematic diagram of determining a storage mode of a current storage period shown according to a schematic embodiment;

[0048] Figure 5 This is a schematic diagram of the structure of an ordered linked list in an embodiment of the present application;

[0049] Figure 6 This is a schematic diagram of a hash table structure in an embodiment of the present application;

[0050] Figure 7 is a schematic diagram showing an update storage mode according to an exemplary embodiment;

[0051] Figure 8 This is a schematic diagram of the structure of a service flow storage device in an embodiment of the present application;

[0052] Figure 9 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0055] The technical solution of the present invention is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0056] When the power system is attacked by traffic-type attacks such as replay and DDoS, the business traffic collected by the threat collection and analysis system will increase dramatically, seriously consuming the system resources of the threat collection and analysis system. If the business traffic solidification rate is not high, a large amount of business traffic will make it impossible to store all the collected business data, and the larger the traffic, the lower the solidification rate.

[0057] Especially when the threat collection and analysis system is deployed at the main station of the network and provincial level, if all substations are attacked by replay, the business traffic of the main station will increase exponentially compared with the usual time. The high-frequency I / O overhead makes the related storage methods unable to achieve efficient storage of the protocol data of huge traffic, and the storage efficiency is low.

[0058] Based on the above problems, a business traffic storage method is provided in the embodiment of the present application, which can be applied to any scenario with unstable business traffic, such as business traffic of the power 104 protocol. In the specific implementation, data collection, analysis and storage are carried out collaboratively through three threads of collection, analysis and storage, and the storage mode is dynamically adjusted based on the collection speed and the working state of the disk IO, and the collection speed is dynamically adjusted based on the storage speed and the working state of the disk IO, so that the collection, analysis and storage can reach a balanced state according to the actual application situation, thereby improving the efficiency of business traffic storage.

[0059] Figure 1 FIG. 1 is a schematic diagram of an implementation environment of a service flow storage method according to an exemplary embodiment. Figure 1 In this implementation environment, at least one collection device 101, an analysis device 102 and a storage device 103 may be included, which will be described in detail below.

[0060] At least one collection device 101 collects business traffic based on the collection thread according to the collection speed of the current collection cycle, and writes the collected business traffic into the first cache; wherein the collection speed of the current collection cycle is determined according to the storage speed and the working state of the disk IO; the storage speed and the working state of the disk IO are the storage speed and the working state of the disk IO of the latest storage cycle that can be obtained;

[0061] At least one analysis device 102, based on the analysis thread, reads the service traffic in the first cache for analysis, generates service data, and writes the data into the second cache;

[0062] At least one storage device 103 reads the business data in the second cache based on the storage thread and writes it into the database according to the storage mode of the current storage cycle; wherein the storage mode of the current storage cycle is determined according to the acquisition speed and the working state of the disk IO; the acquisition speed is the acquisition speed of the latest acquisition cycle that can be obtained currently; the working state of the disk IO is the working state of the disk IO of the previous storage cycle.

[0063] The acquisition device 101 , the analysis device 102 and the storage device 103 are directly or indirectly connected to each other via wired or wireless communication, which is not limited in the embodiment of the present disclosure.

[0064] Those skilled in the art will be aware that the number of the above devices can be more or less. For example, there can be only one of the above computer devices, or there can be dozens or hundreds of the above terminals, or even more. The embodiments of the present disclosure do not limit the number of devices.

[0065] See Figure 2 , Figure 2 which is a flowchart of a service traffic storage method shown according to an exemplary embodiment. The specific steps are as follows:

[0066] In step 201, based on the acquisition thread, service traffic is acquired according to the acquisition speed of the current acquisition period, and the acquired service traffic is written into the first buffer; wherein, the acquisition speed of the current acquisition period is determined according to the storage speed and the working state of the disk IO; the storage speed and the working state of the disk IO are the storage speed and the working state of the disk IO of the latest storage period that can be obtained.

[0067] In step 202, based on the analysis thread, the service traffic in the first buffer is read for analysis to generate service data, which is then written into the second buffer.

[0068] In step 203, based on the storage thread, the service data in the second buffer is read and written into the database according to the storage mode of the current storage period; wherein, the storage mode of the current storage period is determined according to the acquisition speed and the working state of the disk IO; the acquisition speed is the acquisition speed of the latest acquisition period that can be obtained currently; the working state of the disk IO is the working state of the disk IO of the previous storage period.

[0069] In this embodiment, data acquisition, analysis, and storage are carried out collaboratively by three threads of acquisition, analysis, and storage. The storage mode is dynamically adjusted based on the acquisition speed and the working state of the disk IO, and the acquisition speed is dynamically adjusted based on the storage speed and the working state of the disk IO, enabling acquisition, analysis, and storage to reach a balanced state according to the actual application situation, thereby improving the efficiency of service traffic storage.

[0070] In the embodiments of the present application, the storage of the service traffic of the 104 protocol is taken as an example:

[0071] First, a brief introduction to the 104 protocol is given below:

[0072] The IEC104 protocol uses TCP port 2404 for communication, and the 104 protocol payload APDU is located after the TCP header of the IP packet. The APDU format is shown in Table 1, and Table 1 is the schematic content of the APDU format.

[0073]

[0074] Table 1

[0075] Three frame types are defined in the IEC 104 protocol, namely U frames, S frames, and I frames.

[0076] See Table 2, which is a schematic diagram of the content included in U frames. U frame: A control function frame that only contains the APCI part and has a length of 6 bytes.

[0077]

[0078] Table 2

[0079] STARTDT: Used to start the transmission. Only when this command is sent will the other party send data, and only when this command is sent does it mean that the local program will parse the data sent over.

[0080] STOPDT: Used to stop the transmission. Sending this command means that there is no need to send data anymore, and the local party will no longer parse any received data.

[0081] TESTFR: Used to send a timed greeting when both parties are idle and have nothing to say, indicating that the other party is still in a normal service state.

[0082] See Table 3, which is a schematic diagram of the content included in S frames. S frame: An acknowledgment frame of the I frame that only contains the APCI part and has a length of 6 bytes.

[0083]

[0084] Table 3

[0085] The master station can send S frames at a certain frequency. For example, it can answer one S frame after receiving 8 I frames, or it can answer one S frame for every 1 I frame received. It is related to specific parameters in the 104 protocol.

[0086] I frame: An information frame used for information interaction, including function messages such as telemetry, telemetry, remote control, remote regulation, remote pulse, time synchronization, testing, and resetting.

[0087] The analysis of the service traffic based on the 104 protocol can include the following steps:

[0088] Identify the 104 protocol and extract the sending and receiving IPs and 104 protocol APDU messages in the traffic.

[0089] The analysis thread, based on the 104 protocol parsing plugin template, identifies I frames, U frames, and S frames and determines whether the APDU message is abnormal.

[0090] The analysis thread organizes information such as the sending and receiving IPs, the type of APDU message, and whether it is normal, and writes them together into the protocol cache.

[0091] The storage thread includes merging, compressing, packing, and writing business data to the disk database.

[0092] In some examples, determining the acquisition speed of the current acquisition cycle according to the storage speed and the working state of disk I / O includes:

[0093] Obtaining the storage speed of the latest storage cycle and the working state of disk I / O based on the storage thread;

[0094] Determining whether the storage speed is greater than a preset storage speed threshold;

[0095] In response to the storage speed having reached the preset storage speed limit value, notifying the acquisition thread to reduce the acquisition speed as the acquisition speed of the current acquisition cycle;

[0096] In response to the storage speed being greater than the preset storage speed threshold, not reaching the preset storage speed limit value, and the working state of disk I / O being a saturated working state, notifying the acquisition thread to reduce the acquisition speed as the acquisition speed of the current acquisition cycle;

[0097] In response to the storage speed not being greater than the preset storage speed threshold and the working state of disk I / O being a non-saturated working state, notifying the acquisition thread to increase the acquisition speed as the acquisition speed of the current acquisition cycle.

[0098] In some examples, determining the storage mode of the current storage cycle according to the acquisition speed and the working state of disk I / O includes:

[0099] Obtaining the acquisition speed of the latest acquisition cycle and the working state of disk I / O in the previous storage cycle; wherein, the acquisition speed is calculated by the acquisition thread at the end of the latest acquisition cycle;

[0100] If the acquisition speed increases and the working state of disk I / O does not reach the saturated working state, adjusting the storage mode to a storage mode with a faster storage speed as the storage mode of the current storage cycle;

[0101] If the acquisition speed decreases, adjusting the storage mode to a storage mode with a smaller storage speed as the storage mode of the current storage cycle.

[0102] In some examples, the method further includes:

[0103] Calculating the storage speed based on the storage thread cycle;

[0104] Comparing the calculated storage speed of the current storage cycle with the acquisition speed of the latest acquisition cycle that can be obtained;

[0105] In response to the acquisition speed of the latest acquisition cycle being greater than the storage speed of the current storage cycle, adjusting the storage mode to a storage mode with a greater storage speed;

[0106] In response to the acquisition speed in the latest acquisition cycle being less than the storage speed in the current storage cycle, adjust the storage mode to a storage mode with a smaller storage speed.

[0107] In some examples, the acquisition speed includes: a first acquisition speed, a second acquisition data, and a third acquisition speed; among them, the first acquisition speed is less than the second acquisition speed; the second acquisition speed is less than the third acquisition speed;

[0108] The storage mode includes: a first storage mode, a second storage mode, and a third storage mode; among them, the first storage mode includes a first compression mode and a first packaging mode; the second storage mode includes a second compression mode and a second packaging mode; the third storage mode includes a third compression mode and a third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode.

[0109] In some examples,

[0110] During the storage process, when compressing based on the first compression mode, the stored content of the service data is the content of the original APDU frame;

[0111] When compressing based on the second compression mode, optimize the U-frame and S-frame;

[0112] When compressing based on the third compression mode, do not store the APDU content of the U-frame and S-frame, and optimize the I-frame.

[0113] In some examples,

[0114] During the storage process, when packing based on the first packing mode, write each piece of service data into the disk database respectively;

[0115] When packing based on the second packing mode, store the ordered linked list with a capacity greater than the preset ordered linked list capacity value into the disk database, and clear the content of the corresponding ordered linked list; the ordered linked list stores multiple pieces of service data;

[0116] When packing based on the third packing mode, store the hash linked list with a storage capacity greater than the preset hash linked list capacity value into the disk database, and clear the content of the corresponding hash linked list; the hash linked list adds service data to different hash linked lists according to the source and destination IP.

[0117] In the embodiments of the present application, the acquisition process has its own acquisition cycle, and the storage process has its own storage cycle. The acquisition cycle and the storage cycle can be the same or different. Generally, the cycle of the acquisition process starts from the acquisition traffic, and the storage cycle starts from the start of storing service data; the current acquisition cycle refers to the cycle in which the service traffic is currently being acquired, and the entire cycle from the start to the end of this acquisition cycle; the current storage cycle refers to the cycle in which the service data is currently being stored, and the entire cycle from the start to the end of this storage cycle.

[0118] Figure 3 It is a schematic diagram showing the acquisition speed of determining the current acquisition cycle according to an exemplary embodiment. The specific steps are as follows:

[0119] In step 301, based on the storage thread, obtain the storage speed of the latest storage cycle and the working state of the disk I / O.

[0120] Based on the storage thread, at the end of a storage cycle, the storage speed of this storage cycle will be calculated, and the working state of the disk I / O of this storage cycle will be obtained.

[0121] In step 302, determine whether the storage speed is greater than the preset storage speed threshold.

[0122] In step 303, in response to the storage speed having reached the preset storage speed limit value, notify the acquisition thread to reduce the acquisition speed as the acquisition speed of the current acquisition cycle. End this process.

[0123] Among them, the preset storage speed threshold is less than the preset storage speed limit value.

[0124] In the embodiments of the present application, the acquisition speed will not be adjusted little by little. Mainly, three gears of acquisition speeds are set, such as the first acquisition speed, the second acquisition data, and the third acquisition speed; among them, the first acquisition speed is less than the second acquisition speed; the second acquisition speed is less than the third acquisition speed; the first acquisition speed is set to 500 Mbps, the second acquisition speed is set to 800 Mbps, and the third acquisition speed is set to 1 Gbps. In specific implementation, it is not limited to the number of acquisition speeds, nor to the setting of the above specific speed values.

[0125] In step 304, in response to the storage speed being greater than the preset storage speed threshold, not reaching the preset storage speed limit value, and the working state of the disk I / O being the saturated working state, notify the acquisition thread to reduce the acquisition speed as the acquisition speed of the current acquisition cycle. End this process.

[0126] In steps 303 and 304, the acquisition speed is reduced as the acquisition speed for the current acquisition cycle. Here, it is usually reduced from the current acquisition speed to the lowest speed closest to it. For example, it is reduced from the third acquisition speed to the second acquisition speed, or from the second acquisition speed to the first acquisition speed. If the current speed is already the lowest acquisition speed, the acquisition speed is not further reduced.

[0127] In step 305, in response to the storage speed being not greater than the preset storage speed threshold and the working state of the disk IO being in a non-saturated state, the acquisition thread is notified to increase the acquisition speed as the acquisition speed for the current acquisition cycle.

[0128] Increasing the acquisition speed as the acquisition speed for the current acquisition cycle. Here, it is usually increased from the current acquisition speed to the highest speed closest to it. For example, it is increased from the second acquisition speed to the third acquisition speed, or from the first acquisition speed to the second acquisition speed. If the current speed is already the highest acquisition speed, the acquisition speed is not further increased.

[0129] The acquisition speed is adjusted according to the storage effect of the storage thread: when the storage thread adjusts to a high-speed storage mode and the disk IO is still in an over-saturated state or a saturated state, the acquisition thread needs to be notified to reduce the acquisition speed; when the storage mode of the storage thread is the first storage mode and the disk IO is still in an unsaturated state, the acquisition speed needs to be increased.

[0130] Figure 4 It is a schematic diagram showing a method for determining the storage mode of the current storage cycle according to an exemplary embodiment. The specific steps are as follows:

[0131] In step 401, the acquisition speed of the latest acquisition cycle and the working state of the disk IO in the previous storage cycle are obtained; among them, the acquisition speed is calculated by the acquisition thread at the end of the latest acquisition cycle.

[0132] Obtaining the acquisition speed of the latest acquisition cycle means obtaining the acquisition speed of the acquisition cycle that is closest to the current time and has ended.

[0133] In step 402, if the acquisition speed increases and the working state of the disk IO does not reach the saturated state, the storage mode is adjusted to a storage mode with a faster storage speed as the storage mode for the current storage cycle. This process ends.

[0134] Here, the increase in the acquisition speed does not mean that any change will trigger an update. Instead, if the acquisition speed changes from the original first acquisition speed to a speed greater than or equal to the second acquisition speed, or from the original second acquisition speed to a speed greater than or equal to the third acquisition speed, it is determined that the acquisition speed has increased.

[0135] In the embodiments of the present application, three storage modes are set for the storage mode, but are not limited to this limitation. Specifically:

[0136] The first storage mode, the second storage mode, and the third storage mode; among them, the first storage mode includes the first compression mode and the first packaging mode; the second storage mode includes the second compression mode and the second packaging mode; the third storage mode includes the third compression mode and the third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode.

[0137] During the storage process, refer to Table 4 for the storage format of business data. Table 4 is the schematic content of the business data storage format.

[0138] Source IP Destination IP Service data type Service data stored content 4 bytes 4 bytes 1 byte

[0139] Table 4

[0140] Among them, the definition of 1 byte of the business data type is shown in Table 5. Table 5 is the content of the 1-byte definition of the business data type.

[0141]

[0142] Table 5

[0143] Among them, the first storage mode can also be called the standard storage mode, the first compression mode can be called the standard compression mode, and the first packaging mode can be called the standard packaging mode.

[0144] When compressing based on the first compression mode, the stored content of the business data is the content of the original APDU frame;

[0145] When packaging based on the first packaging mode, each piece of business data is written into the disk database respectively;

[0146] When compressing based on the second compression mode, optimize the U-frame and S-frame; that is, do not store the APDU content of the U-frame.

[0147] For the S-frame, construct a hash linked list with the source IP and destination IP as indexes, and store the C2 and C3 fields of each S-frame in the hash table.

[0148] In the low-speed mode, in the linked list with the same hash value, for every 20 S-frames, only store the C2 and C3 fields of the last S-frame in the specified storage content.

[0149] When packaging based on the second packaging mode, store the ordered linked list with a capacity greater than the preset ordered linked list capacity value in the disk database, and clear the content of the corresponding ordered linked list; the ordered linked list stores multiple pieces of business data;

[0150] In this packing mode, the storage thread creates an ordered linked list for each piece of business data that is ready. When the capacity of the ordered linked list is greater than the preset capacity value of the ordered linked list, the threshold is adjusted according to the memory size and can be set to 100MB, but is not limited to this value. The content of the linked list is written to the disk database in an orderly manner, and the content of the linked list is cleared.

[0151] See Figure 5 , Figure 5 is a schematic structural diagram of an ordered linked list in an embodiment of the present application. Each storage unit corresponds to a piece of business data. The content stored in each piece of business data includes the source IP, destination IP, business data type, business data storage content. The total number of storage units is the total number of pieces of business data, and the storage type is an ordered linked list.

[0152] When compressing based on the third compression mode, the APDU content of U frames and S frames is not stored, and the I frames are optimized.

[0153] Since the length of the APDU of the 104 protocol does not exceed 256 bytes, the following optimization method is formulated:

[0154] Use one byte T to represent N consecutive 1s or 0s, where the high bit represents whether the consecutive ones are 1 or 0, and the last 7 bits represent the value of N. See Table 6. Table 6 shows the content represented by one byte T in the third compression mode.

[0155]

[0156] Table 6

[0157] Convert C1 / C2 / C3 / C4 in APCI together with the ASDU data into binary and stringify them. The total length is (8 * APDU length L). Starting from C1, count the number of consecutive 0s or 1s in turn. When the cumulative count is equal to 127 or the consecutive sequence is interrupted by 1 or 0, add a new byte and continue to count the number of consecutive 0s or 1s in the remaining string until the ASDU is traversed.

[0158] See Table 7. Table 7 shows the content before and after format adjustment in the third compression mode.

[0159]

[0160]

[0161] Table 7

[0162] When packing based on the third packing mode, the hash linked list with a storage capacity greater than the preset hash linked list capacity value is stored in the disk database, and the content of the corresponding hash linked list is cleared; the hash linked list adds business data to different hash linked lists according to the source and destination IPs.

[0163] In this packaging mode, the storage thread builds multiple hash linked lists from the prepared protocol data according to the source and destination IPs. When the capacity of the entire hash linked list is greater than the preset hash linked list capacity value, the content of the hash linked list is written into the disk database in an orderly manner, and the content of the hash linked list is cleared.

[0164] The storage mode is adjusted to a storage mode with a greater storage speed, that is, the storage mode is adjusted from the first storage mode to the second storage mode, and the storage mode is adjusted from the second storage mode to the third storage mode. If it is already the third storage mode, no mode adjustment is performed.

[0165] See Figure 6 , Figure 6 which is a schematic diagram of a hash linked list structure in an embodiment of the present application. Figure 6 In, the storage type is a hash linked list, and the total number of storage units refers to the number of business data stored. Different storage units store business data corresponding to different source IPs and destination IPs, and the same storage unit stores business data corresponding to the same source IP and destination IP.

[0166] In step 403, if the acquisition speed becomes smaller, the storage mode is adjusted to a storage mode with a smaller storage speed as the storage mode for the current storage cycle.

[0167] Here, the acquisition speed becoming smaller does not mean that it is updated as soon as there is a little change. Instead, if it was originally at the third acquisition speed and the current acquisition speed is less than or equal to the second acquisition speed; or if it was originally at the second acquisition speed and the current acquisition speed is less than or equal to the first acquisition speed, then it is determined that the acquisition speed has become smaller.

[0168] The storage mode is adjusted to a storage mode with a smaller storage speed, that is, the storage mode is adjusted from the third storage mode to the second storage mode, and the storage mode is adjusted from the second storage mode to the first storage mode. If it is already the first storage mode, no mode adjustment is performed.

[0169] In specific implementation, the storage thread can also determine whether to update the storage mode according to the storage speed and the latest acquisition speed that can be obtained. The specific implementation is as follows:

[0170] Figure 7 is a schematic diagram showing an update of the storage mode according to an exemplary embodiment. The specific steps are as follows:

[0171] In step 701, the storage speed is calculated based on the storage thread cycle.

[0172] In step 702, the storage speed of the currently calculated storage cycle is compared with the acquisition speed of the latest acquisition cycle that can be obtained.

[0173] The acquisition speed of the latest acquisition cycle that can be obtained, that is, the acquisition speed of the last ended acquisition cycle currently. The acquisition thread will calculate the acquisition speed of the acquisition thread after an acquisition thread ends and push it to the storage thread.

[0174] In step 703, in response to the acquisition speed of the latest acquisition cycle being greater than the storage speed of the current storage cycle, adjust the storage mode to the storage mode with a greater storage speed. End this process.

[0175] If the current storage mode is the first storage mode, adjust it to the second storage mode; if the current storage mode is the second storage mode, adjust it to the third storage mode; if the current storage mode is the third storage mode, no further adjustment is made.

[0176] In step 704, in response to the acquisition speed of the latest acquisition cycle being less than the storage speed of the current storage cycle, adjust the storage mode to the storage mode with a smaller storage speed.

[0177] If the current storage mode is the third storage mode, adjust it to the second storage mode; if the current storage mode is the second storage mode, adjust it to the first storage mode; if the current storage mode is the first storage mode, no further adjustment is made.

[0178] In the embodiments of the present application, the acquisition speed is adjusted based on the working status of the storage speed and disk I / O, the storage mode is adjusted based on the working status of the acquisition speed and disk I / O, and the storage mode is adjusted based on the storage speed and acquisition speed. This mechanism of dynamic - feedback - adjustment is used for traffic acquisition and protocol storage. By comparing the size of the acquired traffic with the system load situation, the storage speed is adjusted in a timely manner; at the same time, the overhead of system resources during the storage process is monitored, and the acquisition speed is adjusted in turn, ultimately enabling the acquisition and storage to work at a green wave speed. It can improve the storage efficiency while saving resources. The combination of multiple packaging modes and compression modes can adapt to storage functions with different scenarios and speed requirements.

[0179] During the compression process for the three compression modes, binary serialization processing is performed on the I - frame part of the service data. One byte is used to represent consecutive multiple 0s or consecutive multiple 1s, effectively shortening the length of the service data. For the S - frame part of the service data, using the characteristics of the service itself, only the last frame of consecutive multiple S - frames is recorded, effectively reducing the number of S - frames.

[0180] In the embodiments of the present application, it is applied to the scenario of 104 protocol traffic storage, which can greatly improve the solidification speed of 104 protocol traffic. When the power system is subjected to traffic-based attacks such as replay attacks and DDoS attacks, the protocol solidification speed can be dynamically adjusted according to the size of the real-time traffic. At the same time, the present invention can adjust the traffic collection speed in real time according to the disk and system overhead during solidification, and can effectively keep the system in a normal and available operating level. Thus, the efficient solidification of protocol traffic is further maximized.

[0181] Based on the same inventive concept, an embodiment of the present application also provides a service traffic storage device. Refer to Figure 8 , Figure 8 which is a schematic structural diagram of the service traffic storage device in the embodiments of the present application. The service traffic storage device includes:

[0182] A collection unit 801, configured to collect service traffic based on a collection thread according to the collection speed of the current collection period, and write the collected service traffic into a first cache; wherein, the collection speed of the current collection period is determined according to the storage speed of the storage thread in the previous storage period and the working state of the IO.

[0183] An analysis unit 802, configured to analyze the service traffic read from the first cache based on an analysis thread, generate service data, and write it into a second cache.

[0184] A storage unit 803, configured to read protocol packets from the second cache based on a storage thread and write them into a database according to the storage mode of the current storage period; wherein, the storage mode of the current storage period is determined according to the collection speed of the previous collection period and the working state of the disk IO.

[0185] In another embodiment,

[0186] When the storage unit 803 is configured to determine the collection speed of the current collection period according to the storage speed and the working state of the disk IO, it obtains the storage speed and the working state of the disk IO in the latest storage period based on the storage thread; determines whether the storage speed is greater than a preset storage speed threshold; in response to the storage speed having reached the preset storage speed limit value, notifies the collection thread to reduce the collection speed as the collection speed of the current collection period; in response to the storage speed being greater than the preset storage speed threshold, not reaching the preset storage speed limit value, and the working state of the disk IO being a saturated working state, notifies the collection thread to reduce the collection speed as the collection speed of the current collection period; in response to the storage speed not being greater than the preset storage speed threshold, and the working state of the disk IO being a non-saturated working state, notifies the collection thread to increase the collection speed as the collection speed of the current collection period.

[0187] In another embodiment,

[0188] The storage unit 803 is configured to, when determining the storage mode of the current storage cycle according to the acquisition speed and the working state of the disk IO, obtain the acquisition speed of the latest acquisition cycle and the working state of the disk IO of the previous storage cycle; wherein, the acquisition speed is calculated by the acquisition thread at the end of the latest acquisition cycle; if the acquisition speed increases and the working state of the disk IO does not reach the saturated working state, adjust the storage mode to a storage mode with a faster storage speed as the storage mode of the current storage cycle; if the acquisition speed decreases, adjust the storage mode to a storage mode with a smaller storage speed as the storage mode of the current storage cycle.

[0189] In another embodiment,

[0190] The storage unit 803 is configured to execute calculating the storage speed based on the storage thread cycle; comparing the storage speed of the calculated current storage cycle with the acquisition speed of the latest acquisition cycle that can be obtained; in response to the acquisition speed of the latest acquisition cycle being greater than the storage speed of the current storage cycle, adjusting the storage mode to a storage mode with a greater storage speed; in response to the acquisition speed of the latest acquisition cycle being less than the storage speed of the current storage cycle, adjusting the storage mode to a storage mode with a smaller storage speed.

[0191] In another embodiment,

[0192] The acquisition speed includes: a first acquisition speed, a second acquisition data, and a third acquisition speed; wherein, the first acquisition speed is less than the second acquisition speed; the second acquisition speed is less than the third acquisition speed;

[0193] The storage mode includes: a first storage mode, a second storage mode, and a third storage mode; wherein, the first storage mode includes a first compression mode and a first packaging mode; the second storage mode includes a second compression mode and a second packaging mode; the third storage mode includes a third compression mode and a third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode.

[0194] In another embodiment,

[0195] During the storage process, when compressing based on the first compression mode, the stored content of the service data is the original APDU frame content;

[0196] When compressing based on the second compression mode, optimize the U-frame and S-frame;

[0197] When compressing based on the third compression mode, the APDU content of U-frames and S-frames is not stored, and I-frames are optimized.

[0198] In another embodiment,

[0199] During the storage process, when packing based on the first packing mode, each piece of service data is written into the disk database respectively;

[0200] When packing based on the second packing mode, an ordered linked list with a capacity greater than the preset ordered linked list capacity value is stored in the disk database, and the content of the corresponding ordered linked list is cleared; the ordered linked list stores multiple pieces of service data;

[0201] When packing based on the third packing mode, a hash linked list with a storage capacity greater than the preset hash linked list capacity value is stored in the disk database, and the content of the corresponding hash linked list is cleared; the hash linked list adds service data to different hash linked lists according to the source and destination IPs.

[0202] The units in the above embodiments can be integrated into one body or separately deployed; they can be combined into one unit or further split into multiple sub-units.

[0203] In another embodiment, an electronic device is further provided, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the service traffic storage method are implemented.

[0204] In another embodiment, a computer-readable storage medium is further provided, on which computer instructions are stored. When the instructions are executed by the processor, the steps in the service traffic storage method are performed.

[0205] Figure 9 This is a schematic diagram of the physical structure of the electronic device provided by the embodiments of the present invention. As Figure 9 shown, the electronic device may include: a processor (Processor) 910, a communication interface (Communications Interface) 920, a memory (Memory) 930, and a communication bus 940. Among them, the processor 910, the communication interface 920, and the memory 930 complete mutual communication through the communication bus 940. The processor 910 can call the logical instructions in the memory 930 to execute the following method:

[0206] Based on the acquisition thread, service traffic is acquired according to the acquisition speed of the current acquisition cycle, and the acquired service traffic is written into the first cache; wherein, the acquisition speed of the current acquisition cycle is determined according to the storage speed and the working state of the disk IO; the storage speed and the working state of the disk IO are the storage speed and the working state of the latest storage cycle that can be obtained;

[0207] Based on the analysis thread, read the service traffic in the first cache for analysis, generate service data, and write it into the second cache;

[0208] Based on the storage thread, read the service data in the second cache and write it into the database according to the storage mode of the current storage cycle; wherein, the storage mode of the current storage cycle is determined according to the acquisition speed and the working state of the disk IO; the acquisition speed is the acquisition speed of the latest acquisition cycle that can be currently obtained; the working state of the disk IO is the working state of the disk IO in the previous storage cycle.

[0209] In addition, when the logical instructions in the above-mentioned memory 930 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. And the aforementioned storage medium includes: USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical disks, etc., which can store program codes.

[0210] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated. The components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative efforts.

[0211] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the technical solution, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disks, optical disks, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments.

[0212] The flowcharts and block diagrams in the accompanying drawings of the present application illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments disclosed in the present application. In this regard, each block in the flowchart or block diagram may represent a module, a program segment, or a part of code, and the above-mentioned module, program segment, or part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in the order marked in different drawings. For example, two consecutively represented blocks may actually be executed substantially in parallel, and they may sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, as well as the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0213] Those skilled in the art can understand that the features described in various embodiments and / or claims of the present application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in the present application. In particular, without departing from the spirit and teachings of the present application, the features described in various embodiments and / or claims of the present application can be combined and / or combined in various ways, and all such combinations and / or combinations fall within the scope of the present application.

[0214] Specific embodiments are used in this article to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention, and is not used to limit the present application. For those skilled in the art, changes can be made in the specific implementation manners and application scopes according to the ideas, spirits, and principles of the present invention. Any modifications, equivalent replacements, improvements, etc. made by them shall be included in the scope of protection of the present application.

Claims

1. A method for storing service traffic, characterized in that, the method includes: Collecting service traffic based on a collection thread according to the collection speed of the current collection cycle, and writing the collected service traffic into a first cache; wherein, the collection speed of the current collection cycle is determined according to the storage speed and the working state of disk I / O; the storage speed and the working state of disk I / O are the storage speed and the working state of disk I / O of the latest storage cycle that can be obtained; Based on an analysis thread, reading the service traffic in the first cache for analysis, generating service data, and writing it into a second cache; Based on a storage thread, reading the service data in the second cache and writing it into a database according to the storage mode of the current storage cycle; wherein, the storage mode of the current storage cycle is determined according to the collection speed and the working state of disk I / O; the collection speed is the collection speed of the latest collection cycle that can be obtained currently; the working state of disk I / O is the working state of disk I / O of the previous storage cycle; wherein, the collection speed includes: a first collection speed, a second collection speed, and a third collection speed; wherein, the first collection speed is less than the second collection speed; the second collection speed is less than the third collection speed; The storage mode includes: a first storage mode, a second storage mode, and a third storage mode; wherein, the first storage mode includes a first compression mode and a first packaging mode; the second storage mode includes a second compression mode and a second packaging mode; the third storage mode includes a third compression mode and a third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode; During the storage process, when compressing based on the first compression mode, the stored content of the service data is the content of the original APDU frame; When compressing based on the second compression mode, optimizing the U-frame and S-frame; When compressing based on the third compression mode, not storing the APDU content of the U-frame and S-frame, and optimizing the I-frame.

2. The method according to claim 1, characterized in that, Determining the collection speed of the current collection cycle according to the storage speed and the working state of disk I / O includes: Based on the storage thread, obtaining the storage speed and the working state of disk I / O of the latest storage cycle; Determining whether the storage speed is greater than a preset storage speed threshold; In response to the storage speed having reached the preset storage speed limit value, notifying the collection thread to reduce the collection speed as the collection speed of the current collection cycle; In response to the storage speed being greater than the preset storage speed threshold, not reaching the preset storage speed limit value, and the working state of disk I / O being a saturated working state, notifying the collection thread to reduce the collection speed as the collection speed of the current collection cycle; In response to the storage speed not being greater than the preset storage speed threshold, and the working state of disk I / O being a non-saturated working state, notifying the collection thread to increase the collection speed as the collection speed of the current collection cycle.

3. The method according to claim 1, characterized in that, Determine the storage mode of the current storage cycle according to the acquisition speed and the working state of disk I / O, including: Obtain the acquisition speed of the latest acquisition cycle and the working state of disk I / O in the previous storage cycle; wherein, the acquisition speed is calculated by the acquisition thread at the end of the latest acquisition cycle; If the acquisition speed increases and the working state of disk I / O does not reach the saturated working state, adjust the storage mode to a storage mode with a faster storage speed as the storage mode of the current storage cycle; If the acquisition speed decreases, adjust the storage mode to a storage mode with a smaller storage speed as the storage mode of the current storage cycle.

4. The method according to claim 1, wherein, the method further includes: Calculate the storage speed based on the storage thread cycle; Compare the storage speed of the calculated current storage cycle with the acquisition speed of the latest acquisition cycle that can be obtained; In response to the acquisition speed of the latest acquisition cycle being greater than the storage speed of the current storage cycle, adjust the storage mode to a storage mode with a greater storage speed; In response to the acquisition speed of the latest acquisition cycle being less than the storage speed of the current storage cycle, adjust the storage mode to a storage mode with a smaller storage speed.

5. The method according to claim 1, wherein, During the storage process, when packing based on the first packing mode, write each piece of service data into the disk database respectively; When packing based on the second packing mode, store the ordered list with a capacity greater than the preset ordered list capacity value into the disk database, and empty the content of the corresponding ordered list; The ordered list stores multiple pieces of service data; When packing based on the third packing mode, store the hash list with a storage capacity greater than the preset hash list capacity value into the disk database, and empty the content of the corresponding hash list; The hash list adds service data to different hash lists according to the source and destination IPs.

6. A service traffic storage device, wherein, the device includes: An acquisition unit, configured to execute the acquisition of service traffic based on an acquisition thread according to the acquisition speed of the current acquisition cycle, and write the acquired service traffic into a first cache; wherein, the acquisition speed of the current acquisition cycle is determined according to the storage speed of the storage thread in the previous storage cycle and the working state of I / O; wherein, the acquisition speed includes: a first acquisition speed, a second acquisition speed, and a third acquisition speed; wherein, the first acquisition speed is less than the second acquisition speed; the second acquisition speed is less than the third acquisition speed; An analysis unit, configured to execute the analysis of the service traffic read from the first cache based on an analysis thread, generate service data, and write it into a second cache; A storage unit, configured to execute writing the protocol packets read from the second cache into a database according to the storage mode of the current storage cycle based on a storage thread; wherein, the storage mode of the current storage cycle is determined according to the acquisition speed of the previous acquisition cycle and the working state of the disk IO; wherein, the storage mode includes: a first storage mode, a second storage mode, and a third storage mode; wherein, the first storage mode includes a first compression mode and a first packaging mode; the second storage mode includes a second compression mode and a second packaging mode; the third storage mode includes a third compression mode and a third packaging mode; the speed of storing data in the first storage mode is less than the speed of storing data in the second storage mode, and the speed of storing data in the second storage mode is less than the speed of storing data in the third storage mode; during storage, when compressing based on the first compression mode, the stored content of the service data is the content of the original APDU frame; when compressing based on the second compression mode, the U-frame and S-frame are optimized; when compressing based on the third compression mode, the APDU content of the U-frame and S-frame is not stored, and the I-frame is optimized.

7. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein, when the processor executes the program, the method described in any one of claims 1-5 is implemented.

8. A computer-readable storage medium, on which a computer program is stored, wherein, when the program is executed by a processor, the method described in any one of claims 1-5 is implemented.

Images