Flow statistic information transfer system, flow statistic information transfer method, and flow statistic information transfer program

Abstract

A network monitoring system includes a collection unit and a binarization unit. The collection unit collects flow statistics information. The binarization unit converts the flow statistics information to binary data. The binarization unit then transfers the binary data through a transmission path.

Description

[Technical Field]

[0001] The present disclosure relates to a flow statistic information transfer system, a flow statistic information transfer method, and a flow statistic information transfer program. [Background technology]

[0002] Carriers obtain flow statistics to visualize network traffic, using technologies such as xFlow and packet encapsulation.

[0003] xFlow is a general term used to refer to various network monitoring protocols. Examples of network monitoring protocols include NetFlow, IPFIX (IP Flow Information Export), and sFlow (registered trademark). xFlow technology samples traffic transmitted over a network and then aggregates and analyzes the traffic. For example, NetFlow (RFC3954) performs the following operations: (1) sampling packets; (2) deriving flow statistics from header information; and (3) transmitting the flow statistics.

[0004] Packet encapsulation techniques involve embedding a packet in the payload of another packet and then transporting the encapsulated packet across a network. For example, packet encapsulation techniques include format conversion techniques that decapsulate the packet and then allow analysis of the inner packet. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2019-097069 [Patent Document 2] Patent Publication No. 2021-090161 [Non-patent literature]

[0006] [Non-Patent Document 1] “RFC3954 Cisco Systems NetFlow Services Export Version 9” [online], [Retrieved January 19, 2023], Internet<https: / / www.rfc-editor.org / rfc / rfc3954> Summary of the Invention [Problem to be solved by the invention]

[0007] However, in the above prior art, it may be difficult to reduce the amount of transferred data in the flow statistics information.

[0008] Therefore, the present disclosure provides a flow statistic information transfer system, a flow statistic information transfer method, and a flow statistic information transfer program that can reduce the amount of data transferred in flow statistic information. [Means for solving the problem]

[0009] In one aspect of the present disclosure, a flow statistics information transfer system includes a collection unit that collects flow statistics information, a conversion unit that converts the flow statistics information into binary data, and a transfer unit that transfers the binary data through a transmission path. [Effects of the Invention]

[0010] The flow statistics information transfer system can reduce the amount of data transferred in the flow statistics information. [Brief explanation of the drawings]

[0011] [Figure 1]Figure 1 shows an example of the challenges associated with flow statistics forwarding. [Figure 2] FIG. 2 is a block diagram of an example environment for flow statistics information transfer. [Figure 3] FIG. 3 shows an overview of one flow statistics information transfer process according to the present disclosure. [Figure 4] FIG. 4 is a block diagram of an example configuration of a network monitoring system according to the present disclosure. [Figure 5] FIG. 5 is a sequence diagram showing an example of a process for transmitting flow statistical information as binary data. [Figure 6] FIG. 6 illustrates an example of binarizing flow statistics according to the present disclosure. [Figure 7] FIG. 7 shows an example of JSON-formatting binary data according to the present disclosure. [Figure 8] FIG. 8 shows an example of the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Several embodiments of the present disclosure are described in the accompanying drawings and the following description, but the present invention is not limited to these embodiments. The various features of these embodiments can be combined in various ways, provided that these features are not mutually inconsistent. Like reference numerals refer to like elements.

[0013] 〔table of contents〕 The following explanation is divided into nine sections: 1. Introduction 2. Environment for Transferring Flow Statistics Information 3. Overview of flow statistics information transfer process 4. Configuring a network monitoring system 5. Sequence diagram of flow statistics information transfer process 6.Effects 7. Hardware Configuration 8. Summary of embodiments 9. Addendum

[0014] 1. Introduction The amount of communication traffic on carrier networks is increasing year by year. The amount of data required for traffic visualization is also increasing. In addition, there is a need to obtain more detailed flow statistics information. As a result, carriers are being asked to reduce bandwidth.

[0015] Figure 1 illustrates an example of a problem associated with flow statistics transfer, problem 10. Problem 10 is that required bandwidth increases as operators acquire more detailed flow statistics.

[0016] As shown in FIG. 1, the xFlow converter 11 collects header samples and xFlow packets from the carrier network 12. The xFlow converter 11 generates flow statistics based on the inner packets. For example, the flow statistics include flow information, routing, and topology data. The flow statistics are sent to a group of visualization APIs (Application Programming Interfaces) 13. The visualization APIs 13 process the flow statistics to visualize traffic.

[0017] The xFlow converter 11 may be implemented by a router, and the visualization APIs 13 may be implemented by a server.

[0018] To obtain more detailed flow information, the xFlow converter 11 can collect header sample information, and therefore can send more detailed flow statistics information to the visualization APIs 13.

[0019] As can be seen from the example in Figure 1, the xFlow conversion device 11 transmits flow statistics information through a transmission path between the xFlow conversion device 11 and the message queue 14 of the visualization API group 13. The xFlow conversion device 11 sends the flow statistics information in JSON (JavaScript (registered trademark) Object Notation) (i.e., text format). However, JSON data requires a large amount of data to be transferred. This increases the bandwidth required for the transmission section.

[0020] As described above, the required bandwidth increases according to the amount of flow statistics information. To solve the above problem, the network monitoring system according to the present disclosure performs one or more flow statistics information transfer processes described below.

[0021] [2. Environment for Transferring Flow Statistics Information] First, the environment for flow statistics information transfer will be described with reference to FIG.

[0022] 2 is a block diagram of an environment 1, which is an example of an environment for flow statistics information transfer. As shown in FIG. 2, the environment 1 includes a network monitoring system 100, a monitoring line 200, and a network 300. The network monitoring system 100 is an example of a flow statistics information transfer system.

[0023] The network monitoring system 100 is a system that performs processing for transferring flow statistics information. In this specification, such processing is referred to as a flow statistics information transfer process. An overview of one flow statistics information transfer process is explained in Section 3. Then, the details of the flow statistics information transfer process are explained using a sequence diagram in Section 5.

[0024] The network monitoring system 100 includes one or more data processing devices, such as one or more servers, one or more personal computers (PCs), one or more network devices, etc. An example of the configuration of the network monitoring system 100 will be described in Section 4.

[0025] The monitoring line 200 is, for example, a WAN (Wide Area Network) line, an Internet line, etc. The monitoring line 200 connects the network monitoring system 100 and the network 300 together.

[0026] The network 300 is a monitored network, such as a WAN.

[0027] [3. Overview of flow statistics information transfer process] Next, an overview of one flow statistics information forwarding process will be described with reference to Figure 3. Note that this overview is not intended to limit the present invention or the embodiments described in the following sections.

[0028] 3 shows an overview 20 of one flow statistics transfer process according to the present disclosure. The overview 20 shows the binarization of flow statistics and the Jsonification of this binary data as new processes.

[0029] 3, the network monitoring system 100 includes an xFlow conversion device 21 and a visualization API group 13. The visualization API group 13 includes a JSON registration device 22.

[0030] The xFlow converter 21 may have the same functions as the xFlow converter 11 described above with reference to Fig. 1. The same description of the xFlow converter 11 will be omitted here. A newly added function of the xFlow converter 21 is the binarization of text data. The xFlow converter 21 converts text data into binary data so that the text data can be converted to JSON later.

[0031] The JSON registration device 22 is a newly added device that converts binary data into data in JSON format.

[0032] In the example of FIG. 3, the xFlow conversion device 21 reduces the amount of data transferred by using binary transmission. The xFlow conversion device 21 transmits binary data to the JSON registration device 22. The JSON registration device 22 converts the binary data into JSON after the binary data has passed through the transmission section. This JSON data is sent to the message queue 14. In this way, the xFlow conversion device 21 can reduce the bandwidth required for the transmission section.

[0033] [4. Network monitoring system configuration] Next, an example of the configuration of the network monitoring system 100 will be described with reference to FIG.

[0034] Fig. 4 is a block diagram of an example configuration of a network monitoring system 100 according to the present disclosure. As shown in Fig. 4, the network monitoring system 100 includes a communication unit 110, a control unit 120, and a storage unit 130. The network monitoring system 100 may also include an input unit (e.g., a keyboard, a mouse) that receives input from an administrator of the network monitoring system 100. The network monitoring system 100 may also include an output unit (e.g., a liquid crystal display, an organic EL (Electro Luminescence) display) that displays information to the administrator.

[0035] (Communication unit 110) The communication unit 110 is implemented by network devices such as a NIC (Network Interface Card), an optical fiber cable, an L2 (Layer 2) switch, an L3 (Layer 3) switch, a router, etc. The communication unit 110 is connected to a monitoring line 200. The communication unit 110 can send and receive data to and from the network 300 via the monitoring line 200.

[0036] (control unit 120) The control unit 120 is implemented by a data processing device and various programs stored in a storage device. The data processing device is, for example, a processor such as a central processing unit (CPU), a micro processing unit (MPU), or a general purpose graphic processing unit (GPGPU). The control unit 120 can be implemented as a controller for controlling multiple operations of the network monitoring system 100. For example, when one or more processors execute a program (multiple instructions) by using a random access memory (RAM) as a working area, the one or more processors perform multiple operations.

[0037] The control unit 120 can receive input data for the flow statistic information transfer process from an external device. The control unit 120 can store data such as the input data, data used in the flow statistic information transfer process, and output data of the flow statistic information transfer process in the storage unit 130. The control unit 120 can obtain such data from the storage unit 130 as needed.

[0038] (Storage unit 130) The storage unit 130 is implemented by a semiconductor memory such as a RAM or a flash memory, a magnetic disk such as a hard disk, or an optical disk. The storage unit 130 can store various programs and various data.

[0039] 4, the control unit 120 includes a collection unit 121, a binarization unit 122, a JSON conversion unit 123, and a visualization unit 124. The binarization unit 122 is an example of a conversion unit, a transfer unit, and an acquisition unit. The JSON conversion unit 123 is an example of a reception unit, an inverse conversion unit, and an acquisition unit. The data processing performed by each unit will be described below.

[0040] (Collection Department 121) The collector 121 collects flow statistics information from the network 300. The xFlow conversion device 21 described above with reference to Fig. 3 is an example of the collector 121. For example, the collector 121 has a function similar to that of a flow exporter.

[0041] (Binarization unit 122) The binarization unit 122 converts the flow statistics information into binary data. The binarization unit 122 transfers the binary data to the JSON conversion unit 123 through a transmission path (for example, the transmission section in FIG. 3). The xFlow conversion device 21 described above with reference to FIG. 3 is an example of the binarization unit 122.

[0042] (JSON conversion section 123) The JSON conversion unit 123 receives the binary data through the transmission path. The JSON conversion unit 123 converts the binary data back into flow statistical information. The JSON registration device 22 described above with reference to FIG. 3 is an example of the JSON conversion unit 123. The flow statistical information is transmitted from the JSON conversion unit 123 to the visualization unit 124.

[0043] (Visualization section 124) The visualization unit 124 visualizes traffic using flow statistical information. The visualization API group 13 described above with reference to Fig. 3 is an example of the visualization unit 124. For example, the visualization unit 124 has a function similar to that of a flow collector.

[0044] [5. Details of flow statistics information transfer process] An overview of one flow statistics information forwarding process has been explained above with reference to Figure 3. The details of the flow statistics information forwarding process will be explained in this section using sequence diagrams.

[0045] A sequence diagram of an example of flow statistic information transfer processing will be described with reference to FIG. 5. The example of flow statistic information transfer processing includes processing for transmitting flow statistic information as binary data. The processing for transmitting flow statistic information as binary data is performed, for example, by the xFlow conversion device 21 and JSON registration device 22 in FIG. 3. The xFlow conversion device 21 is an example of the binarization unit 122 in FIG. 4. The JSON registration device 22 is an example of the JSONization unit 123 in FIG. 4.

[0046] 5 is a sequence diagram showing process P100, which is an example of a process for transmitting flow statistics as binary data. Before process P100 starts, the xFlow conversion device 21 collects header samples and xFlow packets from the network 300. The xFlow conversion device 21 extracts the inner packet from the encapsulated packet.

[0047] The xFlow conversion device 21 transmits the config information to the JSON registration device 22 (step S101). The config information is stored in the xFlow conversion device 21.

[0048] The JSON registration device 22 receives the config information (step S102). The JSON registration device 22 reads the config information. As a result, the config information is copied from the xFlow conversion device 21 to the JSON registration device 22.

[0049] The xFlow conversion device 21 creates flow statistical data in JSON format based on the inner packet (step S103).

[0050] The xFlow conversion device 21 creates binary data using the data extracted from the flow statistical data (step S104).

[0051] 6 shows a binarization example 30 of flow statistical information according to the present disclosure. The binarization example 30 shows the data structure in the process for binarizing flow statistical data. The flow statistical data is JSON data.

[0052] The xFlow conversion device 21 obtains KEY information from the JSON data. The KEY information includes multiple keys. A key is also called a variable name. The xFlow conversion device 21 also obtains other data associated with these keys.

[0053] The xFlow converter 21 binarizes the JSON data. In the example of FIG. 6, the key "fct_id" is replaced with the binary value "1" based on the config information. Similarly, the value size "2 Bytes" is replaced with the binary value "2", and the value "256" is replaced with the binary value 256. The KEY information and other data are transmitted from the xFlow converter 21 to the JSON registration device 22 using the config information.

[0054] As shown in Figure 6, multiple keys such as fct_id, inner_Src_IP, and GTP_TEID are replaced with binary values ​​such as 1, 2, and 3. Typically, JSON data has approximately 50 keys. For example, when this JSON data is sent using HTTP (Hypertext Transfer Protocol), the total amount of transferred data is 1,784 bytes. On the other hand, when the JSON data is converted to binary data, the total amount of transferred data is 300 bytes. The xFlow conversion device 21 can significantly compress the amount of transmitted data by simplifying the KEY information.

[0055] Returning to FIG. 5, the xFlow conversion device 21 transmits the binary data to the JSON registration device 22 (step S105).

[0056] The JSON registration device 22 receives the binary data (step S106).

[0057] The JSON registration device 22 creates JSON data using a combination of binary data and config information (step S107).

[0058] 7 shows a JSON representation 40, which is an example of JSON-izing binary data according to the present disclosure. The JSON representation 40 shows the data structure in the process of JSON-izing binary data. The JSON registration device 22 generates JSON data by combining binary data with configuration information.

[0059] 7, the JSON registration device 22 reads the config information and then generates JSON data based on the binary data. For example, the binary value "1" is replaced with the key "fct_id". Similarly, the binary value "2" is replaced with "2Byte", and the binary value of 256 is replaced with the value "256".

[0060] Returning to FIG. 5, the JSON registration device 22 transmits the JSON data to the message queue 14 (step S108).

[0061] The message queue 14 receives the JSON data (step S109).

[0062] [6. Effects] The xFlow conversion device 21 can significantly reduce the amount of data to be transferred by transferring binary data before JSON data is generated. As a result, the bandwidth required for the transmission section is significantly reduced.

[0063] [7. Hardware Configuration] 8 is a diagram showing an example of a computer hardware configuration, ie, a computer 1000. The systems and methods described in this specification are implemented by the computer 1000, for example.

[0064] Computer 1000 is an example of a computer that implements network monitoring system 100 by executing a program. Computer 1000 includes memory 1010 and a CPU 1020. Computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0065] The memory 1010 includes a ROM (Read Only Memory) 1011 and a RAM 1012. The ROM 1011 stores, for example, a boot program such as a BIOS (Basic Input Output System). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium (for example, a magnetic disk or an optical disk) can be inserted into the disk drive 1100. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0066] The hard disk drive 1090 stores an OS 1091, an application program 1092, a program module 1093, and program data 1094. The programs executed by the computer 1000 define multiple operations of the network monitoring system 100. These programs may be implemented as program modules 1093 written in code executable by the computer 1000. The program modules 1093 are stored in, for example, the hard disk drive 1090. For example, the hard disk drive 1090 stores the program modules 1093 for executing processes similar to the functions of the components of the network monitoring system 100. The hard disk drive 1090 may be replaced with an SSD (Solid State Drive).

[0067] The hard disk drive 1090 can store a flow statistics information transfer program for the flow statistics information transfer process. The hard disk drive 1090 may store a computer program product including a flow statistics information transfer program (a plurality of instructions). When executed, the flow statistics information transfer program performs one or more of the methods described above.

[0068] The setting data used in the various processes described above may be implemented as program data 1094. The setting data is stored in, for example, the memory 1010 or the hard disk drive 1090. The CPU 1020 loads the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as needed. Then, the CPU 1020 performs the various processes described above.

[0069] The program module 1093 and the program data 1094 may be stored in a removable storage medium instead of the hard disk drive 1090. The CPU 1020 may load the program module 1093 and the program data 1094 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected to the computer 1000 via a network (such as a LAN or WAN). In this case, the CPU 1020 may load the program module 1093 and the program data 1094 via the network interface 1070.

[0070] 8. Summary of Embodiments As described above, network monitoring system 100 includes collector 121 and binarizer 122. In at least one embodiment, collector 121 collects flow statistics. In at least one embodiment, binarizer 122 converts the flow statistics into binary data. Binarizer 122 then forwards the binary data over a transmission path.

[0071] As described above, the network monitoring system 100 includes a JSONizer 123. In at least one embodiment, the JSONizer 123 receives binary data over a transmission path and then converts the binary data back into flow statistics.

[0072] In some embodiments, the flow statistics information is in JSON format.

[0073] In some embodiments, the binarizer 122 and the JSONizer 123 obtain configuration information that specifies mappings between multiple keys of the flow statistics information in JSON format and multiple predetermined binary values. The configuration information is, for example, the config information described above with reference to FIG. 7.

[0074] In some embodiments, the binarizer 122 replaces multiple keys of the flow statistics with multiple predetermined binary values, respectively, based on the configuration information. In other words, converting the flow statistics into binary data may include replacing multiple keys of the flow statistics with multiple predetermined binary values, respectively, based on the configuration information.

[0075] In some embodiments, the JSON conversion unit 123 replaces a plurality of predetermined binary values ​​with a plurality of keys of the flow statistics, respectively, based on the configuration information. In other words, converting the binary data back into the flow statistics may include replacing a plurality of predetermined binary values ​​with a plurality of keys of the flow statistics, respectively, based on the configuration information.

[0076] In some embodiments, the binarizer 122 replaces multiple key values ​​associated with multiple keys with multiple binary values ​​corresponding to the multiple key values. In other words, converting the flow statistics information into binary data may include replacing multiple key values ​​associated with multiple keys with multiple binary values ​​corresponding to the multiple key values. In some embodiments, the JSONizer 123 replaces these binary values ​​with multiple key values. In other words, converting the binary data back into flow statistics information may include replacing these binary values ​​with multiple key values.

[0077] [9. Addendum] Finally, the above description is supplemented with other embodiments. Various embodiments have been described above with reference to the drawings. These embodiments are exemplary, and the above description is not intended to limit the present disclosure to these embodiments. The features described in this specification can be implemented in various ways, including modifications and improvements based on the knowledge of those skilled in the art.

[0078] (various variations) In this specification, some processes have been described as being performed automatically. Some of these processes may be performed manually. Some other processes have been described as being performed manually. All or part of these other processes may be performed automatically using known methods.

[0079] Various implementations of the network monitoring system 100 are described herein or illustrated in the drawings. Some implementations relate to information including various data, data processing procedures, specific names, or parameters. Such implementations may be arbitrarily modified unless otherwise specified. For example, the various data are not limited to the data illustrated in the drawings.

[0080] Components of the system and device are shown in the drawings. The illustrated components conceptually illustrate the functions of the system and device. The components are not necessarily physically configured as shown in the drawings. The components may be integrated or distributed, and the specific forms of the system and device are not limited to the forms shown in the drawings. All or part of the system and device may be functionally or physically integrated or distributed depending on various loads and usage situations.

[0081] (Terms that represent components) The term "module," "section," "-er suffix," or "-or suffix" can be read as "unit," "means," "circuit," etc. For example, a communication module, a control module, and a storage module can be read as a communication unit, a control unit, and a storage unit, respectively.

[0082] (Configuration of control unit) The configuration of the control unit 120 shown in Fig. 4 is exemplary, and data processing described with respect to a particular unit may not necessarily be performed by that particular unit. For example, the binarization unit 122 may perform the data processing described with respect to the collection unit 121. Furthermore, the control unit 120 may include other units not shown in Fig. 4. The other units may perform the data processing described with respect to the control unit 120.

[0083] (Data Processing Device) The data processing device described for the control unit 120 is not limited to the specific hardware described above, and may be, for example, various types of computers or integrated circuits such as an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a GPGPU (General Purpose Graphic Processing Unit). [Explanation of symbols]

[0084] 1 Environment 100 Network Monitoring System 110 Communications Department 120 control section 121 Collection Department 122 Binary conversion section 123 JSON conversion section 124 Visualization section 130 Storage section 200 Monitoring line 300 Network

Claims

1. A flow statistics information transfer system comprising an xFlow conversion device and a JSON (JavaScript Object Notation) registration device, The xFlow conversion device includes: an acquisition unit that acquires configuration information that specifies mapping between a plurality of keys of flow statistics information in JSON format and a plurality of predetermined binary values; a transmitter that transmits the configuration information to the JSON registration device; a collection unit that collects flow statistics information using xFlow technology, the flow statistics information being in JSON format; a conversion unit that converts the flow statistical information collected by the collection unit into binary data based on the configuration information; a transfer unit that transfers the binary data to the JSON registration device through a transmission path between an xFlow conversion device and a message queue of a group of visualization APIs (Application Programming Interfaces), the group of visualization APIs processing the flow statistics information to visualize traffic; Equipped with The JSON registration device: a receiving unit that receives the binary data through the transmission path; an inverse conversion unit that inversely converts the binary data into the flow statistical information based on the configuration information; a sending unit that sends the flow statistics information to the message queue; Equipped with Flow statistics forwarding system.

2. the conversion unit replaces a plurality of keys of the flow statistics information with the plurality of predetermined binary values ​​based on the configuration information, The inverse conversion unit replaces the plurality of predetermined binary values ​​with a plurality of keys of the flow statistics information based on the configuration information, respectively. The flow statistics information transfer system according to claim 1 .

3. the conversion unit replaces a plurality of key values ​​associated with the plurality of keys with a plurality of binary values ​​corresponding to the plurality of key values; The inverse conversion unit replaces the plurality of binary values ​​with the plurality of key values. The flow statistics information transfer system according to claim 1 .

4. 1. A computer-implemented method for transferring flow statistics information, comprising: an acquisition step of acquiring configuration information specifying mapping between a plurality of keys of flow statistics information in JavaScript Object Notation (JSON) format and a plurality of predetermined binary values ​​by an xFlow conversion device; a sending step of sending the configuration information to a JSON registrar by the xFlow conversion device; a collecting step of collecting flow statistics information by the xFlow conversion device using xFlow technology, wherein the flow statistics information is in JSON format; a conversion step of converting the flow statistics information collected by the collection step into binary data by the xFlow conversion device based on the configuration information; a transfer step in which the xFlow converter transfers the binary data to the JSON registration device through a transmission path between the xFlow converter and a message queue of a group of visualization application programming interfaces (APIs), the group of visualization APIs processing the flow statistics information to visualize traffic; a receiving step of receiving the binary data through the transmission path by the JSON registration device; an inverse conversion step of inversely converting the binary data into the flow statistics information based on the configuration information by the JSON registration device; a sending step of sending the flow statistics information to the message queue by the JSON registration device; A flow statistics transfer method including:

5. A flow statistic information transfer program for causing a computer to function as the flow statistic information transfer system according to any one of claims 1 to 3.

Images