A message collection method and system

Abstract

This application relates to the field of data analysis technology, and in particular to a message acquisition method and system. The method is applied to a message acquisition system, which includes at least two collectors. Each collector includes a first thread for acquiring messages and a second thread for processing messages. The method includes: if the first collector receives a target message, the first thread of the first collector obtains the inner 5-tuple information of the target message; based on the inner 5-tuple information of the target message, the first thread of the first collector determines whether the target collector for processing the target message is the first collector; if the first thread of the first collector determines that the target collector for processing the target message is not the first collector, the first thread of the first collector forwards the target message to the second collector, so that the second thread of the second collector processes the target message.

Description

Technical Field

[0001] This application relates to the field of data analysis technology, and in particular to a message acquisition method and system. Background Technology

[0002] To obtain a complete packet forwarding path, the Encapsulated Remote Switch Port Analyzer (ERSPAN) technique is typically used. In this technique, all switches in the network are configured to send packets to the data collector via the IP protocol. The collector then associates packets with the same five-tuple to obtain a complete packet forwarding path. When the data volume exceeds the processing capacity of a single data collector, it is necessary to horizontally scale up the number of data collectors to achieve greater collection and processing capabilities. However, this can lead to the original packets being distributed across different data collectors, causing the collectors to be unable to calculate a complete forwarding path. Summary of the Invention

[0003] This application provides a message acquisition method and apparatus.

[0004] In a first aspect, this application provides a message acquisition method applied to a message acquisition system, the message acquisition system comprising at least two collectors, each collector comprising a first thread for acquiring messages and a second thread for processing messages; the method comprising:

[0005] If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message;

[0006] The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet.

[0007] If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

[0008] Optionally, the method further includes:

[0009] If the first thread of the first collector determines that the target collector used to process the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector.

[0010] The second thread of the first collector processes the target message.

[0011] Optionally, each collector in the message acquisition system has a pre-set mapping relationship between hash values ​​and collectors; the first thread of the first collector determines whether the target collector for processing the target message is the first collector based on the inner five-tuple information of the target message, including:

[0012] The first thread of the first collector performs a hash calculation on the inner five-tuple information of the target message to obtain the target hash value;

[0013] The first thread of the first collector determines the target collector for processing the target message based on the target hash value and the mapping relationship between each hash value and the collector;

[0014] The first thread of the first collector determines whether the target collector is the first collector.

[0015] Optionally, the step of the second thread of the first collector processing the target packet includes:

[0016] The second thread of the first collector performs concatenation session processing on target packets with the same inner 5-tuple information to obtain the forwarding path of the target packet in the forwarding network.

[0017] Optionally, the method further includes:

[0018] The first collector receives packets forwarded by the first thread of other collectors in the packet collection system;

[0019] The first collector sends the received message to the second thread of the first collector;

[0020] The second thread of the first collector processes the received message.

[0021] Secondly, this application provides a message acquisition system, which includes at least two collectors, each collector including a first thread for acquiring messages and a second thread for processing messages;

[0022] If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message;

[0023] The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet.

[0024] If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

[0025] Optionally, if the first thread of the first collector determines that the target collector for processing the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector.

[0026] The second thread of the first collector processes the target message.

[0027] Optionally, each collector in the message acquisition system has a pre-set mapping relationship between hash values ​​and collectors; the first thread of the first collector determines whether the target collector for processing the target message is the first collector based on the inner five-tuple information of the target message, including:

[0028] The first thread of the first collector performs a hash calculation on the inner five-tuple information of the target message to obtain the target hash value;

[0029] The first thread of the first collector determines the target collector for processing the target message based on the target hash value and the mapping relationship between each hash value and the collector;

[0030] The first thread of the first collector determines whether the target collector is the first collector.

[0031] Optionally, the step of the second thread of the first collector processing the target packet includes:

[0032] The second thread of the first collector performs concatenation session processing on target packets with the same inner 5-tuple information to obtain the forwarding path of the target packet in the forwarding network.

[0033] Optionally, the first collector receives packets forwarded by the first thread of other collectors in the packet collection system;

[0034] The first collector sends the received message to the second thread of the first collector;

[0035] The second thread of the first collector processes the received message.

[0036] Thirdly, embodiments of this application provide a message acquisition device, which is applied to a collector, and the message acquisition device includes:

[0037] Memory, used to store program instructions;

[0038] A processor is configured to invoke program instructions stored in the memory and execute the steps of the method as described in any one of the first aspects above, according to the obtained program instructions.

[0039] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to perform the steps of the method as described in any of the first aspects above.

[0040] In summary, the message acquisition method provided in this application is applied to a message acquisition system, which includes at least two collectors. Each collector includes a first thread for acquiring messages and a second thread for processing messages. The method includes: if the first collector receives a target message, the first thread of the first collector obtains the inner 5-tuple information of the target message; the first thread of the first collector determines, based on the inner 5-tuple information of the target message, whether the target collector for processing the target message is the first collector; if the first thread of the first collector determines that the target collector for processing the target message is not the first collector, the first thread of the first collector forwards the target message to the second collector, so that the second thread of the second collector processes the target message.

[0041] Using the message collection method provided in this application, the first thread of each collector receives a message, parses and processes it, performs a hash calculation on the inner five-tuple, and distributes the message to the second process of the corresponding collector based on the hash value. For messages whose hash value does not match their own, they are forwarded to the corresponding collector. This achieves accurate distribution based on the original message. In this way, by leveraging the load capacity of the device, load balancing is implemented in software, allowing for clustering and horizontal scaling while ensuring accurate message distribution. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments of this application or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings of the embodiments of this application.

[0043] Figure 1 A detailed flowchart of a message acquisition method provided for an embodiment of this application;

[0044] Figure 2This application provides a schematic diagram of the structure of a message acquisition system according to an embodiment of the present application.

[0045] Figure 3 This application provides a schematic diagram of a message forwarding path process.

[0046] Figure 4 This is a schematic diagram of a message acquisition device provided in an embodiment of this application. Detailed Implementation

[0047] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to limit the application. The singular forms “a,” “the,” and “the” as used in this application and claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to any and all possible combinations comprising one or more of the associated listed items.

[0048] It should be understood that although the terms first, second, third, etc., may be used to describe various information in embodiments of this application, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" may also be interpreted as "when," "when," or "in response to a determination."

[0049] To analyze the complete forwarding path of packets in a forwarding network, ERSPAN technology is typically used. All switches are configured to forward packets to the data collectors via the IP protocol. Currently, one implementation involves configuring OSPF technology on the edge switches of the data collection system. These edge switches connect to multiple data collectors, and each edge switch distributes packets based on the IP address of the received packets, ensuring that packets with the same IP address are distributed to the same data collector. However, edge switches can only distribute packets based on the outer source IP address. The inner original 5-tuple encapsulated by ERSPAN contains the actual data that reconstructs the packet's forwarding path in the forwarding network. Packets with identical original 5-tuples may still be distributed to different data collectors, making it impossible to calculate the complete forwarding path of the original packet in the forwarding network.

[0050] Furthermore, in order to calculate the complete forwarding path of the original packet in the forwarding network, a distributed data collection and centralized processing approach can be adopted, that is, to put the different packets to be processed in the original five-tuples onto the same processing node for unified processing. However, a single processing point also has a performance limit during unified processing, and the amount of computation for unified processing will increase exponentially.

[0051] This application addresses the issue that edge switches cannot distribute data based on the inner five-tuple of a packet, and provides a packet acquisition method and system. For example, see [link to relevant documentation]. Figure 1 The diagram shown is a detailed flowchart of a message acquisition method provided in an embodiment of this application. This method is applied to a message acquisition system, which includes at least two collectors. Each collector includes a first thread for acquiring messages and a second thread for processing messages. The method includes the following steps:

[0052] Step 100: If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message.

[0053] In practical applications, ERSPAN technology is used in the forwarding network. Each forwarding device in the forwarding network encapsulates the received packets and forwards them to the packet acquisition system through ERSPAN. Specifically, the packet acquisition system can distribute the received ERSPAN-encapsulated packets to each collector based on preset rules.

[0054] So, when the first collector receives the target message, which is encapsulated using ERSPAN technology and sent to the first collector, it calls the local first thread to perform message parsing and processing to obtain the inner five-tuple information of the target message.

[0055] Step 110: The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet.

[0056] In this embodiment of the application, each collector in the message acquisition system is pre-configured with a mapping relationship between each hash value and the collector.

[0057] For example, assuming the message collection system includes collector 1 and collector 2, the mapping relationship between the pre-set hash value on each collector and the collector can be: hash value 0 maps to collector 1, hash value 1 maps to collector 2.

[0058] In this embodiment of the application, when the first thread of the first collector determines whether the target collector for processing the target packet is the first collector based on the inner five-tuple information of the target packet, a preferred implementation is as follows: the first thread of the first collector performs a hash calculation on the inner five-tuple information of the target packet to obtain a target hash value; the first thread of the first collector determines the target collector for processing the target packet based on the target hash value and the mapping relationship between each hash value and the collector; the first thread of the first collector determines whether the target collector is the first collector.

[0059] Specifically, after the first collector obtains the inner five-tuple information of the target message, it performs a hash calculation on the obtained inner five-tuple information to obtain the corresponding hash value, and then determines the target collector to process the target message based on the hash value.

[0060] Step 120: If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

[0061] In other words, if it is determined that the target packet is not a packet that this collector (the first collector) needs to process, then it is necessary to determine another collector (such as the second collector) to process the target packet, and then forward the target packet to the second collector for packet processing based on the IP address exposed by the second collector.

[0062] Specifically, after receiving the target message sent by the first collector, the second collector can directly use its local second thread to process the target message.

[0063] In other words, in this embodiment of the application, each collector includes two threads: a first thread (pre-processing thread) for collecting / distributing packets, which can perform load balancing. The packet collection system can send packets with the same inner 5-tuple to the same collector for packet processing based on the first thread of each collector; and a second thread (processing thread) for processing the packets.

[0064] Furthermore, in this embodiment of the application, if the first thread of the first collector determines that the target collector for processing the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector; the second thread of the first collector processes the target message.

[0065] If the first collector determines that the target packet is one it needs to process, it can process it locally without forwarding it to other collectors. Specifically, the second thread of the first collector can process the target packet.

[0066] In this embodiment of the application, when the second thread of the first collector processes the target packet, a preferred implementation is as follows: the second thread of the first collector performs concatenation session processing based on the target packets with the same inner 5-tuple information received, to obtain the forwarding path of the target packet in the forwarding network.

[0067] In this embodiment, the second thread processes the message and performs session processing based on messages with the same inner 5-tuple information. This is an existing related technology and will not be described in detail here.

[0068] Furthermore, the first collector will also receive packets forwarded by other collectors in the packet collection system. For example, after receiving a packet sent by another collector, the second thread of the first collector will process the packet.

[0069] This achieves load balancing for the data collectors. Each collector can distribute data based on the inner 5-tuple information of the packet, assigning packets with the same inner 5-tuple information to the same collector. This allows each collector to accurately calculate the actual forwarding path for each packet. When the data collectors in the packet acquisition system are horizontally expanded, it is only necessary to update the preset mapping relationship between the hash values ​​and the collectors themselves. Each collector can still fully calculate the forwarding path for each packet that needs to be processed.

[0070] This application also provides a message acquisition system, for example, see [link to relevant documentation]. Figure 2 The diagram shown is a schematic diagram of a message acquisition system provided in an embodiment of this application. The message acquisition system includes at least two collectors (e.g., collector 1 and collector 2), and each collector includes a first thread for acquiring messages and a second thread for processing messages.

[0071] If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message;

[0072] The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet.

[0073] If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

[0074] Optionally, if the first thread of the first collector determines that the target collector for processing the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector.

[0075] The second thread of the first collector processes the target message.

[0076] Optionally, each collector in the message acquisition system has a pre-set mapping relationship between hash values ​​and collectors; the first thread of the first collector determines whether the target collector for processing the target message is the first collector based on the inner five-tuple information of the target message, including:

[0077] The first thread of the first collector performs a hash calculation on the inner five-tuple information of the target message to obtain the target hash value;

[0078] The first thread of the first collector determines the target collector for processing the target message based on the target hash value and the mapping relationship between each hash value and the collector;

[0079] The first thread of the first collector determines whether the target collector is the first collector.

[0080] Optionally, the step of the second thread of the first collector processing the target packet includes:

[0081] The second thread of the first collector performs concatenation session processing on target packets with the same inner 5-tuple information to obtain the forwarding path of the target packet in the forwarding network.

[0082] Optionally, the first collector receives packets forwarded by the first thread of other collectors in the packet collection system;

[0083] The first collector sends the received message to the second thread of the first collector;

[0084] The second thread of the first collector processes the received message.

[0085] The message acquisition method provided in this application embodiment will be described in detail below with reference to specific application scenarios. For example, see [link to relevant documentation]. Figure 3The diagram illustrates a message forwarding path process according to an embodiment of this application. A Spine device is connected to a forwarding network, with two Leaf devices connected to it. Each Leaf device is connected to two collectors. Each collector includes a pre-processing thread for load balancing and a processing thread for concatenating multiple identical sessions. After receiving the encapsulated message from the forwarding network, the Spine device distributes the message based on its outer five-tuple, sending the encapsulated message to Leaf1 or Leaf2. If Leaf1 receives message 1 distributed by the Spine device, it will... Message 1 is sent to Collector 1. The front-end thread of Collector 1 obtains the inner five-tuple information of Message 1 and determines itself as the collector to process Message 1 based on the inner five-tuple information of Message 1. Then, it sends Message 1 to the processing thread for processing. If Leaf1 receives Message 3 distributed by Spine device, it sends Message 3 to Collector 1. The front-end thread of Collector 1 obtains the inner five-tuple information of Message 3 and determines itself as Collector 2 as the collector to process Message 1 based on the inner five-tuple information of Message 3. Then, it sends Message 3 to the processing thread of Collector 2 for processing.

[0086] In this way, even if the analyzers are in different networks, they can still complete the on-demand distribution and load sharing of packets based on the front-end thread, so that packets with the same inner five-tuple information can be distributed to the processing thread of the same collector for processing.

[0087] These units can be one or more integrated circuits configured to implement the above methods, such as one or more Application Specific Integrated Circuits (ASICs), one or more digital signal processors (DSPs), or one or more Field Programmable Gate Arrays (FPGAs). Alternatively, when one of these units is implemented using processing element scheduler code, the processing element can be a general-purpose processor, such as a Central Processing Unit (CPU) or other processor capable of calling program code. Furthermore, these units can be integrated together to form a system-on-a-chip (SOC).

[0088] Furthermore, regarding the message acquisition device provided in this application embodiment, from a hardware perspective, the hardware architecture diagram of the message acquisition device can be found in [reference needed]. Figure 3 As shown, the message acquisition device may include: a memory 30 and a processor 31.

[0089] The memory 30 is used to store program instructions; the processor 31 calls the program instructions stored in the memory 30 and executes the above method embodiment according to the obtained program instructions. The specific implementation method and technical effect are similar, and will not be described again here.

[0090] Optionally, this application also provides a message acquisition device, including at least one processing element (or chip) for performing the above method embodiments.

[0091] Optionally, this application also provides a program product, such as a computer-readable storage medium storing computer-executable instructions for causing the computer to perform the above-described method embodiments.

[0092] Here, a machine-readable storage medium can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, etc. For example, a machine-readable storage medium can be: RAM (Random Access Memory), volatile memory, non-volatile memory, flash memory, storage drives (such as hard disk drives), solid-state drives, any type of storage disk (such as optical discs, DVDs, etc.), or similar storage media, or combinations thereof.

[0093] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which can take the form of a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email sending and receiving device, game console, tablet computer, wearable device, or any combination of these devices.

[0094] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0095] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, embodiments of this application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0096] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0097] Furthermore, these computer program instructions can also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in the process. Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0098] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0099] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A message acquisition method, characterized in that, The method is applied to a message acquisition system, which includes at least two acquirers, each of which includes a first thread for acquiring messages and a second thread for processing messages; the method includes: If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message; The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet. If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

2. The method as described in claim 1, characterized in that, The method further includes: If the first thread of the first collector determines that the target collector used to process the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector. The second thread of the first collector processes the target message.

3. The method as described in claim 1, characterized in that, The message acquisition system includes a pre-set mapping relationship between each hash value and the acquisition unit on each acquisition unit; the first thread of the first acquisition unit determines whether the target acquisition unit for processing the target message is the first acquisition unit based on the inner five-tuple information of the target message, including the following steps: The first thread of the first collector performs a hash calculation on the inner five-tuple information of the target message to obtain the target hash value; The first thread of the first collector determines the target collector for processing the target message based on the target hash value and the mapping relationship between each hash value and the collector; The first thread of the first collector determines whether the target collector is the first collector.

4. The method according to any one of claims 1-3, characterized in that, The steps for the second thread of the first collector to process the target packet include: The second thread of the first collector performs concatenation session processing on target packets with the same inner 5-tuple information to obtain the forwarding path of the target packet in the forwarding network.

5. The method as described in claim 4, characterized in that, The method further includes: The first collector receives packets forwarded by the first thread of other collectors in the packet collection system; The first collector sends the received message to the second thread of the first collector; The second thread of the first collector processes the received message.

6. A message acquisition system, characterized in that, The message acquisition system includes at least two collectors, each collector including a first thread for acquiring messages and a second thread for processing messages; If the first collector receives the target message, the first thread of the first collector obtains the inner quintuple information of the target message; The first thread of the first collector determines whether the target collector used to process the target packet is the first collector based on the inner five-tuple information of the target packet. If the first thread of the first collector determines that the target collector for processing the target packet is not the first collector, then the first thread of the first collector forwards the target packet to the second collector, so that the second thread of the second collector can process the target packet.

7. The system as described in claim 6, characterized in that, If the first thread of the first collector determines that the target collector used to process the target message is the first collector, then the first thread of the first collector sends the target message to the second thread of the first collector. The second thread of the first collector processes the target message.

8. The system as described in claim 6, characterized in that, The message acquisition system includes a pre-set mapping relationship between each hash value and the acquisition unit on each acquisition unit; the first thread of the first acquisition unit determines whether the target acquisition unit for processing the target message is the first acquisition unit based on the inner five-tuple information of the target message, including the following steps: The first thread of the first collector performs a hash calculation on the inner five-tuple information of the target message to obtain the target hash value; The first thread of the first collector determines the target collector for processing the target message based on the target hash value and the mapping relationship between each hash value and the collector; The first thread of the first collector determines whether the target collector is the first collector.

9. The system according to any one of claims 6-8, characterized in that, The steps for the second thread of the first collector to process the target packet include: The second thread of the first collector performs concatenation session processing on target packets with the same inner 5-tuple information to obtain the forwarding path of the target packet in the forwarding network.

10. The system as described in claim 9, characterized in that, The first collector receives packets forwarded by the first thread of other collectors in the packet collection system; The first collector sends the received message to the second thread of the first collector; The second thread of the first collector processes the received message.

Images