Data processing method and apparatus

Abstract

A data processing method and apparatus, relating to the technical field of communications, and used for reducing the power consumption of a network device. The specific solution comprises: a first network device receives a plurality of data streams, and parses each data stream to obtain quintuple information; on the basis of the quintuple information and a filtering model, the first network device filters the plurality of data streams, wherein the filtering model is generated by a second network device on the basis of the quintuple information of the plurality of data streams, and the filtering model comprises quadruple information or triple information; and the first network device sends the plurality of filtered data streams to a third network device. The present application is used in a process where a network device processes data streams.

Description

Data processing methods and apparatus

[0001] This application claims priority to Chinese Patent Application No. 202410946324.8, filed on July 12, 2024, entitled “Method and Apparatus for Data Processing”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of communication technology, and in particular to a data processing method and apparatus. Background Technology

[0003] With the rapid development of the Internet, network devices need to handle more and more traffic, thus placing increasingly stringent demands on their performance.

[0004] Taking a server as an example, after receiving a data packet, the server can forward it to other corresponding routers, terminals, or servers based on the packet's five-tuple information. This five-tuple information can include the source address, source port, destination address, destination port, and protocol number. Similarly, data packets can also carry four-tuple or three-tuple information. A four-tuple can include the source address, destination address, destination port, and protocol number, while a three-tuple can include the destination address, destination port, and protocol number. However, network devices that forward data packets rely on network resources such as bandwidth, buffer space, and the processing power of network devices. When the required network resources exceed the currently available network resources, network congestion occurs.

[0005] However, with the continuous updates and iterations of mobile communications, the increase in data traffic will be even more rapid. How to control the data traffic of network devices and reduce the power consumption of network devices has become an urgent problem to be solved. Summary of the Invention

[0006] This application provides a data processing method and apparatus that reduces the power consumption of network devices.

[0007] To achieve the above objectives, the embodiments of this application adopt the following technical solutions.

[0008] In a first aspect, embodiments of this application provide a data processing method, which includes: a first network device receiving multiple data streams and parsing each data stream to obtain quintuple information; the first network device filtering the multiple data streams based on the quintuple information and a filtering model, wherein the filtering model is generated by a second network device based on the quintuple information of the multiple data streams and includes quadruple information or triplet information; and the first network device sending the filtered multiple data streams to a third network device.

[0009] Therefore, in the data processing method provided in this application embodiment, multiple data streams can be filtered based on the five-tuple information and the filtering model. Among them, the data streams that match the five-tuple information and the four-tuple information or the three-tuple information in the filtering model can be considered as low-value traffic. The first network device can filter this part of low-value traffic, which can reduce the number of data streams sent to the third network device, reduce the traffic sent by the first network device to the third network device, and reduce the power consumption of the first network device.

[0010] In one possible design, the first network device filters multiple data streams based on 5-tuple information and a filtering model. This includes filtering the first data stream if its 5-tuple information matches the 4-tuple information in the filtering model, and / or filtering the second data stream if its 5-tuple information matches the 3-tuple information in the filtering model. Therefore, data streams corresponding to the 4-tuple or 3-tuple information in the filtering model can be understood as low-value traffic, i.e., data streams that need to be filtered. When the 5-tuple information of a data stream matches the 4-tuple or 3-tuple information, that data stream is also low-value traffic and can be filtered to reduce the number of data streams sent to the third network device.

[0011] In one possible design, before the first network device sends the filtered multiple data streams to the third network device, the method further includes: the first network device compressing the filtered multiple data streams. Thus, by compressing the multiple data streams, the traffic sent by the first network device to the third network device is reduced, thereby reducing the power consumption of the first network device.

[0012] In one possible design, before the first network device filters multiple data streams based on the five-tuple information and the filtering model, the method further includes: the first network device performing protocol parsing on the multiple data streams to obtain the protocol type corresponding to each data stream; and the first network device filtering the multiple data streams based on the protocol type. Therefore, filtering multiple data streams by protocol type can also reduce the number of data streams sent from the first network device to the third network device, thereby reducing the traffic sent from the first network device to the third network device and reducing the power consumption of the first network device.

[0013] In one possible design, the first network device filters multiple data streams based on protocol type, including: if the protocol type corresponding to a third data stream among the multiple data streams meets a preset rule, the first network device filters the third data stream. Therefore, data streams corresponding to protocol types that meet the preset rule can be understood as low-value traffic, and thus these data streams can be filtered to reduce the number of data streams sent to the third network device.

[0014] In one possible design, the data processing method further includes: the first network device receiving a filtering model sent by the second network device.

[0015] Secondly, embodiments of this application provide a data processing method, which includes: a second network device receiving multiple data streams and parsing each data stream to obtain quintuple information; the second network device aggregating the quintuple information from the multiple data streams to obtain quadruple information or triple information; the second network device generating a filtering model based on the quadruple information or triple information; and the second network device sending the filtering model to a first network device.

[0016] Therefore, in the data processing method provided in this application embodiment, the second network device can learn and identify low-value traffic based on multiple data streams, convert the five-tuple information of low-value traffic into four-tuple information or three-tuple information, generate a filtering model based on the four-tuple information or three-tuple information, and synchronize the filtering model to the first network device, so as to support the first network device to filter multiple data streams based on the filtering model, thereby reducing the number of data streams sent by the first network device to the third network device and reducing the power consumption of the first network device.

[0017] In one possible design, before the second network device generates a filtering model based on quadruple or triplet information, the method further includes: the second network device acquiring the number of data packets for each data stream, and obtaining sub-stream label information for each data stream based on the number of data packets for each data stream and a first preset threshold. The sub-stream label information is used to indicate the type of data stream. The second network device filters multiple data streams based on the sub-stream label information for each data stream. Thus, multiple data streams can be classified into "large streams" or "small streams" based on the number of data packets in each data stream and the first preset threshold, where "large streams" can be understood as high-value traffic and "small streams" can be understood as low-value traffic.

[0018] In one possible design, the second network device generates a filtering model based on quadruple or triplet information. This includes: the second network device further filtering multiple data streams based on a second preset threshold, where the second preset threshold is related to the number of data streams corresponding to the five-tuple information; the second network device sorting the filtered data streams in descending order of the number of data streams corresponding to the five-tuple information; and generating a filtering model based on the quadruple or triplet information of the top-ranked data streams. Thus, the second network device can filter multiple data streams based on the second preset threshold and the number of data streams corresponding to the five-tuple information to obtain low-value traffic, thereby generating the filtering model.

[0019] Thirdly, embodiments of this application also provide a data processing method, the method comprising: a third network device receiving multiple filtered data streams; the third network device decompressing the multiple data streams to obtain multiple decompressed data streams; the third network device generating a document based on the multiple decompressed data streams and sending the document.

[0020] Fourthly, embodiments of this application also provide a first network device, which includes a receiving module, a filtering module, and a sending module. The receiving module receives multiple data streams and parses each data stream to obtain quintuple information. The filtering module filters the multiple data streams based on the quintuple information and a filtering model, which is generated by a second network device based on the quintuple information of the multiple data streams. The filtering model includes quadruple information or triplet information. The sending module sends the filtered multiple data streams to a third network device.

[0021] In one possible design, the filtering module is specifically used to: filter the first data stream if the quintuple information corresponding to the first data stream among multiple data streams matches the quadruple information in the filtering model; and / or filter the second data stream if the quintuple information corresponding to the second data stream among multiple data streams matches the triple information in the filtering model.

[0022] In one possible design, the first network device also includes a compression module for compressing the filtered multiple data streams.

[0023] In one possible design, the first network device further includes a parsing module for parsing multiple data streams to obtain the protocol type corresponding to each data stream. A filtering module is also used to filter the multiple data streams based on their protocol types.

[0024] In one possible design, the filtering module is specifically used to filter the third data stream if the protocol type of the third data stream among multiple data streams meets the preset rules.

[0025] In one possible design, the receiving module is also used to receive the filtering model sent by the second network device.

[0026] The beneficial effects of the fourth aspect can be found in the explanation of the first aspect.

[0027] Fifthly, embodiments of this application also provide a second network device, which includes a receiving module, an aggregation module, a generation module, and a sending module. The receiving module is used to receive multiple data streams and parse each data stream to obtain 5-tuple information. The aggregation module is used to aggregate the 5-tuple information from multiple data streams to obtain 4-tuple information or 3-tuple information. The generation module is used to generate a filtering model based on the 4-tuple information or 3-tuple information. The sending module is used to send the filtering model to the first network device.

[0028] In one possible design, the second network device further includes an acquisition module and a filtering module. The acquisition module is used to: acquire the number of data packets in each data stream, and based on the number of data packets in each data stream and a first preset threshold, obtain sub-stream label information for each data stream, the sub-stream label information being used to indicate the type of data stream. The filtering module is also used to: filter multiple data streams based on the sub-stream label information of each data stream.

[0029] In one possible design, the generation module is specifically used to: filter multiple data streams based on a second preset threshold, the second preset threshold being related to the number of data streams corresponding to the five-tuple information; sort the filtered multiple data streams in descending order of the number of data streams corresponding to the five-tuple information; and generate a filtering model based on the four-tuple or three-tuple information of the top-ranked multiple data streams.

[0030] The beneficial effects of the fifth aspect can be found in the explanation of the second aspect.

[0031] Sixthly, embodiments of this application provide a third network device, which includes a receiving module, a decompression module, and a generating module. The receiving module is used to receive multiple filtered data streams, the decompression module is used to decompress the multiple data streams to obtain decompressed multiple data streams, and the generating module is used to generate documents based on the decompressed multiple data streams and send the documents.

[0032] In a seventh aspect, embodiments of this application provide a computer-readable storage medium including computer instructions that, when executed on an electronic device, cause the electronic device to perform the data processing method in any of the above aspects and any possible implementations.

[0033] Eighthly, embodiments of this application provide a computer program product that, when run on a computer or processor, causes the computer or processor to perform the data processing method in any of the above aspects and any possible implementations.

[0034] It is understood that any of the first network device, second network device, third network device, computer-readable storage medium or computer program product provided above can be applied to the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects in the corresponding methods, and will not be repeated here.

[0035] These or other aspects of this application will become more readily apparent in the following description. Attached Figure Description

[0036] Figure 1 is a statistical chart of traffic and server quantity provided in an embodiment of this application;

[0037] Figure 2 is a flowchart of a sampling technique provided in an embodiment of this application;

[0038] Figure 3 is a flowchart of a data processing method provided in an embodiment of this application;

[0039] Figure 4 is a flowchart of a data stream matching process provided in an embodiment of this application;

[0040] Figure 5 is a flowchart of another data processing method provided in an embodiment of this application;

[0041] Figure 6 is a flowchart of a five-tuple information aggregation method provided in an embodiment of this application;

[0042] Figure 7 is a flowchart of a raw data stream processing method provided in an embodiment of this application;

[0043] Figure 8 is an interaction diagram of a first network device and a second network device provided in an embodiment of this application;

[0044] Figure 9 is a schematic diagram of the composition of a first network device provided in an embodiment of this application;

[0045] Figure 10 is a schematic diagram of the composition of a second network device provided in an embodiment of this application. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0047] Hereinafter, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this embodiment, unless otherwise stated, "a plurality of" means two or more.

[0048] The embodiments of this application use terms such as "first" and "second" to distinguish objects with similar names, functions, or effects. Those skilled in the art will understand that the terms "first" and "second" do not limit the quantity or order of execution. The term "coupling" is used to indicate an electrical connection, including direct connection via wires or terminals or indirect connection via other devices. Therefore, "coupling" should be considered as a broad type of electronic communication connection.

[0049] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0050] As network traffic increases, so does the number of servers. Figure 1 illustrates the traffic and server count at different times, with server size increasing at a rate of 5-6 gigabits per second (Gbps) per server. For example, between 2018 and 2022, traffic increased from 1800 Gbps to 5000 Gbps, and the number of servers increased from 570 to 770. However, server hardware, footprint, power consumption, and maintenance costs are relatively high. Considering these factors, the number of servers cannot be increased indefinitely, meaning large-scale investment in customer experience management (CEM) systems is no longer feasible. Therefore, a reasonable sampling scheme needs to be established on the data source side to reduce the overall platform configuration of the CEM management system. For example, an intelligent data collection technology can reduce the amount of data to be collected and the number of frequency analyses, eliminating the need to obtain the entire network information. It can process and store data only for faulty users or those on a very important person (VIP) whitelist using different strategies, dynamically increasing the amount of data collected in problem areas, the number of escalation events, or the number of special events.

[0051] To reduce the number of documents, a sampling technique is proposed, the process of which is shown in Figure 2. Specifically, the sampling process includes: parsing data packets, VIP user filtering, custom rule filtering, and sampling. VIP user filtering and custom rule filtering are commonly used techniques in network devices; additionally, sampling can reduce the number of documents.

[0052] However, using VIP user filtering, customized rule filtering, and sampling can all lead to business losses and affect network metric analysis. Furthermore, a full-scale data collection approach results in a large amount of low-value traffic wasting the customer experience management system's computing resources and reducing the product's performance.

[0053] Therefore, this application provides a data processing method that proposes a new filtering model. This filtering model can filter low-value traffic from multiple data streams, thereby reducing the number of data streams transmitted by network devices, reducing the traffic transmitted by network devices, and reducing the power consumption of network devices.

[0054] Each data stream, after being parsed by the probe, generates a record, which is a document. Documents generated from low-value data streams are low-value documents. The following section uses a call traffic model as an example to provide a detailed introduction to low-value documents, as shown in Table 1. Table 1 displays the document volume, document volume percentage, traffic volume, and traffic percentage for different document types. Document types can include other, website (web), file access (FA), domain name system (DNS), instant messaging (IM), and streaming.

[0055] Table 1

[0056] Table 2 shows the number of documents and the corresponding traffic volume in the "other" data stream. Specifically, the data streams can be categorized according to the number of data packets: less than 1, less than 3, less than 5, less than 10, less than 20, and more than 20.

[0057] Table 2

[0058] As can be seen from Tables 1 and 2, data streams with a total number of packets less than 20 account for 61.93% of the total number of documents in the "other" category, but the corresponding traffic only accounts for 0.179% of the total traffic. Therefore, these documents can be considered as low-value documents.

[0059] Filtering out these low-value documents can reduce the processing load on network devices. While reducing overall traffic by 0.179%, it can reduce the number of documents in the "other" category by 61.93%. Furthermore, since network devices also need to store flow tables in memory to process these documents, filtering them can reduce the flow table size by approximately 40%, resulting in a reduction of network device memory usage by more than 30%.

[0060] In the above scenario, the data processing method provided in this application embodiment can be applied to network devices, which can be communication devices such as switches, routers, bridges, gateways, servers, or terminal devices. This application embodiment does not limit the specific form of the communication device.

[0061] The data processing method provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0062] This application provides a data processing method, as shown in Figure 3, which is a flowchart of a data processing method provided in this application. The method is applied to a first network device and includes the following steps.

[0063] S301. The first network device receives multiple data streams and parses each data stream to obtain quintuple information.

[0064] For example, the multiple data streams can be data streams obtained by the first network device through mirroring technology, wherein mirroring technology is a technology that copies data packets received on an ingress port of the network device or data packets conforming to specified rules to a target port.

[0065] For example, each data stream may include multiple data packets. The first network device can parse the protocol of the data packets in the data stream to obtain the 5-tuple information. The protocol of the data packets may include Internet Protocol (IP), Transmission Control Protocol (TCP), and User Datagram Protocol (UDP), etc. The 5-tuple information may include source address, source port, destination address, destination port, and protocol number. For example, the 5-tuple information could be 192.168.1.1 1000 TCP 121.14.88.76 100, meaning a network device with a source address of 192.168.1.1 connects to a network device with a destination address of 121.14.88.76 and a destination port of 100 via source port 1000 and TCP.

[0066] S302, The first network device filters multiple data streams based on 5-tuple information and a filtering model.

[0067] The filtering model is generated by the second network device based on the five-tuple information of multiple data streams, and the filtering model includes four-tuple information or three-tuple information.

[0068] Prior to S302, the data processing method may also include: the first network device receiving and storing the filtering model sent by the second network device.

[0069] For example, the first network device can be understood as a front-end processor, and the second network device can be understood as a probe (model learning machine). The probe has the ability to learn the low-value traffic characteristic model, which can transform the five-tuple information of low-value traffic into four-tuple information or three-tuple information, and generate a filtering model that is synchronized to the front-end processor. That is to say, the filtering model sent by the second network device includes multiple four-tuple information or three-tuple information, and the data stream corresponding to the four-tuple information or three-tuple information is low-value traffic.

[0070] For example, the first network device may load a filtering model, which can be understood as including multiple matching terms and action terms. For instance, when the input of the filtering model matches at least one of the multiple matching terms, the action term may instruct the first network device to process the input according to the processing method indicated by the action term. For example, the matching term may include quadruple information or triple information, and the action term may instruct the first network device to filter the input when the input of the filtering model matches the quadruple information or triple information.

[0071] Optionally, S302 may include: if the five-tuple information corresponding to the first data stream among multiple data streams matches the four-tuple information in the filtering model, the first network device filters the first data stream.

[0072] For example, the first network device removes the source port from the five-tuple information of the first data stream to obtain four-tuple information. If the four-tuple information of the first data stream matches the four-tuple information in the filtering model, i.e., the first data stream is low-value traffic, the first network device can mark the first data stream as discardable and discard all data packets of the first data stream in subsequent operations. If the four-tuple information of the first data stream does not match the four-tuple information in the filtering model, i.e., the first data stream is high-value traffic, the first network device can perform subsequent processing on the first data stream.

[0073] And / or, S302 may include: if the quintuple information corresponding to the second data stream in the multiple data streams matches the triple information in the filtering model, the first network device filters the second data stream.

[0074] For example, the first network device removes the source address and source port from the five-tuple information of the second data stream to obtain the three-tuple information. If the three-tuple information of the second data stream matches the three-tuple information in the filtering model, i.e., the second data stream is low-value traffic, the first network device can mark the second data stream as discardable and discard all packets of the second data stream in subsequent operations. If the three-tuple information of the second data stream does not match the three-tuple information in the filtering model, i.e., the second data stream is high-value traffic, the first network device can perform further processing on the second data stream.

[0075] Therefore, the first network device can traverse each data stream to filter low-value data streams from multiple data streams. Specifically, the first network device obtains the five-tuple information of each data stream, removes the source port to obtain the four-tuple information, and removes the source address and source port to obtain the three-tuple information. The first network device then matches the obtained four-tuple or three-tuple information against the four-tuple or three-tuple information in the filtering model sequentially. As shown in Figure 4, the data stream matching process may include: S401, matching the four-tuple information of the data stream with the four-tuple information in the filtering model; if the four-tuple information of the data stream does not match the four-tuple information in the filtering model, proceed to S402; if the four-tuple information of the data stream matches the four-tuple information in the filtering model, proceed to S403. S402. Match the triple information of the data stream with the triple information in the filtering model. If the triple information of the data stream and the triple information in the filtering model do not match, proceed to S404. If the triple information of the data stream and the triple information in the filtering model match, proceed to S403. S403. Mark the data stream as discardable and discard all data packets in the data stream. S404. Perform subsequent processing on the data stream.

[0076] S303, The first network device sends multiple filtered data streams to the third network device.

[0077] For example, the third network device can be a service processor that performs subsequent processing on the data stream, such as parsing the data stream to generate documents.

[0078] Therefore, by filtering low-value traffic from multiple data streams through the first network device, the number of data streams sent to the third network device can be reduced, thus reducing the traffic sent from the first network device to the third network device and reducing the power consumption of the first network device.

[0079] Optionally, as shown in Figure 5, before S303, the method may further include S304: the first network device compresses the filtered multiple data streams.

[0080] For example, the first network device can compress multiple data streams using uplink data compression (UDC) technology. Specifically, UDC technology is mainly used for data packets with a large amount of repetitive content. By compressing redundant information between consecutive packets, it reduces the amount of data that needs to be transmitted, thereby saving data and reducing the power consumption of the network device. It is understood that the first network device can also compress multiple data streams in other ways, and this application embodiment does not limit this.

[0081] Optionally, referring to Figure 5, before S302, the method may further include S305: the first network device performs protocol parsing on multiple data streams to obtain the protocol type corresponding to each data stream; and S306: the first network device filters the multiple data streams based on the protocol type.

[0082] For example, protocol types may include Internet Control Message Protocol (ICMP), Network Time Protocol (NTP), Hypertext Transfer Protocol (HTTP), or Secure Sockets Layer (SSL), etc.

[0083] For example, the first network device can perform protocol parsing on the data stream to obtain the protocol type corresponding to the data stream. The first network device can also filter multiple data streams based on their protocol types. For instance, if the protocol type corresponding to a data stream is a low-value protocol, the data stream is considered low-value traffic, and the first network device can filter it. If the protocol type corresponding to a data stream is a high-value protocol, the data stream is considered high-value traffic, and the first network device can send it to a third network device for further processing. Therefore, filtering multiple data streams by protocol type can reduce the number of data streams sent from the first network device to the third network device, thereby reducing the traffic sent from the first network device to the third network device and lowering the power consumption of the first network device.

[0084] Optionally, S306 may include: if the protocol type corresponding to the third data stream among multiple data streams meets the preset rules, the first network device filters the third data stream.

[0085] For example, the preset rule could be: the protocol type is ICMP, NTP, or other custom protocols. That is, if the protocol type corresponding to the third data stream is ICMP, NTP, or other custom protocols, then the protocol type of the third data stream meets the preset rule, meaning the third data stream is low-value traffic, and the first network device can filter the third data stream. If the protocol type corresponding to the third data stream does not meet the preset rule, the first network device can perform subsequent processing on the third data stream.

[0086] Referring again to Figure 5, which also shows a flowchart of a data processing method applied to a second network device, the data processing method includes the following steps.

[0087] S307. The second network device receives multiple data streams and parses each data stream to obtain quintuple information.

[0088] For example, these multiple data streams can also be data streams acquired by the second network device through mirroring technology, and these multiple data streams can be understood as experimental data or learning data. Additionally, the second network device can also parse each data stream to obtain quintuple information.

[0089] S308, the second network device aggregates the five-tuple information from multiple data streams to obtain four-tuple information or three-tuple information.

[0090] For example, since the source address and source port are relatively easy to change, the characteristics of the source address and source port can be weakened in the filtering model. As shown in Figure 6, the aggregation process performed by the second network device may include selecting the source address, source port, destination address, destination port, and protocol number as the basis for aggregation to obtain a four-tuple of information, and then selecting the destination address, destination port, and protocol number again for aggregation to obtain a three-tuple of information.

[0091] As shown in Table 3, the table displays information for flow identifiers of quadruples and triplet. In addition, Table 3 also shows the flow type and the number of sample flows for quintuples corresponding to the quadruple and triplet information. Among them, "1" in the flow type represents "large flow" and "0" represents "small flow".

[0092] Table 3

[0093] S309, The second network device generates a filtering model based on quadruple or triple information.

[0094] For example, the second network device can establish a triple / quadruple model table based on quadruple information or triple information, and a filtering model can be generated based on the triple / quadruple model table.

[0095] S3010, The second network device sends the filtering model to the first network device.

[0096] For example, the second network device can learn and identify low-value traffic based on multiple data streams, convert the five-tuple information of low-value traffic into four-tuple information or three-tuple information, generate a filtering model based on the four-tuple information or three-tuple information, and synchronize the filtering model to the first network device to support the first network device to filter multiple data streams based on the filtering model, thereby reducing the number of data streams sent by the first network device to the third network device and reducing the power consumption of the first network device.

[0097] Optionally, prior to S309, the method further includes: the second network device acquiring the number of data packets for each data stream, and obtaining sub-stream label information for each data stream based on the number of data packets for each data stream and a first preset threshold. The second network device filters multiple data streams based on the sub-stream label information. The sub-stream label information is used to indicate the type of data stream.

[0098] For example, the second network device can build a hash table using 5-tuples as keys and count the number of data packets included in each data stream. The data stream type can include "large stream" and "small stream," where "large stream" includes more data packets and "small stream" includes fewer data packets. As shown in Figure 7, comparing the original data stream with a first preset threshold yields data streams with small stream label information. Specifically, if the number of data packets in a data stream is greater than the first preset threshold, the data stream is a "large stream," and the corresponding small stream label information is "1"; if the number of data packets in a data stream is less than or equal to the first preset threshold, the data stream is a "small stream," and the corresponding small stream label information is "0."

[0099] For example, the first preset threshold can be determined based on the distribution of the number of data streams and the traffic volume of data streams of different sizes in the analysis of historical data streams. In one possible example, the first preset threshold could be 20.

[0100] For example, the partitioning of data streams satisfies the following relationship: F smallpacket (thresh)=P smallpacket (X≤thresh) F bigpacket (thresh)=P bigpacket (X>thresh), where thresh = 20

[0101] Where X represents the number of data packets in the data stream, thresh represents the first preset threshold, Psmallpacket(X≤thresh) represents a data stream whose number of data packets is less than or equal to the first preset threshold, and Pbigpacket(X>thresh) represents a data stream whose number of data packets is greater than the first preset threshold.

[0102] Optionally, S309 may include: the second network device further filters the data streams corresponding to the quadruplet information or triplet information based on a second preset threshold, wherein the second preset threshold is related to the number of data streams corresponding to the quintuplet information.

[0103] For example, continuing to refer to Figure 7, the data stream with small stream label information and the second preset threshold can be compared to obtain the initial quadruple information or triple information.

[0104] For example, the second preset threshold can be the number of quintuple sample streams. In one possible example, the second preset threshold can be 10. Thus, if the number of quintuple sample streams corresponding to quaternary or ternary information is less than 10, the data stream corresponding to that quaternary or ternary information can be filtered. If the number of quintuple sample streams corresponding to quaternary or ternary information is greater than or equal to 10, the data stream corresponding to that quaternary or ternary information can be retained. It is understood that the second preset threshold can also be other values, and this embodiment of the application does not limit this.

[0105] For example, based on Table 3, the information of quadruplets and triplets filtered based on the second preset threshold is shown in Table 4.

[0106] Table 4

[0107] As can be seen from Table 4, the second network device filters the information of quadruples and triplets corresponding to a sample flow count of 2 for quintuples.

[0108] Optionally, S309 may further include: the second network device sorting the filtered multiple data streams in descending order of the number of data streams corresponding to the five-tuple information, and generating a filtering model based on a preset number of four-tuple or three-tuple information that are ranked first.

[0109] For example, continuing to refer to Figure 7, after sorting the initial quadruplet or triplet information in descending order, a quadruplet or triplet information model library can be obtained. Specifically, the number of data streams corresponding to the quintuple information can be understood as weights. Sorting them in descending order gives higher weights to data streams ranked earlier and lower weights to data streams ranked later. Therefore, the second network device can filter out data streams ranked later and generate a filtering model based on a preset number of quadruplet or triplet information items ranked earlier.

[0110] Therefore, the interaction between the first network device and the second network device is shown in Figure 8. The first network device receives packets, i.e., receives data packets from multiple data streams. The first network device also performs parsing, i.e., parsing the data streams to obtain 5-tuple information. The first network device also filters low-value traffic and other traffic based on a filtering model. The first network device builds a flow table based on the filtered data streams and sends this flow table to the second network device. The second network device receives packets and generates a flow table based on the 5-tuple information of the data streams. The second network device also generates a filtering model based on a learning model and sends the filtering model to the first network device. Thus, the second network device can continuously learn and iterate the filtering model to further reduce the power consumption of the first network device.

[0111] Referring again to Figure 5, which also shows a flowchart of a data processing method applied to a third network device, the data processing method includes the following steps.

[0112] S3011, the third network device receives multiple filtered data streams.

[0113] S3012, the third network device decompresses multiple data streams to obtain multiple decompressed data streams.

[0114] For example, the third network device can decompress multiple data streams based on the decompression method corresponding to the compression method of the first network device to obtain multiple decompressed data streams.

[0115] S3013, the third network device generates a document based on the decompressed multiple data streams and sends the document.

[0116] For example, the third network device will generate a record, or document, after parsing the data stream. This document can be a user-formatted document. The third network device can then send the generated user-formatted document to a third-party system.

[0117] It is understood that, in order to achieve the above functions, the electronic device includes hardware and / or software modules that perform the respective functions. Based on the algorithmic steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is implemented in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0118] This embodiment can divide the electronic device into functional modules according to the above method example. For example, each function can be divided into its own functional modules, or two or more functions can be integrated into one processing module. The integrated modules can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0119] With each functional module divided according to its corresponding function, Figure 9 shows a possible compositional schematic diagram of the first network device 900 involved in the above embodiments. As shown in Figure 9, the first network device 900 may include: a receiving module 901, a filtering module 902, and a sending module 903.

[0120] The receiving module 901 can be used to support the first network device 900 in performing the above-described steps S301, and / or other processes used in the technology described herein.

[0121] The filtering module 902 can be used to support the first network device 900 in performing the above steps S302 and S306, and / or other processes used in the technology described herein.

[0122] The transmitting module 903 can be used to support the first network device 900 in performing the above-described steps S303, and / or other processes used in the techniques described herein.

[0123] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0124] The first network device 900 provided in this embodiment is used to execute the above-described data processing method, and thus can achieve the same effect as the above-described implementation method.

[0125] When using integrated units, the first network device 900 may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the operations of the first network device 900, for example, it can support the first network device 900 in executing the steps performed by the receiving module 901, the filtering module 902, and the transmitting module 903. The storage module can be used to support the first network device 900 in storing program code and data. The communication module can be used to support communication between the first network device 900 and other devices, such as communication with wireless access devices.

[0126] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination of functions that implement computing capabilities, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0127] With each functional module divided according to its corresponding function, Figure 10 shows a possible compositional schematic diagram of the second network device 1000 involved in the above embodiment. As shown in Figure 10, the second network device 1000 may include: a receiving module 1001, an aggregation module 1002, a generation module 1003, and a sending module 1004.

[0128] The receiving module 1001 can be used to support the second network device 1000 in performing the above-described steps S307, etc., and / or other processes used in the technology described herein.

[0129] The aggregation module 1002 can be used to support the second network device 1000 in performing the above-described steps S308, and / or other processes for the technology described herein.

[0130] The generation module 1003 can be used to support the second network device 1000 in performing the above-described steps S309, and / or other processes used in the techniques described herein.

[0131] The sending module 1004 can be used to support the second network device 1000 in performing the above steps S3010, etc., and / or other processes used in the technology described herein.

[0132] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0133] The second network device 1000 provided in this embodiment is used to execute the above-described data processing method, and therefore can achieve the same effect as the above-described implementation method.

[0134] When using integrated units, the second network device 1000 may include a processing module, a storage module, and a communication module. The processing module can be used to control and manage the operations of the second network device 1000, for example, to support the second network device 1000 in executing the steps performed by the receiving module 1001, the aggregation module 1002, the generation module 1003, and the sending module 1004. The storage module can be used to support the second network device 1000 in storing program code and data. The communication module can be used to support communication between the second network device 1000 and other devices, such as communication with wireless access devices.

[0135] This application also provides an electronic device, including one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code, including computer instructions. When the one or more processors execute the computer instructions, the electronic device performs the aforementioned method steps to implement the data processing method described in the above embodiments.

[0136] Embodiments of this application also provide a computer storage medium storing computer instructions. When the computer instructions are executed on an electronic device, the electronic device performs the aforementioned method steps to implement the data processing method described in the above embodiments.

[0137] Embodiments of this application also provide a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement the data processing method performed by the electronic device in the above embodiments.

[0138] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory. The memory is used to store computer execution instructions. When the apparatus is running, the processor can execute the computer execution instructions stored in the memory to cause the chip to perform the data processing methods performed by the electronic devices in the above-described method embodiments.

[0139] In this embodiment, the electronic device, computer storage medium, computer program product or chip are all used to execute the corresponding method provided above. Therefore, the beneficial effects that can be achieved can be referred to the beneficial effects of the corresponding method provided above, and will not be repeated here.

[0140] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0141] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0144] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, essentially or in other words, the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0145] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data processing method, characterized in that, The method includes: The first network device receives multiple data streams and parses each data stream to obtain quintuple information; The first network device filters the multiple data streams based on the five-tuple information and the filtering model. The filtering model is generated by the second network device based on the five-tuple information of the multiple data streams. The filtering model includes four-tuple information or three-tuple information. The first network device sends the filtered multiple data streams to the third network device.

2. The method according to claim 1, characterized in that, The first network device filters the multiple data streams based on the five-tuple information and the filtering model, including: If the five-tuple information corresponding to the first data stream among the plurality of data streams matches the four-tuple information in the filtering model, the first network device filters the first data stream; And / or, if the quintuple information corresponding to the second data stream among the plurality of data streams matches the triple information in the filtering model, the first network device filters the second data stream.

3. The method according to claim 1 or 2, characterized in that, Before the first network device sends the filtered plurality of data streams to the third network device, the method includes: The first network device compresses the filtered multiple data streams.

4. The method according to any one of claims 1-3, characterized in that, Before the first network device filters the multiple data streams based on the five-tuple information and the filtering model, the method further includes: The first network device performs protocol parsing on the multiple data streams to obtain the protocol type corresponding to each data stream; The first network device filters the multiple data streams based on the protocol type.

5. The method according to claim 4, characterized in that, The first network device filters the plurality of data streams based on the protocol type, including: If the protocol type of the third data stream among the plurality of data streams meets the preset rules, the first network device filters the third data stream.

6. A data processing method, characterized in that, The method includes: The second network device receives multiple data streams and parses each data stream to obtain quintuple information; The second network device generates a filtering model based on the five-tuple information of the multiple data streams, wherein the filtering model includes four-tuple information or three-tuple information; The second network device sends the filtering model to the first network device.

7. The method according to claim 6, characterized in that, The second network device generates a filtering model based on the five-tuple information of the multiple data streams, including: The second network device aggregates the five-tuple information from the multiple data streams to obtain the four-tuple information or the three-tuple information; and generates the filtering model based on the four-tuple information or the three-tuple information.

8. The method according to claim 7, characterized in that, Before the second network device generates a filtering model based on the four-tuple information or the three-tuple information, the method further includes: The second network device obtains the number of data packets in each data stream, and based on the number of data packets in each data stream and a first preset threshold, obtains the small stream label information for each data stream, wherein the small stream label information is used to indicate the type of data stream; The second network device filters the multiple data streams based on the sub-stream label information of each data stream.

9. The method according to claim 7, characterized in that, The second network device generates a filtering model based on the quadruple or triple information, including: The second network device further filters the plurality of data streams based on a second preset threshold, the second preset threshold being related to the number of data streams corresponding to the five-tuple information; The second network device sorts the filtered multiple data streams in descending order of the number of data streams corresponding to the five-tuple information, and generates a filtering model based on the four-tuple information or the three-tuple information of the multiple data streams with the highest sorted number.

10. A first network device, characterized in that, include: The receiving module is used to receive multiple data streams and parse each data stream to obtain quintuple information; A filtering module is used to filter the multiple data streams based on the five-tuple information and a filtering model. The filtering model is generated by a second network device based on the five-tuple information of the multiple data streams, and the filtering model includes four-tuple information or three-tuple information. The sending module is used to send the filtered multiple data streams to a third network device.

11. The first network device according to claim 10, characterized in that, The filtering module is specifically used for: If the quintuple information corresponding to the first data stream in the plurality of data streams matches the quadruple information in the filtering model, then the first data stream is filtered; and / or, If the quintuple information corresponding to the second data stream among the plurality of data streams matches the triple information in the filtering model, the second data stream is filtered.

12. The first network device according to claim 10 or 11, characterized in that, The first network device also includes a compression module. The compression module is used to compress the filtered multiple data streams.

13. A second network device, characterized in that, include: The receiving module is used to receive multiple data streams and parse each data stream to obtain quintuple information; A generation module is used to generate a filtering model based on the five-tuple information of the multiple data streams, wherein the filtering model includes four-tuple information or three-tuple information. The sending module is used to send the filtering model to the first network device.

14. The second network device according to claim 13, characterized in that, The generation module is specifically used to: aggregate the five-tuple information of the multiple data streams to obtain the four-tuple information or the three-tuple information; and generate the filtering model based on the four-tuple information or the three-tuple information.

15. The second network device according to claim 14, characterized in that, The second network device also includes an acquisition module and a filtering module; The acquisition module is used to acquire the number of data packets in each data stream, and based on the number of data packets in each data stream and a first preset threshold, to obtain the sub-stream tag information of each data stream, wherein the sub-stream tag information is used to indicate the type of data stream; The filtering module is used to filter the multiple data streams based on the sub-stream label information of each data stream.

16. The second network device according to claim 14, characterized in that, The generation module is used to generate the filtering model based on the quadruple information or the triple information, including: The multiple data streams are filtered based on a second preset threshold, which is related to the number of data streams corresponding to the five-tuple information. The filtered multiple data streams are sorted in descending order of the number of data streams corresponding to the five-tuple information. A filtering model is generated based on the four-tuple information or the three-tuple information of the multiple data streams with the highest preset number of sorted data streams.

17. A communication system, characterized in that, The communication system includes a first network device and a second network device; The second network device is configured to generate a filtering model based on the five-tuple information of multiple data streams, and send the filtering model to the first network device. The filtering model includes four-tuple information or three-tuple information. The first network device is configured to receive the plurality of data streams and parse each data stream to obtain quintuple information; Based on the quintuple information and the filtering model, the multiple data streams are filtered, and the filtered multiple data streams are sent to the third network device.

18. The communication system according to claim 17, characterized in that, The first network device is configured to filter the plurality of data streams based on the five-tuple information and the filtering model, including: If the quintuple information corresponding to the first data stream in the plurality of data streams matches the quadruple information in the filtering model, then the first data stream is filtered; and / or, If the quintuple information corresponding to the second data stream among the plurality of data streams matches the triple information in the filtering model, the second data stream is filtered.

19. The communication system according to claim 17 or 18, characterized in that, The second network device is used to generate a filtering model based on five-tuple information from multiple data streams, including: The five-tuple information from the multiple data streams is aggregated to obtain the four-tuple information or the three-tuple information; the filtering model is generated based on the four-tuple information or the three-tuple information.

20. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-5 or any one of claims 6-9.

21. A computer program product, characterized in that, When the computer program product is run on a computer or processor, it causes the computer or processor to perform the method according to any one of claims 1-5 or any one of claims 6-9.

Images