A data aggregation method, apparatus, electronic device, and storage medium

Abstract

This application discloses a data aggregation method, data aggregation device, electronic device, and storage medium, relating to the field of computer technology. It can improve the query performance of data after it is stored in the database, and reduce the difficulty and cost of querying. The specific solution includes: acquiring network data stream traffic data; generating at least one first data log and at least one second data log based on the traffic data; aggregating all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log to obtain at least one first aggregated log; and aggregating all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log.

Description

Technical Field

[0001] This application relates to the field of computers, and more particularly to a data aggregation method and apparatus, electronic device and storage medium. Background Technology

[0002] As network traffic becomes increasingly complex, analyzing, troubleshooting, and gathering evidence for enterprise-level network traffic is becoming more and more difficult. When troubleshooting, enterprise or operations personnel often lack a clear understanding of the actual network traffic situation at each interface and the IP (Internet Protocol) distribution, making traffic analysis extremely challenging.

[0003] In the relevant solutions, in order to conduct traffic analysis, it is necessary to first backtrack and analyze the historical data of network traffic. However, due to the extremely large quantity of historical network traffic data, the query performance is low, and the time and technical costs of retrieving the target historical data from a large amount of historical network traffic data are extremely high. Summary of the Invention

[0004] This application provides a data aggregation method that can improve data query performance and reduce query difficulty and cost.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a data aggregation method is provided. The method includes: acquiring traffic data of a network data stream; wherein the traffic data includes a link identifier; generating at least one first data log and at least one second data log based on the traffic data; wherein the at least one first data log includes all first indicator information from the traffic data; and the at least one second data log includes all second indicator information from the traffic data; aggregating all the first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log, to obtain at least one first aggregated log; the first aggregated log includes at least one first data log; and aggregating all the second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log, to obtain at least one second aggregated log; the second aggregated log includes at least one second data log.

[0007] In conjunction with the first aspect, in another possible implementation, generating at least one first data log and at least one second data log based on traffic data includes: dividing traffic data in the network data stream into multiple sub-traffic data according to preset rules; generating a third data log and a fourth data log corresponding to the sub-traffic data based on the sub-traffic data; the third data log belongs to the first data log, and the fourth data log belongs to the second data log.

[0008] In conjunction with the first aspect, in another possible implementation, a third data log corresponding to the sub-traffic data is generated based on the sub-traffic data, including: combining the first indicator information within each unit of time in the sub-traffic data to obtain the third data log corresponding to each unit of time.

[0009] In conjunction with the first aspect, in another possible implementation, generating a fourth data log corresponding to the sub-traffic data based on the sub-traffic data includes: merging the first traffic data in the sub-traffic data to obtain at least one flow data; the first traffic data is the traffic data of a first message with the same five-tuple information; the five-tuple information includes: source address information, source port information, destination address information, destination port information, and communication protocol; generating at least one sub-data log based on the flow data; the sub-data log belongs to the fourth data log.

[0010] In conjunction with the first aspect, in another possible implementation, generating at least one sub-data log based on the streaming data includes: determining the duration of the session to which the first message corresponding to the streaming data is located when the session has ended; generating a sub-data log when the duration is less than or equal to a preset duration; and generating multiple sub-data logs based on the preset duration when the duration is greater than the preset duration.

[0011] In conjunction with the first aspect, in another possible implementation, based on the link identifier corresponding to the first data log, all the first data logs are aggregated according to the first aggregation rule to obtain at least one first aggregated log, including: aggregating the first data logs in ascending order of multiple time granularities to obtain the first aggregated log corresponding to each time granularity; and / or, aggregating the first data logs according to the link identifier to obtain the first aggregated data log corresponding to the link identifier.

[0012] In conjunction with the first aspect, in another possible implementation, based on the link identifier corresponding to the second data log, the second data log is aggregated according to the second aggregation rule to obtain at least one second aggregated log, including: aggregating the second data log according to the link identifier to obtain the second aggregated data log corresponding to the link identifier; and / or, aggregating the second data log according to the quintuple information to obtain the second aggregated data log corresponding to the quintuple information.

[0013] In conjunction with the first aspect, in another possible implementation, a display instruction is received; wherein the display instruction is used to request the display of target traffic data, and the display instruction includes a query index and a display format for the target traffic data; the query index includes a first index or a second index; the first index includes at least one or more of the following: target link identifier, target time period; the second index includes at least one or more of the following: target link identifier, target quintuple information, target time period; the target traffic data is queried according to the query index; and the target traffic data is displayed according to the display format.

[0014] Secondly, a data aggregation device is provided, which has the function of implementing the method of the first aspect described above. This function can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the aforementioned function.

[0015] Thirdly, a data aggregation device is provided, comprising a data acquisition module for acquiring traffic data of a network data stream; wherein the traffic data includes a link identifier; a log generation module for generating at least one first data log and at least one second data log based on the traffic data acquired by the data acquisition module; wherein the at least one first data log includes all first indicator information in the traffic data; and the at least one second data log includes all second indicator information in the traffic data; a first aggregation module for aggregating all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log generated by the log generation module, to obtain at least one first aggregated log; the first aggregated log includes at least one first aggregated log; and a second aggregation module for aggregating all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log generated by the log generation module, to obtain at least one second aggregated log; the second aggregated log includes at least one second aggregated log.

[0016] Fourthly, a data aggregation device is provided, comprising: a processor and a memory; the memory is used to store computer execution instructions, and when the first device is running, the processor executes the computer execution instructions stored in the memory to cause the first device to perform the data aggregation method provided in any of the first aspects above.

[0017] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, enable the computer to perform the data aggregation method provided in any of the preceding aspects.

[0018] In a sixth aspect, a computer program product containing instructions is provided, which, when run on a data aggregation device, enables the data aggregation device to execute the data aggregation method provided in any of the first aspects above.

[0019] In a seventh aspect, an apparatus (e.g., a system-on-a-chip) is provided, comprising a processor for supporting a data aggregation device in implementing the functions described in the first aspect above. In one possible design, the apparatus further comprises a memory for storing program instructions and data necessary for the data aggregation device. When the apparatus is a system-on-a-chip, it may be composed of chips or may include chips and other discrete devices.

[0020] Based on the technical solution provided in the embodiments of this application, firstly, network traffic data and link information corresponding to network traffic are obtained; secondly, a first data log and a second data log are generated according to the network traffic data and link information; then, the first data log and the second data log are aggregated according to the link information and a preset aggregation rule; finally, the aggregated first data log and the second data log are stored in the database.

[0021] In summary, because the technical solution provided in this application embodiment can aggregate network traffic data based on various indicators such as the source and historical changes of network traffic during the network traffic data entry stage, it improves the query performance of network traffic data, reduces the query difficulty and query cost, and provides data support for fault analysis. Attached Figure Description

[0022] Figure 1 This application provides a schematic diagram of the software architecture of a data aggregation device.

[0023] Figure 2 This application provides a flowchart of a data aggregation method. Figure 1 ;

[0024] Figure 3 This application provides a flowchart of a data aggregation method. Figure 2 ;

[0025] Figure 4 This application provides a flowchart of a data aggregation method. Figure 3 ;

[0026] Figure 5 This application provides a flowchart of a data aggregation method. Figure 4 ;

[0027] Figure 6 This application provides a flowchart of a data aggregation method. Figure 5 ;

[0028] Figure 7 This application provides a flowchart of a data aggregation method. Figure 6 ;

[0029] Figure 8 This application provides a flowchart of a data aggregation method. Figure 7 ;

[0030] Figure 9 This application provides a flowchart of a data aggregation method. Figure 8 ;

[0031] Figure 10 A schematic diagram illustrating another display method provided in an embodiment of this application;

[0032] Figure 11 A schematic diagram of a summary statistical log provided in an embodiment of this application;

[0033] Figure 12 A schematic diagram of a session log provided in an embodiment of this application;

[0034] Figure 13 This is a schematic diagram illustrating the display of query data provided in an embodiment of this application;

[0035] Figure 14 This is a schematic diagram of another data aggregation device provided in an embodiment of this application;

[0036] Figure 15 This is a schematic diagram of the structure of another data aggregation device provided in an embodiment of this application. Detailed Implementation

[0037] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0038] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0039] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0040] The terms “comprising” and “having” and any variations thereof in this application are intended to cover but not exclude inclusion, for example, a product or device that includes a series of components is not necessarily limited to all the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0041] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0042] Based on the exemplary embodiments described in this application, all other embodiments obtained by those skilled in the art without inventive effort are within the scope of protection of the appended claims. Furthermore, although the disclosures in this application are presented by way of one or more exemplary examples, it should be understood that each aspect of these disclosures can constitute a complete implementation on its own. It should be noted that the brief descriptions of terminology in this application are merely for the convenience of understanding the embodiments described below, and are not intended to limit the implementation of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0043] First, the terms used in this application are explained as follows:

[0044] DPDK stands for Data Plane Development Kit. It's a collection of libraries and toolkits typically used for packet reception and distribution.

[0045] Stream creation: The process of merging traffic data from a session into a single data stream.

[0046] Application identification: The process of identifying which application's client the data stream originates from.

[0047] Summary Statistics Log: A data log containing summary statistics. These statistics include the number of data packets in the network data stream, the size of the data packets in the network data stream, the type of data packets in the network data stream, and the transmission speed of the network data stream.

[0048] Session: refers to the process of communication between one terminal and another terminal.

[0049] Session Log: A data log containing session information. Session information includes source port information, source address information, destination port information, destination address information, session name information, traffic information, bit rate, and other related information.

[0050] Links: These include physical links and logical links. A physical link refers to a physical line formed under the action of physical layer devices (including transmission media, physical interfaces, and transceivers, etc.) and corresponding physical layer communication protocols. It is permanent and cannot be deleted (unless physically dismantled). A logical link, on the other hand, is a logical link established by two communicating parties under the action of data link layer devices and corresponding communication protocols when data communication is required. It can be permanent (such as an Ethernet link in a local area network) or non-permanent (such as a link in a wide area network). Whether it is permanent depends on the specific data link layer service type. Unless otherwise specified, all links mentioned in this application are logical links.

[0051] Data aggregation: Data aggregation is the process of collecting data and representing it in a specific format. The purposes of data aggregation include statistical analysis. Data aggregation is a representation of information and data, involving the searching, collection, and presentation of data and information in a specific format to achieve specific business objectives or processes and / or for manual analysis.

[0052] Current traffic analysis solutions typically require operations personnel to capture packets at each node and then perform comparative analysis. However, in high-traffic scenarios, the efficiency of packet capture analysis is low due to the sheer number of data packets. Furthermore, packet capture analysis demands a high level of technical skill from operations personnel, increasing the cost of traffic analysis.

[0053] In other related solutions, in order to perform traffic analysis, it is necessary to first backtrack and analyze the historical data of network traffic. However, due to the extremely large quantity of historical network traffic data, the query performance is low, and the time and technical costs of retrieving the target historical data from a large amount of historical network traffic data are extremely high.

[0054] To address the aforementioned issues, this application proposes a data aggregation method applicable to data aggregation devices. The method includes acquiring traffic data from a network data stream, generating a first data log and a second data log based on the traffic data, aggregating the first data log according to a first aggregation rule, and aggregating the second data log according to a second aggregation rule. In other words, this solution improves data query performance and reduces query difficulty and cost.

[0055] Figure 1 This is a schematic diagram illustrating the composition of a data aggregation system used in an exemplary embodiment of a data aggregation method. (Refer to...) Figure 1 As shown, the data aggregation system includes server 01 and server 02. It should be understood that... Figure 1 The servers shown are merely illustrative; depending on the implementation requirements, there can be any number of servers. For example, server 01 could be a server cluster composed of multiple servers.

[0056] The data aggregation method provided in this application can be executed on server 01. For example, server 01 can acquire network data stream traffic data and generate a first data log and a second data log based on the traffic data. Finally, the first data log and the second data log are aggregated into a first aggregated log and a second aggregated log. After obtaining the first aggregated log and the second aggregated log, server 01 can send the first aggregated log and the second aggregated log to server 02. After receiving the first aggregated log and the second aggregated log, server 02 can store them in its own memory or database. Of course, server 01 can also store the first aggregated log and the second aggregated log in its own memory; this application does not impose specific limitations on this.

[0057] In some embodiments, server 01 may also have a display, which can receive target traffic data query instructions from clients or other ports, query target traffic data in its own memory or database according to the target traffic data query instructions, and finally display the target traffic data through the display.

[0058] The video data involved in this application may be data authorized by the user or fully authorized by all parties.

[0059] The methods in the following embodiments can be implemented in a data aggregation device having the above-described hardware and software structures. The data aggregation methods provided in the embodiments of this application are described below.

[0060] Reference Figure 2 As shown in the embodiments of this application, a data aggregation method is provided, applied to a data aggregation device. The data aggregation device has the function of generating logs based on traffic data. The data aggregation device can be server 01 or a part thereof in the above embodiments. The method may include S210-S240:

[0061] S210. Obtain network data stream traffic data.

[0062] Among them, network data stream is a data stream containing transmitted data during network transmission. A network data stream consists of multiple data packets, and each data packet contains a message.

[0063] The traffic data includes link identifiers. Traffic data comprises all network transmission-related information contained in the network data stream during network transmission, and link identifiers are used to indicate at least one logical link within the network data stream. For example, traffic data may include information about the transmission source, the destination address, the transmitted content, and the transmission protocol.

[0064] In some examples, combined Figure 1As shown, S210 can specifically involve server 01 obtaining traffic data from a network data stream via the Internet. This traffic data can include information such as link identifier, number of data packets in the network data stream, data packet size, data packet type, transmission speed, bit rate, number of sessions, and session duration.

[0065] In some embodiments, due to the extremely large magnitude of network data flow traffic, it is necessary to aggregate the traffic data in the form of logs to facilitate subsequent searching or analysis of the network data flow traffic data. That is, traffic data with the same attributes are aggregated into one or more log tables according to specific aggregation rules. Aggregating traffic data into multiple log tables facilitates the subsequent query and analysis stages by allowing the traffic data to be quickly retrieved from the log tables. Based on this, after step S210, steps S220, S230, and S240 are executed.

[0066] S220. Generate at least one first data log based on the traffic data.

[0067] At least one first data log includes all first indicator information from the traffic data. The first indicator information is summary statistics of the network data flow. The summary statistics may include: the number of data packets in the network data flow, the size of the data packets in the network data flow, the type of data packets in the network data flow, the transmission speed of the network data flow, etc.

[0068] In some examples, combined Figure 1 As shown, after acquiring traffic data from the network data stream, server 01 generates a summary statistics log table (first data log) every unit of time. This summary statistics log table includes summary statistics of the network data stream within the unit of time. It is understood that in practical applications, server 01 can adjust the generation time of the summary statistics log table according to requirements; for example, it can generate a summary statistics log table every minute. This embodiment of the application does not limit the generation time of the first data log.

[0069] S220. Generate at least one second data log based on the traffic data.

[0070] At least one second data log includes all second indicator information from the traffic data. The second indicator information is session information of the network data stream. Session information may include: source port information, source address information, destination port information, destination address information, session name information, application name information, traffic information, bit rate, and other related information.

[0071] In some examples, combined Figure 1As shown, S220 can specifically involve server 01 generating a session log table (second data log) after acquiring traffic data from the network data stream. Specifically, server 01 summarizes all second indicator information from the traffic data under the same session onto at least one log table to generate the session log table.

[0072] It is understood that there is no specific order or sequence in which server 01 generates the first data log and the second data log. Server 01 may generate only the first data log or the second data log, or it may generate both the first data log and the second data log at the same time. This application embodiment does not impose any special limitation on the generation order of the first data log and the second data log.

[0073] In practical applications, the format of the first data log and the second data log is usually CSV (comma-separated values, a plain text format). However, it is understood that the embodiments of this application do not impose any special limitations on the format of the first data log and the second data log.

[0074] In some examples, to improve the efficiency of generating and aggregating logs based on traffic data, the data aggregation device distributes the traffic data in the network data stream to different processing threads and executes subsequent steps when executing S210. Based on this, combined with Figure 2 , refer to Figure 3 As shown, S220 includes S310 and S320.

[0075] S310. Divide the traffic data in the network data stream into multiple sub-traffic data according to preset rules.

[0076] Among them, the preset rule is the rule for dividing traffic data into sub-traffic data. For example, the preset rule can be to divide the traffic data into multiple sub-traffic data evenly according to the size of the traffic data, or it can be to divide the traffic data into multiple sub-traffic data according to the indicator information in the traffic data.

[0077] In some examples, combined Figure 1 As shown, S310 can specifically be that there are multiple processing threads in server 01. After server 01 completes DPDK packet reception (acquiring traffic data in network data stream), it determines the total size of the packets in all data packets, and then distributes the packets to multiple processing threads so that the packet sizes received by multiple processing threads are approximately the same (divided into multiple sub-traffic data according to preset rules).

[0078] In some embodiments, after receiving a message, a processing thread continues to generate data logs (the aforementioned first data log and second data log) based on the first indicator information and the second indicator information of the message. Multiple processing threads generating data simultaneously can improve efficiency. Based on this, S320 and S330 are executed after S310.

[0079] S320. Generate a third data log corresponding to the sub-flow data based on the sub-flow data.

[0080] The third data log is a subset of the first data log; that is, the third data log is one of the first data logs. Specifically, sub-traffic data belongs to traffic data. The first data log includes all first indicator information from the traffic data, and the third data log includes all first indicator information from the sub-traffic data.

[0081] In some embodiments, combined with Figure 3 ,refer to Figure 4 As shown, generating a third data log may include S410.

[0082] S410. Combine the first indicator information within each unit of time in the sub-traffic data to obtain the third data log corresponding to each unit of time.

[0083] In some examples, combined Figure 1 As shown, S410 can specifically be that, for a certain processing thread in server 01, after obtaining sub-traffic data, a summary statistics log table (third data log) is generated every unit time. The summary statistics log table includes summary statistics information (first indicator information) of the packets corresponding to the sub-traffic data within a unit time.

[0084] In the S410 technical solution, the first indicator information within each unit of time in the sub-traffic data is combined to obtain the third data log corresponding to each unit of time. It is evident that the third data log contains all the first indicator information from the sub-traffic data, ensuring the integrity of the traffic data and reducing the packet loss rate when generating the first data log.

[0085] S330. Generate a fourth data log corresponding to the sub-flow data based on the sub-flow data.

[0086] The fourth data log is a subset of the second data log; that is, the fourth data log is one of the second data logs. Specifically, sub-traffic data belongs to traffic data, the second data log includes all second indicator information from the traffic data, and the fourth data log includes all second indicator information from the sub-traffic data.

[0087] It is understood that S320 and S330 have no logical order. The data aggregation device can execute S320 first and then S330, or S330 first and then S320, or both S320 and S330 can be executed simultaneously. This application embodiment does not impose any special restrictions on the execution order of S320 and S330.

[0088] In some examples, combined Figure 1 As shown, S330 can specifically involve a processing thread in server 01 generating a session log table (fourth data log) after acquiring sub-traffic data. Specifically, this processing thread summarizes all second indicator information from the traffic data within the same session onto at least one log table to generate the session log table.

[0089] In some embodiments, combined with Figure 4 ,refer to Figure 5 As shown, the generation of the fourth data log includes S510 and S520.

[0090] S510. Merge the first flow data in the sub-flow data to obtain at least one flow data.

[0091] The first traffic data is the traffic data of the first message with the same five-tuple information. The five-tuple information includes source address information, source port information, destination address information, destination port information and communication protocol. The first message is the message with the same five-tuple information, and the flow data is the traffic data of the first message.

[0092] In some examples, combined Figure 1 As shown, S510 can specifically involve, for a specific processing thread in server 01, after acquiring the sub-traffic data, merging the first traffic data and using the merged first traffic data as the stream data. Since the same 5-tuple information can identify a session, each stream data corresponds to one session, meaning that the first packets corresponding to the same stream data belong to the same session. For a specific session (or a specific stream data), the corresponding session log table can be generated using the second indicator information in the session. Based on this, S520 is executed after S510.

[0093] S520. Generate at least one sub-data log based on the streaming data.

[0094] The sub-data log belongs to the fourth data log, and the fourth data log belongs to the second data log. Specifically, traffic data can be divided into multiple sub-traffic data, and each sub-traffic data can be further divided into multiple stream data based on the 5-tuple information. The sub-data log includes all the second indicator information in the stream data, the fourth data log includes all the second indicator information in the sub-traffic data, and the second data log includes all the second indicator information in the traffic data.

[0095] In some examples, combined Figure 1 As shown, S520 can specifically involve, for a specific stream of data in a specific processing thread of server 01, first determining the second indicator information of the stream data, and then generating at least one session log table based on the second indicator information. For a specific processing thread of server 01, a session is determined based on each stream of data, and a session log table corresponding to each session is generated based on the second indicator information.

[0096] Taking application name information as the second indicator as an example, the application name information of the streaming data is determined by identifying the association between the application identifier or 5-tuple information in the streaming data and the application name information. For each streaming data, at least one session log table is generated based on the application name information.

[0097] In the technical solutions S510 to S520 described above, the data aggregation device merges the traffic inter-data of first packets with identical five-tuple information in the sub-traffic data to obtain flow data, and then generates at least one sub-data log based on the flow data. It can be seen that each flow data generates a sub-data log containing all the second indicator information of the flow data, and the fourth data log composed of the sub-data logs also contains all the second indicator information from the sub-traffic data, further ensuring the integrity of the traffic data and further reducing the packet loss rate when generating the first data log.

[0098] In some examples, due to the long session duration, the corresponding streaming data is also large, resulting in long sub-data logs generated from the streaming data. For the second metric information within these long sub-data logs, query performance remains low. For instance, when querying the target second metric information, even if the target sub-data log is identified, the target second metric information still needs to be determined from a large number of target sub-data logs. Based on this, combined with... Figure 5 , refer to Figure 6 As shown, S520 includes S610 to S630.

[0099] S610. If the session to which the first message corresponding to the stream data belongs has ended, determine the duration of the session to which the first message belongs.

[0100] In some examples, for a session, the session message contains a session start field and a session end field to indicate the start and end of the current session. The data aggregation device can determine the end of the session upon detecting the session end field.

[0101] In some examples, if the session to which the first packet corresponding to the streaming data belongs has ended, the start and end times of the first packet are determined to determine the duration of the session to which the first packet belongs. In other examples, the data aggregation device may be configured with a timer that starts counting at the beginning of the session and stops counting at the end of the session to determine the duration of the session to which the first packet belongs. This application does not impose any special limitations on the method for determining the duration of the session to which the first packet belongs.

[0102] In some examples, if the session to which the first message corresponding to the streaming data belongs has not yet ended, the data aggregation device waits for the session to end and determines the duration of the session to which the first message belongs after the session ends.

[0103] S620. If the duration is less than or equal to the preset duration, generate a sub-data log.

[0104] In some examples, combined Figure 1 As shown, S620 can specifically be, taking a duration of 120s and a preset duration of 180s as an example, for a certain stream of data in a certain processing thread in server 01, if it is determined that the duration of the session in which the first message corresponding to the stream of data is located is 120s, which is less than the preset duration of 180s, then a sub-data log is generated.

[0105] S630. If the duration exceeds the preset duration, generate multiple sub-data logs based on the preset duration.

[0106] In some examples, combined Figure 1 As shown, S630 can specifically be implemented as follows: taking a duration of 800s and a preset duration of 180s as an example, for a certain stream of data in a certain processing thread of server 01, if the duration of the session corresponding to the first packet of this stream of data is determined to be 800s, which is greater than the preset duration of 180s, then a sub-data log is generated every 180s, and a sub-data log is generated again in the remaining 80s, ultimately generating five sub-data logs. The session duration of the packets corresponding to the first four sub-data logs is 180s, and the session duration of the packet corresponding to the fifth sub-data log is 80s, meaning a sub-data log is generated every preset duration.

[0107] In the technical solutions corresponding to S610 to S630 above, the duration of the session is determined after the session ends. For sessions with shorter durations, only one sub-data log is generated, while for sessions with longer durations, a sub-data log is generated every preset duration. By setting a preset duration and dividing the session in which the first packet is located into segments to generate sub-data logs according to the preset duration, it is ensured that a single sub-data log is not too complex, thereby further improving the query performance of traffic data.

[0108] In practical applications, the same stream data may exist in different processing threads. For example, the first stream data in the first processing thread and the second stream data in the second processing thread have the same five-tuple information. In this case, the first stream data and the second stream data are essentially the same stream data. After generating sub-data logs, each processing thread merges the sub-data logs of the same stream data in different processing threads.

[0109] In the technical solutions corresponding to S510 and S520 above, traffic data is divided into multiple sub-traffic data according to preset rules, and then a third data log and a fourth data log are generated based on the sub-traffic data. It can be seen that this technical solution improves the efficiency of data log generation by dividing traffic data into multiple sub-traffic data and generating data logs in parallel.

[0110] In some examples, after acquiring the first data log and the second aggregated data log, the data aggregation device needs to aggregate the first data log and the second data log to improve the query performance of traffic data. Based on this, S230 and S240 are executed after S220.

[0111] S230. Based on the link identifier corresponding to the first data log, aggregate all the first data logs according to the first aggregation rule to obtain at least one first aggregated log.

[0112] The first aggregated log includes at least one first data log, and the first aggregation rule is a preset rule followed when aggregating the first data log into the first aggregated log. For example, the first preset rule can be to aggregate the first aggregated log according to the time granularity based on the link identifier to generate at least one first aggregated log, or it can be to aggregate the first data log according to the type of network data packet based on the link identifier to generate at least one first aggregated log.

[0113] In some examples, combined Figure 1 As shown, S230 can specifically involve server 01, after obtaining at least one summary statistics log table, aggregating the summary statistics log table into at least one first aggregation log based on the link and according to the first aggregation rule. For example, the network data flow traffic data obtained by server 01 includes a first link identifier, a second link identifier, and a third link identifier. Based on the link identifier and according to the time granularity, the summary statistics log table is aggregated into the first aggregation log corresponding to the first link, the first aggregation log corresponding to the second link, and the first aggregation log corresponding to the third link.

[0114] In some examples, combined Figure 6 Reference Figure 7 As shown, S230 may include S710 and S720.

[0115] S710. Aggregate the first data log in order of increasing time granularity to obtain the first aggregated log corresponding to each time granularity.

[0116] In some examples, combined Figure 1 As shown, S710 can specifically be configured with server 01 having six timers with time granularities of 1s, 10s, 1min, 10min, 1h, and 1d. Each timer has an aggregation priority, with the order being 1s > 10s > 1min > 10min > 1h > 1d. Every 10s, server 01 aggregates the summary statistics log table (the first data log generated based on a unit time) and generates a summary statistics log table based on 10s. Every minute, server 01 aggregates the six summary statistics log tables based on 10s and generates a summary statistics log table based on 1min, referred to as the minute table. Every 10 minutes, server 01 aggregates the minute tables and generates a summary statistics log table based on 10 minutes, referred to as the 10-minute table. Every hour, server 01 aggregates the six 10-minute tables and generates a summary statistics log table based on 1h, referred to as the hour table. Every 24 hours, server 01 aggregates the 24-hour tables and generates a summary statistics log table based on 1 day, referred to as the daily table.

[0117] S720. Aggregate the first data log according to the link identifier to obtain the first aggregated data log corresponding to the link identifier.

[0118] In some examples, combined Figure 1 As shown, S720 can specifically be that server 01 aggregates the summary statistics log table into at least one first aggregated log according to the link identifier. It can be understood that server 01 can execute either S710 or S720, or it can execute both S710 and S720. At the same time, there is no logical order between S710 and S720, and the implementation of this application does not impose any special restrictions on the execution order of S710 and S720.

[0119] In the aforementioned technical solutions S710 and S720, the first data logs are aggregated according to multiple time granularities in ascending order to obtain the first aggregated logs corresponding to each time granularity, and / or the first data logs are aggregated according to the link identifier to obtain the first aggregated data logs corresponding to the link identifier. It is evident that this technical solution can effectively and reasonably aggregate the first data logs, increasing the performance of traffic data querying.

[0120] S240. Based on the link identifier corresponding to the second data log, aggregate all the second data logs according to the second aggregation rule to obtain at least one second aggregated log.

[0121] The second aggregated log includes at least one second data log, and the second aggregation rule is a preset rule followed when aggregating the first data log into the second aggregated log. For example, the second preset rule can be to aggregate the second aggregated log based on the link identifier and application name information to generate at least one second aggregated log, or it can be to aggregate the second data log based on the link identifier and five-tuple information to generate at least one second aggregated log.

[0122] In some examples, combined Figure 1 As shown, S240 can specifically involve server 01, after acquiring at least one session log table, aggregating the session log tables into at least one second aggregated log based on the link identifier and according to the second aggregation rule. For example, the network data flow traffic data acquired by server 01 contains a first link identifier, a second link identifier, and a third link identifier. Based on the link identifier and according to the five-tuple information, the session log tables are aggregated into the second aggregated log corresponding to the first link, the second aggregated log corresponding to the second link, and the second aggregated log corresponding to the third link.

[0123] In some examples, combined Figure 7 Reference Figure 8 As shown, S230 may include S810 and S820.

[0124] S810. Aggregate the second data log based on the link identifier to obtain the second aggregated data log corresponding to the link identifier. In some examples, this is combined with... Figure 1 As shown, S810 can specifically be that server 01 aggregates the session log table into at least one second aggregated log according to the link identifier.

[0125] S820. Aggregate the second data log based on the quintuple information to obtain the second aggregated data log corresponding to the quintuple information. In some examples, this is combined with... Figure 1 As shown, S820 can specifically be that server 01 aggregates the second data logs based on the 5-tuple information, and aggregates the second data logs with the same 5-tuple information into a second data log.

[0126] It is understood that server 01 may execute either S810 or S820, or it may execute both S810 and S820. At the same time, there is no logical order between S810 and S820. This application embodiment does not impose any special restrictions on the execution order of S810 and S820.

[0127] In the above technical solutions S810 and S820, the second data logs are aggregated according to the link identifier to obtain the second aggregated data logs corresponding to the link identifier, and / or the second data logs are aggregated according to the 5-tuple information to obtain the second aggregated data logs corresponding to the 5-tuple information. It is evident that this technical solution can effectively and reasonably aggregate the second data logs, further improving the performance of traffic data query.

[0128] It is understood that the data aggregation device can execute S230 first and then S240, or it can execute S240 first and then S230, or it can execute S230 and S240 simultaneously. This application does not impose any special restrictions on the execution order of S230 and S240.

[0129] It is understandable that in practical applications, the first aggregation rule and the second aggregation rule may be the same or different. For example, both the first and second data logs can be aggregated based on the link identifier at a time granularity, in which case the first and second aggregation rules are the same; the first data log can be aggregated based on the link identifier at a time granularity, and the second data log can be aggregated based on the link identifier at a 5-tuple information, in which case the first and second aggregation rules are different. This application does not make any special limitation on whether the first and second aggregation rules are the same.

[0130] In the technical solutions corresponding to S210 to S240 above, firstly, the data aggregation device acquires traffic data from the network data stream; secondly, the data aggregation device generates at least one first data log and at least one second data log based on the traffic data; finally, the data aggregation device aggregates the first data log and the second data log into a first aggregated log and a second aggregated log. It can be seen that, in this technical solution, on the one hand, the data aggregation device can aggregate traffic data from the network data stream into aggregated logs according to different rules, enabling the query performance and efficiency of traffic data to be improved and the query cost reduced by first querying the target aggregated log and then querying the target traffic data during the traffic data query stage; on the other hand, since the number of aggregated logs is usually much smaller than the order of magnitude of the traffic data, this technical solution can reduce the order of magnitude of data being entered into the database during the data entry stage, fully utilizing the parallel storage capacity of the database and improving data entry efficiency.

[0131] In some embodiments, the data aggregation device can display data query results, based on which, combined with Figure 2 , refer to Figure 9 As shown, after S240, S910 to S930 are executed.

[0132] S910, Receive display commands.

[0133] The display instruction is used to request the display of target traffic data. The display instruction includes the query index and display format of the target traffic data. The query index includes a first index or a second index. The first index includes at least one or more of the following: target link identifier, target time period. The second index includes at least one or more of the following: target link identifier, target quintuple information, target time period.

[0134] For example, the display format can include tables, bar charts, and pie charts.

[0135] Combination Figure 1 As shown, S910 can specifically be a server 01 receiving a display command, which can be sent by a client or by another terminal.

[0136] S920. Query the target traffic data based on the query index.

[0137] In some examples, reference Figure 1 As shown, S920 can specifically be, taking the query index as the first link, the first quintuple information, and 0 to 1 as an example, the server 02 queries the second aggregate log of the session flow where the first packet corresponding to the first quintuple information in the first link is located from 0 to 1 according to the query index, and further queries the target traffic data under the second aggregate log.

[0138] S930. Display the target flow data according to the display format.

[0139] In some examples, combined Figure 1 As shown, after querying the target traffic data, server 01 displays the target traffic data on the monitor according to the display instructions.

[0140] In the aforementioned technical solutions S910 to S930, a display command is first received, then the target traffic data is queried according to the query index in the display command, and finally the target traffic data is displayed in the specified display format. It is evident that this technical solution enables the visualization of traffic data, providing a convenient way for customers or maintenance personnel to understand the status of network data flow.

[0141] Understandably, in practical applications, it is necessary to combine... Figure 1 As shown, the functions that server 01 can perform can also be performed by server 02. In this case, the technical solution provided in the aforementioned embodiments can be implemented by server 02. Of course, server 01 and server 02 can also perform different steps of the technical solution provided in the aforementioned embodiments, depending on the actual situation. In this case, the aforementioned data aggregation device can be a combination of functional modules or functional devices in server 01 and server 02 that execute the technical solution provided in this application. This application does not impose specific limitations in this regard.

[0142] In one example embodiment of this application, a data aggregation device is used as the server, a first data log is used as a summary statistical log, and a second data log is used as a session log. The first aggregation rule is based on unit time aggregation, and the second aggregation rule is based on five-tuple information aggregation. Taking a preset time of 180 seconds as an example, the data aggregation method provided in this application may include, for example... Figure 10 The process shown is for reference only. Figure 11 As shown, the data aggregation method provided in this application may specifically include:

[0143] 1. DPDK receives packets.

[0144] For example, the server receives data packets and link IDs from network data streams via DPDK.

[0145] 2. Distribute data packets to different processing threads.

[0146] For example, the server distributes data packets evenly to different processing threads so that each processing thread receives approximately the same number of data packets.

[0147] 3. Generate summary statistics log.

[0148] For example, each processing thread generates a summary statistics log based on the packets in the data packet, on a second-by-second basis. This summary statistics log records traffic statistics information for the current second. The server starts a 1-second timer; every second, the timer triggers an insert action to insert the summary statistics log into the database. For details on the summary statistics log, please refer to [link / reference]. Figure 11 As shown, 111 is a line graph showing the change of network traffic peak over time, and 112 is a summary statistical information (first indicator information) of various items from 11:36 to 11:41, including: 2,332,618 data packets, 1.47 GB of bytes, 450,095 sessions, 59,203 peak data packets, and 31.08 MB of peak traffic.

[0149] 4. Establish flow.

[0150] For example, each processing thread treats messages with the same five-tuple information as a stream, and each stream corresponds to a session.

[0151] 5. Application recognition.

[0152] For example, each processing thread identifies the application for each stream and generates the application name information corresponding to each stream.

[0153] 6. Short connection sessions generate session logs.

[0154] For example, if the session duration of each processing thread is less than or equal to 180 seconds, it is considered a short connection session, and a session log is generated for each short connection session. The short connection session log is then inserted into the database. The server starts a timer; at regular intervals, the timer triggers an insert action, inserting the generated short connection session log into the database.

[0155] 7. Generate session logs for long-lived connection sessions.

[0156] For example, if each processing thread session lasts longer than 180 seconds, it is considered a long-connection session. A long-connection session generates a session log every 180 seconds. This long-connection session log is then inserted into the database. Alternatively, the server can start a timer; at regular intervals, the timer will trigger an insert action, inserting the generated short-connection session logs into the database.

[0157] 8. Aggregate summary statistics logs based on link identifiers.

[0158] For example, the management device corresponding to the database aggregates summary statistics logs based on the link ID.

[0159] 9. Aggregate summary statistics logs based on time granularity. The time granularities are 1s, 10s, 1min, 10min, 1h, and 1d. Each time granularity has an aggregation priority, with the order being 1s > 10s > 1min > 10min > 1h > 1d. Every 10 seconds, the database aggregates the summary statistics logs and generates a summary statistics log based on 10s. Every minute, the database aggregates six summary statistics logs based on 10s and generates a summary statistics log based on 1min, referred to as the minute table. Every 10 minutes, the database aggregates the minute logs and generates a summary statistics log based on 10 minutes, referred to as the 10-minute table. Every hour, the database aggregates six 10-minute logs and generates a summary statistics log based on 1h, referred to as the hour table. Every 24 hours, the database aggregates 24 hours and generates a summary statistics log based on 1d, referred to as the day table. See Table 1 for details.

[0160] Table 1: Correspondence between query time granularity and the queried table.

[0161] Query time granularity <10s <1min <10min <1h <1d >1d Query table Original table 10s watch Minute watch 10-minute watch hour watch Tianbiao

[0162] Specifically, refer to the summary statistics log. Figure 11 As shown, 111 is a line graph showing the change of network traffic peak over time, and 112 is a summary statistical information (first indicator information) of various items from 11:36 to 11:41, including: 2,332,618 data packets, 1.47 GB of bytes, 450,095 sessions, 59,203 peak data packets, and 31.08 MB of peak traffic.

[0163] 10. Aggregate session logs based on link identifiers.

[0164] For example, the database management device aggregates session logs based on link IDs. Session logs from different links are stored in different folders and session tables.

[0165] 11. Aggregate session logs based on quintuple information.

[0166] For example, the database management device aggregates session logs based on the five-tuple information. Specifically, the session logs refer to... Figure 12 As shown, 121 is the communication protocol tab, and 122 is the session information (second indicator information) for each session under the TCP protocol, including: source IP, destination IP, source port, destination port, application name, uplink traffic, downlink traffic, and other information.

[0167] 12. Query the target data.

[0168] For example, the target data can be queried in the database management device by querying the aggregated summary statistics log or session log.

[0169] 13. Display target data.

[0170] For example, the database management device communicates with the web interface, returning the queried data for visualization. The web interface primarily displays real-time graphs, pie charts of traffic percentages, and bar charts. Specifically, a pie chart might be used to display the traffic percentages of ten TCP sessions for reference. Figure 13 As shown.

[0171] 14. End the process.

[0172] The foregoing mainly describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the above functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0173] This application embodiment can divide the data aggregation device into functional modules according to the above method example. For example, each function can be divided into its own functional module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0174] Reference Figure 14 As shown in the figure, this application provides a data aggregation device, which may include a data acquisition module 141, a log generation module 142, a first aggregation module 143, and a second aggregation module 144.

[0175] Specifically, the data acquisition module 141 is used to acquire traffic data of the network data stream; wherein the traffic data includes a link identifier; the log generation module 142 is used to generate at least one first data log and at least one second data log based on the traffic data acquired by the data acquisition module; wherein the at least one first data log includes all first indicator information in the traffic data; and the at least one second data log includes all second indicator information in the traffic data; the first aggregation module 143 is used to aggregate all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log generated by the log generation module, to obtain at least one first aggregated log; the first aggregated log includes at least one first aggregated log; the second aggregation module 144 is used to aggregate all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log generated by the log generation module, to obtain at least one second aggregated log; the second aggregated log includes at least one second aggregated log.

[0176] In one possible implementation, at least one first data log and at least one second data log are generated based on the traffic data. The log generation module 142 is specifically used to: divide the traffic data in the network data stream into multiple sub-traffic data according to a preset rule; generate a third data log and a fourth data log corresponding to the sub-traffic data based on the sub-traffic data; the third data log belongs to the first data log, and the fourth data log belongs to the second data log.

[0177] In one possible implementation, a third data log corresponding to the sub-traffic data is generated based on the sub-traffic data. Specifically, the log generation module 142 is used to combine the first indicator information within each unit of time in the sub-traffic data to obtain the third data log corresponding to each unit of time.

[0178] In one possible implementation, a fourth data log corresponding to the sub-traffic data is generated based on the sub-traffic data. The log generation module 142 is specifically used to: merge the first traffic data in the sub-traffic data to obtain at least one flow data; the first traffic data is the traffic data of a first message with the same five-tuple information; the five-tuple information includes: source address information, source port information, destination address information, destination port information and communication protocol; generate at least one sub-data log based on the flow data; the sub-data log belongs to the fourth data log.

[0179] In one possible implementation, at least one sub-data log is generated based on the streaming data. The log generation module 142 is specifically used to: determine the duration of the session to which the first message to which the streaming data belongs when the session has ended; generate a sub-data log when the duration is less than or equal to a preset duration; and generate multiple sub-data logs based on the preset duration when the duration is greater than the preset duration.

[0180] In one possible implementation, based on the link identifier corresponding to the first data log, all the first data logs are aggregated according to the first aggregation rule to obtain at least one first aggregated log. The first aggregation module 143 is specifically used to: aggregate the first data logs in order of multiple time granularities from small to large to obtain the first aggregated log corresponding to each time granularity; and / or, aggregate the first data logs according to the link identifier to obtain the first aggregated data log corresponding to the link identifier.

[0181] In one possible implementation, the second data log is aggregated according to the second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log. The second aggregation module 144 is specifically used to aggregate the second data log according to the link identifier to obtain the second aggregated data log corresponding to the link identifier; and / or, to aggregate the second data log according to the quintuple information to obtain the second aggregated data log corresponding to the quintuple information.

[0182] In one possible implementation, the data aggregation device further includes: receiving a display instruction; wherein the display instruction is used to request the display of target traffic data, and the display instruction includes a query index and a display format for the target traffic data; the query index includes a first index or a second index; the first index includes at least one or more of the following: target link identifier, target time period; the second index includes at least one or more of the following: target link identifier, target quintuple information, target time period; querying the target traffic data according to the query index; and displaying the target traffic data according to the display format.

[0183] It should be understood that the division of units or modules (hereinafter referred to as units) in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, all units in the device can be implemented in software through processing element calls; all units can be implemented in hardware; or some units can be implemented in software through processing element calls, and some units can be implemented in hardware.

[0184] For example, each unit can be a separate processing element, or it can be integrated into a chip within the device. Alternatively, it can be stored in memory as a program, and its function can be called and executed by a processing element within the device. Furthermore, these units can be integrated together in whole or in part, or they can be implemented independently. The processing element here can also be called a processor, which can be an integrated circuit with signal processing capabilities. In implementation, each step of the above method or each of the above units can be implemented through integrated logic circuits in the processor element or through software calls by the processing element.

[0185] In one example, the unit in the above device may be one or more integrated circuits configured to implement the above methods, such as one or more ASICs, or one or more DSPs, or one or more FPGAs, or a combination of at least two of these integrated circuit forms.

[0186] For example, when the units in the device can be implemented through a processing element scheduler, the processing element can be a general-purpose processor, such as a CPU or other processor capable of calling programs. Alternatively, these units can be integrated together to form a system-on-a-chip (SoC).

[0187] In one implementation, the units that implement the corresponding steps in the above method can be implemented in the form of a processing element scheduler. For example, the device may include a processing element and a storage element, wherein the processing element calls a program stored in the storage element to execute the data aggregation method of the above method embodiments. The storage element may be a storage element located on the same chip as the processing element, i.e., an on-chip storage element.

[0188] In another implementation, the program used to perform the above method can be located on a storage element on a different chip than the processing element, i.e., an off-chip storage element. In this case, the processing element calls or loads the program from the off-chip storage element onto the on-chip storage element to call and execute the data aggregation method of the above method embodiment.

[0189] Reference Figure 15As shown in the illustration, this application embodiment also provides a data aggregation device, including a display 151 configured to display target traffic data in a display format; a communicator 153 configured to acquire traffic data of a network data stream; and a processor 152 coupled to the display and the communicator respectively. The processor is configured to: generate at least one first data log and at least one second data log based on the traffic data; wherein, at least one first data log includes all first indicator information in the traffic data; at least one second data log includes all second indicator information in the traffic data; aggregate all first data logs according to a first aggregation rule based on the link identifier corresponding to the first data log to obtain at least one first aggregated log; the first aggregated log includes at least one first data log; and aggregate all second data logs according to a second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log; the second aggregated log includes at least one second data log.

[0190] In one possible implementation, the processor 152 is configured to divide the traffic data in the network data stream into multiple sub-traffic data according to a preset rule; generate a third data log and a fourth data log corresponding to the sub-traffic data based on the sub-traffic data; the third data log belongs to the first data log, and the fourth data log belongs to the second data log.

[0191] In one possible implementation, at least one first data log and at least one second data log are generated based on the traffic data. The processor 152 is configured to divide the traffic data in the network data stream into multiple sub-traffic data according to a preset rule; and to generate a third data log and a fourth data log corresponding to the sub-traffic data based on the sub-traffic data; the third data log belongs to the first data log, and the fourth data log belongs to the second data log.

[0192] In one possible implementation, a third data log corresponding to the sub-traffic data is generated based on the sub-traffic data. The processor 152 is configured to combine the first indicator information within each unit of time in the sub-traffic data to obtain the third data log corresponding to each unit of time.

[0193] In one possible implementation, a fourth data log corresponding to the sub-traffic data is generated based on the sub-traffic data. The processor 152 is configured to merge the first traffic data in the sub-traffic data to obtain at least one flow data. The first traffic data is the traffic data of a first message with the same five-tuple information. The five-tuple information includes: source address information, source port information, destination address information, destination port information, and communication protocol. At least one sub-data log is generated based on the flow data. The sub-data log belongs to the fourth data log.

[0194] In one possible implementation, at least one sub-data log is generated based on the streaming data. The processor 152 is configured to determine the duration of the session to which the first message belongs when the session to which the first message to which the streaming data belongs has ended; generate a sub-data log when the duration is less than or equal to a preset duration; and generate multiple sub-data logs based on the preset duration when the duration is greater than the preset duration.

[0195] In one possible implementation, based on the link identifier corresponding to the first data log, all the first data logs are aggregated according to the first aggregation rule to obtain at least one first aggregated log. The processor 152 is configured to aggregate the first data logs in ascending order of multiple time granularities to obtain a first aggregated log corresponding to each time granularity; and / or, aggregate the first data logs according to the link identifier to obtain a first aggregated data log corresponding to the link identifier.

[0196] In one possible implementation, the second data log is aggregated according to a second aggregation rule based on the link identifier corresponding to the second data log to obtain at least one second aggregated log. The processor 152 is configured to aggregate the second data log according to the link identifier to obtain a second aggregated data log corresponding to the link identifier; and / or to aggregate the second data log according to the quintuple information to obtain a second aggregated data log corresponding to the quintuple information.

[0197] In one possible implementation, the processor 152 is further configured to receive a display instruction; wherein the display instruction is used to request the display of target traffic data, and the display instruction includes a query index and a display format for the target traffic data; the query index includes a first index or a second index; the first index includes at least one or more of the following: target link identifier, target time period; the second index includes at least one or more of the following: target link identifier, target quintuple information, target time period; the processor queries the target traffic data according to the query index; and the processor displays the target traffic data according to the display format.

[0198] This application also provides a data aggregation device, which may include a display screen, a memory, and one or more processors. The display screen, memory, and processors are coupled. The memory stores computer program code, which includes computer instructions. When the processor executes the computer instructions, the data aggregation device can perform various functions or steps performed by the data aggregation device (such as a television) in the above method embodiments.

[0199] For example, this application also provides a chip that can be applied to the aforementioned data aggregation device or server. The chip includes one or more interface circuits and one or more processors; the interface circuits and processors are interconnected via lines; the processor receives and executes computer instructions from the memory of the data aggregation device through the interface circuits to implement the methods described in the above method embodiments.

[0200] This application also provides a computer-readable storage medium storing computer program instructions (or instructions). When the computer program instructions are executed by a data aggregation device, the data aggregation device can implement the data aggregation method described above.

[0201] This application also provides a computer program product, including computer instructions for running as described above in the data aggregation device. When the computer program product runs in the data aggregation device, the data aggregation device can implement the data aggregation method described above.

[0202] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0204] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0205] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0206] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product, such as a program. This software product is stored in a program product, such as a computer-readable storage medium, and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks.

[0207] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data aggregation method, characterized in that, The method includes: Obtain traffic data from a network data stream; wherein the traffic data includes a link identifier; At least one first data log and at least one second data log are generated based on the traffic data; wherein, the at least one first data log includes all first indicator information in the traffic data, and the first indicator information is summary statistical information of the network data flow; the at least one second data log includes all second indicator information in the traffic data, and the second indicator information is session information of the network data flow; Based on the link identifier corresponding to the first data log, all the first data logs are aggregated according to the first aggregation rule to obtain at least one first aggregated log; the first aggregated log includes at least one first data log; wherein, the first data logs are aggregated in order of multiple time granularities from small to large to obtain a first aggregated log corresponding to each time granularity; and / or, the first data logs are aggregated according to the link identifier to obtain a first aggregated data log corresponding to the link identifier; Based on the link identifier corresponding to the second data log, all the second data logs are aggregated according to the second aggregation rule to obtain at least one second aggregated log; the second aggregated log includes at least one second data log; wherein, the second data logs are aggregated according to the link identifier to obtain the second aggregated data log corresponding to the link identifier; and / or, the second data logs are aggregated according to the 5-tuple information to obtain the second aggregated data log corresponding to the 5-tuple information.

2. The data aggregation method according to claim 1, characterized in that, Generate at least one first data log and at least one second data log based on the traffic data, including: The traffic data in the network data stream is divided into multiple sub-traffic data according to a preset rule; A third data log and a fourth data log corresponding to the sub-traffic data are generated based on the sub-traffic data; the third data log belongs to the first data log, and the fourth data log belongs to the second data log.

3. The data aggregation method according to claim 2, characterized in that, The step of generating a third data log corresponding to the sub-traffic data based on the sub-traffic data includes: The first indicator information within each unit of time in the sub-traffic data is combined to obtain the third data log corresponding to each unit of time.

4. The data aggregation method according to claim 2, characterized in that, The step of generating a fourth data log corresponding to the sub-traffic data based on the sub-traffic data includes: The first flow data in the sub-flow data is merged to obtain at least one flow data; the first flow data is the flow data of the first message with the same five-tuple information; the five-tuple information includes: source address information, source port information, destination address information, destination port information and communication protocol; At least one sub-data log is generated based on the streaming data; the sub-data log belongs to the fourth data log.

5. The data aggregation method according to claim 4, characterized in that, The step of generating at least one sub-data log based on the streaming data includes: If the session to which the first packet corresponding to the streaming data belongs has ended, determine the duration of the session to which the first packet belongs; If the duration is less than or equal to the preset duration, a sub-data log is generated; If the duration exceeds a preset duration, multiple sub-data logs are generated based on the preset duration.

6. The data aggregation method according to claim 1, characterized in that, The method further includes: Receive a display instruction; wherein the display instruction is used to request the display of target traffic data, and the display instruction includes a query index and display format of the target traffic data; the query index includes a first index or a second index; the first index includes at least one or more of the following: target link identifier, target time period; the second index includes at least one or more of the following: target link identifier, target quintuple information, target time period; Query the target traffic data according to the query index; The target traffic data is displayed in the manner described.

7. A data aggregation device, characterized in that, The device includes: The data acquisition module is used to acquire traffic data of the network data stream; wherein, the traffic data includes a link identifier; A log generation module is used to generate at least one first data log and at least one second data log based on the traffic data acquired by the data acquisition module; wherein, the at least one first data log includes all first indicator information in the traffic data, and the first indicator information is summary statistical information of the network data stream; the at least one second data log includes all second indicator information in the traffic data, and the second indicator information is session information of the network data stream; A first aggregation module is configured to aggregate all the first data logs according to a first aggregation rule based on the link identifier corresponding to the first data logs generated by the log generation module, to obtain at least one first aggregated log; the first aggregated logs include at least one first aggregated log; wherein, the first data logs are aggregated in ascending order of multiple time granularities to obtain a first aggregated log corresponding to each time granularity; and / or, the first data logs are aggregated according to the link identifier to obtain a first aggregated data log corresponding to the link identifier; The second aggregation module is used to aggregate all the second data logs according to the link identifier corresponding to the second data log generated by the log generation module, and obtain at least one second aggregated log; the second aggregated log includes at least one second aggregated log; wherein, the second data logs are aggregated according to the link identifier to obtain the second aggregated data log corresponding to the link identifier; and / or, the second data logs are aggregated according to the 5-tuple information to obtain the second aggregated data log corresponding to the 5-tuple information.

8. An electronic device, characterized in that, include: processor; Memory for storing the executable instructions of the processor; The processor is configured to execute the data aggregation method according to any one of claims 1-6 by executing the executable instructions.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions that, when executed by a processor of the data aggregation device, enable the data aggregation device to perform the data aggregation method as described in any one of claims 1-6.

Images