Traffic statistical method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a traffic statistical method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of cloud. The method comprises the following steps: in a process of periodically sampling a message sent by a network card, for each sampling period, acquiring a first count of a preset counter at the end of a previous sampling period of the sampling period; updating the first count based on a preset value to obtain a second count at the beginning of the sampling period; sampling according to the second count; and performing flow statistics on the messages sent by the network card in the sampling period according to the messages sampled in the sampling period. According to the embodiment of the invention, the counter is introduced, peak clipping and valley filling are realized, and the increase of the calculated flow error caused by incapability of sampling in some sampling periods or insufficient sampling in some sampling periods in a burst flow scene is avoided.

Description

Technical Field

[0001] The present application relates to technical fields such as cloud technology, cloud computing, smart transportation, and big data. The present application relates to a traffic statistics method, device, electronic device, storage medium, and program product. Background Art

[0002] With the development of network technology, the amount of data transmitted on the network is also increasing. In some scenarios in this field, it is usually necessary to collect traffic statistics of the transmitted data to analyze network behavior and diagnose problems.

[0003] When the traffic packet sending speed is uniform, according to the collection logic, the traffic collected by a certain flow in a single sampling period is relatively uniform and stable, and the calculated traffic can truly and accurately reflect the actual traffic.

[0004] However, if the system network resources are limited, or if the business logic requires burst traffic when sending messages, the traffic collected in each sampling period for a certain flow will be uneven. In extreme cases, no traffic may be collected in multiple sampling periods, resulting in a large error between the calculated traffic and the actual traffic. Summary of the Invention

[0005] The embodiments of the present application provide a traffic statistics method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can improve the accuracy of traffic statistics.

[0006] According to one aspect of an embodiment of the present application, a traffic statistics method is provided, the method comprising:

[0007] In the process of periodically sampling the messages sent by the network card, for each sampling period, obtaining a first count of a preset counter at the end of a previous sampling period of the sampling period;

[0008] updating the first count according to a first preset value to obtain a second count at the beginning of the sampling period;

[0009] Sampling is performed using the second count as a target sampling number of the sampling period, and reducing the second count by a second preset value each time a message is obtained by sampling;

[0010] According to each message sampled in the sampling period, traffic statistics are performed on the messages sent by the network card in the sampling period.

[0011] According to another aspect of an embodiment of the present application, a traffic statistics device is provided, the device comprising:

[0012] A historical count acquisition module is used to acquire, for each sampling period, a first count of a preset counter at the end of a previous sampling period of the sampling period during the process of periodically sampling messages sent by the network card;

[0013] a count updating module, configured to update the first count according to a first preset value to obtain a second count at the beginning of the sampling period;

[0014] a sampling module, configured to perform sampling using the second count as a target sampling number for the sampling period, and to reduce the second count by a second preset value each time a message is obtained through sampling;

[0015] The statistics module is used to perform flow statistics on the messages sent by the network card during the sampling period according to the messages sampled during the sampling period.

[0016] As an optional embodiment, the sampling module includes:

[0017] The message sending number submodule is used to obtain the number of messages sent by the network card to the cache during the sampling period;

[0018] a sampling rate determination submodule, configured to obtain a message sampling rate of the sampling period according to the number of messages and the second count;

[0019] A sampling submodule is configured to perform sampling from the cache according to the message sampling rate and the second count.

[0020] As an optional embodiment, the counting update module includes:

[0021] an accumulation submodule, configured to accumulate the first preset value based on the first count to obtain a second initial count;

[0022] The count determination submodule is configured to use the second initial count as the second count if the second initial count does not exceed a preset threshold, and use the preset threshold as the second count if the second initial count exceeds a preset threshold.

[0023] As an optional embodiment, the statistics module includes:

[0024] A classification submodule, configured to count the amount of sampled data belonging to each message type in each message obtained by sampling according to at least one message type;

[0025] The classification statistics submodule is used to obtain the data volume of each message type sent by the network card within the sampling period based on the message sampling rate and the sampled data volume of each message type.

[0026] As an optional embodiment, a method for obtaining at least one message type includes any one of the following:

[0027] Performing statistics on at least one quintuple corresponding to each sampled message to obtain the at least one message type;

[0028] Counting at least one service identifier corresponding to each message obtained by the sampling to obtain the at least one message type;

[0029] Statistics are collected on at least one quality of service (QoS) priority corresponding to each sampled message to obtain the at least one message type.

[0030] As an optional embodiment, the classification submodule includes:

[0031] A type extraction unit, configured to extract the message type of the corresponding message from each sampled message;

[0032] An association unit is used to, for each message obtained by sampling, based on the message type of each message and in accordance with a pre-configured association relationship between the message type and the sampled data volume, accumulate the data volume of each message to the sampled data volume associated with the message type of each message.

[0033] As an optional embodiment, the message sending submodule includes:

[0034] a sequence number determination unit, configured to obtain a first sequence number of a last message sent by the network card at the end of the sampling period, and a second sequence number of a last message sent by the network card at the end of a sampling period preceding the sampling period;

[0035] The difference calculation unit is used to obtain the number of messages sent by the network card to the cache in the sampling period according to the difference between the first message and the second message.

[0036] According to another aspect of an embodiment of the present application, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the above-mentioned traffic statistics method.

[0037] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the above-mentioned traffic statistics method is implemented.

[0038] According to one aspect of an embodiment of the present application, a computer program product is provided, including a computer program, which implements the above-mentioned traffic statistics method when executed by a processor.

[0039] The technical solution provided by the embodiment of the present application pre-sets a counter, and the count of the counter at the beginning of each sampling period represents the maximum number of messages sampled in the corresponding sampling period. The first count of the preset counter at the end of the previous sampling period of the sampling period is obtained, and the first count is updated with a preset value to obtain the second count at the beginning of the current sampling period. Therefore, when the corresponding number of messages is not collected in a sampling period, the remaining count will be accumulated to the next sampling period to achieve peak shaving and valley filling. When a burst of messages is sent, the number of messages that can be collected in the corresponding sampling period will also increase, avoiding the problem of increased error in the estimated traffic due to failure to collect messages in some sampling periods or insufficient collection in some sampling periods in burst traffic scenarios, thereby improving the accuracy of traffic statistics. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A schematic diagram of the system architecture for implementing the traffic statistics method provided in an embodiment of the present application;

[0041] Figure 2 A schematic diagram of the inventive concept of a traffic statistics method provided in an embodiment of the present application;

[0042] Figure 3 A flow chart of a traffic statistics method provided in an embodiment of the present application;

[0043] Figure 4 A flow chart of a traffic statistics method provided in an embodiment of the present application;

[0044] Figure 5 A flow chart of a traffic statistics method provided in an embodiment of the present application;

[0045] Figure 6 A flow chart of a sampling process and a flow statistics process provided in an embodiment of the present application;

[0046] Figure 7 A schematic diagram of the sampling error rate of the traffic statistics method provided by the related art;

[0047] Figure 8 A schematic diagram of the sampling error rate of the traffic statistics method provided in an embodiment of the present application;

[0048] Figure 9 A schematic diagram of the structure of a flow statistics device provided in an embodiment of the present application;

[0049] Figure 10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0051] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an" and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to that the element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0052] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0053] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide or local area network (WAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0054] Cloud computing is a computing model that distributes computing tasks across a resource pool consisting of a large number of computers, enabling various application systems to access computing power, storage space, and information services as needed. The network that provides these resources is called the "cloud." To users, these resources appear infinitely scalable and can be accessed at any time, used on demand, expanded at any time, and paid for on a per-use basis. As a provider of basic cloud computing capabilities, a cloud computing resource pool (referred to as a cloud platform, generally referred to as an IaaS (Infrastructure as a Service) platform) will be established. Various types of virtual resources will be deployed in the resource pool for external customers to choose and use. The cloud computing resource pool mainly includes: computing devices (virtualized machines, including operating systems), storage devices, and network devices. According to the logical function division, the PaaS (Platform as a Service) layer can be deployed on the IaaS (Infrastructure as a Service) layer, and the SaaS (Software as a Service) layer can be deployed on the PaaS layer. SaaS can also be deployed directly on IaaS. PaaS is a platform for software operation, such as databases and web containers. SaaS is a variety of business software, such as web portals and SMS mass senders. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

[0055] When the traffic packet transmission rate is uniform, the traffic collected for a particular flow in a single sampling cycle is relatively uniform and stable according to the collection logic, and the calculated traffic can accurately reflect the actual traffic. However, if the system network resources are limited, or if the business logic requires bursty traffic during packet transmission, the traffic collected for a particular flow in each sampling cycle may be uneven. In extreme cases, no traffic may be collected for multiple sampling cycles, resulting in a significant discrepancy between the calculated and actual traffic.

[0056] The traffic statistics method, device, electronic device, computer-readable storage medium, and computer program product provided in this application are intended to solve the above technical problems in the prior art.

[0057] The following describes several exemplary embodiments to illustrate the technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application. It should be noted that the following embodiments can refer to, draw on, or combine with each other, and the same terms, similar features, and similar implementation steps in different embodiments will not be repeated.

[0058] Figure 1 This is a schematic diagram of the implementation environment of a traffic statistics method provided by this application. Figure 1 As shown, the implementation environment includes: an electronic device 101, which can be a server, a terminal, or a cloud computing center device, etc.

[0059] The electronic device 101 may be configured with an application for executing the traffic statistics method of the present application. The application may be an independent application or a program plug-in installed in an independent application. For example, the application may be a management platform with a traffic statistics function. Of course, the management platform may also have functions such as traffic analysis and traffic billing. The present application does not limit this.

[0060] The electronic device 101 may adopt a Linux system, and the electronic device 101 includes a Linux-based kernel and a network card; wherein the Linux-based kernel may send a message to be sent to the network card, and the message is sent out via the network card.

[0061] In one possible scenario, the network card sends the message to the cache, and as long as there is a message in the cache, it will notify the corresponding application to process the message without waiting for the cache to be full.

[0062] The embodiment of the present application adopts periodic sampling. Taking into account the uneven characteristic of packet sending, which is that every message may be cached in the previous sampling period, while there may be many messages in the next sampling period, the embodiment of the present application pre-sets a counter. The count of the counter at the beginning of each sampling period represents the maximum number of messages sampled in the corresponding sampling period, and when the corresponding number of messages is not collected in a sampling period, the remaining count will be accumulated to the next sampling period, thereby achieving peak shaving and valley filling. When there is a sudden sending of messages, the number of messages that can be collected in the corresponding sampling period will also increase, avoiding the problem of increased error in the estimated traffic due to failure to collect messages in some sampling periods or insufficient collection in some sampling periods in burst traffic scenarios.

[0063] It should be noted that a server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server or server cluster that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The above-mentioned networks may include, but are not limited to, wired networks and wireless networks, wherein the wired networks include local area networks, metropolitan area networks, and wide area networks, and the wireless networks include Bluetooth, Wi-Fi, and other networks that enable wireless communication.

[0064] The terminal can be a smart phone (such as an Android phone, iOS phone, etc.), a tablet computer, a laptop computer, a digital broadcast receiver, a MID (Mobile Internet Devices), a PDA (Personal Digital Assistant), a desktop computer, a vehicle-mounted terminal (such as a vehicle-mounted navigation terminal, a vehicle-mounted computer, etc.), a smart speaker, a smart watch, etc. The specific requirements can also be determined based on the actual application scenario and are not limited here.

[0065] See Figure 2 , which exemplarily shows a schematic diagram of the inventive concept of the traffic statistics method provided by an embodiment of the present application. As shown in the figure, the embodiment of the present application sets a counter. The count of the counter at the beginning of each sampling period represents the target number of samples planned to be collected in the sampling period. The count of the counter increases by a first preset value at the beginning of each sampling period. The count will decrease with each collected message. If the target number of samples is not collected in the sampling period, the remaining number of the counter will be accumulated to the next sampling period. During each message collection, it will be determined whether the count has dropped to a preset value (for example, 0). When the preset value is reached, the collection of the sampling period is stopped. If the sending time of the collected message coincides with the sampling time, the collection of the sampling period is also stopped. Based on all the messages collected in the period, traffic statistics are calculated for the messages sent by the network card in the sampling period. The embodiment of the present application achieves peak shaving and valley filling by introducing a counter. When a large number of messages are sent suddenly, the number of messages collected in the corresponding sampling period is large, and it is not limited to collecting a fixed number of messages in each sampling period. This avoids the increase in the estimated traffic error caused by the failure to collect or insufficient collection in some sampling periods in burst traffic scenarios.

[0066] The present application provides a method for counting traffic, such as Figure 3 As shown, the method includes:

[0067] S101 . In a process of periodically sampling messages sent by a network card, for each sampling period, obtain a first count of a preset counter at the end of a previous sampling period of the sampling period.

[0068] In this application, if there is a message to be sent in the electronic device, the kernel of the electronic device can send the message to the network card, and the network card can send it. The electronic device can also sample the message to use the sampling situation to calculate the flow statistics of the message actually sent by the network card.

[0069] When the kernel of the electronic device sends a message to the network card, the electronic device can copy the message passing through the network card to the cache so as to sample the message stored in the cache. In some embodiments, as soon as the cache stores the message, the corresponding application will be notified to process it, without having to wait until the cache is full.

[0070] It should be noted that the sampling period is a time window used to define packets. At the end of each sampling period, the number of packets actually sent within that sampling period can be counted. For example, a sampling period can be 30ms long. For example, the first sampling period can be a time interval from 1ms to 30ms. At the end of 30ms, the number of packets sent by the network card between 1ms and 30ms can be counted.

[0071] The embodiment of the present application indicates the maximum number of samples sampled in the sampling period through the count of the counter. Every time a message is collected, the count of the counter will be reduced. If the number of messages sampled in a sampling period does not reach the maximum number, it means that the count of the counter is still remaining. The embodiment of the present application will obtain the count at the end of the previous sampling period (that is, the first count) and add a first preset value to the count to obtain the count at the beginning of the current sampling period (that is, the second count).

[0072] S102: Update the first count based on a first preset value to obtain a second count at the beginning of the sampling period.

[0073] In some embodiments, the present invention accumulates the first count at the end of the previous sampling period with a fixed first preset value in each sampling period. The present invention does not limit the preset value, and it can be 300, 500, etc. For example, if the first count at the end of the previous sampling period is 100 and the preset value is 300, then the second count at the beginning of the current sampling period is 400, which means that a maximum of 400 messages are sampled in the current sampling period.

[0074] S103 : Sampling is performed with the second count as the target sampling number of the sampling period, and the second count is reduced by a second preset value each time a message is obtained through sampling.

[0075] In the embodiment of the present application, sampling is performed from the cache according to the second count in the present sampling period. When sampling, sampling can be performed based on a preset quantity interval. For example, if sampling is performed with a quantity interval of 3, it means that there are 3 messages between the messages obtained from two adjacent samples.

[0076] In the embodiment of the present application, each time a message is sampled from the cache, the second count is reduced by a second preset value. The embodiment of the present application does not limit the size of the second preset value, for example, it can be 1, that is, each time a message is sampled, the second count is reduced by 1.

[0077] It should be understood that there are two situations when sampling from the cache with the second count:

[0078] In case 1, there are a large number of packets in the cache. When the second count reaches 0 while packets are being sampled from the sampling process, sampling in this sampling period is stopped even though there are still packets remaining in the cache.

[0079] In case 2, the number of messages in the cache is small, and the second count has not been cleared yet. The sending time of the sampled message is the same as the current sampling time, indicating that the latest message in the cache has also been collected, and the sampling of the sampling period is also stopped (assuming that there are no new messages stored at the end of the sampling period).

[0080] For example, whenever an original message is sampled, the electronic device may start collecting data of the IP header of the original message from the start bit of the original message, and the collection ends when the collection of the IP header data is completed.

[0081] During actual data collection, the electronic device can set a maximum data length for collection, which is the maximum amount of data that can be collected from a single original message. While collecting data from the IP header of the message, the electronic device can count the length of data collected from the original message in real time. If the collected data length does not exceed the maximum data length, the electronic device continues collecting data until the IP header data is collected, concluding the collection for the original message. If the collected data length exceeds the maximum data length, the collection ends.

[0082] The electronic device may configure the target sampling data length as a target length threshold, and the target length threshold may be configured based on needs, which is not limited in this application.

[0083] In one possible scenario, a VxLAN (Virtual eXtensible Local Area Network) overlay network includes a Layer 2 IP header. Therefore, a target length threshold can be set to a value no less than the Layer 2 IP header's data size. For example, the target length threshold could be 135 bytes. This allows parsing of the VxLAN packet down to the inner transport layer, resulting in the VxLAN packet's quintuple.

[0084] The IP header includes the packet data volume of the packet. For each sampled packet, the total packet data volume of each packet can be obtained by counting, that is, the total sampled data volume.

[0085] S104: Perform traffic statistics on the messages sent by the network card during the sampling period according to the messages sampled during the sampling period.

[0086] For example, the electronic device may further, based on the total sampled data volume, categorize and calculate traffic statistics for different types of messages actually sent during the sampling period. Alternatively, the electronic device may also, based on the total sampled data volume, calculate the total data volume of messages actually sent during the sampling period.

[0087] The traffic statistics method of the embodiment of the present application takes into account the uneven packet sending characteristic that every message may be cached in the previous sampling period, while there may be many messages in the next sampling period. The embodiment of the present application pre-sets a counter, and the count of the counter at the beginning of each sampling period represents the maximum number of messages sampled in the corresponding sampling period. The first count of the preset counter at the end of the previous sampling period of the sampling period is obtained, the first count is updated with a preset value, and the second count at the beginning of the current sampling period is obtained. Therefore, when a sampling period does not collect the corresponding number of messages, the remaining count will be accumulated to the next sampling period to achieve peak shaving and valley filling. When a burst of messages is sent, the number of messages that can be collected in the corresponding sampling period will also increase, avoiding the problem of increased error in the estimated traffic due to failure to collect messages in some sampling periods or insufficient collection in some sampling periods in burst traffic scenarios.

[0088] On the basis of the above embodiments, as an optional embodiment, in order to better deal with burst traffic and avoid certain burst traffic from being unable to be collected, the embodiment of the present application further optimizes the traffic statistics method, does not perform continuous packet collection, but collects one packet every R packets according to the sampling rate R, such as Figure 4 As shown, step S103 may further include steps S1031 to S1033:

[0089] S1031. Obtain the number of messages sent by the network card to the cache during the sampling period;

[0090] S1032. Obtain a message sampling rate of the sampling period according to the number of messages and the second count;

[0091] S1033: Sampling from the cache according to the message sampling rate and the second count.

[0092] In an embodiment of the present application, for each sampling period, the electronic device may count the messages stored in the cache to obtain the number of messages sent by the network card during the sampling period. For example, based on the timestamps of each message in the cache (used to record the time when the message was sent) and the sampling period, the electronic device may use the total number of original messages with timestamps within the sampling period as the number of messages corresponding to the sampling period.

[0093] In this step, the electronic device can obtain the number of messages sent by the network card during the sampling period by calling a statistics interface. The statistics interface can be an interface for counting messages sent by the network card. For example, the statistics interface can be called based on the sampling period to obtain the number of messages sent within the sampling period, and the number can be used as the number of messages sent by the network card to the cache during the sampling period.

[0094] In the embodiment of the present application, the electronic device may obtain the message sampling rate of the sampling period based on the number of messages and the second count.

[0095] For example, the electronic device may use the following formula 1 to calculate the message sampling rate:

[0096] R=N / n Formula 1

[0097] Wherein, R represents the message sampling rate of the sampling period, N represents the number of messages sent by the issuing network card to the cache in the sampling period, and n represents the second count at the beginning of the sampling period.

[0098] It should be noted that the number of sampling messages in the current sampling period is determined by combining the previous sampling period, thereby affecting the message sampling rate of the current sampling period; the first message sampling rate is combined with the sampling situation of the previous sampling period; if the expected number of sampling messages in the previous sampling period is not achieved, it can be added to the number of sampling messages in the current sampling period, so that the message sampling rate of the current sampling period is smaller, that is, the sampling interval of the current sampling period is shortened, thereby increasing the number of sampled messages; thereby, the message sampling rate of each sampling period can be more flexibly and effectively adjusted in combination with actual conditions, so as to make it more in line with actual conditions, thereby improving the flexibility and accuracy of sampling.

[0099] For example, the electronic device may sample the messages in the cache whose timestamps belong to the sampling period according to the obtained message sampling rate. For example, if the message sampling rate is 10, the electronic device may obtain one message from every ten messages in the cache.

[0100] In a possible implementation, the electronic device may sample the IP header of the message and perform statistics on the sampled message based on the information carried in the IP header. Exemplarily, sampling from the cache according to the message sampling rate may include S201 to S202:

[0101] S201. Sample the IP headers of the original messages within the sampling period according to the message sampling rate to obtain the sampled IP headers of the original messages.

[0102] S202: Based on the packet data volume extracted from the IP header of the sampled original packet, obtain the sampled data volume of the sampled packet by statistics.

[0103] For example, every time a message is sampled, the electronic device may start collecting data of the IP header of the original message from the start bit of the message, and the collection ends when the data collection of the IP header is completed.

[0104] During actual data collection, the electronic device can set a maximum data length for collection. The maximum sampled data length is the maximum amount of data that can be collected from a single message. While collecting data from the message's IP header, the electronic device can count the length of data collected for the message in real time. If the collected data length does not exceed the maximum data length, the electronic device continues collecting data until the IP header data is collected, and then the collection for the original message ends. If the collected data length exceeds the maximum data length, the collection ends.

[0105] The electronic device may configure the maximum sampling data length as a target length threshold, and the target length threshold may be configured based on needs, which is not limited in this application.

[0106] In one possible scenario, a VxLAN (Virtual eXtensible Local Area Network) overlay network includes a Layer 2 IP header. Therefore, a target length threshold can be set to a value no less than the Layer 2 IP header's data size. For example, the target length threshold could be 135 bytes. This allows parsing of the VxLAN packet down to the inner transport layer, resulting in the VxLAN packet's quintuple.

[0107] The IP header includes the packet data volume of the packet. For each sampled packet, the total packet data volume of each packet can be obtained by counting, that is, the total sampled data volume.

[0108] Based on the above embodiments, as an optional embodiment, the present embodiment sets a preset threshold based on the processing performance of the electronic device. The preset threshold represents the maximum number of samples allowed by the electronic device within one cycle under the limitation of its own performance. The implementation of step S102 may include S1021, S1022, and one of S1023-1 and S1023-2:

[0109] S1021. Accumulate the preset value based on the first count to obtain a second initial count;

[0110] S1022. Determine whether the second initial count exceeds a preset threshold. If not, execute S1023-1. If yes, execute S1023-2.

[0111] S1023-1. Use the second initial count as the second count;

[0112] S1023-2. Use the preset threshold as the second count.

[0113] The embodiment of the present application pre-configures the extreme value of the number of samples in the sampling period, that is, the preset threshold value, so that if the second initial count obtained by adding the preset value to the first count does not exceed the preset threshold value, the second initial count is used as the second count. If the second initial count exceeds the preset threshold value, the preset threshold value is used as the second count. For example, if the preset threshold value is 2000, the first count at the end of the previous sampling period is 1000, and the preset value is 300, the second initial count is 1300. Since 1300 is less than 2000, the second count is 1300. For another example, if the preset threshold value is 2000, the first count at the end of the previous sampling period is 1800, and the preset value is 300, the second initial count is 2100. Since 2100 is greater than 2000, the second count is still set to 2000. By setting a preset threshold value for the second count, the maximum collection amount of the sampling period will not increase indefinitely, and the collection power consumption of the sampling period is controlled at a relatively balanced level.

[0114] Based on the above embodiments, as an optional embodiment, performing traffic statistics on the messages sent by the network card during the sampling period according to the messages sampled during the sampling period includes:

[0115] According to at least one message type, counting the amount of sampled data belonging to each message type in each message obtained by sampling;

[0116] Based on the message sampling rate and the sampled data volume of each message type, the data volume of each message type sent by the network card within the sampling period is obtained.

[0117] For example, the electronic device may further categorize and calculate traffic statistics for different types of messages based on the total sampled data volume of each message and the message types of messages actually sent during the sampling period. Alternatively, the electronic device may also calculate the total data volume of messages actually sent during the sampling period based on the total sampled data volume.

[0118] In a possible implementation, when the electronic device samples messages based on the message sampling rate, it can also statistically sample the amount of sampled data corresponding to each message type obtained by classification to improve statistical accuracy.

[0119] Based on the above embodiments, as an optional embodiment, according to at least one message type, counting the amount of sampled data belonging to each message type in each sampled message includes:

[0120] Extracting the message type of the corresponding message from each sampled message;

[0121] For each message obtained by sampling, based on the message type of each message and in accordance with the pre-configured association relationship between the message type and the sampled data amount, the data amount of each message is accumulated to the sampled data amount associated with the message type of each message.

[0122] For example, the message type may be extracted from the IP header of the sampled message.

[0123] For example, the electronic device may associate and store each message type and the corresponding sampled data volume for each message type in a map. Based on this, whenever the electronic device samples a message, it may search the map for the sampled data volume associated with the message type of the message, and add the message data volume of the message to the sampled data volume found.

[0124] Based on the above embodiments, as an optional embodiment, in a possible implementation, the message type may be determined by any one of a quintuple of the message, a service identifier, or a quality of service (QoS) priority; accordingly, a method for obtaining the at least one message type includes any one of the following methods 1 to 3:

[0125] Method 1: Count at least one quintuple corresponding to the sampled original message to obtain the at least one message type;

[0126] Method 2: Count at least one service identifier corresponding to the sampled original message to obtain the at least one message type;

[0127] Mode 3: Count at least one quality of service (QoS) priority corresponding to the sampled original message to obtain the at least one message type.

[0128] Exemplarily, the electronic device may also extract at least one of a quintuple, a service identifier, or a QoS (Quality of Service) priority of the original message from the IP header of the sampled original message.

[0129] In method 1, the message type is determined by the quintuple of the original message. Different quintuples correspond to different message types, and each message type is represented by a quintuple. Each quintuple corresponding to each message in the sampled original message can be used as the message type. Based on this, the data volume of messages with different quintuples sent by the network card can be counted separately.

[0130] In method 2, the message type can be determined by the service identifier of the original message. Different service identifiers correspond to different message types, and each message type corresponds to a service identifier. Therefore, the service identifiers corresponding to each message in the sampled original message can be used as the message type. Based on this, the data volume of messages with different service identifiers sent by the network card can be counted separately.

[0131] In method 3, the message type can be determined by the QoS priority of the original message. Different QoS priorities correspond to different message types, and each message type corresponds to a QoS priority. Therefore, the QoS priorities corresponding to each message in the sampled original message can be used as each message type. Based on this, the amount of data sent by messages of different QoS priorities sent by the network card can be counted separately. The embodiment of the present application provides a variety of ways to obtain message types, which can obtain message types based on information of different dimensions, thereby improving the diversity of ways to obtain message types.

[0132] Based on the above embodiments, as an optional embodiment, obtaining the number of packets sent by the network card to the cache during the sampling period includes:

[0133] Obtaining the first sequence number of the last message sent by the network card at the end of the sampling period, and the second sequence number of the last message sent by the network card at the end of the sampling period before the sampling period;

[0134] The number of packets sent by the network card to the cache during the sampling period is obtained according to the difference between the first sequence number and the second sequence number.

[0135] When sending messages, the network card of the embodiment of the present application will mark each message with a sequence number according to the order in which the messages are sent. Therefore, the embodiment of the present application can obtain the number of messages sent to the cache by the network card during the sampling period by subtracting the first sequence number of the last message sent by the network card at the end of the sampling period and the second sequence number of the last message sent at the end of the previous sampling period, thereby ensuring the accuracy of determining the number of messages and improving the efficiency of determining the number of messages.

[0136] The following combination Figure 5 The execution flow shown here introduces the traffic statistics process of this application:

[0137] like Figure 5As shown, the traffic statistics process of this application may include:

[0138] 1. Initialize sampling parameters:

[0139] Set the sampling filter mode to the outbound direction of the network card (outbound and (not broadcast and not multicast));

[0140] Set the maximum length of collected data to the target length threshold. For example, in a VxLAN network scenario based on an overlay network, set the maximum length of collected data to 135 bytes.

[0141] 2. Set the sampling timestamp accuracy, sampling timeout, and cache size:

[0142] The sampling timestamp precision is set to microseconds; the sampling timeout is 10ms. The sampling timeout refers to the waiting time from the start time of the sampling cycle to the start of sampling the message within a sampling cycle. For example, if no original message appears after waiting for 10ms from 1ms to 30ms in a sampling cycle, that is, the message cannot be sampled, the waiting time is stopped and the sampling process corresponding to the sampling cycle ends. The cache size is set to 30M.

[0143] 3. Set the maximum number of maps to be stored;

[0144] Taking the map storing five-tuple traffic statistics as an example, the maximum number of maps storing five-tuple traffic statistics is 4096;

[0145] 4. Get the statistics of the messages currently sent by the network card:

[0146] last_tx_packets and last_tx_bytes, respectively, indicate the sequence number of the last message in the previous sampling period and the number of bytes of the last message;

[0147] 5. Enter the periodic sampling loop:

[0148] First, obtain the start_time timestamp of this cycle; it is used to control the duration of a single sampling cycle to be within 30ms;

[0149] 6. Add a second preset value to the first count n' of the counter at the end of the previous sampling period to obtain the second count n of the counter at the beginning of the current sampling period;

[0150] Set n = n' + 300, and determine whether n is greater than 2000. If n is greater than 2000, set n = 2000.

[0151] 7. Initialize the total amount of sampled data collected in the current cycle:

[0152] Initialize the total amount of sampled data collected in the current cycle mb = 0;

[0153] 8. Get the statistics of the messages sent to the network card in the current cycle:

[0154] tx_packets: indicates the first sequence number of the last packet in the current sampling period;

[0155] tx_bytes: indicates the number of bytes of the last message in the current sampling cycle;

[0156] Calculate the number of packets to be processed in the current cycle, where N = tx_packets – last_tx_packets, where last_tx_packets represents the second sequence number of the last packet in the previous sampling cycle. Mb = tx_bytes – last_tx_byte, where the total amount of data actually sent by the network card is saved, with last_tx_packets being tx_packets and last_tx_bytes being tx_bytes.

[0157] 9. According to the number of messages N and the second count n, obtain the message sampling rate R=N / n of the sampling period, and perform sampling according to the message sampling rate:

[0158] ①: Start sampling the messages in the cache, that is, collect one message every R messages;

[0159] ②: Whenever a message is collected, if n>1, set n=n-1 and parse the five-tuple of the sampled message;

[0160] ③: Search the map for the sampled data volume Nb associated with the corresponding quintuple, and add the data volume of the sampled message to the sampled data volume Nb associated with the quintuple;

[0161] ④: Add the data volume of the sampled message to the total sampled data volume mb corresponding to the current cycle;

[0162] ⑤: Compare the message timestamp with the current time. If the message timestamp is greater than or equal to the current time, it means that the current cycle has ended, and the sampling process of the current cycle ends;

[0163] 10. Calculate the traffic sampling rate: Rb = Mb / mb;

[0164] 11. Count the data volume of each five-tuple actually sent by the network card:

[0165] ①: Traverse the quintuple map, extract the correlation between each quintuple and the amount of sampled data, and calculate the total amount of data sent by each quintuple during the sampling period: S = Rb * Nb;

[0166] ②: Report to the analysis server: Use the quintuple as the key and the traffic flow S as the value, record a piece of data corresponding to each quintuple, and report each piece of data to the traffic analysis server;

[0167] 12. Control the current cycle duration within 30ms:

[0168] Get the timestamp end_time again;

[0169] Calculate the time consumption of this cycle tcap (ms) = end_time-start_time, sleep (30-tcap); based on this, the time consumption of packet capture in this cycle can be controlled within 30ms, thereby indirectly controlling the CPU consumption of this acquisition program at a low level.

[0170] 13. Repeat step 5.

[0171] Below is Figure 6 The flowchart shown in FIG. 1 further introduces the sampling process and traffic statistics process of this application. Figure 6 As shown, taking a 30ms sampling period as an example, for the current period, the first count of a preset counter at the end of the previous sampling period is obtained; the first count is updated according to a first preset value to obtain a second count at the beginning of the sampling period; the second count is used as the target sampling number for the sampling period, and samples are taken from the cache, and the second count is reduced by a second preset value each time a message is sampled; based on each message sampled during the sampling period, traffic statistics are collected for the messages sent by the network card during the sampling period. Based on this, the increase in estimated traffic errors caused by the failure to collect data in certain sampling periods or insufficient data collection in certain sampling periods in burst traffic scenarios is avoided.

[0172] The following combination Figure 7 and Figure 8 , the sampling error rates of the related technologies and the embodiments of the present application are analyzed. Figure 7 is the sampling error rate of the traffic statistics method in the related art; Figure 8 It is the error rate of traffic statistics using the method of the embodiment of the present application.

[0173] like Figure 7 As shown, in the related art, the sampling error rate of the Linux network card is relatively large, and can reach the order of 100 at most. Figure 8The error rate of traffic statistics using the method of the embodiment of the present application is as follows: Figure 8 As shown, when the method of the present application is used for sampling and traffic statistics, the accuracy of the kernel-based traffic collection is greatly improved, and the error rate is greatly reduced, from the original average error rate of about 50% to an average of less than 5%. The accuracy of the method of the present application is significantly improved, which meets the statistical needs of the scenario of massive server deployment, without intruding into the business and without modifying the network card configuration, and the need to accurately count the traffic and burst traffic scenarios.

[0174] The embodiment of the present application provides a flow statistics device, such as Figure 9 As shown, the traffic statistics device may include: a history counting acquisition module 901, a counting update module 902, a sampling module 903 and a statistics module 904, wherein:

[0175] The historical count acquisition module 901 is configured to acquire, for each sampling period, a first count of a preset counter at the end of a previous sampling period of the sampling period during the periodic sampling of messages sent by the network card;

[0176] A count updating module 902 is configured to update the first count according to a first preset value to obtain a second count at the beginning of the sampling period;

[0177] a sampling module 903 configured to take the second count as a target sampling number for the sampling period, perform sampling from the cache, and decrement the second count by a second preset value each time a message is sampled;

[0178] The statistics module 904 is configured to perform traffic statistics on the messages sent by the network card during the sampling period based on the messages sampled during the sampling period.

[0179] As an optional embodiment, the sampling module includes:

[0180] The message sending number submodule is used to obtain the number of messages sent by the network card to the cache during the sampling period;

[0181] a sampling rate determination submodule, configured to obtain a message sampling rate of the sampling period according to the number of messages and the second count;

[0182] A sampling submodule is configured to perform sampling from the cache according to the message sampling rate and the second count.

[0183] As an optional embodiment, the counting update module includes:

[0184] an accumulation submodule, configured to accumulate the first preset value based on the first count to obtain a second initial count;

[0185] The count determination submodule is configured to use the second initial count as the second count if the second initial count does not exceed a preset threshold, and use the preset threshold as the second count if the second initial count exceeds a preset threshold.

[0186] As an optional embodiment, the statistics module includes:

[0187] A classification submodule, configured to count the amount of sampled data belonging to each message type in each message obtained by sampling according to at least one message type;

[0188] The classification statistics submodule is used to obtain the data volume of each message type sent by the network card within the sampling period based on the message sampling rate and the sampled data volume of each message type.

[0189] As an optional embodiment, a method for obtaining at least one message type includes any one of the following:

[0190] Performing statistics on at least one quintuple corresponding to each sampled message to obtain the at least one message type;

[0191] Counting at least one service identifier corresponding to each message obtained by the sampling to obtain the at least one message type;

[0192] Statistics are collected on at least one quality of service (QoS) priority corresponding to each sampled message to obtain the at least one message type.

[0193] As an optional embodiment, the classification submodule includes:

[0194] A type extraction unit, configured to extract the message type of the corresponding message from each sampled message;

[0195] An association unit is used to, for each message obtained by sampling, based on the message type of each message and in accordance with a pre-configured association relationship between the message type and the sampled data volume, accumulate the data volume of each message to the sampled data volume associated with the message type of each message.

[0196] As an optional embodiment, the message sending submodule includes:

[0197] a sequence number determination unit, configured to obtain a first sequence number of a last message sent by the network card at the end of the sampling period, and a second sequence number of a last message sent by the network card at the end of a sampling period preceding the sampling period;

[0198] A difference calculation unit is configured to obtain the number of messages sent by the network card to the cache during the sampling period based on the difference between the first message and the second message. The apparatus of the embodiment of the present application can execute the method provided by the embodiment of the present application, and its implementation principles are similar. The actions performed by each module in the apparatus of each embodiment of the present application correspond to the steps in the method of each embodiment of the present application. For a detailed functional description of each module of the apparatus, please refer to the description of the corresponding method shown above, and will not be repeated here.

[0199] An embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory. The processor executes the computer program to implement the steps of the traffic statistics method. Compared with the related art, the following can be achieved:

[0200] By taking into account the uneven packet sending characteristic in each sampling period, that is, every message may be cached in the previous sampling period, while there may be many messages in the next sampling period, the embodiment of the present application pre-sets a counter, and the count of the counter at the beginning of each sampling period represents the maximum number of messages sampled in the corresponding sampling period, obtains the first count of the preset counter at the end of the previous sampling period of the sampling period, updates the first count with a preset value, and obtains the second count at the beginning of the current sampling period, so that when a sampling period does not collect the corresponding number of messages, the remaining count will be accumulated to the next sampling period, realizing peak shaving and valley filling. When there is a sudden sending of messages, the number of messages that can be collected in the corresponding sampling period will also increase, avoiding the problem of increased error in the estimated traffic due to failure to collect messages in some sampling periods or insufficient collection in some sampling periods in burst traffic scenarios.

[0201] In an alternative embodiment, an electronic device is provided, such as Figure 10 As shown, Figure 10 The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, for example, via a bus 4002. Optionally, the electronic device 4000 may further include a transceiver 4004, which may be used for data exchange between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the number of transceivers 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present application.

[0202] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, and the like.

[0203] Bus 4002 may include a path for transmitting information between the aforementioned components. Bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, for example. Bus 4002 may be divided into an address bus, a data bus, a control bus, and so on. For ease of illustration, bus 4002 is represented by a single thick line in the figure, but this does not indicate that there is only one bus or only one type of bus.

[0204] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation here.

[0205] The memory 4003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the computer program stored in the memory 4003 to implement the steps shown in the above method embodiment.

[0206] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0207] An embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0208] The terms "first," "second," "third," "fourth," "1," "2," and the like (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in an order other than that shown or described in the drawings.

[0209] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0210] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A traffic statistics method, characterized in that: include: In the process of periodically sampling the messages sent by the network card, for each sampling period, obtaining a first count of a preset counter at the end of a previous sampling period of the sampling period; updating the first count according to a first preset value to obtain a second count at the beginning of the sampling period; Sampling is performed using the second count as a target sampling number of the sampling period, and reducing the second count by a second preset value each time a message is obtained by sampling; According to each message sampled in the sampling period, traffic statistics are performed on the messages sent by the network card in the sampling period.

2. The method according to claim 1, characterized in that The performing sampling by taking the second count as the target sampling number of the sampling period includes: Obtain the number of packets sent by the network card to the cache during the sampling period; Obtaining a message sampling rate of the sampling period according to the number of messages and the second count; Sampling is performed from the cache according to the message sampling rate and the second count.

3. The method according to claim 1, characterized in that The updating of the first count according to the first preset value to obtain the second count at the beginning of the sampling period includes: Accumulate the first preset value based on the first count to obtain a second initial count; If the second initial count does not exceed the preset threshold, the second initial count is used as the second count; if the second initial count exceeds the preset threshold, the preset threshold is used as the second count.

4. The method according to any one of claims 1 to 3, characterized in that The performing flow statistics on the messages sent by the network card within the sampling period according to the messages sampled during the sampling period includes: According to at least one message type, counting the amount of sampled data belonging to each message type in each message obtained by sampling; Based on the message sampling rate and the sampled data volume of each message type, the data volume of each message type sent by the network card within the sampling period is obtained.

5. The method according to claim 4, characterized in that The method for obtaining the at least one message type includes any one of the following: Performing statistics on at least one quintuple corresponding to each sampled message to obtain the at least one message type; Counting at least one service identifier corresponding to each message obtained by the sampling to obtain the at least one message type; Statistics are collected on at least one quality of service (QoS) priority corresponding to each sampled message to obtain the at least one message type.

6. The method according to claim 4, characterized in that The counting of the sampled data amount belonging to each message type in each message obtained by sampling according to at least one message type includes: Extracting the message type of the corresponding message from each sampled message; For each message obtained by sampling, based on the message type of each message and in accordance with the pre-configured association relationship between the message type and the sampled data amount, the data amount of each message is accumulated to the sampled data amount associated with the message type of each message.

7. The method according to claim 2, characterized in that The obtaining the number of messages sent by the network card to the cache during the sampling period includes: Obtaining the first sequence number of the last message sent by the network card at the end of the sampling period, and the second sequence number of the last message sent by the network card at the end of the sampling period before the sampling period; The number of packets sent by the network card to the cache during the sampling period is obtained according to the difference between the first sequence number and the second sequence number.

8. A flow statistics device, characterized in that: include: A historical count acquisition module is used to acquire, for each sampling period, a first count of a preset counter at the end of a previous sampling period of the sampling period during the process of periodically sampling messages sent by the network card; a count updating module, configured to update the first count according to a first preset value to obtain a second count at the beginning of the sampling period; a sampling module, configured to perform sampling using the second count as a target sampling number for the sampling period, and to reduce the second count by a second preset value each time a message is obtained through sampling; The statistics module is used to perform flow statistics on the messages sent by the network card during the sampling period according to the messages sampled during the sampling period.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the traffic statistics method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the traffic statistics method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the traffic statistics method according to any one of claims 1 to 7 is implemented.

Images