Statistical quantity accumulation method and system and statistical quantity storage method and system

Abstract

The invention provides a statistical magnitude accumulation method and system. The statistical magnitude accumulation method comprises the steps that S1, a threshold value t is set; s2, setting a counting register based on a threshold value t; s3, receiving a data stream, and accumulating statistical values through a counting register according to frames of the data stream; s4, judging whether the accumulated statistical value is equal to a threshold value t or not; if the result is yes, sending a statistical signal and a statistical increment, resetting the counting register, and ending; if not, the step S3 is executed again. According to the invention, a fast computing system is realized in a register accumulator mode, the requirement for copying a plurality of storage systems at a rear stage is eliminated, the design complexity of the system is greatly simplified, and the overhead cost of the system is also reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of testing instruments, and in particular, relates to a statistical accumulation method and system and a storage method and system, and in particular, to a high-speed cumulative statistical storage method and system. Background Art

[0002] Ethernet tester, Ethernet tester is a special test instrument used to generate Ethernet network data traffic and inject it into the device under test, and obtain information about the device under test by analyzing the network data traffic status returned from the device under test.

[0003] The tag field refers to a byte sequence defined for the purpose of testing. It contains information such as timestamp, sequence number, data stream number, etc. It is added to the end of the test frame when the Ethernet tester constructs the test frame.

[0004] Ethernet test frame: refers to the Ethernet frame specially generated by the Ethernet tester for testing purposes. In most cases, these test frames have a label field to achieve testing purposes.

[0005] Ethernet testers are widely used in the research and development of network equipment, the selection of network equipment, the production of network equipment, the construction of network deployment, etc. With the increasing enrichment and enhancement of network equipment functions, network testers have gradually realized the testing function of multiple data streams, that is, realizing multiple logical data flows on one physical port, sharing the sending and receiving functions of one physical port.

[0006] like Figure 1 As shown in the above example, there are n data streams in total, and each data stream has a logical test frame data stream, namely p0, p1, and p2. In order to distinguish these frames, a test domain structure is defined at the end of each frame, such as Figure 2 As shown, the data stream number uniquely identifies a specific data stream, and the receiver uses this value to classify various data stream statistics.

[0007] In order to achieve the purpose of testing, it is a common requirement to perform cumulative statistics on certain characteristics of network packets. For example, statistics on the number of sent and received data stream frames. For this type of statistics, a significant feature is that it does not focus on the characteristics of a single network packet, but only on the cumulative amount.

[0008] Statistics also need to be stored, whether it is stored in a volatile storage device, such as DDR, or a non-volatile storage device, such as an SSD hard drive. To ensure the correctness of the accumulated data, the storage logic process requires three steps to complete: Step S1: read out the old value; Step S2: accumulate the new value; Step S3: write back.

[0009] That is to say, Figure 3 As shown in the figure, a typical storage system for cumulative statistics needs to have two main parts: the part for analyzing and extracting the required statistics and the part for accumulating and writing back to the storage.

[0010] As network rates increase, a prominent problem is that the analysis and extraction of statistics is faster than the accumulation and storage. For example, in a 100G packet processing system implemented with FPGA, in the case of short packets, that is, 64 bytes, an average of one packet comes every 2 clock cycles, but reading the original count, accumulating, and writing back to storage requires at least 3 cycles, so a method is needed to reduce the statistical count rate transmitted to the back-end storage. For the components shown in the figure, in the case of high-speed short packets, statistics extraction comes every two cycles on average, but statistics accumulation and writing back require 3 cycles. When the network port rate increases to 400G or 1000G, the situation will be even worse.

[0011] US9197574B2 proposes a technical solution, which is to copy several identical statistical storage queues, and distribute the statistical count values ​​to be transmitted to different statistical storage queues according to certain rules. For example, the storage backend sets three identical storage queues, and divides the statistical counts equally into these three queues according to certain rules. Then the data rate of each queue is only 1 / 3 of the original data rate. Figure 4 shown.

[0012] The disadvantage of the above solution is that the storage queue that needs to be replicated increases the cost and complexity of the system. It is even not feasible in some cases. For example, due to chip yield considerations, the chip area cannot be made too large, so the number of pins that the chip can support is limited. Since replicating the storage queue requires occupying multiple chip pins, it is not feasible due to the limitation of the number of chip pins.

[0013] In other words, the current classic Ethernet transmission, the communication channel rate has reached up to 100Gbps or even 400Gbps, and the challenge to hardware design is getting bigger and bigger. In the field of network, it is a common requirement to count the characteristics of network packets. It is very common to implement network processing systems with general-purpose CPUs or FPGAs as the core. For such systems, the problem of mismatch between the rate of reaching and extracting statistics and the rate of accumulating and storing statistics is becoming more and more prominent as the network line speed increases. Some method must be used to reduce the statistical counting rate seen by the subsequent storage system.

[0014] Patent document US9197574B2 proposes a method of replicating multiple statistical systems and distributing statistical quantities to multiple statistical systems to achieve this purpose. The disadvantage of this solution is that it significantly increases the cost and complexity of the system. Summary of the invention

[0015] In view of the defects in the prior art, the object of the present invention is to provide a statistical accumulation method and system and a storage method and system.

[0016] A statistical accumulation method provided by the present invention includes:

[0017] Step S1: setting a threshold value t;

[0018] Step S2: Based on the threshold value t, set the counting register;

[0019] Step S3: receiving a data stream, and accumulating statistical values ​​through a counting register according to frames of the data stream;

[0020] Step S4: determine whether the accumulated statistical value is equal to the threshold value t; if the result is yes, send a statistical signal and a statistical increment, and clear the counting register, and end; if the result is no, re-execute step S3.

[0021] Preferably, in step S2:

[0022] The threshold value t is expressed as follows:

[0023] t=2 m

[0024] Wherein, the superscript m is an integer, m=4 to 6;

[0025] The counting register has m bits.

[0026] Preferably, in step S3: for each incoming frame of the data stream, the statistical value is increased by one.

[0027] Preferably, in step S3: when the frames of the data stream arrive in sequence at a ratio of 1:1, a statistical command is used to make the statistical values ​​reach the threshold value t in sequence;

[0028] When the data stream stops being sent, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

[0029] A method for storing statistics provided by the present invention includes:

[0030] Step A1: Initialize configuration and set parameters; receive data streams and perform statistics on them to obtain statistical results;

[0031] Step A2: buffering statistical results;

[0032] Step A3: Accumulate and write back the buffered statistical results, and send them to the off-chip memory for storage;

[0033] The step A1 adopts the statistical accumulation method.

[0034] A statistical quantity accumulation system provided by the present invention comprises:

[0035] Refresh control unit 1: set threshold t;

[0036] Refresh control unit 2: based on threshold t, set counting register;

[0037] Refresh control unit three: receiving data streams, and accumulating statistical values ​​through counting registers according to frames of the data streams;

[0038] Refresh control unit 4: determine whether the accumulated statistical value is equal to the threshold t; if the result is yes, send the statistical signal and the statistical increment, and clear the counting register, and end; if the result is no, re-execute the refresh control unit 3.

[0039] Preferably, in the second refresh control unit:

[0040] The threshold value t is expressed as follows:

[0041] t=2 m

[0042] Wherein, the superscript m is an integer, m=4 to 6;

[0043] The counting register has m bits.

[0044] Preferably, in the refresh control unit three: for each incoming frame of the data stream, the statistical value is increased by one.

[0045] Preferably, in the refresh control unit 3: when the frames of the data stream arrive in sequence at a ratio of 1:1, the statistical commands are used to make the statistical values ​​reach the threshold value t in sequence;

[0046] When the data stream stops being sent, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

[0047] A statistical storage system provided according to the present invention includes:

[0048] Refresh control module: initialize configuration, set parameters; receive data stream, perform statistics on it, and obtain statistical results;

[0049] Refresh buffer module: buffer statistical results;

[0050] On-chip register accumulation module: accumulates and writes back the buffered statistical results, and sends them to the off-chip memory for storage.

[0051] The refresh control module includes the statistical quantity accumulation system.

[0052] Compared with the prior art, the present invention has the following beneficial effects:

[0053] 1. The present invention divides the statistical values ​​into two levels, fast and slow, and can complete the high-speed statistical storage with a single storage system. The use of register accumulators is relatively simple in technical implementation, and for complex logic systems such as FPGA, the operation is also relatively simple.

[0054] 2. The present invention realizes a fast computing system through a register accumulator, eliminating the need for subsequent replication of multiple storage systems, greatly simplifying the complexity of system design and reducing system overhead costs.

[0055] 3. The present invention can realize high-speed calculation and storage of cumulative statistics without increasing storage queues, that is, it can complete the correct storage of cumulative statistics in real time without significantly increasing system costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0057] Figure 1 A schematic diagram of multiple data streams sharing one physical port in the prior art;

[0058] Figure 2 It is a schematic diagram of the test frame label domain of the prior art;

[0059] Figure 3 A schematic diagram of the statistical accumulation steps of the prior art;

[0060] Figure 4 A schematic diagram of multi-queue write-back in the prior art;

[0061] Figure 5 A schematic diagram of a block diagram provided by the present invention;

[0062] Figure 6 A schematic diagram of the cumulative and flushing counts provided by the present invention. DETAILED DESCRIPTION

[0063] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several changes and improvements can also be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0064] This proposal is based on the following observations:

[0065] In actual network system architecture, there are a large number of systems designed with FPGA or general-purpose CPU as the core. In addition, in the system, when the network rate is getting higher and higher, for example, 100 Gbps and above, the arrival rate of statistical counts is higher than the speed that the backend storage system can handle.

[0066] Based on productivity considerations, the chip area and thus the number of pins that the chip can support is relatively limited;

[0067] As chip technology advances, the computing and storage resources inside the chip will become more and more;

[0068] In real Ethernet test scenarios, packet loss, disorder, and other abnormal situations rarely occur, and most packets arrive in order;

[0069] The segmented accumulation of cumulative statistical counts does not affect the accuracy of the final value;

[0070] Hardware register operations, such as addition, can be completed in just one clock cycle;

[0071] In the world of network testing, the sending of test frames is known and controlled.

[0072] Network packet, that is, network data packet.

[0073] Multi-stream statistics is a common statistical item. For example, in a test, the user configures 1024 streams. These 1024 streams are distinguished by the stream numbers in the label field mentioned above. For each stream, the total number of sent and received packets needs to be counted. The total number of sent and received packets for each stream is represented by a 32-bit, 48-bit, 64-bit, or 128-bit count value. These count values ​​ultimately need to be stored in the RAM on the FPGA or in an off-chip memory outside the FPGA chip, such as DDR. Both the on-chip RAM of the FPGA and the off-chip memory outside the FPGA chip require multiple cycles to complete an accumulation-writeback. In the following, the off-chip memory outside the FPGA chip is used to illustrate the problem.

[0074] like Figure 5 As shown, this solution adds the following hardware devices to realize off-chip storage of high-speed statistical values: on-chip register accumulation module, refresh control module, refresh buffer module;

[0075] The on-chip register accumulation module realizes fast accumulation of statistics, and can perform one addition per clock. In addition, the on-chip register accumulation module is shorter. For example, a 5-bit or 6-bit register can achieve 32 or 64 accumulations without overflow.

[0076] A refresh control module is used to control when to pass the value of the register accumulation unit backward to achieve accumulation and storage in the off-chip memory, and to clear the value of the register;

[0077] The refresh buffer module is used to buffer the register accumulation value that needs to be processed by the off-chip memory. For example, assuming that the DDR controller needs 20 cycles to complete a read-accumulation-write back, then this buffer unit is needed to buffer the value to be processed from the register accumulation unit.

[0078] According to the previous description, the problem that needs to be solved is to match the arrival rate of statistics with the statistics that the back-end memory can process. This solution adopts a principle similar to sampling and needs to ensure that under various boundary conditions, the back-end memory sees a statistic arrival rate that is not higher than its processing capacity.

[0079] A threshold t register is implemented in the refresh module. The user sets that when a statistical value reaches a certain threshold t, a statistical command is sent to the backend, and the corresponding increment is also sent.

[0080] Assume that the threshold t is t=2 m , set a statistical m-bit counting register for each data stream; generally speaking, preferably, m=4-6; for a certain data stream, the statistical value is increased by one for each incoming frame. If the current accumulated statistical value plus 1 is equal to the threshold t, a statistical signal and a statistical increment are sent to the subsequent refresh buffer module, and the m-bit counting register is cleared.

[0081] For network testing scenarios, there are some special edge cases that need to be considered:

[0082] 1. When the frames of each data stream arrive in sequence at a ratio of 1:1, their register accumulated values ​​will also reach the threshold value t in sequence. Since the preset rule is that when the register value of a data stream reaches t, a statistical command is sent to the subsequent module. In this case, a burst of statistical commands will appear, and their sending interval is equal to the arrival interval of network packets.

[0083] The refresh buffer can solve this problem. It can be implemented as a FIFO structure, and its depth can be designed as long as there is no loss of statistical commands.

[0084] Second, when the arrival of network packets is particularly sparse, a large number of counts will also appear, accumulating in the register accumulation value, waiting to be refreshed to the subsequent module;

[0085] To this end, the software controls either lowering the threshold t or forcing a refresh periodically;

[0086] 3. When the test is completed, all network packets stop being sent. It is possible that one or more data streams will never reach the threshold t. In this case, since the sending is known, the software can also know when the sending stops. A self-clearing forced refresh control register is set in the refresh module, and the software controls to force all current statistical values ​​to be refreshed to the subsequent modules;

[0087] Aiming at the application characteristics of network testing, this solution adds a fast register accumulator and a corresponding refresh trigger mechanism and device to realize fast and slow two-level statistical counting, thereby realizing that a set of off-chip memory can achieve high-speed cumulative statistical value counting.

[0088] Specifically, fast counting means that the register can count once per cycle; slow counting means that the register value is refreshed once, and the external memory can realize a read-accumulation-write back, which is a very slow operation compared to register accumulation. For example, if the register is 5 bits, the register can accumulate 32 times before refreshing to the subsequent external memory. Every time the register accumulates 32 times, the external register accumulates once.

[0089] A hardware-based solution provided according to the present invention includes:

[0090] Step 1: Initialize the configuration and set parameters such as refresh threshold t;

[0091] Step 2: Receiving network packets;

[0092] Step 3: Extracting network packet statistics;

[0093] Step 4: Select the accumulator register;

[0094] Step 5: Register accumulation; (fast, can be completed in a single clock cycle);

[0095] Step 6: Check whether it needs to be refreshed to the subsequent accumulation and write-back modules; (slow, requiring multiple cycles to complete).

[0096] That is to say, a statistical accumulation method provided by the present invention includes:

[0097] Step S1: setting a threshold value t;

[0098] Step S2: Based on the threshold value t, set the counting register;

[0099] Step S3: receiving a data stream, and accumulating statistical values ​​through a counting register according to frames of the data stream;

[0100] Step S4: determine whether the accumulated statistical value is equal to the threshold value t; if the result is yes, send a statistical signal and a statistical increment, and clear the counting register, and end; if the result is no, re-execute step S3.

[0101] Specifically, in step S2:

[0102] The threshold value t is expressed as follows:

[0103] t=2 m

[0104] Wherein, the superscript m is an integer, m=4 to 6;

[0105] The counting register has m bits.

[0106] Specifically, in the step S3: for each incoming frame of the data stream, the statistical value is increased by one;

[0107] When the frames of the data stream arrive in sequence at a ratio of 1:1, the statistical commands are used to make the statistical values ​​reach the threshold t in sequence;

[0108] When the data stream stops being sent, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

[0109] A method for storing statistics provided by the present invention includes:

[0110] Step A1: Initialize configuration and set parameters; receive data streams and perform statistics on them to obtain statistical results;

[0111] Step A2: buffering statistical results;

[0112] Step A3: Accumulate and write back the buffered statistical results, and send them to the off-chip memory for storage.

[0113] The step A1 adopts the statistical accumulation method.

[0114] The present invention also provides a statistical accumulation system, which can be implemented by executing the process steps of the statistical accumulation method, that is, those skilled in the art can understand the statistical accumulation method as a preferred implementation of the statistical accumulation system.

[0115] A statistical quantity accumulation system provided by the present invention comprises:

[0116] Refresh control unit 1: set threshold t;

[0117] Refresh control unit 2: based on threshold t, set counting register;

[0118] Refresh control unit three: receiving data streams, and accumulating statistical values ​​through counting registers according to frames of the data streams;

[0119] Refresh control unit four: determine whether the accumulated statistical value is equal to the threshold value t; if the result is yes, send the statistical signal and the statistical increment, and clear the counting register, and end; if the result is no, re-execute the refresh control unit three.

[0120] Specifically, in the refresh control unit 2:

[0121] The threshold value t is expressed as follows:

[0122] t=2 m

[0123] Wherein, the superscript m is an integer, m=4 to 6;

[0124] The counting register has m bits.

[0125] Specifically, in the refresh control unit three: for data stream, each time a frame comes, the statistical value is increased by one;

[0126] When the frames of the data stream arrive in sequence at a ratio of 1:1, the statistical commands are used to make the statistical values ​​reach the threshold t in sequence; when the data stream stops sending, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

[0127] The present invention also provides a statistical storage system, which can be implemented by executing the process steps of the statistical storage method, that is, those skilled in the art can understand the statistical storage method as a preferred implementation of the statistical storage system.

[0128] A statistical storage system provided according to the present invention includes:

[0129] Refresh control module: initialize configuration, set parameters; receive data stream, perform statistics on it, and obtain statistical results;

[0130] Refresh buffer module: buffer statistical results;

[0131] On-chip register accumulation module: accumulates and writes back the buffered statistical results, and sends them to the off-chip memory for storage.

[0132] The refresh control module executes the accumulation system of the statistics.

[0133] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art can make various changes or modifications within the scope of the claims, which does not affect the essence of the present invention. In the absence of conflict, the embodiments of the present application and the features in the embodiments can be combined with each other arbitrarily.

Claims

1. A statistical accumulation method, characterized in that: include: Step S1: setting a threshold value t; Step S2: Based on the threshold value t, set the counting register; Step S3: receiving a data stream, and accumulating statistical values ​​through a counting register according to frames of the data stream; Step S4: determine whether the accumulated statistical value is equal to the threshold value t; if the result is yes, send a statistical signal and a statistical increment, and clear the counting register, and end; if the result is no, re-execute step S3.

2. The statistical accumulation method according to claim 1, characterized in that: In step S2: The threshold value t is expressed as follows: t＝2 m Wherein, the superscript m is an integer, m=4 to 6; The counting register has m bits.

3. The method for storing statistics according to claim 2, characterized in that: In step S3: for each incoming frame of the data stream, the statistical value is increased by one.

4. The method for storing statistics according to claim 3, characterized in that: In step S3: when the frames of the data stream arrive in sequence at a ratio of 1:1, the statistical commands are used to make the statistical values ​​reach the threshold t in sequence; When the data stream stops being sent, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

5. A method for storing statistics, characterized in that: include: Step A1: Initialize configuration and set parameters; receive data streams and perform statistics on them to obtain statistical results; Step A2: buffering statistical results; Step A3: Accumulate and write back the buffered statistical results, and send them to the off-chip memory for storage; The step A1 adopts the statistical accumulation method described in claim 1.

6. A statistical accumulation system, characterized in that: include: Refresh control unit 1: set threshold t; Refresh control unit 2: based on threshold t, set counting register; Refresh control unit three: receiving data streams, and accumulating statistical values ​​through counting registers according to frames of the data streams; Refresh control unit four: determine whether the accumulated statistical value is equal to the threshold value t; if the result is yes, send the statistical signal and the statistical increment, and clear the counting register, and end; if the result is no, re-execute the refresh control unit three.

7. The statistical quantity accumulation system according to claim 6, characterized in that: In the refresh control unit 2: The threshold value t is expressed as follows: t＝2 m Wherein, the superscript m is an integer, m=4 to 6; The counting register has m bits.

8. The statistical quantity accumulation system according to claim 7, characterized in that: In the refresh control unit three: for data stream, each time a frame comes, the statistical value is increased by one.

9. The statistical accumulation system according to claim 8, characterized in that: In the refresh control unit 3: when the frames of the data stream arrive in sequence at a ratio of 1:1, the statistical commands are used to make the statistical values ​​reach the threshold value t in sequence; When the data stream stops being sent, the statistical value cannot reach the threshold t, and the statistical value is forced to be refreshed.

10. A statistical storage system, characterized in that: include: Refresh control module: initialize configuration, set parameters; receive data stream, perform statistics on it, and obtain statistical results; Refresh buffer module: buffer statistical results; On-chip register accumulation module: accumulates and writes back the buffered statistical results, and sends them to the off-chip memory for storage. The refresh control module includes the statistical accumulation system of claim 6.

Images