Window caching device and method for data frame

Abstract

The invention provides a window caching device and method for data frames, which are applied to network flow analysis, and comprises a window used for managing the life cycle of the data frames, and marking the data frames in the window as valid for analysis and reference; the two-way data transmission queue comprises a sending queue and a recycling queue, the production end collects data frames and stores the data frames in a usable container, and after a window value is distributed to the container, the window value is transmitted to the consumption end through the sending queue; and the consumption end receives the data frame and submits the data frame to the window for management, and when the window value corresponding to the container is not in the window, the container occupied by the data frame is released and transmitted to the production end through the recovery queue. According to the method, the memory of the data frame is fully utilized, the data copying frequency of the byte content of the data frame is reduced to the minimum, finally, the system overhead can be reduced, and the overall system performance is improved; traceability and application of certain historical data can be realized, and certain traceability services are solved; the input rate of data frames may also be controlled.

Description

Technical Field

[0001] The present invention relates to the field of network traffic analysis, and in particular to a window buffering device and method for a data frame. Background Art

[0002] In the field of traffic analysis technology, based on the trend of increasing network data traffic, data analysis systems are required to extract a large amount of useful information from massive data. This requires that the data analysis system not only has high data collection efficiency and performance, but also has methods to properly process the collected data. In a full-traffic collection system, if the received data is not properly processed, it will eventually affect system performance and memory overhead, resulting in poor overall system performance.

[0003] When a data frame is obtained from a network device, it is usually allocated memory for the data frame based on the block memory structure and handed over to the analyzer. At this time, the analyzer usually returns these block memories after analysis. This method is to return the block memory corresponding to the data frame in time, but this will bring some problems. (a) There is no ability to trace historical data frames, and (b) because these block memories are not effectively used, a lot of data generated during the analysis process must be reallocated and copied, and this value is a reference to the original value, such as file streams. Many file streams are directly derived from the byte content in the original frame. Therefore, this method undoubtedly increases the burden on the system and fails to achieve resource tracing and reuse. Summary of the invention

[0004] In view of the problems existing in the prior art, a data frame window buffering device and method are provided, which control the validity of data through a window moving mechanism.

[0005] The first aspect of the present invention provides a window buffer device for data frames, which is applied to network traffic analysis, and includes: Window, used to manage the life cycle of data frames. Data frames within the window are marked as valid for analysis reference; Bidirectional data transmission queue, including sending queue and recycling queue; Among them, the production end collects data frames and stores them in an available container, allocates a window value to the container, and transmits them to the consumer end through a sending queue; the consumer end receives the data frames and hands them over to the window management. When the window value corresponding to the container is not within the window, the container occupied by the data frame is released and transmitted to the production end through a recycling queue.

[0006] As a preferred solution, the window value allocated by the production end to the container is atomically incremented.

[0007] As a preferred solution, the window has an interval size determined by a start value and an end value, and has a first-in-first-out principle. The data frames in the container corresponding to the window value that leaves the window interval become invalid.

[0008] As a preferred solution, in the window buffer device, the input rate of the data frame is controlled by controlling the interval size of the window.

[0009] As a preferred solution, the consumer end temporarily stores the data frames in the window.

[0010] As a preferred solution, the container stores multiple data frames, that is, multiple data frames share one window value.

[0011] As a preferred solution, before releasing the container occupied by the data frame, the consumer notifies the referenced data frame that it is about to expire, and the referenced data frame is processed according to the policy and released after the policy processing is completed.

[0012] A second aspect of the present invention provides a window caching method for a data frame, comprising: The production end collects data frames from the network device and stores them in a usable container, assigns a window value to the container, and transmits it to the consumer end; The consumer side temporarily stores the data frames in the container and establishes a window. The window controls the life cycle of the data frame. When the window value is within the window interval, the corresponding data frame is valid; when the window value leaves the window interval, the corresponding data frame becomes invalid and the consumer side releases the corresponding container.

[0013] As a preferred solution, the interval size of the window is determined by a set start value and an end value, and the input rate of the data frame is controlled by controlling the interval size of the window.

[0014] As a preferred solution, before releasing the container occupied by the data frame, the consumer notifies the referenced data frame that it is about to expire, and the referenced data frame is processed according to the policy and released after the policy processing is completed.

[0015] Compared with the prior art, the beneficial effects of adopting the above technical solution are: 1. Make full use of the memory of the data frame and reduce the frequency of data copying of the byte content of the data frame to a minimum, which can ultimately reduce system overhead and improve overall system performance; 2. It can realize the tracing and application of certain historical data and solve certain traceability businesses; 3. Control the data frame input rate of the production end. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 FIG. 4 is a schematic diagram of a window in an embodiment of the present invention.

[0017] Figure 2 Schematic diagram of a bidirectional data transmission queue in an embodiment of the present invention.

[0018] Figure 3 The figure is a flow chart of window buffering of data frames in one embodiment of the present invention. DETAILED DESCRIPTION

[0019] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0020] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0021] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0022] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0023] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The embodiment of the present invention proposes a window cache device for data frames, which is mainly used in the field of network traffic analysis and is mainly implemented through windows and bidirectional data transmission queues. Among them, the window is used to manage the life cycle of the data frame, and the data frame in the window is marked as valid for analysis reference; the bidirectional data transmission queue includes a sending queue and a recycling queue. The production end collects data frames and stores them in a usable container, and after allocating a window value to the container, it is transmitted to the consumer end through the sending queue; the consumer end receives the data frame and hands it over to the window for management. When the window value corresponding to the container is not within the window, the container occupied by the data frame is released and transmitted to the production end through the recycling queue.

[0025] In this embodiment, the data frame is an original data frame obtained from a network device, such as a data frame collected from a network card, with the frame as the basic unit.

[0026] For details, please refer to Figure 1 , the window consists of a period ( Figure 2 The interval is composed of the start and end values ​​of the window. The data in the window is a referenced data frame. When the window moves, the data that leaves the window becomes invalid. Based on the window mechanism, data frames within the window range can be safely referenced, and their contents can be directly analyzed and used. For example, pictures, document extraction, field extraction analysis, data frame storage and other scenarios can be directly referenced without re-applying for memory and copying, which can improve system resource reuse and save system overhead (including CPU and memory resources). At the same time, based on the characteristics of the window itself, the data frames in the window can be regarded as caches. As long as the data frames in the window are not released, the life cycle of these data frames will be extended. Based on the extended life cycle, historical data within the cycle can be traced and applied. For example, a session detects relevant information only in the Nth frame, and it is necessary to trace and analyze the M frame messages before the Nth frame again.

[0027] Please refer to Figure 2 The bidirectional data transmission queue has a bidirectional transmission characteristic, which consists of a sending queue A and a recycling queue B. The production end writes the data frame into the sending queue A through a container, and the consumer end takes it out and writes it into the window. When the data is no longer needed, the consumer end writes the container storing the data frame into the recycling queue B. After the production end obtains the container, it rewrites the subsequent data frame. In one embodiment, the window value allocated by the production end to the container is atomically incremented. Each container can store multiple data frames, that is, multiple data frames share a window value. In this way, the window capacity can be expanded.

[0028] It should be noted that, generally, the data frames in the window will be used by the referencer. Therefore, before releasing the container occupied by the corresponding data frame, the consumer notifies the referencer that the referenced data frame is about to expire. The referencer will process it according to the policy and release it after the policy processing is completed.

[0029] By adjusting the window size, you can control the input rate of data frames. Specifically, if the consumer wants to reduce the data input rate, you can increase the window size to control the number of returned containers (that is, the number of block memory objects). If the producer cannot get an available container, it will continue to wait until an available container is available. The longer the waiting time, the slower the data production and the slower the data frame input rate. Reduce the production rate When available system memory and other resources are insufficient, control the peak value of traffic input and reduce resource usage. If it is not a continuous traffic input scenario (such as local disk data), you can ultimately control the speed of data production and ensure that data is not lost.

[0030] Compared with the traditional solution of allocating memory for data frames and returning these blocks of memory after analysis, the present invention uses a bidirectional data transmission queue to enable the data frames transmitted by the queue to be reused, and the data frames after the life cycle is extended by the data frame window can be safely used during their life cycle. This is of great significance to business support, while the prior art is to copy on demand during a single use, that is, for the same business, the number of copies is significantly reduced, because based on the present invention, within the life cycle of the data frame, many analysis businesses can be completed normally, so each data reference of the analysis business can reduce one data copy, such as field output, field detection and other businesses. In the prior art, if the present device is missing, for multiple data frames in the session, the frames before the current data frame have long been invalid, and the detection and other businesses have just begun to use the content of the previous data frame, so the prior art is bound to have a large number of copies, which will eventually consume some system resources.

[0031] Please refer to Figure 3 This embodiment also proposes a window caching method for a data frame, including: Step 1: The production end collects data frames from the network device and stores them in an available container, assigns a window value to the container, and transmits it to the consumer end.

[0032] In this embodiment, the window value allocated by the production end to the container is atomically incremented, and multiple data frames can share one window value to increase the window capacity. The window value is used to subsequently determine whether the corresponding data frame is within the window.

[0033] Step 2: The consumer temporarily stores the data frames in the container and establishes a window. The window controls the life cycle of the data frames. When the window value is within the window interval, the corresponding data frame is valid; when the window value leaves the window interval, the corresponding data frame becomes invalid and the consumer releases the corresponding container.

[0034] In actual applications, the window interval size is determined by the set start value and end value, and can be adjusted according to demand. Generally, when a window is established, the initial state window size is 0. As the consumer continues to receive data frames, the window slowly increases until it reaches the set window size.

[0035] Data frames that exceed the window range are marked as invalid frames. That is, when a new container containing data frames is inserted, according to the first-in-first-out principle of the window, the container that first enters the window may be marked as invalid, and the corresponding data frame is also marked as invalid. At this time, there is a notification mechanism to inform the reference that this data frame is about to expire. The reference must have its own processing strategy after receiving this notification. After the message notification is completed and the references of each frame have completed the processing according to the strategy, the container should be returned. At this time, through the two-way transmission mechanism of the data queue, the consumer end transmits the container back through the return queue, and finally the production end rewrites new data for reuse.

[0036] The embodiment of the present invention further proposes a system, including a memory and a processor, wherein the memory stores a computer program corresponding to the aforementioned window caching method for a data frame that can be loaded and executed by the processor.

[0037] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer readable medium, and the computer program includes a program code for executing the method shown in the flowchart.

[0038] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0039] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0040] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0041] As another aspect, the present application further provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the window caching method of the data frame described in the above embodiment.

[0042] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the window caching method of the data frame described in the above embodiment.

[0043] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0044] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software, or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0045] For those skilled in the art, the specific meanings of the above terms in the present invention can be understood in specific situations; the drawings in the embodiments are used to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0046] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limitations on the present application. Ordinary technicians in this field can change, modify, replace and modify the above embodiments within the scope of the present application.

Claims

1. A window buffer device for a data frame, characterized in that: Applied to network traffic analysis, including: Window, used to manage the life cycle of data frames. Data frames within the window are marked as valid for analysis reference; Bidirectional data transmission queue, including sending queue and recycling queue; Among them, the production end collects data frames and stores them in an available container, allocates a window value to the container, and transmits them to the consumer end through a sending queue; the consumer end receives the data frames and hands them over to the window management. When the window value corresponding to the container is not within the window, the container occupied by the data frame is released and transmitted to the production end through a recycling queue.

2. The window buffer device for data frames according to claim 1, characterized in that: The window value allocated by the producer to the container is atomically incremented.

3. The window buffer device for data frames according to claim 1, characterized in that: The window has an interval size determined by a start value and an end value, and has a first-in-first-out principle. The data frames in the container corresponding to the window value that leaves the window interval become invalid.

4. The window buffer device for data frames according to claim 1, characterized in that: In the window buffer device, the input rate of the data frame is controlled by controlling the interval size of the window.

5. The window buffer device for data frames according to claim 1, characterized in that: The consumer end temporarily stores data frames in the window.

6. The window buffer device for data frames according to claim 1, characterized in that: The container stores multiple data frames, that is, the multiple data frames share one window value.

7. The window buffer device for data frames according to claim 1, characterized in that: Before releasing the container occupied by the data frame, the consumer notifies the referenced data frame that it is about to become invalid, and the referenced data frame is processed according to the policy, and then released after the policy processing is completed.

8. A window caching method for a data frame, characterized in that: include: The production end collects data frames from the network device and stores them in a usable container, assigns a window value to the container, and transmits it to the consumer end; The consumer side temporarily stores the data frames in the container and establishes a window. The window controls the life cycle of the data frame. When the window value is within the window interval, the corresponding data frame is valid; when the window value leaves the window interval, the corresponding data frame becomes invalid and the consumer side releases the corresponding container.

9. The window caching method for data frames according to claim 8, characterized in that: The interval size of the window is determined by the set start value and end value. By controlling the interval size of the window, the input rate of the data frame is controlled.

10. The window buffering method of data frame according to claim 8, characterized in that: Before releasing the container occupied by the data frame, the consumer notifies the referenced data frame that it is about to become invalid, and the referenced data frame is processed according to the policy, and then released after the policy processing is completed.

Images