DPI message storage management system and method

Abstract

The invention relates to a DPI message storage management system and method. The system mainly comprises a DPI device, a full-flow retention device and a storage device cluster, wherein the DPI device is used for collecting and analyzing messages, outputting session field information required by the full-flow retention device, outputting message index information and outputting original messages added with locators; and the full-flow retention equipment is used for receiving session field information and storing the session field information in the database, receiving message index information and storing the message index information in the database, receiving an original message added with a locator and transferring the original message to the storage equipment cluster so as to provide a message searching function. According to the invention, a part of function requirements of the full-flow retention device can be transferred to the DPI device for realization, so that the full-flow retention device releases the part of performance pressure, and the storage capability of single-machine message storage of the full-flow retention device can be improved or the storage cost of each Gbps message can be reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of message storage management, and in particular to a DPI message storage management system and method. Background Art

[0002] With the rapid development of the Internet, the amount of network data has shown explosive growth. The analysis and retention of full-flow data is becoming increasingly important for network security analysis, troubleshooting, traffic behavior analysis and other fields. For the analysis of full-flow messages, Deep Packet Inspection (DPI) devices are usually used (DPI devices are network devices with business data flow identification and control capabilities) to analyze and store the corresponding XDR call records (DR is the abbreviation of Detailed Record. XDR call records are session-level detailed records of the signaling process and business transmission process generated after processing the full amount of Internet data, including all the user's Internet access information). For the retention of full-flow messages, full-flow retention devices store full-flow messages for post-analysis, security audits, troubleshooting, compliance checks and business insights.

[0003] The main working process of the full-flow retention device is as follows: the diversion device diverts the data packets that need to be retained to the full-flow retention device. The full-flow retention device analyzes and processes the packets and stores them in real time. When users search for packets, they log in to the corresponding full-flow retention device as needed to extract the packets.

[0004] In order to ensure the efficiency of searching for messages, the full-flow retention device needs to perform protocol identification, application identification and business identification on the collected messages, extract the five-tuple information in the message and store the above message information. At present, the maximum capacity of the mainstream full-flow storage device to analyze and store messages is 40Gbps. With the explosive growth of network data volume, how to improve the storage capacity of the full-flow retention device or reduce the storage cost of each Gbps message is a problem we need to consider.

[0005] In view of this, how to overcome the defects of the existing technology and how to meet the needs of improving the storage capacity of single-machine message storage of full-flow retention equipment or reducing the storage cost of each Gbps message has become an important technical problem that needs to be urgently solved in the industry. Summary of the invention

[0006] In view of the defects or improvement requirements in the prior art: how to improve the storage capacity of a single-machine message storage of a full-flow retention device or reduce the storage cost of each Gbps message. The present invention provides a DPI message storage management system and method, which transfers part of the functional requirements of a full-flow retention device to a DPI device for implementation, so that the full-flow retention device releases this part of the performance pressure, thereby improving the storage capacity of a single-machine message storage of a full-flow retention device or reducing the storage cost of each Gbps message.

[0007] The present invention adopts the following technical solution:

[0008] In a first aspect, the present invention provides a DPI message storage management system, including a DPI device, a full-flow retention device, and a storage device cluster, wherein:

[0009] The DPI device is used to collect and analyze messages, output session field information required by the full flow retention device, output message index information, and output the original message with the locator added;

[0010] The full-flow retention device is used to receive session field information and store it in a database, receive message index information and store it in a database, receive original messages with locators added and transfer them to a storage device cluster to provide a message search function.

[0011] In some embodiments, the DPI device includes a field analysis module, wherein:

[0012] The field analysis module is used to analyze the fields that can be directly referenced from the DPI session field and the fields that need to be obtained through different field operations from the session field template provided by the full-flow retention device, and output the statistical session field information to the full-flow retention main module of the full-flow retention device;

[0013] The field analysis module is also used to generate a unique ID for the session field information according to the session. The unique ID is generated by the time of the first packet of the session+device ID+internal sequence number.

[0014] In some embodiments, the DPI device includes a storage message processing module, wherein:

[0015] The stored message processing module is used to obtain a message storage strategy from a full-flow retention device to determine the flow that needs to be stored and the storage method, wherein the storage method includes one or more of full-flow storage, truncated storage, and selective truncation storage by flow; wherein the full-flow storage includes: storing all messages without truncation; the truncation storage includes: performing stage processing on the message and storing only the preset bytes of the head; the selective truncation storage by flow includes: storing all the first several messages of each flow, truncating the subsequent messages, and retaining only the preset bytes of the head;

[0016] The stored message processing module is also used to add a unique ID, a start character and a stop character to the message and then output it to the full flow retention main module of the full flow retention device;

[0017] The stored message processing module is also used to generate message index information and send the message index information to the full-flow retention main module of the full-flow retention device, wherein the message index information includes one or more of an ID, a sequence number, whether to truncate, a truncation position, and a storage position, wherein the ID is a unique ID obtained from the field analysis module; the sequence number is the sequence number of the message in the session flow; whether to truncate includes: judging whether the message needs to be truncated according to the message storage policy; the truncation position includes: judging the truncation position of the message according to the message storage policy; the storage position is left blank and filled in by the full-flow retention main module of the full-flow retention device.

[0018] In some embodiments, the full flow retention device includes a full flow retention main module, wherein:

[0019] The full-flow retention main module is used to generate a message storage strategy according to user needs and synchronize it in real time to the storage message processing module of the DPI device; generate a session field template according to user needs and synchronize it in real time to the field analysis module of the DPI device.

[0020] The full flow retention main module is also used to receive the session field information sent by the field analysis module from the field analysis module of the DPI device and store it in the local database;

[0021] The full-flow retention main module is also used to receive messages from the storage message analysis module of the DPI device and transfer them to the storage device cluster according to the storage server write policy;

[0022] The full flow retention main module is also used to receive message index information and backfill the storage location field in the message index information, and store it in the local database after the backfilling is completed.

[0023] In some embodiments, the full-flow retention main module is further used to receive messages from the storage message analysis module of the DPI device and transfer them to the storage device cluster according to the storage server write policy, specifically including:

[0024] The received message is first stored in the local memory. When the message cache reaches the preset amount, it is written to the disk file of the storage device cluster according to the storage server write policy.

[0025] The file overwrite technology is used to overwrite the message. Before overwriting, the number of files for each storage device in the storage device cluster is planned in advance; after each storage device writes and / or overwrites the preset number of files, it switches to the next storage device.

[0026] In some embodiments, the full flow retention device includes a storage device monitoring module, wherein:

[0027] The storage device monitoring module is used to monitor one or more of the disk size, disk availability, abnormal file writing, disk health status, file name, number of files and file type information of each storage device in real time; if the disk dynamically increases or decreases, the maximum available space and writable amount of the storage device are dynamically adjusted, and the storage server write policy is updated, and the information is passed to the alarm module; if the disk cannot be written and / or a disk health alarm occurs, the node is taken offline, the storage server write policy is updated, and the information is passed to the alarm module; if unplanned file names and / or file types appear, the unplanned file names and / or file types are passed to the alarm module; if the number of files decreases or increases, the information is passed to the alarm module.

[0028] In some embodiments, the full flow retention device includes a storage device management module, wherein:

[0029] The storage device management module is used to generate a storage server write strategy based on one or more of the number of storage devices, the maximum available disk space and the maximum number of writable files of each storage device, the size of each message file in the full flow retention main module, the device binding time, the device status, and the maximum disk cycle write times in the storage device monitoring module, and write messages to the storage devices one by one according to the storage server write strategy. After each storage device has written the specified number of messages, it starts writing to the next storage device. When all storage devices have been written, it loops to the first storage device to write again;

[0030] When an individual storage device goes offline abnormally, rearrange the writing order of the storage device;

[0031] When the offline storage device comes online again, its write order is rearranged according to its offline time;

[0032] If the offline time of a storage device is within the preset range, and the messages it stores are later than those of some other storage devices, then the overwriting of this storage device will begin after the overwriting of these other storage devices is completed; before the storage device goes offline for maintenance, if there are messages to be stored, a label will be added to the corresponding storage device, so that when data is subsequently stored, tasks with a corresponding complete reading time period that is longer than its historical online and offline time period will be given priority in the corresponding storage device.

[0033] In some embodiments, each storage device calculates the disk health according to the maximum disk cycle write times and the actual message write times provided by the hard disk manufacturer, and maintains the storage device according to the disk health; for storage devices with low health, the maximum number of writable files is actively reduced, and the files exceeding the number are transferred to other storage devices, which is achieved by overwriting the earliest message; wherein, the calculation of the disk health includes:

[0034] Get the maximum number of cyclic writes MaxWrites, the actual storage space ActualCapacity, the size of each message file PacketSize, and the maximum number of message writes TotalMaxWrites;

[0035] TotalMaxWrites=MaxWrites*ActualCapacity / PacketSize;

[0036] Assume that the actual number of writes that have occurred is CurrentWrites, then the percentage of disk health HealthPercentage is: CurrentWrites / TotalMaxWrites*100%.

[0037] In some embodiments, the full-flow retention device includes a message search module and an alarm module, wherein:

[0038] The message search module is used to provide a query page and a query interface for a third party to call or directly access; the message search module retrieves the unique ID, storage location and message timestamp of the corresponding message in the index table of the database according to the message search condition selected by the user, and then searches in the corresponding storage device and message file to match the message. If the message is in multiple storage devices or multiple message files, it is matched in a multi-threaded parallel manner; after finding all the messages, check again whether they meet the user's requirements, remove the messages that do not meet the requirements, and strip the unique ID, start character, and end character, and finally merge all the messages before outputting them to the user;

[0039] The alarm module is used to notify the user of the alarm information through one or more of a pop-up window, an email, and a sound after receiving the alarm information.

[0040] In a second aspect, the present invention further provides a DPI message storage management method, which is applied to the DPI message storage management system described in the first aspect, comprising:

[0041] The DPI device collects and analyzes the packets, outputs the session field information required by the full-flow retention device, outputs the packet index information, and outputs the original packets with the locator added;

[0042] The full-flow retention device receives session field information and stores it in the database, receives message index information and stores it in the database, receives original messages with locators added and transfers them to the storage device cluster to provide message search function.

[0043] Compared with the prior art, the present invention provides a DPI message storage management system and method, which has the beneficial effect of transferring part of the functional requirements of the full-flow retention device to the DPI device for implementation, so that the full-flow retention device releases this part of the performance pressure, thereby improving the storage capacity of the single-machine message storage of the full-flow retention device or reducing the storage cost of each Gbps message. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 A schematic diagram of a module of a DPI message storage management system provided in Example 1 of the present invention;

[0046] Figure 2 A schematic diagram of a sample session statistics field output by the field analysis module provided in Embodiment 1 of the present invention;

[0047] Figure 3 A schematic diagram of a sample message output by the storage message processing module provided in Embodiment 1 of the present invention to the full-flow retention main module;

[0048] Figure 4 A schematic diagram of a sample of message index information generated by a message processing module for storing messages provided in Embodiment 1 of the present invention;

[0049] Figure 5 A schematic diagram of an example of a storage location field in a backfill message index information of a full flow retention main module provided in Embodiment 1 of the present invention;

[0050] Figure 6 An example diagram of a storage server write strategy provided in Example 1 of the present invention;

[0051] Figure 7 A flowchart of a DPI message storage management method provided in Example 1 of the present invention;

[0052] Figure 8 A schematic diagram of the structure of a DPI message storage management device provided in Example 2 of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but do not limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, without departing from the concept of the present invention, several variations and improvements can also be made. These all belong to the protection scope of the present invention. It should be noted that, if there is no conflict, the various features in the embodiments of the present invention can be combined with each other, all within the protection scope of this application. In addition, although the functional module division may be performed in the device schematic diagram, and the logical order may be shown in the flow chart, in some cases, it may be different from the module division in the device, or the steps shown or described in the order in the flow chart may be performed.

[0054] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in this specification and in the specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. In addition, the technical features involved in each embodiment of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0055] Unless the context requires otherwise, throughout the specification and claims, the term "including" is to be interpreted as open inclusion, that is, "including, but not limited to". In the description of the specification, the terms "one embodiment", "some embodiments", "exemplary embodiments", "examples", "specific examples" or "some examples" and the like are intended to indicate that specific features, structures, materials or characteristics associated with the embodiment or example are included in at least one embodiment or example of the present disclosure. The schematic representation of the above terms does not necessarily refer to the same embodiment or example. In addition, the specific features, structures, materials or characteristics may be included in any one or more embodiments or examples in any appropriate manner, that is, although they may be carried in the embodiments or examples of the above terms due to reasons such as the order and position of appearance, it is not limited to that they can be carried in combination by one embodiment or example.

[0056] The main working process of the full-flow retention device is as follows: the diversion device diverts the data packets that need to be retained to the full-flow retention device. The full-flow retention device analyzes and processes the packets and stores them in real time. When users search for packets, they log in to the corresponding full-flow retention device as needed to extract the packets.

[0057] In order to ensure the efficiency of searching for messages, the full-flow retention device needs to perform protocol identification, application identification, and business identification on the collected messages, extract the five-tuple information in the message and store the above message information. This part of the capability can be 100% covered by the DPI device. At present, the maximum capacity of the mainstream full-flow storage device to analyze and store messages is 40Gbps. If this part of the functional requirements is transferred to the DPI device for implementation, the full-flow retention device releases this part of the performance pressure, which can improve the storage capacity of the full-flow retention device for single-machine message storage or reduce the storage cost of each Gbps message.

[0058] Since DPI devices have excellent message collection, analysis, and processing capabilities, a single machine can support up to the Tbps level. If the message analysis capabilities of DPI devices are reused, the full-flow retention device is only responsible for message storage and search. This will greatly reduce the performance loss of the full-flow retention device and can fully save costs.

[0059] In the embodiment of the present invention, the DPI device collects and analyzes the message, outputs the session information required by the full flow retention device, outputs the message index information, and outputs the original message with the locator added. The full flow retention device receives the session information and stores it in the database, receives the message index information and stores it in the database, receives the message that needs to be stored and transfers it to the storage device, and provides the message search function.

[0060] In order to make the purpose, technical scheme and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0061] Embodiment 1:

[0062] like Figure 1As shown, an embodiment of the present invention provides a DPI message storage management system, which includes a DPI device, a full-flow retention device, and a storage device cluster, wherein the DPI device includes a field analysis module and a storage message processing module in addition to the original functions of the DPI device; the full-flow retention device includes a full-flow retention main module, a storage device monitoring module, a storage device management module, and an alarm module in addition to the local database and the original functions of the full-flow retention device; the storage device cluster includes a number of storage devices, such as storage device 1, storage device 2, storage device N, etc. Based on the above architecture, the DPI device is used to collect and analyze messages, output the session field information required by the full-flow retention device, output the message index information, and output the original message with the locator added; the full-flow retention device is used to receive the session field information and store it in the database, receive the message index information and store it in the database, receive the original message with the locator added and transfer it to the storage device cluster to provide a message search function.

[0063] Through the above settings, part of the functional requirements of the full-flow retention device are transferred to the DPI device for implementation, so that the full-flow retention device releases this part of the performance pressure, thereby improving the storage capacity of the single-machine message storage of the full-flow retention device or reducing the storage cost of each Gbps message. The following describes the functions of each module in the system in more detail to illustrate the solution of this embodiment in more detail.

[0064] In some embodiments, the DPI device includes a field analysis module, wherein: the field analysis module is used to analyze the fields that can be directly referenced from the DPI session field and the fields that need to be obtained through different field operations from the session field template provided by the full-flow retention main module of the full-flow retention device, and output the statistical session field information to the full-flow retention main module of the full-flow retention device.

[0065] In addition, in order to facilitate message search, the field analysis module is also used to generate a unique ID for the session field information by session. The unique ID is used to associate the session table, index table, and original message. The unique ID is generated by the time of the first packet of the session + device ID + internal sequence number.

[0066] Example of session statistics fields in output Figure 2 shown.

[0067] In some embodiments, the DPI device includes a stored message processing module, wherein: the stored message processing module is used to obtain a message storage strategy from a full-flow retention main module of a full-flow retention device to determine the flow that needs to be stored and the storage method, and the storage method includes one or more of full-flow storage, truncated storage, and selective truncation storage by flow, etc.; wherein, the full-flow storage includes: storing all messages without truncation; the truncated storage includes: performing stage processing on the message, storing only preset bytes of the header, for example, storing only 128 / 256 / 512 bytes of the header, etc.; the selective truncation storage by flow includes: storing all the first several messages of each flow, truncating subsequent messages, and retaining only preset bytes of the header, for example, retaining only 128 / 256 / 512 bytes of the header, etc.

[0068] The stored message processing module is also used to add a unique ID, a start character and a stop character to the message and then output it to the full flow retention main module of the full flow retention device. Figure 3 shown.

[0069] All messages in each flow use the same unique ID.

[0070] The stored message processing module is also used to generate message index information and send the message index information to the full-flow retention main module of the full-flow retention device, wherein the message index information includes one or more of ID, sequence number, whether to truncate, truncation position and storage position, wherein the ID is a unique ID obtained from the field analysis module; the sequence number is the sequence number of the message in the session flow, and the first message sequence number is 1; whether to truncate includes: judging whether the message needs to be truncated according to the message storage policy, 0 means no truncation, and 1 means truncation; the truncation position includes: judging the truncation position of the message according to the message storage policy, that is, marking how many bytes of the header of the message are retained (128 / 256 / 512 bytes); the storage position is left blank and filled in by the full-flow retention main module of the full-flow retention device. For reference, see Example Figure 4 shown.

[0071] In some embodiments, the full-flow retention device includes a full-flow retention main module, wherein: the full-flow retention main module is used to generate a message storage strategy according to user needs and synchronize it in real time to the storage message processing module of the DPI device; generate a session field template according to user needs and synchronize it in real time to the field analysis module of the DPI device.

[0072] The full-flow retention main module is also used to receive session field information sent by the field analysis module from the field analysis module of the DPI device and store it in the local database.

[0073] The full-flow retention main module is also used to receive messages from the storage message analysis module of the DPI device and transfer them to the storage device cluster according to the storage server write strategy; the specific transfer method includes:

[0074] (1) In order to reduce disk IO and improve writing speed, the received messages are first stored in the local memory. When the messages are cached to a preset amount (such as 1GB), they are written to the disk files of the storage device cluster according to the storage server write strategy. In order to reduce the number of stored files, the total write size of each file is set to 10GB (which can be customized as needed).

[0075] (2) Since large-scale file deletion will damage the disk, this module uses file overwriting technology to overwrite the message instead of directly deleting it. Before overwriting, it is necessary to plan the number of files for each storage device in the storage device cluster in advance and manage the files accurately.

[0076] (3) Usually, the actual storage space of a 1TB hard disk is 931GB, which can write 93 10GB message files in total. Since the performance will drop sharply when the disk is about to be full, a part of the disk space must be reserved. Theoretically, 93 files can be written. In order to ensure the disk performance, the actual maximum number of files written is 70. That is, each storage device needs to switch to the next storage device after writing / overwriting a preset number (for example, 70) files.

[0077] (4) The written message file is composed of [file number_time stamp of the first message to be stored in the written file_time stamp of the last message to be stored in the written file_write packet thread number.suffix].

[0078] Example: 00001_1792321329_1792324329_01.gncap

[0079] Note: A 1TB hard disk is planned to write 70 files. The value range of the first field is 00001-00070; the second field is the timestamp of the first original message in the written message; the third field is the timestamp of the last original message in the written message; the fourth field is the number of the packet writing thread, which is generally 4 or 8; gncap indicates that the message format is a unique message format of the company.

[0080] The full flow retention main module is also used to receive message index information and backfill the storage location field in the message index information, and store it in the local database after the backfilling is completed.

[0081] The storage location field information includes: storage server number_file number.

[0082] Example: Reference Figure 5As shown, it means that the message is stored in the 00001 message of the storage server 01.

[0083] In some embodiments, the full-flow retention device includes a storage device monitoring module, which supports binding storage devices and monitors the survival and availability of storage devices in real time. When a storage device is abnormal (such as memory usage continuously exceeding a threshold, interface DOWN, device inaccessibility, and other server-level problems), the node is offline, the storage server write strategy is updated, and the information is passed to the alarm module.

[0084] The storage device monitoring module is used to monitor one or more of the information such as disk size, disk availability, abnormal file writing, disk health status, file name, number of files and file type of each storage device in real time; if the disk increases or decreases dynamically, the maximum available space and writable amount of the storage device are dynamically adjusted, and the storage server write strategy is updated, and the information is passed to the alarm module; if the disk cannot be written, the node is offline, the storage server write strategy is updated, and the information is passed to the alarm module; if a disk health alarm occurs, the node is offline, the storage server write strategy is updated, and the information is passed to the alarm module, and the disk health alarm can be read through the baseboard management controller (baseboard management controller, abbreviated as: BMC) program interface of the device; if an unplanned file name and / or file type appears, the unplanned file name and / or file type is passed to the alarm module; if the number of files decreases or increases (such as: a storage device that has been polled for a round should have 70 files, but there are only 50 files), the information is passed to the alarm module.

[0085] In some embodiments, the full-flow retention device includes a storage device management module, wherein: the storage device management module is used to generate a storage server write strategy based on one or more of the number of storage devices, the maximum available disk space and the maximum number of writable files of each storage device, the size of each message file in the full-flow retention main module, the device binding time in the storage device monitoring module, the device status, the maximum number of disk cycle writes consulted from the disk manufacturer, and the like, and write messages to the storage devices one by one according to the storage server write strategy. After each storage device has written the specified number of messages, it starts to write to the next storage device. When all storage devices are written, they cycle to the first storage device to write again, and so on.

[0086] Storage Server Write Policy Example Reference Figure 6 As shown (storage server is storage device).

[0087] Note: Taking a 1TB hard disk with 931GB available as an example, writing 70 10GB files and overwriting 23 10GB files completes one disk write.

[0088] When an individual storage device goes offline abnormally, the writing order of the storage device is rearranged to ensure that the storage message always writes data stably.

[0089] When the offline storage device comes online again, its writing order is rearranged according to its offline time; if all storage devices have completed a cycle, that is, the message of the offline storage device is earlier than the messages of all storage devices, then the storage device is set as the next storage device to be written.

[0090] If the offline time of the storage device is within the preset range (the preset range is an empirical value of a short time, that is, if the storage device is offline for a short time), and the messages stored in it are later than those of some other storage devices, then the storage device will be rewritten after the overwriting of these other storage devices is completed; further, before the storage device enters offline maintenance, if there are messages to be stored, the corresponding storage device is labeled, so that when the data is subsequently stored, the tasks whose corresponding complete reading time period is longer than its historical offline time period will be arranged in priority to the corresponding storage device. For example, if storage device 1 is offline, it cannot be sent for maintenance immediately at this time. There will be a period of continuous work before it is sent for maintenance. During this time, when storing messages, if the processing time of some messages is longer than the offline time period of storage device 1, then these messages will be arranged to be stored in storage device 1 first; on the contrary, if the processing time of some messages is less than the offline time period of storage device 1, then these messages will be arranged to be stored in other storage devices except storage device 1 first.

[0091] In some embodiments, each storage device calculates the disk health based on the maximum number of disk cycle writes provided by the hard disk manufacturer and the actual number of message writes, and maintains the storage device based on the disk health; for example, if the maximum number of cycle writes provided by the manufacturer of a 1TB hard disk is 500 times, and the actual storage space of a 1TB hard disk is 931GB, then the maximum number of message writes (calculated based on 10GB for each message file) is: 500*931GB / 10GB=46550 times. For storage devices with low health, the maximum number of writable files is actively reduced, and the files exceeding the number are transferred to other storage devices, which is achieved by overwriting the earliest message.

[0092] The calculation method of disk health is as follows. First, several parameters need to be clarified: the maximum number of cycle writes MaxWrites: provided by the hard disk manufacturer, 500 times for a 1TB hard disk; the actual storage space ActualCapacity: 931GB for a 1TB hard disk; the size of each packet file PacketSize: assuming that each packet file is 10GB; the maximum number of packet writes TotalMaxWrites: MaxWrites*ActualCapacity / PacketSize.

[0093] Based on these parameters, we can calculate the maximum number of times a message can be written:

[0094] TotalMaxWrites=MaxWrites*ActualCapacity / PacketSize;

[0095] TotalMaxWrites=500*931GB / 10GB=46550 times;

[0096] Now that we have the maximum number of message writes, we can set the health level based on it. Let the actual number of writes that have occurred be CurrentWrites, then the percentage of disk health HealthPercentage is: CurrentWrites / TotalMaxWrites*100%.

[0097] Then the levels are divided according to the health percentage:

[0098] Normal: HealthPercentage < 70%;

[0099] Good: 70% ≤ HealthPercentage < 80%;

[0100] General: 80% ≤ HealthPercentage < 90%;

[0101] Poor: 90% ≤ HealthPercentage < 100%;

[0102] Overdue: HealthPercentage ≥ 100%;

[0103] When the disk health reaches poor or expired status, the information will be sent to the alarm module once a day.

[0104] In some embodiments, the full-flow retention device includes a message search module, which can directly use the original functions of the full-flow retention device, so it is not necessary to set up a separate module. The message search module is used to provide a query page and a query interface for third-party calls or direct access; the message search module retrieves the unique ID, storage location and message timestamp of the corresponding message in the index table of the database according to the message search conditions selected by the user, and then searches in the corresponding storage device and message file to match the message. If the message is in multiple storage devices or multiple message files, it is matched in a multi-threaded parallel manner; after finding all the messages, check again whether they meet the user's requirements, remove the messages that do not meet the requirements, and strip the unique ID, start character, and end character, and finally merge all the messages before outputting them to the user.

[0105] In some embodiments, the full-flow retention device includes an alarm module, wherein: the alarm module is used to inform the user of the alarm information through one or more of pop-up windows, emails, sounds, etc. after receiving the alarm information.

[0106] Based on the above system, an embodiment of the present invention further provides a DPI message storage management method, which is applied to the above DPI message storage management system, referring to Figure 7 As shown, the DPI message storage management method includes the following steps:

[0107] Step 100: The DPI device collects and analyzes the message, outputs the session field information required by the full-flow retention device, outputs the message index information, and outputs the original message with the locator added.

[0108] Step 200: The full-flow retention device receives the session field information and stores it in the database, receives the message index information and stores it in the database, receives the original message with the locator added and transfers it to the storage device cluster to provide a message search function.

[0109] Specifically, the expansion process corresponding to this method has been described in the detailed functions of each module of the above system, and will not be repeated here.

[0110] To sum up, the embodiments of the present invention provide a DPI message storage management system and method, which has the beneficial effect of: transferring part of the functional requirements of the full-flow retention device to the DPI device for implementation, so that the full-flow retention device releases this part of the performance pressure, thereby improving the storage capacity of the single-machine message storage of the full-flow retention device or reducing the storage cost of each Gbps message.

[0111] Embodiment 2:

[0112] Based on the DPI message storage management system and method provided in the above embodiment 1, the present invention also provides a DPI message storage management device that can be used to implement the above system and method, such as Figure 8 , which is a schematic diagram of the device architecture of an embodiment of the present invention. The DPI message storage management device of this embodiment includes one or more processors 21 and a memory 22. Figure 8 A processor 21 is taken as an example.

[0113] The processor 21 and the memory 22 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0114] The memory 22, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer executable programs and modules, such as the DPI message storage management method in Example 1. The processor 21 executes various functional applications and data processing of the DPI message storage management device by running the non-volatile software programs, instructions and modules stored in the memory 22, that is, implements the DPI message storage management method in Example 1.

[0115] The memory 22 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 may optionally include a memory remotely arranged relative to the processor 21, and these remote memories may be connected to the processor 21 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0116] The program instructions / modules are stored in the memory 22, and when executed by one or more processors 21, the DPI message storage management method in the above embodiment 1 is executed, for example, the above described Figure 7 The steps shown.

[0117] The above-mentioned product can execute the method provided in the embodiment of the present application, and has the functional modules and beneficial effects corresponding to the execution method. For technical details not fully described in this embodiment, please refer to the method provided in the embodiment of the present application.

[0118] It should be noted that the device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0119] Through the description of the above implementation methods, ordinary technicians in this field can clearly understand that each implementation method can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Ordinary technicians in this field can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing related hardware through a computer program, and the program can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0120] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them. Under the concept of the present invention, the technical features in the above embodiments or different embodiments may also be combined, the steps may be implemented in any order, and there are many other changes in different aspects of the present invention as described above, which are not provided in detail for the sake of simplicity. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A DPI message storage management system, characterized in that: It includes DPI devices, full-flow retention devices, and storage device clusters, including: The DPI device is used to collect and analyze messages, output session field information required by the full flow retention device, output message index information, and output the original message with the locator added; The full-flow retention device is used to receive session field information and store it in a database, receive message index information and store it in a database, receive original messages with locators added and transfer them to a storage device cluster to provide a message search function.

2. The DPI message storage management system according to claim 1, characterized in that: The DPI device includes a field analysis module, wherein: The field analysis module is used to analyze the fields that can be directly referenced from the DPI session field and the fields that need to be obtained through different field operations from the session field template provided by the full-flow retention device, and output the statistical session field information to the full-flow retention main module of the full-flow retention device; The field analysis module is also used to generate a unique ID for the session field information according to the session. The unique ID is generated by the time of the first packet of the session+device ID+internal sequence number.

3. The DPI message storage management system according to claim 2, characterized in that: The DPI device includes a storage message processing module, wherein: The stored message processing module is used to obtain a message storage strategy from a full-flow retention device to determine the flow that needs to be stored and the storage method, wherein the storage method includes one or more of full-flow storage, truncated storage, and selective truncation storage by flow; wherein the full-flow storage includes: storing all messages without truncation; the truncation storage includes: performing stage processing on the message and storing only the preset bytes of the head; the selective truncation storage by flow includes: storing all the first several messages of each flow, truncating the subsequent messages, and retaining only the preset bytes of the head; The stored message processing module is also used to add a unique ID, a start character and a stop character to the message and then output it to the full flow retention main module of the full flow retention device; The stored message processing module is also used to generate message index information and send the message index information to the full-flow retention main module of the full-flow retention device, wherein the message index information includes one or more of an ID, a sequence number, whether to truncate, a truncation position, and a storage position, wherein the ID is a unique ID obtained from the field analysis module; the sequence number is the sequence number of the message in the session flow; whether to truncate includes: judging whether the message needs to be truncated according to the message storage policy; the truncation position includes: judging the truncation position of the message according to the message storage policy; the storage position is left blank and filled in by the full-flow retention main module of the full-flow retention device.

4. The DPI message storage management system according to claim 1, characterized in that: The full flow retention device comprises a full flow retention main module, wherein: The full-flow retention main module is used to generate a message storage strategy according to user needs and synchronize it in real time to the storage message processing module of the DPI device; generate a session field template according to user needs and synchronize it in real time to the field analysis module of the DPI device. The full flow retention main module is also used to receive the session field information sent by the field analysis module from the field analysis module of the DPI device and store it in the local database; The full-flow retention main module is also used to receive messages from the storage message analysis module of the DPI device and transfer them to the storage device cluster according to the storage server write policy; The full flow retention main module is also used to receive message index information and backfill the storage location field in the message index information, and store it in the local database after the backfilling is completed.

5. The DPI message storage management system according to claim 4, characterized in that: The full-flow retention main module is also used to receive messages from the storage message analysis module of the DPI device and transfer them to the storage device cluster according to the storage server write strategy, which specifically includes: The received message is first stored in the local memory. When the message cache reaches the preset amount, it is written to the disk file of the storage device cluster according to the storage server write policy. The file overwrite technology is used to overwrite the message. Before overwriting, the number of files for each storage device in the storage device cluster is planned in advance; after each storage device writes and / or overwrites the preset number of files, it switches to the next storage device.

6. The DPI message storage management system according to claim 4, characterized in that: The full flow retention device includes a storage device monitoring module, wherein: The storage device monitoring module is used to monitor one or more of the disk size, disk availability, abnormal file writing, disk health status, file name, number of files and file type information of each storage device in real time; if the disk dynamically increases or decreases, the maximum available space and writable amount of the storage device are dynamically adjusted, and the storage server write policy is updated, and the information is passed to the alarm module; if the disk cannot be written and / or a disk health alarm occurs, the node is taken offline, the storage server write policy is updated, and the information is passed to the alarm module; if unplanned file names and / or file types appear, the unplanned file names and / or file types are passed to the alarm module; if the number of files decreases or increases, the information is passed to the alarm module.

7. The DPI message storage management system according to claim 6, characterized in that: The full-flow retention device includes a storage device management module, wherein: The storage device management module is used to generate a storage server write strategy based on one or more of the number of storage devices, the maximum available disk space and the maximum number of writable files of each storage device, the size of each message file in the full flow retention main module, the device binding time, the device status, and the maximum disk cycle write times in the storage device monitoring module, and write messages to the storage devices one by one according to the storage server write strategy. After each storage device has written the specified number of messages, it starts writing to the next storage device. When all storage devices have been written, it loops to the first storage device to write again; When an individual storage device goes offline abnormally, the writing order of the storage device is rearranged; When the offline storage device comes online again, its write order is rearranged according to its offline time; If the offline time of a storage device is within the preset range, and the messages it stores are later than those of some other storage devices, then the overwriting of this storage device will begin after the overwriting of these other storage devices is completed; before the storage device goes offline for maintenance, if there are messages to be stored, a label will be added to the corresponding storage device, so that when data is subsequently stored, tasks with a corresponding complete reading time period that is longer than its historical online and offline time period will be given priority in the corresponding storage device.

8. The DPI message storage management system according to claim 7, characterized in that: Each storage device calculates the disk health based on the maximum disk cycle write times and the actual message write times provided by the hard disk manufacturer, and maintains the storage device based on the disk health. For storage devices with low health, the maximum number of writable files is actively reduced, and the files exceeding the number are transferred to other storage devices by overwriting the earliest messages. The calculation of disk health includes: Get the maximum number of cyclic writes MaxWrites, the actual storage space ActualCapacity, the size of each message file PacketSize, and the maximum number of message writes TotalMaxWrites; TotalMaxWrites=MaxWrites*ActualCapacity / PacketSize; Assume that the actual number of writes that have occurred is CurrentWrites, then the percentage of disk health HealthPercentage is: CurrentWrites / TotalMaxWrites*100%.

9. The DPI message storage management system according to any one of claims 1 to 8, characterized in that: The full-flow retention device includes a message search module and an alarm module, wherein: The message search module is used to provide a query page and a query interface for a third party to call or directly access; the message search module retrieves the unique ID, storage location and message timestamp of the corresponding message in the index table of the database according to the message search condition selected by the user, and then searches in the corresponding storage device and message file to match the message. If the message is in multiple storage devices or multiple message files, it is matched in a multi-threaded parallel manner; after finding all the messages, check again whether they meet the user's requirements, remove the messages that do not meet the requirements, and strip the unique ID, start character, and end character, and finally merge all the messages before outputting them to the user; The alarm module is used to notify the user of the alarm information through one or more of a pop-up window, an email, and a sound after receiving the alarm information.

10. A DPI message storage management method, applied to the DPI message storage management system according to any one of claims 1 to 9, characterized in that: include: The DPI device collects and analyzes the packets, outputs the session field information required by the full-flow retention device, outputs the packet index information, and outputs the original packets with the locator added; The full-flow retention device receives session field information and stores it in the database, receives message index information and stores it in the database, receives original messages with locators added and transfers them to the storage device cluster to provide message search function.

Images