A method and device for dynamically switching data storage for a network traffic analysis system

Abstract

The present invention discloses a data storage dynamic switching method and device for a network traffic analysis system, and belongs to the field of computer data processing. The data storage dynamic switching device is set up in the database unit of the network traffic analysis system, and the data storage dynamic switching device at least includes: a data interaction unit, a data operation agent unit and a plurality of data storage units. The present invention determines the data structure type of network traffic data through the data interaction unit, and calls the corresponding data operation layer to declare the data structure type and operation content to the data operation agent unit. The data operation agent unit matches a suitable data storage unit according to the data type and operation content; then the driver program and the operation SDK of the matched data storage unit are uniformly processed and converted to complete the actual operation of the data, so that the network traffic analysis system can realize the dynamic switching of multiple databases while completing the processing of massive data.

Description

Technical Field

[0001] The present invention relates to the field of computer data processing, and in particular to a data storage dynamic switching method and device for a network traffic analysis system. Background Art

[0002] The network traffic analysis system is used to monitor and analyze data traffic in the network, which mainly involves two aspects: massive data processing and switching of multiple databases.

[0003] Traditional network traffic analysis systems have many problems, such as low storage and processing efficiency, insufficient storage capacity, or difficulty in switching some large data storages, which affects system services.

[0004] The data storage unit in the traditional network traffic analysis system refers to a multifunctional storage system or software that can store and manage various types of data. They provide efficient and reliable data storage and access mechanisms, allowing users to easily store, retrieve, update and delete data. Common storage units include relational databases (such as MySQL, Oracle), non-relational databases (such as MongoDB, Redis), columnar databases (such as Apache Doris, ClickHouse), etc. The data storage unit in the network traffic analysis system needs to be able to store and process large or massive data with good performance.

[0005] The Chinese patent with the announcement number CN102479211B discloses a system and method for processing massive data based on a database. The system includes: a database for storing raw data; a data acquisition device for extracting data from the database according to set conditions and transmitting the extracted data to a data processing device; a data processing device for processing the received data in an odd-even rotation manner according to a set dimension and transmitting the processed data to a data storage device; and a data storage device for storing the received processed data in a classified manner.

[0006] However, these data storage units (data storage devices in CN102479211B) belong to common storage unit relational databases (such as MySQL, Oracle), all of which use dedicated connection drivers (i.e., Driver hereinafter) and operation SDKs (i.e., Template hereinafter), and cannot implement operations used across data storage units.

[0007] In order to realize the operation of storing data across data storage units (databases), a Chinese patent with announcement number CN103530427B discloses a dynamic switching method and device based on multiple databases, wherein the method includes: when the computer application system is started, reading the relevant configuration file; the configuration file includes more than one key-value pair, wherein the key is the database identifier and the value is the method name for accessing the database; setting the database identifier in the configuration file in the thread pool of the computer application system; when the method of the computer application system is called, intercepting the method, obtaining the database identifier corresponding to the method from the thread pool, and returning the intercepted method; executing the method using the obtained database identifier. However, for non-relational databases (such as MongoDB and Redis), the storage unit cannot use the technical solution of this patent; the dynamic switching method and device of the database involved in this patent can only intercept and switch the storage processing implemented based on the standard JDBC (Java Database Connectivity, which is the standard interface for connecting and operating databases in Java), such as: MySQL, SQLserver, DB2, etc., and such data storage units cannot process and store massive data and query them efficiently.

[0008] In summary, the network traffic analysis system in the prior art cannot dynamically switch different data storage units when processing massive amounts of data. Summary of the invention

[0009] The purpose of the present invention is to overcome the problem existing in the existing network traffic analysis system: it is impossible to realize the dynamic switching of data storage and the processing of massive data at the same time; to provide a method and device for dynamic switching of data storage for a network traffic analysis system, so that the network traffic analysis system can realize dynamic switching of different data storage units while processing massive data.

[0010] In order to achieve the above-mentioned object of the invention, the present invention provides the following technical solutions:

[0011] The present invention provides a data storage dynamic switching device for a network traffic analysis system. The data storage dynamic switching device is set up in a database unit of the network traffic analysis system, and the data storage dynamic switching device at least includes: a data interaction unit, a data operation proxy unit and a plurality of data storage units. The plurality of data storage units are used to be responsible for real data storage services. The data interaction unit is used to receive instructions sent by a network traffic collection and processing unit of the network traffic analysis system, and is also used to receive instructions sent by a network traffic data analysis and display unit of the network traffic analysis system, and call the data operation layer configured by the data interaction unit according to the received instructions; wherein the instructions at least include the data structure type of the network traffic data. The data operation layer declares the data structure type and the operation content to the data operation proxy unit according to the instructions. When the declaration information is received, the data operation proxy unit matches a suitable data storage unit according to the data type and the operation content; and then performs unified processing and conversion on the driver (Driver) and the operation SDK (Template) of the matched data storage unit to complete the actual operation of the data.

[0012] According to a preferred implementation, the data structure types include at least: structured data, semi-structured data, unstructured data and metadata.

[0013] According to a preferred implementation, the data operation layer includes at least several data storage DAO interfaces, which may be structured data storage DAO interfaces, semi-structured data storage DAO interfaces, unstructured storage DAO interfaces, and metadata storage DAO interfaces.

[0014] According to a preferred implementation, the data interaction unit determines the data structure type of the relevant network traffic data according to the flag in the instruction, and calls the data storage DAO interface that matches the data structure type according to the data structure type.

[0015] According to a preferred implementation, the data operation proxy unit is provided with a public data interface for receiving declaration information of each data storage DAO interface.

[0016] According to a preferred implementation, the data operation agent unit matches a suitable data storage unit according to the data type and operation content, including at least: reading the configuration information of the data storage unit of the network traffic analysis system, obtaining the type and key parameters of the data storage unit configured in the network traffic analysis system; matching the data storage unit according to the connection URI (Uniform Resource Identifier) ​​in the declaration information.

[0017] According to a preferred embodiment, after completing the matching of the data storage unit, the data operation agent unit loads the driver of the matched data storage unit to determine whether the matched data storage unit is available. If the matched data storage unit is available, the data operation agent unit loads the matched data storage unit with relevant configurations to generate a database that can meet the data storage requirements of the corresponding structure type.

[0018] According to a preferred implementation, the data operation proxy unit performs unified processing and conversion on the driver (Driver) and operation SDK (Template) of the matched data storage unit to complete the actual operation on the data.

[0019] The present invention also provides a method for dynamically switching data storage for a network traffic analysis system, the method for dynamically switching data storage for a network traffic analysis system at least comprising:

[0020] Dividing data in a network traffic analysis system into at least two data structure types;

[0021] Configuring a plurality of data storage units for the network traffic analysis system to meet the need for storing data of different data structures;

[0022] Constructing a data interaction unit including at least two different data operation layers; establishing associations between data structure types and data operation layers, and assigning unique data structure types to different data operation layers;

[0023] A data operation proxy unit is constructed, and the data operation proxy unit is used to uniformly process and transform the driver program and the operation SDK of the data storage unit to complete the actual operation of the data.

[0024] According to a preferred embodiment, the working steps of the data storage dynamic switching method at least include:

[0025] Step 1: The data interaction unit calls the data operation layer that matches the data structure type in the network traffic analysis system;

[0026] Step 2, the data operation layer declares the data structure type and operation content to the data operation agent unit;

[0027] Step 3: The data operation agent unit matches a suitable data storage unit according to the declaration information, and generates a database that can meet the data storage requirements of the corresponding structure type.

[0028] Compared with the prior art, the present invention has the following beneficial effects:

[0029] The present invention determines the data structure type of network traffic data through a data interaction unit, and calls the corresponding data operation layer to declare the data structure type and operation content to the data operation proxy unit. The data operation proxy unit matches a suitable data storage unit according to the data type and operation content; then the driver and operation SDK of the matched data storage unit are uniformly processed and converted to complete the actual operation of the data, so that the network traffic analysis system can realize dynamic switching of multiple databases while completing the processing of massive data, so that the network traffic analysis system can process different types of data, improve the data processing efficiency of the network traffic analysis system, and relational databases (such as MySQL, Oracle), non-relational databases (such as MongoDB, Redis), column databases (such as Apache Doris, ClickHouse), etc. can all be dynamically switched through the present invention. The present invention can enable common storage units to realize dynamic switching of databases without affecting the business operation of the network traffic analysis system. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the composition of the network traffic analysis system;

[0031] Figure 2 A schematic diagram of the architecture of a network traffic analysis system using the data storage dynamic switching method provided by the present invention;

[0032] Figure 3 The figure is a schematic diagram of the working process of the data storage dynamic switching method provided by the present invention. DETAILED DESCRIPTION

[0033] The present invention is further described in detail below in conjunction with test examples and specific implementation methods. However, this should not be understood as the scope of the above subject matter of the present invention being limited to the following embodiments, and all technologies realized based on the content of the present invention belong to the scope of the present invention.

[0034] Example 1

[0035] This embodiment provides a data storage dynamic switching device for a network traffic analysis system. The data storage dynamic switching device is set up in the database unit of the network traffic analysis system, and the data storage dynamic switching device includes at least: a data interaction unit, a data operation agent unit and several data storage units. Several data storage units are used to be responsible for real data storage services. The data interaction unit is used to receive instructions sent by the network traffic collection and processing unit of the network traffic analysis system, and is also used to receive instructions sent by the network traffic data analysis and display unit of the network traffic analysis system, and call the data operation layer configured by the data interaction unit according to the received instructions; wherein the instructions at least include the data structure type of the network traffic data. The data operation layer declares the data structure type and operation content to the data operation agent unit according to the instructions. When the declaration information is received, the data operation agent unit matches the appropriate data storage unit according to the data type and operation content; then the driver (Driver) and the operation SDK (Template) of the matched data storage unit are uniformly processed and converted to complete the actual operation of the data.

[0036] Common storage units include relational databases (such as MySQL, Oracle), non-relational databases (such as MongoDB, Redis), columnar databases (such as Apache Doris, ClickHouse), etc., which can all be dynamically switched through this embodiment.

[0037] In this embodiment, the data structure type of network traffic data is determined by the data interaction unit, and the corresponding data operation layer is called to declare the data structure type and operation content to the data operation proxy unit. The data operation proxy unit matches the appropriate data storage unit according to the data type and operation content; then the driver program and operation SDK of the matched data storage unit are uniformly processed and converted to complete the actual operation of the data, so that the network traffic analysis system can realize dynamic switching of multiple databases while completing massive data processing, so that the network traffic analysis system can process different types of data, improve the data processing efficiency of the network traffic analysis system, and enable common storage units such as relational databases (such as MySQL, Oracle), non-relational databases (such as MongoDB, Redis), and columnar databases (such as Apache Doris, ClickHouse) to realize dynamic switching without affecting the business operation of the network traffic analysis system.

[0038] Example 2

[0039] This embodiment is a further improvement of Embodiment 1, and repeated contents are not repeated here. In this embodiment, the data structure types at least include: structured data, semi-structured data, unstructured data and metadata.

[0040] Preferably, the data operation layer includes at least: a structured data storage DAO interface, a semi-structured data storage DAO interface, an unstructured storage DAO interface, and a metadata storage DAO interface.

[0041] Preferably, the data interaction unit determines the data structure type of the relevant network traffic data according to the flag in the instruction, and calls the data storage DAO interface that matches the data structure type according to the data structure type.

[0042] Preferably, the data operation proxy unit is provided with a public data interface for receiving declaration information of each data storage DAO interface.

[0043] Preferably, the data operation agent unit matches a suitable data storage unit according to the data type and operation content, including at least: reading the configuration information of the data storage unit of the network traffic analysis system, obtaining the type and key parameters of the data storage unit configured in the network traffic analysis system; matching the data storage unit according to the connection URI (Uniform Resource Identifier) ​​in the declaration information.

[0044] Preferably, after completing the matching of the data storage unit, the data operation agent unit loads the driver of the matched data storage unit to determine whether the matched data storage unit is available. If the matched data storage unit is available, the data operation agent unit loads the matched data storage unit with relevant configurations to generate a database that can meet the data storage requirements of the corresponding structure type.

[0045] Preferably, the data operation proxy unit performs unified processing and conversion on the driver program (Driver) and the operation SDK (Template) of the matched data storage unit to complete the actual operation on the data.

[0046] Example 3

[0047] This embodiment is a further improvement of Embodiment 1 and Embodiment 2, and the repeated contents are not repeated here. This embodiment provides a data storage dynamic switching device for a network traffic analysis system. Figure 1 The network traffic analysis system can be composed of a network traffic collection and processing unit, a database unit, and a network traffic data analysis and display unit. The network traffic collection and processing unit collects network traffic data and stores it in the database unit. The network traffic data analysis and display unit queries and retrieves relevant traffic data from the database unit.

[0048] The data storage dynamic switching device provided in this embodiment is set up in the database unit of the network traffic analysis system.

[0049] See also Figure 2 ,The data storage dynamic switching device includes: a data interaction unit, a data operation agent unit and a number of data storage units.

[0050] The data interaction unit is respectively connected to the network flow collection and processing unit and the network flow data analysis and display unit for communication.

[0051] When the network traffic collection and processing unit stores the collected network traffic data in the database unit, it sends a storage instruction to the data exchange unit. The storage instruction may include: a flag for indicating the data structure type information of the network traffic data to be stored.

[0052] When the network traffic data analysis and display unit queries or calls the network traffic data from the database unit, it sends a query instruction to the data interaction unit. The query instruction may include: a flag for indicating the data structure type information of the network traffic data to be queried or called.

[0053] Data structure types can include structured data, semi-structured data, unstructured data, and metadata.

[0054] Structured data is data organized according to a predefined schema, usually stored in a relational database or spreadsheet. It has well-defined fields and data types and is easy to query and analyze. For example, rows and columns in a table are typical representations of structured data.

[0055] Semi-Structured Data is data that is partially structured, but does not have a strict predefined schema. It contains some tags or labels that make it easier to process and organize. Semi-structured data can be represented using different formats (such as XML, JSON), and fields can be added, deleted, or modified flexibly. For example, the element tag in an HTML document is an example of semi-structured data.

[0056] Unstructured data is data without a clear structure or pattern. It usually exists in a free form and cannot be directly incorporated into traditional relational databases for processing. Unstructured data can be in the form of text files, images, audio, video, etc. For example, social media posts, email content, and image files are all unstructured data.

[0057] Metadata is data that describes data. It provides information about the properties, structure, format, and meaning of data. Metadata can help understand and manage data, including data source, creation date, owner, data type, etc. For example, metadata in a photo can include camera model, shooting time, aperture value, etc.

[0058] Several data storage units are responsible for real data storage services, and different data storage units are used to store data with different data structures. Preferably, in this embodiment, the number of data storage units can be set to 8 (DB1-DB8).

[0059] Preferably, each data storage unit (DB1~DB8) can support storage processing of multiple types of data, and different data storage units can respectively meet the needs of storing data of different data structures. Preferably, each data storage unit (DB1~DB8) can be a conventional database such as a relational database (such as MySQL, Oracle), a non-relational database (such as MongoDB, Redis), a columnar database (such as Apache Doris, ClickHouse), etc.

[0060] The data interaction unit is configured with several data operation layers. Preferably, the data operation layer is a key component of the model part in the MVC (Model-View-Controller) design pattern.

[0061] Preferably, in this embodiment, the data operation layer adopts the DAO mode (Data Access Object Pattern).

[0062] The DAO pattern separates data access logic from business logic and encapsulates data access operations in a dedicated class. Through the DAO pattern, the rest of the application (such as controllers and business logic layers) can be independent of the specific implementation details of data storage, such as database type, SQL query statements, etc.

[0063] The functions of the DAO model include:

[0064] Encapsulate database access details: The DAO pattern encapsulates details such as database connection creation, SQL statement execution, and result mapping, so that the business logic layer does not need to interact directly with the database, thereby reducing the coupling of the code.

[0065] Improve code reusability: Different business logics may need to access the same data table. Through the DAO mode, these business logics can share the same database access code, improving code reusability.

[0066] Enhance code maintainability: When the database structure changes, only the relevant code of the DAO mode needs to be modified, without modifying the code of the business logic layer or the controller layer, thus simplifying maintenance work.

[0067] Support data persistence: The DAO pattern is not limited to database access, but can also be extended to other types of data persistence mechanisms, such as files, caches, etc., to provide a unified data access interface for applications.

[0068] The implementation of the DAO pattern usually includes the following steps:

[0069] Define DAO interface: define a DAO interface for each data table that needs to be accessed. The interface declares all database operation methods related to the data table, such as adding, deleting, modifying, querying, etc.

[0070] Implement DAO interface: Provide one or more implementation classes for each DAO interface. These implementation classes contain specific database access logic.

[0071] Use DAO: In the business logic layer or controller layer, access data through the DAO interface (rather than its implementation class), so that the DAO implementation class can be replaced without modifying the business logic layer or controller layer code.

[0072] Preferably, in this embodiment, the data operation layer can be different data storage DAO interfaces.

[0073] Preferably, the data operation layer of this embodiment is provided with four DAO interfaces, including: a structured data storage DAO interface, a semi-structured data storage DAO interface, an unstructured storage DAO interface, and a metadata storage DAO interface.

[0074] Preferably, the data interaction unit is configured with a structured data storage DAO interface, a semi-structured data storage DAO interface, an unstructured storage DAO interface and a metadata storage DAO interface.

[0075] The structured data storage DAO interface is used to process structured data; the semi-structured data storage DAO interface is used to process semi-structured data; the unstructured data storage DAO interface is used to process unstructured data; and the metadata storage DAO interface is used to process metadata.

[0076] The data interaction unit determines the data structure type of the relevant network traffic data according to the flag in the query instruction or the storage instruction, and calls the data storage DAO interface that matches the data structure type according to the data structure type.

[0077] The data storage DAO interface only declares the data structure type and operation content, and does not perform actual data processing operations. The actual data processing operations are performed by the data operation proxy unit.

[0078] The data storage DAO interface declares that the data structure type and the operation content can be implemented based on polymorphism in the Java architecture. In the Java architecture, polymorphism is a feature that allows us to use a reference variable of a parent class type to reference objects of different subclass types. In simple terms, polymorphism allows us to call methods of subclass types through references of parent class types.

[0079] The data storage DAO interface can declare the data structure type and operation content through the interface, and implement the actual data processing operation according to the registered specific implementation interface.

[0080] Registration is divided into two parts: dynamic binding and method overriding.

[0081] Dynamic binding: In polymorphism, the method call is determined at runtime rather than at compile time. When a reference variable of the parent class type is used to reference an object of a subclass type and call its method, the subclass method that should be called is actually determined based on the type of the runtime object. This makes the code more flexible and can decide which subclass method to execute at runtime.

[0082] Method Overriding: Polymorphism also relies on the child class overriding the parent class's methods. The child class can redefine the methods inherited from the parent class and provide its own implementation. When the overridden method is called using a reference variable of the parent class type, the child class's method is actually called instead of the parent class's method.

[0083] To implement an interface specifically, you need to override the method of the parent class. The subclass can redefine the method inherited from the parent class and provide its own implementation.

[0084] The data storage DAO interface does not directly operate the Driver and Template.

[0085] Driver: A driver is a software component or library used to interact with a database.

[0086] Taking MongoDB storage unit as an example, in MongoDB, Driver refers to a client library specifically used to communicate with MongoDB database. The driver provides a set of APIs and methods that allow developers to interact with the database through programming languages ​​​​(such as Java, Python, etc.). The driver is responsible for handling specific data processing operations such as underlying network communication, data conversion, and query execution, so that developers can easily operate the database.

[0087] Template (Operation SDK): Template is an abstraction layer that encapsulates the specific implementation details of the underlying database driver and provides a higher-level interface to simplify database operations.

[0088] Take MongoDB storage unit as an example. In MongoDB, Template refers to MongoTemplate, which is a module provided by Spring Framework for interacting with MongoDB. MongoTemplate provides a set of methods to perform CRUD operations (create, read, update, delete), query data, and support transaction management and aggregation operations. Through MongoTemplate, developers can operate MongoDB database more conveniently without writing complex raw database query code.

[0089] The data operation proxy unit is provided with a public data interface for receiving declaration information of each data storage DAO interface.

[0090] After receiving the data type and operation content declared by the data storage DAO interface, the data operation proxy unit matches the appropriate data storage unit according to the data type and operation content; then the driver (Driver) and operation SDK (Template) of the matched data storage unit are uniformly processed and transformed to complete the actual operation of the data. The unified processing and transformation is to rewrite the methods of the parent class. The subclass can redefine the methods inherited from the parent class and provide its own implementation.

[0091] The matching rule of the data operation proxy unit for the declaration information and the data storage unit is to match the data storage unit according to the connection URI in the declaration information.

[0092] The data operation agent unit can obtain the configuration information of the network traffic analysis system database unit, and determine the data storage DB type and key parameters supported by the database unit, such as: service address, port, authentication information, etc.

[0093] Preferably, when the data operation agent unit matches the data storage unit, if there are multiple data storage units with the same type of data storage capabilities, the data operation agent unit can select the corresponding data storage unit for connection according to the configured priority rules, and can also determine whether to connect according to the storage service availability of the data storage unit, or can connect according to the data storage unit manually selected by the staff.

[0094] Preferably, the data operation agent unit connects different data storage units through dynamic identification and dynamic assembly, and dynamically loads relevant configurations to each data storage unit as needed to generate a database that meets the data storage needs of different structural types as the final operation implementation of the data operation agent.

[0095] Preferably, after receiving the declaration information of the data storage DAO interface, the data operation agent unit reads the configuration information of the data storage unit of the network traffic analysis system, loads the Driver, and connects the data storage unit; dynamically loads relevant configurations for each data storage unit as needed, and generates a database that can complete one or more implementations of four types of data storage processing: structured data, semi-structured data, unstructured data, and metadata.

[0096] If there are multiple data storage units configured in the traffic analysis system, the data operation proxy unit can load the service template of the data storage unit with the highest adaptability and available service according to the implementation priority rules built into the data storage unit.

[0097] Preferably, the configuration information of the data storage unit includes: the type of the data storage unit configured by the network traffic analysis system and key parameters, such as: service address, port, authentication information, etc.

[0098] Preferably, when the traffic analysis system is configured with multiple data storage units having the same type of data storage capacity, when the data operation agent unit selects a specific data storage unit to connect to generate a database, the data storage unit to be connected can be selected manually by a staff member, or can be automatically connected according to the priority of the data storage unit configuration, or can be determined based on the storage service availability of the data storage unit. The priority of the data storage unit configuration can be the priority order of the hardware configuration, or the priority order of the built-in program.

[0099] Preferably, without manually specifying the priority, and when different DBs (data storage units) have the same type of data storage capabilities, the data operation agent unit can connect the data storage unit according to the system priority. Preferably, the system priority can be built-in based on the experience of actual deployment power consumption, stability, and performance indicators, and the matching degree is also the performance feedback of the data storage DB on the above indicators of such data storage.

[0100] Preferably, the method for determining whether the storage service of the data storage unit is available can be: the data operation agent unit initiates the authentication service by loading the Driver program and returns a message of successful authentication. After successful authentication, the data operation agent unit can initiate a connection to the corresponding data storage unit according to the received successful authentication information.

[0101] Preferably, after the data operation proxy unit connects the corresponding data storage unit according to the declaration information of the data storage DAO interface received, it rewrites the method (subclass) of the data storage unit according to the interface declaration information (parent class), so that the data storage unit can complete the storage processing of one or more of the four types of data: structured data, semi-structured data, unstructured data, and metadata.

[0102] See also Figure 1 , the data operation agent unit generates a database that meets the needs of data storage of different structure types, which may include:

[0103] Connect DB1, DB2, and DB3 to generate a database that meets the requirements for structured data storage;

[0104] Connect DB3, DB4, and DB5 to generate a database that meets the requirements of semi-structured data storage;

[0105] Connect DB5, DB6, and DB7 to generate a database that meets the requirements for unstructured data storage;

[0106] Connect DB7 and DB8 to generate a database that meets the requirements of metadata storage.

[0107] Figure 1 DB1-DB8 in the table refers to one or more implementations that can complete four types of data storage processing: structured data, semi-structured data, unstructured data, and metadata. DB1-DB8 can be real database products, which can be either stand-alone databases or databases for cluster services. DB1-DB8 are used to complete real data storage services, and DB1-DB8 in the accompanying drawings are only used to illustrate that this embodiment is configured with multiple different types of data storage units to meet the data storage requirements of different data structures, and it does not limit this embodiment to only be configured with 8 data storage units.

[0108] See also Figure 3 , the workflow of the data storage dynamic switching device can be:

[0109] Step 1: The data interaction unit receives an instruction sent by the network traffic collection and processing unit or the network traffic data analysis and display unit; wherein the instruction includes a flag representing the data structure type information of the relevant network traffic data;

[0110] Step 2: The data interaction unit calls the data storage DAO interface corresponding to the data structure type represented by the instruction according to the flag in the instruction;

[0111] Step 3: The data storage DAO interface declares the data structure type and operation content to the data operation proxy unit through the public interface;

[0112] Step 4: The proxy unit generates a database that meets the storage needs of data of different structural types according to the received declaration information.

[0113] Preferably, taking the storage of structured data and semi-structured data as an example, the workflow of the storage dynamic switching device is described, and the specific process includes:

[0114] S1. The network traffic collection and processing unit collects data packet data, and generates structured data application data statistics (including various network performance indicators of the App) and semi-structured data transaction log data (including network performance indicators and other fields of variable quantity and type) according to the built-in business data logic, and then sends the storage instruction with structured data flag and semi-structured data flag to the data interaction unit.

[0115] S2. The interaction unit calls the structured data storage DAO interface and the semi-structured data storage DAO interface according to the structured data flag and the semi-structured data flag in the storage instruction.

[0116] S3. The structured data storage DAO interface and the semi-structured data storage DAO interface declare the data loading operation and the corresponding data structure type to the data operation proxy unit according to the storage instruction. The structured data storage DAO interface declares the loading operation of structured data, and the semi-structured data storage DAO interface declares the loading operation of semi-structured data.

[0117] S4. After receiving the declaration information of the structured data storage DAO interface and the semi-structured data storage DAO interface, the data operation agent unit reads the configuration information of the data storage unit of the network traffic analysis system, determines the data storage unit that supports structured data storage and semi-structured data storage, and uniformly processes and transforms the driver (Driver) and operation SDK (Template) of the determined data storage unit. Then, it connects to the data storage unit, registers it as the real implementation of the data agent operation, generates a dynamically assembled database, and stores and processes the structured data and semi-structured data.

[0118] Example 4

[0119] This embodiment provides a method for dynamically switching data storage for a network traffic analysis system, and the method for dynamically switching data storage for a network traffic analysis system provided by this embodiment can be implemented by the dynamic switching device for data storage for a network traffic analysis system provided by Embodiment 1, Embodiment 2, and Embodiment 3. The method for dynamically switching data storage for a network traffic analysis system at least includes:

[0120] Dividing data in a network traffic analysis system into at least two data structure types;

[0121] Configuring a network traffic analysis system with a number of data storage units that meet the need for storing data of different data structures;

[0122] Construct a data interaction unit including at least two different data operation layers; establish an association between data structure types and data operation layers, and assign unique data structure types to different data operation layers;

[0123] Construct a data operation proxy unit, and use the data operation proxy unit to uniformly process and transform the driver and operation SDK of the data storage unit to complete the actual operation of the data.

[0124] Preferably, the working steps of the data storage dynamic switching method at least include:

[0125] Step 1: The data interaction unit calls the data operation layer that matches the data type according to the structure type of the data in the network traffic analysis system;

[0126] Step 2: The data operation layer declares the data structure type and operation content to the data operation agent unit;

[0127] Step 3: The data operation agent unit matches the appropriate data storage unit according to the declaration information and generates a database that can meet the data storage requirements of the corresponding structure type.

[0128] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A data storage dynamic switching device for a network traffic analysis system, characterized in that: The data storage dynamic switching device is set up in the database unit of the network traffic analysis system, and the data storage dynamic switching device at least includes: a data interaction unit, a data operation agent unit and a plurality of data storage units; The data storage units are used to store real data. The data interaction unit is used to receive instructions sent by the network traffic collection and processing unit of the network traffic analysis system, and is also used to receive instructions sent by the network traffic data analysis and display unit of the network traffic analysis system, and call the data operation layer configured by the data interaction unit according to the received instructions; wherein the instructions at least include the data structure type of the network traffic data; The data operation layer declares the data structure type and operation content to the data operation agent unit according to the instruction; the data structure type includes at least: structured data, semi-structured data, unstructured data and metadata; the data operation layer includes at least: structured data storage DAO interface, semi-structured data storage DAO interface, unstructured storage DAO interface, metadata storage DAO interface; When the declaration information is received, the data operation proxy unit matches a suitable data storage unit according to the data type and the operation content; then the driver program and the operation SDK of the matched data storage unit are uniformly processed and converted to complete the actual operation of the data; The data operation proxy unit matches the appropriate data storage unit according to the data type and operation content in at least the following ways: Read the configuration information of the data storage unit of the network traffic analysis system, and obtain the type and key parameters of the data storage unit configured in the network traffic analysis system; Matching a data storage unit according to the connection URI in the declaration information; After completing the matching of the data storage unit, the data operation agent unit loads the driver of the matched data storage unit to determine whether the matched data storage unit is available; if the matched data storage unit is available, the data operation agent unit loads the matched data storage unit with relevant configurations to generate a database that can meet the data storage requirements of the corresponding structure type; The method of determining whether the storage service of the data storage unit is available is as follows: the data operation agent unit initiates the authentication service by loading the Driver program and returns a message of successful authentication; After successful authentication, the data operation proxy unit can initiate a connection to the corresponding data storage unit based on the received authentication success information; After the data operation agent unit connects the corresponding data storage unit according to the declaration information of the data storage DAO interface, it rewrites the method of the data storage unit according to the interface declaration information, so that the data storage unit can complete the storage processing of one or more of the four types of data: structured data, semi-structured data, unstructured data, and metadata.

2. The data storage dynamic switching device for a network traffic analysis system according to claim 1, characterized in that: The data interaction unit determines the data structure type of the relevant network traffic data according to the flag in the instruction, and calls the data storage DAO interface that matches the data structure type according to the data structure type.

3. The data storage dynamic switching device for a network traffic analysis system according to claim 1, characterized in that: The data operation proxy unit is provided with a public data interface for receiving declaration information of each data storage DAO interface.

4. A data storage dynamic switching method for a network traffic analysis system, characterized in that: The method for dynamically switching data storage for a network traffic analysis system is implemented by using the device for dynamically switching data storage for a network traffic analysis system as claimed in claim 1, and the method for dynamically switching data storage for a network traffic analysis system at least comprises: Dividing data in a network traffic analysis system into at least two data structure types; Configuring a plurality of data storage units for the network traffic analysis system to meet the need for storing data of different data structures; Constructing a data interaction unit including at least two different data operation layers; establishing associations between data structure types and data operation layers, and assigning unique data structure types to different data operation layers; A data operation proxy unit is constructed, and the data operation proxy unit is used to uniformly process and transform the driver program and the operation SDK of the data storage unit to complete the actual operation of the data.

5. The data storage dynamic switching method for a network traffic analysis system according to claim 4 is characterized in that: The working steps of the data storage dynamic switching method at least include: Step 1: The data interaction unit calls the data operation layer that matches the data structure type in the network traffic analysis system; Step 2, the data operation layer declares the data structure type and operation content to the data operation agent unit; Step 3: The data operation agent unit matches a suitable data storage unit according to the declaration information, and generates a database that can meet the data storage requirements of the corresponding structure type.

Images