Data storage method and device and electronic equipment

Abstract

The invention discloses a data storage method and device and electronic equipment. The method comprises the steps that a log data set sent by collection equipment is received, the log data set comprises multiple pieces of log data, and the multiple pieces of log data are sent in a classified mode according to network protocol types; performing desensitization operation on the log data in the log data set to obtain an updated data set; selecting a target database from a database cluster according to the network protocol type; and storing the updated data set into a local table of the target database. Through the method and the device, the problems of low safety and low storage efficiency of storing the traffic log through a hardware probe in the related technology are solved.

Description

Technical Field

[0001] The present application relates to the field of big data, and more specifically, to a data storage method, device, and electronic device. Background Art

[0002] With the development and popularization of Internet technology, network traffic logs have become an important tool for monitoring network security, information security, data security, optimizing network performance, and preventing and detecting attack behaviors.

[0003] When collecting network traffic logs, traffic collection can be performed based on hardware probes. For the scenario of collecting network traffic logs in a data center, existing technical methods generally use hardware probes to deploy port mirroring of network switches to collect traffic logs through hardware probes.

[0004] However, after the hardware probe collects the network traffic log, it usually stores the network traffic log locally. When storing, the hardware probe saves the original log without desensitizing personal privacy information and sensitive data, resulting in low storage security of user information. In addition, the storage space of the hardware probe itself cannot be easily expanded, which leads to the hardware probe device being unable to store the traffic log well when the amount of data in the network traffic log is large.

[0005] Currently, no effective solution has been proposed to address the problems of low security and storage efficiency of storing traffic logs through hardware probes in related technologies. Summary of the invention

[0006] The present application provides a data storage method, device and electronic device to solve the problems of low security and storage efficiency of storing traffic logs through hardware probes in the related art.

[0007] According to one aspect of the present application, a data storage method is provided. The method includes: receiving a log data set sent by a collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to the network protocol type; performing a desensitization operation on the log data in the log data set to obtain an update data set; selecting a target database from a database cluster according to the network protocol type; and storing the update data set in a local table of the target database.

[0008] Optionally, before desensitizing the log data in the log data set to obtain the updated data set, the method also includes: randomly selecting sample log data from the log data set to obtain multiple sample log data, and identifying whether sensitive information exists in the multiple sample log data; if sensitive information exists in the multiple sample log data, performing the step of desensitizing the log data in the log data set; if no sensitive information exists in the multiple sample log data, determining the log data set as an updated data set.

[0009] Optionally, before desensitizing the log data in the log data set to obtain the updated data set, the method also includes: obtaining the source address of the log data set, and determining the target application to which the log data set is sent based on the source address; judging whether the target application meets preset filtering conditions; if the target application meets the preset filtering conditions, determining the log data set as an updated data set; if the target application does not meet the preset filtering conditions, executing the step of desensitizing the log data in the log data set.

[0010] Optionally, selecting a target database from a database cluster according to a network protocol type includes: determining a database in the database cluster corresponding to the network protocol type to obtain multiple candidate databases; obtaining remaining storage space of each candidate database, and determining a candidate database with the largest remaining storage space as the target database.

[0011] Optionally, a log data set is stored in a first subject space, and a desensitizing operation is performed on the log data in the log data set to obtain an updated data set, including: obtaining each log data in the log data set from the first subject space in turn; determining whether there is sensitive data in the log data; when sensitive data exists in the log data, determining the data type of the sensitive data; determining a desensitizing strategy for changing the sensitive data based on the data type, and desensitizing the log data using the desensitizing strategy to obtain updated data; storing the updated data in a second subject space until all log data in the first subject space is moved to the second subject space; and combining the updated data in the second subject space into an updated data set.

[0012] Optionally, storing the updated data set in the local table of the target database includes: sending the updated data set to the engine table in the target database, and sending the updated data set to the materialized view table through the engine table; parsing the updated data set through the materialized view table to obtain multiple updated data, and storing the multiple updated data in the local table through a distributed table, wherein the distributed table is used to store the updated data in the local table, and store the metadata of the updated data in the metadata table of the target database.

[0013] Optionally, after storing the multiple updated data in the local table, the distributed table sends metadata of each updated data to a metadata table in a first database having the same network protocol type as the target database.

[0014] According to another aspect of the present application, a data storage device is provided. The device includes: a receiving unit, used to receive a log data set sent by a collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to the network protocol type; a desensitizing unit, used to perform a desensitizing operation on the log data in the log data set to obtain an updated data set; a first selection unit, used to select a target database from a database cluster according to the network protocol type; and a storage unit, used to store the updated data set in a local table of the target database.

[0015] According to another aspect of the present invention, a computer program product is provided, including a computer program, which implements a data storage method provided in the aforementioned embodiment of the present application when the computer program is executed by a processor.

[0016] According to another aspect of the present invention, an electronic device is provided, comprising one or more processors and a memory; the memory stores computer-readable instructions, and the processor is used to execute the computer-readable instructions, wherein when the computer-readable instructions are executed, a data storage method provided in the aforementioned embodiment is executed.

[0017] Through this application, the following steps are adopted: receiving a log data set sent by a collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to the network protocol type; performing a desensitization operation on the log data in the log data set to obtain an update data set; selecting a target database from a database cluster according to the network protocol type; and storing the update data set in a local table of the target database. By obtaining the log data set sent by the collection device and performing a desensitization operation on the log data in the log data set, the security of sensitive data is ensured, and the log data is stored in a local table in the target database according to the network protocol type, thereby ensuring the storage efficiency of the data. At the same time, when expansion is required, the database can be directly expanded or multiple databases can be used for data storage in a distributed database manner, thereby achieving the technical effect of improving the security and storage efficiency of log data storage. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1is a flowchart of a data storage method provided according to an embodiment of the present application;

[0020] Figure 2 It is a schematic diagram of a log data storage process containing sensitive data provided in accordance with an embodiment of the present application;

[0021] Figure 3 It is a schematic diagram of a log data storage process that does not contain sensitive data provided according to an embodiment of the present application;

[0022] Figure 4 is a schematic diagram of a data storage process provided according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of an optional log data storage process provided according to an embodiment of the present application;

[0024] Figure 6 is a schematic diagram of another optional log data storage process provided according to an embodiment of the present application;

[0025] Figure 7 is a schematic diagram of a data storage device provided according to an embodiment of the present application;

[0026] Figure 8 It is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] It should be noted that the data storage method, device and electronic device determined in the present disclosure can be used in the field of big data, and can also be used in any field other than the field of big data. The application field of the data storage method, device and electronic device determined in the present disclosure is not limited.

[0031] It should be noted that the collected information, user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) used in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated. Corresponding operation portals are provided for users to choose to authorize or refuse to use. For example, an interface is set up between this system and relevant users or organizations. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or organization through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or organization.

[0032] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0033] For the convenience of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0034] Network traffic log: A log file that records and stores information related to network traffic in a network environment. The log file can contain detailed information about the data flow in the network, such as source address, destination address, transmission protocol, packet size, connection duration, etc.

[0035] Desensitizing technology: refers to data processing technology that blurs or anonymizes data in order to remove sensitive information from the data.

[0036] Apache Kafka, a message middleware, is an open source distributed stream data processing platform designed for data publishing and subscription systems that process large amounts of real-time data streams. It has the characteristics of high data throughput, low latency, horizontal expansion, and support for multiple data sources.

[0037] Kafka Topic: It is a noun that represents a data category in the message middleware Apache Kafka. A data category is represented by a Topic in Kafka.

[0038] Apache Spark: is an open source distributed stream computing big data engine used for unified analysis of large-scale data processing. It can obtain data from the message middleware, parse, calculate statistics, match and replace the obtained data, and support writing data back to the message middleware or database.

[0039] Data warehouse: A messaging system that is usually used to store, manage, query and analyze large amounts of historical data. Compared to traditional data warehouses, real-time data warehouses have greatly improved the timeliness of data processing, updating and querying.

[0040] According to an embodiment of the present application, a data storage method is provided.

[0041] Figure 1 1 is a flow chart of a data storage method provided according to an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:

[0042] Step S101: receiving a log data set sent by a collection device, wherein the log data set includes a plurality of log data, and the plurality of log data are classified and sent according to network protocol types.

[0043] It should be noted that the executor of this embodiment is the message middleware, and the collection device can be a network traffic collection device equipped with a hardware probe. The collection device sends the collected log data set to the message middleware, so that the log data set can be desensitized and stored through the message middleware.

[0044] Specifically, the log data set may include network traffic logs collected by the hardware probe. Each network traffic log can be regarded as a log data. After the hardware probe collects a certain number of traffic logs or traffic logs within a certain time period, the hardware probe packages the traffic logs to obtain a log data set, and sends the log data set to the message middleware, so that the log middleware can process the data in the log data set after receiving the log data set.

[0045] It should be noted that when the collection device receives log data and generates a log data set, it needs to group multiple log data with the same network protocol type into one log data set, so that the log data set corresponds to one network protocol type.

[0046] Step S102: performing a desensitization operation on the log data in the log data set to obtain an update data set.

[0047] Specifically, after the message middleware receives the log data set, it needs to parse the log data set to obtain multiple log data carried in the log data set, and each log data needs to be desensitized to ensure the security of user information.

[0048] It should be noted that when desensitizing log data, it is first necessary to identify whether the log data needs to be desensitized. At this time, a preset keyword set can be obtained, and it can be identified whether the log data contains keywords or data included in the keyword set. If the keywords or data exist, it is determined that the log data needs to be desensitized, thereby ensuring the comprehensiveness and accuracy of the desensitization operation on the log data.

[0049] It should be noted that the desensitizing operation can be performed in the message middleware, or the message middleware can send the log data that needs to be desensitized to the stream computing tool, desensitize the log data through the desensitization processing method set in the stream computing tool, and receive the desensitized data fed back by the stream computing tool, so as to package the desensitized data and obtain an updated data set.

[0050] It should be noted that when the data in the log data set does not contain keywords that need to be desensitized, there is no need to perform a desensitization operation on the log data set, and the data in the log data set can be directly sent to the database.

[0051] Step S103: Select a target database from the database cluster according to the network protocol type.

[0052] Specifically, after obtaining the update data set, the update data set needs to be stored. At this time, the network protocol type of the log data set can be determined first. The network protocol type can be a data transmission protocol, such as HTTP (Hypertext Transfer Protocol), TCP (Transmission Control Protocol), etc., and then the target database required to store the update data set is determined according to the data transmission protocol. Then, the log data under the same network protocol type can be stored in the same database according to the storage requirements to ensure the normal storage of the log data.

[0053] Step S104: store the updated data set in a local table of the target database.

[0054] Specifically, after determining the target database for storing the updated data set, a local table may be generated in the target database, and the updated data set may be stored in the local table of the target database, thereby completing the data storage operation.

[0055] The data storage method provided by the embodiment of the present application is through receiving the log data set sent by the collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to the network protocol type; the log data in the log data set is desensitized to obtain an update data set; the target database is selected from the database cluster according to the network protocol type; and the update data set is stored in the local table of the target database. The problem of low security and storage efficiency of storing traffic logs by hardware probes in the related technology is solved. By obtaining the log data set sent by the collection device and desensitizing the log data in the log data set, the security of sensitive data is guaranteed, and the log data is stored in the local table of the target database according to the network protocol type, so that the storage efficiency of the data can be guaranteed. At the same time, when expansion is needed, the database can be directly expanded or multiple databases can be used for data storage in a distributed database manner, thereby achieving the technical effect of improving the security and storage efficiency of log data storage.

[0056] Optionally, in the data storage method provided in the embodiment of the present application, before desensitizing the log data in the log data set to obtain the updated data set, the method also includes: randomly selecting sample log data from the log data set to obtain multiple sample log data, and identifying whether sensitive information exists in the multiple sample log data; if sensitive information exists in the multiple sample log data, performing the step of desensitizing the log data in the log data set; if sensitive information does not exist in the multiple sample log data, determining the log data set as an updated data set.

[0057] It should be noted that when performing desensitization operations on log data sets, since not all log data sets contain desensitization operations, in order to improve the processing efficiency of log data processing, before desensitizing the data, it is necessary to determine whether there is desensitized data in the log data in the log data set, and then determine whether it is necessary to perform desensitization operations on the log data set, and perform desensitization operations on the log data sets that need to be desensitized, and directly perform storage operations on the log data sets that do not need to be desensitized. Therefore, when desensitizing operations are performed on log data sets through stream computing tools, the data processing volume of the stream computing tools can be reduced, and the efficiency of data desensitization processing can be improved.

[0058] Specifically, when judging whether there is data that needs to be desensitized in the log data in the log data set, since the amount of data in the log data set is large, a certain number of sample log data can be randomly selected from the log data set, and each sample log data can be identified in turn to determine whether there is data that needs to be desensitized in the sample log data. In the case that sensitive information exists in multiple sample log data, it is characterized that there are sample log data that need to be desensitized. At this time, it is determined that the log data set needs to be desensitized.

[0059] It should be noted that when there is no log data that needs to be desensitized in each sample log data, the log data set can be directly stored in the target database without sending the log data set to the stream computing tool, thereby speeding up the storage process of the log data set and improving the efficiency of data storage.

[0060] This embodiment determines whether it is necessary to perform a desensitization operation on the log data set by randomly detecting the logs in the log data set, thereby achieving the technical effect of improving data processing efficiency and storage efficiency.

[0061] Optionally, in the data storage method provided in the embodiment of the present application, before desensitizing the log data in the log data set to obtain the updated data set, the method also includes: obtaining the source address of the log data set, and determining the target application to send the log data set based on the source address; judging whether the target application meets the preset filtering conditions; if the target application meets the preset filtering conditions, determining the log data set as an updated data set; if the target application does not meet the preset filtering conditions, executing the step of desensitizing the log data in the log data set.

[0062] It should be noted that when determining whether it is necessary to perform desensitization operations on a log data set, you can also determine the target application of the log data set, judge whether the log data set carries sensitive data based on the target application, and then determine whether it is necessary to perform desensitization operations on the log data set.

[0063] Specifically, you first need to obtain the source address of the log data set. When the log data set is captured by the network traffic hardware collection probe device, it will first perform a preliminary analysis of the data packet to extract its source address information. The source address is the sender's network location information contained in the log data set, which is used to identify the source of the data packet. Based on the source address, you can determine which application in which device generated the log data set. At this time, you can obtain the target application that generated the log data set, analyze the target application, and then identify whether the log data set needs to be desensitized.

[0064] Optionally, after determining the target application, it is possible to check whether the target application meets preset filtering criteria, where the filtering criteria may include application type, application content, application function, etc., and if the target application does not meet the preset filtering criteria, the target application is characterized as an application that frequently generates log data containing sensitive information. Therefore, all data packets sent by the application must be desensitized to ensure the security of user information.

[0065] This embodiment determines the source application of the log data set and determines whether the log data set needs to be desensitized based on the analysis results of the target application, thereby ensuring the security of user information while maintaining high efficiency of data storage and retrieval.

[0066] Optionally, in the data storage method provided in an embodiment of the present application, selecting a target database from a database cluster according to a network protocol type includes: determining a database in the database cluster corresponding to the network protocol type to obtain multiple candidate databases; obtaining the remaining storage space of each candidate database, and determining the candidate database with the largest remaining storage space as the target database.

[0067] Specifically, when determining the target database for storing the log data set, the network protocol type of the log data set can be identified first, such as HTTP, TCP, etc. Then, the message middleware will find the database designated for storing the network protocol type data from the database cluster, so that the selected databases form a candidate database list, which includes multiple candidate databases.

[0068] After determining the candidate databases, the message middleware can evaluate the remaining storage space of each candidate database. This step can be achieved by real-time monitoring or regular querying of the storage usage of each database in the database cluster. After obtaining the evaluation results of the remaining storage space, the message middleware will automatically select the candidate database with the largest remaining storage space as the target database, thereby ensuring maximum storage efficiency, avoiding storage bottlenecks caused by a single database filling up too quickly, and ensuring that data storage is more evenly distributed, improving the storage utilization of the entire database cluster.

[0069] Furthermore, after determining the target database, the processed log data set (whether it has been desensitized or not) can be written into the target database, usually into the local table of the real-time data warehouse Clickhouse, thereby utilizing the distributed storage characteristics and efficient writing mechanism of the real-time data warehouse to ensure that the data can be stored quickly and securely.

[0070] This embodiment selects the target database according to the network protocol type and the remaining storage space, which can not only intelligently allocate storage space according to the network protocol type of the log data, but also dynamically select the most suitable database in the database cluster for storage, thereby improving the efficiency and flexibility of data storage while ensuring the security of user information.

[0071] Optionally, in the data storage method provided in the embodiment of the present application, a log data set is stored in a first subject space, and a desensitizing operation is performed on the log data in the log data set to obtain an updated data set, including: obtaining each log data in the log data set from the first subject space in turn; determining whether there is sensitive data in the log data; if there is sensitive data in the log data, determining the data type of the sensitive data; determining a desensitizing strategy for changing the sensitive data based on the data type, and desensitizing the log data through the desensitizing strategy to obtain updated data; storing the updated data in a second subject space until all log data in the first subject space is moved to the second subject space; and combining the updated data in the second subject space into an updated data set.

[0072] Specifically, when desensitizing log data, since the message middleware stores the log data set in the first topic space, such as Raw_topic, when desensitizing log data, it is necessary to obtain each log data in the log data set from the first topic space, and after obtaining a log data, determine whether there is sensitive data in the log data, and if there is sensitive data, determine the data type of the sensitive data. For example, the data type may be an ID number, a name, a mobile phone number, etc., and determine a strategy for desensitizing the data based on the data type, and then perform a desensitization operation on the sensitive data in the log data according to the desensitization strategy to obtain updated data, that is, updated log data, and store the updated data in the second topic space, such as Processed_topic, so that after all log data in the first topic space have completed the desensitization operation, the updated data in the second topic space is packaged and processed to obtain an updated data set, thereby completing the desensitization operation on the data.

[0073] It should be noted that when desensitizing sensitive data, the data can be desensitized by regular matching, and regular expressions can be used to describe string features with specific patterns, find text containing string features with specific patterns, and then replace the feature strings in the text containing string features with specific patterns, thereby completing the desensitization operation of sensitive data. This embodiment is not limited to using the above desensitization method to desensitize data, and other desensitization methods are not limited in this embodiment.

[0074] It should be noted that the operation of desensitizing sensitive data can be performed in the message middleware, or the desensitizing operation can be performed through the stream computing tool by interacting the message middleware with the stream computing tool.

[0075] Figure 2 is a schematic diagram of a log data storage process containing sensitive data provided in an embodiment of the present application, such as Figure 2 As shown, when using the stream computing tool to perform the desensitization operation, multiple network traffic log data can be first collected by multiple network traffic collection devices, and the logs are sent to the first topic space Raw_topic in the message middleware by sending the log data set. At this time, the stream computing tool spark consumes the log data set in Raw_topic, and performs a desensitization operation on the log data in the log data set through the processing nodes in the stream computing tool spark to obtain the desensitized log data, and writes the desensitized log data into the second topic space Processed_topic, thereby obtaining an updated log data set, that is, an updated data set, thereby completing the desensitization operation of the log data set.

[0076] Figure 3 is a schematic diagram of a log data storage process that does not contain sensitive data provided by an embodiment of the present application, such as Figure 3 As shown, without the need to desensitize the log data set, after multiple network traffic collection devices collect network traffic log data and send the log to the first topic space Raw_topic in the message middleware by sending the log data set, the log data set can be directly stored in the first topic space Raw_topic, and the log data set can be sent from the first topic space Raw_topic to the target database, thereby completing the data storage operation.

[0077] This embodiment ensures data security by desensitizing sensitive data.

[0078] Optionally, in the data storage method provided in the embodiment of the present application, storing the updated data set in the local table of the target database includes: sending the updated data set to the engine table in the target database, and sending the updated data set to the materialized view table through the engine table; parsing the updated data set through the materialized view table to obtain multiple updated data, and storing the multiple updated data in the local table through a distributed table, wherein the distributed table is used to store the updated data in the local table, and store the metadata of the updated data in the metadata table of the target database.

[0079] Specifically, Figure 4 is a schematic diagram of a data storage process provided according to an embodiment of the present application, such as Figure 4 As shown, when storing the updated data set, it is first necessary to send the updated data set to the engine table in the target database, such as "CK_KafkaEnginTable". The engine table is the interface between Clickhouse and the message middleware Apache Kafka. It can consume data directly from Kafka and then write the data to Clickhouse, so that Clickhouse can interact directly with Kafka without going through an additional intermediate layer, thereby improving the data writing efficiency.

[0080] After sending the updated data set to the engine table, the engine table sends the updated data set to the materialized view table, such as "CK_MaterializedTable". The materialized view table can automatically copy data from one table to another, and can perform aggregation operations, data transformations, etc. during the copy process. The materialized view enables the necessary preprocessing of data to be completed when it is written, thereby speeding up the query speed and reducing the cost of real-time computing. When the materialized view table receives the updated data set, the materialized view table will parse the received updated data set and convert them into structured data, that is, multiple updated data. The parsing process may include converting the log data in JSON format into a row and column structure that Clickhouse can understand, as well as possible other data conversions to ensure that the data can be stored and queried efficiently.

[0081] The parsed updated data is then stored through Clickhouse's distributed table mechanism. A distributed table, such as "CK_all", is a virtual table that does not actually store data, but serves as an interface to access local tables stored on multiple nodes in the cluster. When data is written to a distributed table, the distributed table stores the updated data in a local table, such as "CK_local", completing the data storage operation.

[0082] It should be noted that when the distributed table receives updated data, it will assign the updated data to the local table according to the preset rules. Therefore, when the user needs to query data, the query conditions can be distributed to the local tables in each database through the distributed table. The local table will share the query task to quickly locate and retrieve data, thereby improving the efficiency of data query.

[0083] This embodiment ensures the secure storage, efficient management, and fast retrieval of network traffic log data through the above storage process, while avoiding single point failures and performance bottlenecks through distributed storage and computing, ensuring high availability and high scalability of the system.

[0084] Optionally, in the data storage method provided in the embodiment of the present application, after storing multiple updated data in the local table, the distributed table sends the metadata of each updated data to the metadata table in the first database having the same network protocol type as the target database.

[0085] Specifically, since the main function of distributed tables is to store source data and perform data queries through distributed tables, when data of the same network protocol type is stored in multiple databases, it is necessary to synchronize the storage contents of the metadata tables in the databases corresponding to the same network protocol type, so that when users query logs under a certain network protocol type, they can locate the storage location of the log and perform log acquisition operations by querying the metadata table of any target database.

[0086] Therefore, after the distributed table stores multiple updated data in the local table, it is necessary to synchronize the updated data to the metadata table in the first database with the same network protocol type as the target database, so that the metadata in the metadata tables in each first database is updated synchronously, thereby ensuring the accuracy of the user's query operation.

[0087] This embodiment achieves the technical effect of ensuring the accuracy of user query data by synchronizing data in distributed tables.

[0088] Figure 5 is a schematic diagram of an optional log data storage process provided according to an embodiment of the present application, such as Figure 5 As shown, a process for processing and storing HTTP log data containing sensitive data in network traffic data is provided.

[0089] Step 1: In the hardware environment, configure a network traffic hardware collection probe device in each network domain. This can be a switch port mirror traffic collection, or a network traffic splitter mirror traffic collection.

[0090] Step 2: Configure data sending tasks in all network traffic hardware collection probe devices in the data center, that is, network traffic collection devices, and send the HTTP protocol log data collected by these devices to the message middleware Apache Kafka cluster.

[0091] Step 3: The Kafka cluster creates two topics, HTTP_Raw_topic and HTTP_Processed_topic, for HTTP logs. HTTP_Raw_topic is used to receive HTTP raw logs containing sensitive data from network traffic hardware collection probes, and HTTP_Processed_topic is used to receive desensitized HTTP logs.

[0092] Step 4: Create an HTTP raw log desensitization processing application in the Apache Spark engine: Step 1, initialize SparkSession, configure Kafka docking parameters, preset the schema of json data in HTTP_Raw_topic, and set the regular expression mode according to the sensitive data contained in the existing business HTTP log; Step 2, read the data in HTTP_Raw_topic, parse the json string into Spark's DataFrame data, perform regular matching and replacement on sensitive data, and convert the processed DataFrame data into a json string; Step 3, write the processed json to HTTP_Processed_topic.

[0093] Step 5: Consume the desensitized HTTP_Processed_topic in Clickhouse, parse it and save it to the local table of Clickhouse. The steps are as follows: Step 1, create a local table HTTP_CK_local for HTTP logs in the Clickhouse cluster; Step 2, create a distributed table HTTP_CK_all based on the local table; Step 3, create Clickhouse's Kafka engine table HTTP_CK_KafkaEnginTable for HTTP_Processed_topic of HTTP logs. The Kafka engine table HTTP_CK_KafkaEnginTable consumes the data of HTTP_Processed_topic from the message middleware Kafka; Step 4, create a materialized view table HTTP_CK_MaterializedTable for the Kafka engine table HTTP_CK_KafkaEnginTable. The materialized view table HTTP_CK_MaterializedTable obtains data from the Clickhouse Kafka engine table HTTP_CK_KafkaEnginTable, parses it, and then writes it to the corresponding distributed table HTTP_CK_all created in the second step to complete the storage of data in the real-time data warehouse.

[0094] Figure 6 is a schematic diagram of another optional log data storage process provided according to an embodiment of the present application, such as Figure 6 As shown, a process for processing and storing HTTP log data that does not contain sensitive data in network traffic data is provided.

[0095] Step 1: In the hardware environment, configure a network traffic hardware collection probe device in each network domain. This can be a switch port mirror traffic collection, or a network traffic splitter mirror traffic collection.

[0096] Step 2: Configure data sending tasks in all network traffic hardware collection probes in the data center to send the TCP protocol log data collected by these probes to the message middleware Apache Kafka cluster.

[0097] Step 3: The Kafka cluster creates a topic called TCP_Raw_topic for the TCP log. TCP_Raw_topic is used to receive the TCP protocol raw logs sent from the network traffic hardware collection probe.

[0098] Step 4: Consume the data in TCP_Raw_topic in Clickhouse, parse it and save it to the local table of Clickhouse. The steps are as follows: Step 1, create a local table TCP CK local for TCP logs in the Clickhouse cluster; Step 2, create a distributed table TCP CK all based on the local table; Step 3, create Clickhouse's Kafka engine table TCP_CK_KafkaEnginTable for TCP_Raw_topic of TCP logs. The Kafka engine table tcp_CK_KafkaEnginTable consumes data from TCP_Raw_topic from the message middleware Kafka; Step 4, create a materialized view table TCP_CK_MaterializedTable for the Kafka engine table TCP_CK_KafkaEnginTable. The materialized view table TCP_CK_MaterializedTable obtains data from the Clickhouse Kafka engine table TCP_CK_KafkaEnginTable, parses it, and then writes it to the corresponding distributed table TCP_CK_all created in the second step, completing the storage of TCP protocol logs in the Clickhouse real-time data warehouse.

[0099] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0100] The embodiment of the present application also provides a data storage device. It should be noted that the data storage device of the embodiment of the present application can be used to execute the data storage method provided by the embodiment of the present application. The data storage device provided by the embodiment of the present application is introduced below.

[0101] Figure 7 is a schematic diagram of a data storage device provided according to an embodiment of the present application, such as Figure 7 As shown, the device includes: a receiving unit 71, a desensitizing unit 72, a first selection unit 73, and a storage unit 74.

[0102] The receiving unit 71 is used to receive a log data set sent by a collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to network protocol types.

[0103] The desensitizing unit 72 is used to perform a desensitizing operation on the log data in the log data set to obtain an updated data set.

[0104] The first selection unit 73 is used to select a target database from the database cluster according to the network protocol type.

[0105] The storage unit 74 is used to store the updated data set in a local table of the target database.

[0106] The data storage device provided in the embodiment of the present application receives the log data set sent by the collection device through the receiving unit 71, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to the network protocol type. The desensitization unit 72 performs a desensitization operation on the log data in the log data set to obtain an update data set. The first selection unit 73 selects the target database from the database cluster according to the network protocol type. The storage unit 74 stores the update data set in the local table of the target database. The problem of low security and storage efficiency of storing traffic logs by hardware probes in the related art is solved. By obtaining the log data set sent by the collection device and performing a desensitization operation on the log data in the log data set, the security of sensitive data is guaranteed, and the log data is stored in the local table in the target database according to the network protocol type, so that the storage efficiency of the data can be guaranteed. At the same time, when expansion is needed, the database can be directly expanded or multiple databases can be used for data storage in a distributed database manner, thereby achieving the technical effect of improving the security and storage efficiency of log data storage.

[0107] Optionally, in the data storage device provided in the embodiment of the present application, before performing a desensitizing operation on the log data in the log data set to obtain an updated data set, the device also includes: a second selection unit, used to randomly select sample log data from the log data set to obtain multiple sample log data, and identify whether sensitive information exists in the multiple sample log data; a first execution unit, used to execute the step of desensitizing the log data in the log data set when sensitive information exists in the multiple sample log data; and a first determination unit, used to determine the log data set as an updated data set when sensitive information does not exist in the multiple sample log data.

[0108] Optionally, in the data storage device provided in the embodiment of the present application, before performing a desensitizing operation on the log data in the log data set to obtain an updated data set, the device also includes: an acquisition unit, used to acquire the source address of the log data set, and determine the target application to send the log data set based on the source address; a judgment unit, used to judge whether the target application meets the preset filtering conditions; a second determination unit, used to determine the log data set as an updated data set if the target application meets the preset filtering conditions; and a second execution unit, used to execute the step of performing a desensitizing operation on the log data in the log data set if the target application does not meet the preset filtering conditions.

[0109] Optionally, in the data storage device provided in the embodiment of the present application, the first selection unit 73 includes: a first determination module, used to determine the database corresponding to the network protocol type in the database cluster to obtain multiple candidate databases; a second determination module, used to obtain the remaining storage space of each candidate database, and determine the candidate database with the largest remaining storage space as the target database.

[0110] Optionally, in the data storage device provided in the embodiment of the present application, the log data set is stored in the first subject space, and the desensitizing unit 72 includes: an acquisition module, which is used to acquire each log data in the log data set from the first subject space in turn; a judgment module, which is used to judge whether there is sensitive data in the log data; a third determination module, which is used to determine the data type of the sensitive data when there is sensitive data in the log data; a fourth determination module, which is used to determine the desensitizing strategy for changing the sensitive data according to the data type, and desensitize the log data according to the desensitizing strategy to obtain updated data; a first storage module, which is used to store the updated data in the second subject space until all the log data in the first subject space is moved to the second subject space; and a combination module, which is used to combine the updated data in the second subject space into an updated data set.

[0111] Optionally, in the data storage device provided in the embodiment of the present application, the storage unit 74 includes: a sending module, used to send the updated data set to the engine table in the target database, and send the updated data set to the materialized view table through the engine table; a second storage module, used to parse the updated data set through the materialized view table to obtain multiple updated data, and store the multiple updated data in the local table through the distributed table, wherein the distributed table is used to store the updated data in the local table, and store the metadata of the updated data in the metadata table of the target database.

[0112] Optionally, in the data storage device provided in the embodiment of the present application, after storing multiple updated data in the local table, the distributed table sends metadata of each updated data to a metadata table in a first database having the same network protocol type as the target database.

[0113] The above-mentioned data storage device includes a processor and a memory. The above-mentioned receiving unit 71, desensitizing unit 72, first selection unit 73, storage unit 74, etc. are all stored in the memory as program units, and the processor executes the above-mentioned program units stored in the memory to realize corresponding functions.

[0114] The processor includes a kernel, which calls the corresponding program unit from the memory. One or more kernels can be set, and the problem of low security and storage efficiency of storing traffic logs through hardware probes in related technologies is solved by adjusting kernel parameters.

[0115] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0116] An embodiment of the present invention provides a computer-readable storage medium on which a program is stored. When the program is executed by a processor, the data storage method is implemented.

[0117] An embodiment of the present invention provides a processor, which is used to run a program, wherein the data storage method is executed when the program is running.

[0118] Figure 8 is a schematic diagram of an electronic device provided according to an embodiment of the present application, such as Figure 8 As shown, an embodiment of the present invention provides an electronic device, the electronic device 80 includes a processor, a memory, and a program stored in the memory and executable on the processor, and the processor implements the steps of the above data storage method when executing the program. The device in this article can be a server, a PC, a PAD, a mobile phone, etc.

[0119] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the steps of the above data storage method.

[0120] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0121] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0122] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0123] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0124] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0125] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0126] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0127] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0128] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A data storage method, characterized in that: include: Receiving a log data set sent by a collection device, wherein the log data set includes multiple log data, and the multiple log data are classified and sent according to network protocol types; Performing a desensitization operation on the log data in the log data set to obtain an updated data set; Selecting a target database from a database cluster according to the network protocol type; The updated data set is stored in a local table of the target database.

2. The method according to claim 1, characterized in that Before performing a desensitization operation on the log data in the log data set to obtain an update data set, the method further includes: Randomly selecting sample log data from the log data set to obtain multiple sample log data, and identifying whether sensitive information exists in the multiple sample log data; In the case where the multiple sample log data contain the sensitive information, performing the step of desensitizing the log data in the log data set; In a case where the sensitive information does not exist in the plurality of sample log data, the log data set is determined as the update data set.

3. The method according to claim 1, characterized in that Before performing a desensitization operation on the log data in the log data set to obtain an update data set, the method further includes: Obtaining a source address of the log data set, and determining a target application to send the log data set according to the source address; Determining whether the target application meets the preset screening conditions; In a case where the target application meets the preset screening condition, determining the log data set as the update data set; When the target application does not meet the preset screening condition, the step of performing a desensitizing operation on the log data in the log data set is performed.

4. The method according to claim 1, characterized in that Selecting a target database from a database cluster according to the network protocol type includes: Determine a database in the database cluster corresponding to the network protocol type to obtain a plurality of candidate databases; The remaining storage space of each candidate database is obtained, and the candidate database with the largest remaining storage space is determined as the target database.

5. The method according to claim 1, characterized in that The log data set is stored in the first subject space, and the log data in the log data set is desensitized to obtain an updated data set including: Obtain each piece of log data in the log data set from the first subject space in sequence; Determining whether there is sensitive data in the log data; In the case where the sensitive data exists in the log data, determining the data type of the sensitive data; Determine a desensitization strategy for changing the sensitive data according to the data type, and perform desensitization processing on the log data using the desensitization strategy to obtain updated data; Storing the updated data in a second topic space until all log data in the first topic space is moved to the second topic space; The update data in the second subject space is combined into the update data set.

6. The method according to claim 1, characterized in that Storing the updated data set in a local table of the target database includes: Sending the updated data set to an engine table in the target database, and sending the updated data set to a materialized view table through the engine table; The updated data set is parsed through the materialized view table to obtain multiple updated data, and the multiple updated data are stored in the local table through a distributed table, wherein the distributed table is used to store the updated data in the local table, and store metadata of the updated data in the metadata table of the target database.

7. The method according to claim 6, characterized in that After storing the plurality of updated data in the local table, the distributed table sends metadata of each updated data to a metadata table in a first database having the same network protocol type as the target database.

8. A data storage device, characterized in that: include: A receiving unit, configured to receive a log data set sent by a collection device, wherein the log data set includes a plurality of log data, and the plurality of log data are classified and sent according to network protocol types; A desensitizing unit, used for performing a desensitizing operation on the log data in the log data set to obtain an updated data set; The first selection unit is used to select a target database from the database cluster according to the network protocol type; the storage unit is used to store the update data set in a local table of the target database.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the data storage method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein when the program is run, the data storage method according to any one of claims 1 to 7 is executed.

Images