A log data collection system, collection method and device
By configuring a different collector for each tenant, the problem that a single collector in the cloud security system cannot collect log data of the same IP address device under multiple tenants is solved, which improves data acquisition performance and realizes secure storage of log data.
Patent Information
- Application Number
- CN202211028931.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-08-26
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2042-08-26
AI Technical Summary
In cloud security systems, a single collector cannot effectively collect device log data of the same IP address under multiple tenants, resulting in a bottleneck in the acquisition performance and cannot meet the data volume requirements of a large number of tenants.
Through the management center, a different collector is configured for each tenant and the configured collector identifier is sent to the tenant, so that it collects and stores the log data of network security devices, realizing the log data collection and storage of devices with the same IP address under different tenants. .
It solves the bottleneck of performance of a single collector, improves the log data acquisition performance under a large number of tenants, and ensures the secure storage of log data of devices with the same IP address under different tenants in the log data acquisition system.
Smart Images

Figure CN115484156B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer communication technology, and in particular to a log data collection system, collection method and device. Background Art
[0002] With the development of cloud computing and cloud security, the use of multi-tenants is becoming more and more extensive, and cloud security systems also need to meet the needs of multi-tenant use. The log collection and analysis system involved in the cloud security system generally performs security analysis by collecting log data reported by network devices and terminals. In the case of a single user, the log collection and analysis system has only one collector, and it is deployed on the same server as the log collection system. After different devices are bound to the collector, they report log data to the collector. The collector caches the logs of different devices according to the device IP address, resulting in a collector that can only bind devices with different IP addresses, that is, when multiple devices with the same IP address report log data to the collector, the collector only recognizes the log data of one device.
[0003] As the number of tenants in cloud security scenarios increases, there will be business isolation and device isolation for different tenants, so there will be at least one device with the same IP address. In this case, for example, tenant A and tenant B are based on the same IP address IP1. When tenant A configures the device of IP1 and binds the collector, tenant B can no longer bind the device of IP1 to the collector, resulting in the collector being unable to collect the log data of tenant B, which in turn cannot meet the use of cloud security scenarios. That is, there is currently a problem of large data volume under a large number of tenants, and a single collector has a collection performance bottleneck.
[0004] Therefore, how to collect log data of devices with the same IP address under different tenants and improve the data collection performance under a large number of tenants is one of the technical issues worth considering. Summary of the invention
[0005] In view of this, the present application provides a log data collection system, collection method and device to realize the collection of log data of devices with the same IP address under different tenants, so as to improve the data collection performance under a large number of tenants.
[0006] Specifically, the present application is implemented through the following technical solutions:
[0007] According to a first aspect of the present application, a log data collection system is provided, comprising: a management center, a plurality of collectors and a data storage engine, wherein:
[0008] The management center is used to configure at least one required collector for the tenant after receiving the application request of the tenant; and send the collector identifier of the configured collector to the tenant, wherein different tenants are configured with different collectors;
[0009] The tenant's collector is used to collect log data of the tenant's network security device and send the collected log data to the data storage engine;
[0010] The data storage engine is used to store the received log data.
[0011] According to a second aspect of the present application, a log data collection method is provided, which is applied to a management center, and the method includes:
[0012] After receiving the application request from the tenant, configuring at least one collector required for the tenant;
[0013] Sending the configured collector identifier to the tenant, so that the tenant's collector collects log data of the tenant's network security device, and sends the collected log data to the data storage engine;
[0014] Different tenants are configured with different collectors.
[0015] According to a third aspect of the present application, a log data collection device is provided, which is arranged in a management center, and the device includes:
[0016] A configuration module, configured to configure at least one collector required for the tenant after receiving an application request from the tenant;
[0017] A sending module, used to send the configured collector identifier to the tenant, so that the tenant's collector collects the log data of the tenant's network security device, and sends the collected log data to the data storage engine;
[0018] Different tenants are configured with different collectors.
[0019] According to the fourth aspect of the present application, an electronic device is provided, comprising a processor and a machine-readable storage medium, wherein the machine-readable storage medium stores a computer program that can be executed by the processor, and the processor is prompted by the computer program to execute the method provided in the first aspect of the embodiment of the present application.
[0020] According to the fifth aspect of the present application, a machine-readable storage medium is provided, which stores a computer program. When called and executed by a processor, the computer program prompts the processor to execute the method provided in the first aspect of the embodiment of the present application.
[0021] Beneficial effects of the embodiments of the present application:
[0022] In the log data collection system, collection method and device provided in the embodiment of the present application, after receiving the application request of the tenant, the management center configures at least one required collector for the tenant; and sends the collector identifier of the configured collector to the tenant, so that after the tenant's collector collects the log data of the tenant's network security device, it can send the collected log data to the data storage engine in the log data collection system, so that the data storage engine stores the received log data. Since different tenants are configured with different collectors, the log data of network security devices with the same IP address under different tenants can be safely stored in the log data collection system, thereby solving the problem of collector performance bottleneck existing in a single collector, and at the same time improving the collection performance of log data under a large number of tenants. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] Figure 1 It is a structural diagram of a log data collection system provided by an embodiment of the present application;
[0024] Figure 2 It is a structural diagram of another log data collection system provided in an embodiment of the present application;
[0025] Figure 3 It is a flow chart of a log data collection method provided in an embodiment of the present application;
[0026] Figure 4 It is a structural schematic diagram of a log data collection device provided in an embodiment of the present application;
[0027] Figure 5 It is a hardware structure diagram of an electronic device implementing a log data collection method provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] Here, exemplary embodiments will be described in detail, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are only examples of devices and methods consistent with some aspects of the present application.
[0029] The terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more corresponding listed items.
[0030] It should be understood that although the terms first, second, third, etc. may be used in the present application to describe various information, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present application, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0031] The log data collection system provided by this application is described in detail below.
[0032] See also Figure 1 , Figure 1 This is a structural diagram of a log data collection system provided by the present application, which includes: a management center, multiple collectors and a data storage engine, wherein:
[0033] The management center is used to configure at least one required collector for the tenant after receiving the application request from the tenant; and send the collector identifier of the configured collector to the tenant, wherein different tenants are configured with different collectors;
[0034] The tenant's collector is used to collect log data of the tenant's network security device and send the collected log data to the data storage engine;
[0035] The data storage engine is used to store the received log data.
[0036] Specifically, in order to ensure that devices with the same IP address can store their collected log data in the log data collection system, this embodiment maintains a management center in the log data collection system, and the management center is responsible for managing the log data collection issues of tenants accessing the log data collection system. Specifically, when a tenant expects the log data collection system to manage and analyze the tenant's log data, an application request will be sent to the log data collection system. After receiving the application request, the management center will allocate a collector for collecting the tenant's log data to the tenant, and then send the collector identifier of the allocated collector to the tenant. In this way, when the tenant obtains the collector identifier allocated to it by the management center, it can configure each network security device locally, for example, configure the destination address reported by the log data to the address of the collector, so that after the network security device collects the log data, it sends the collected log data according to the configured destination address.
[0037] When the management center allocates collectors, in order to ensure that the log data of network security devices with the same IP can be stored in the log data collection system and can be stored separately, the management center will allocate collectors based on the principle of assigning different collectors to different tenants. For example, the management center can maintain a collector management list, which records all collector identifiers currently assigned to tenants in the current log data collection system, and records the online status of the collector in the collector management list, as shown in Table 1 below; in addition, the management center can also maintain a correspondence list, which records the correspondence between tenants and assigned collectors, so as to facilitate subsequent searches for collectors assigned to each tenant. Based on this, it can be ensured that network security devices with the same IP addresses under different tenants can report log data to their respective collectors, and because different tenants correspond to different collectors, there will be no log confusion problem for the log data of network security devices with the same IP addresses under different tenants.
[0038] After maintaining the collector management list, when an application request from a new tenant is received and a collector is configured for the new tenant, the collector ID of the collector can be written into the collector management list, and each collector can be monitored by maintaining its online status.
[0039] In addition, the tenant's collector requirements may change. For example, when more collectors are required, or the number of collectors used is reduced, the tenant will send a collector change request to the management center. In this way, when the management center receives the tenant's collector change request, if the collector change request is an increase request, since the number of collectors in the log data collection system provided by the present application is expandable, the management center can send the collector identifier of the newly added collector to the tenant, and write the newly assigned collector identifier into the collector management list; if the collector change request is a deletion request, the deletion request can carry the collector identifier that can be deleted. When the management center receives the deletion request, it can change the online status of the corresponding collector identifier in the collector management list to offline according to the collector identifier that needs to be deleted, and delete the corresponding relationship between the tenant and the collector that needs to be deleted in the corresponding relationship list.
[0040] Table 1
[0041] Collector ID Online Status Collector 1 Online Collector 2 Online Collector 3 Offline …… ……
[0042] Optionally, the tenant may carry application requirement information in the application request, and the application requirement information may include, but is not limited to, the number of collectors required by the tenant, the amount of log data actually required to be collected, and / or the number of network security devices maintained by the tenant, etc. On this basis, when configuring a collector for each tenant, the management center may configure a sufficient number of collectors for the tenant based on the application requirement information carried in the tenant's application, so that the configured collectors can meet the tenant's application requirements.
[0043] In addition, in order to store the log data collected by each tenant's collector, the log data collection system provided in this embodiment may also include a data storage engine. In this way, when each collector collects the log data of the corresponding network security device, the log data can be sent to the data storage engine so that the data storage engine can store the log data, thereby achieving the storage of security log data collected by each network security device under each tenant in the log data collection system.
[0044] By implementing the log data collection system provided by the present application, after receiving the application request from the tenant, the management center configures at least one required collector for the tenant; and sends the collector identifier of the configured collector to the tenant, so that after the tenant's collector collects the log data of the tenant's network security device, it can send the collected log data to the data storage engine in the log data collection system so that the data storage engine can store the received log data. Since different tenants are configured with different collectors, the log data of network security devices with the same IP address under different tenants can be safely stored in the log data collection system, thereby solving the collector performance bottleneck problem existing in a single collector, and at the same time improving the collection performance of log data under a large number of tenants.
[0045] It should be noted that the collector can be set up on the cloud, that is, each collector unit is deployed together with the management center in a network. In this way, the tenant's network security device needs to communicate with the collector in the public network, which may put forward higher requirements on the network security of the tenant's local network. In view of this, this embodiment also proposes a method for deploying the collector, that is, the collector can also be set in the local network of each tenant. By deploying the collector in the local network of the tenant, the tenant's local network security device only needs to interact with the collector deployed in the local network, that is, the collected log data is sent to the collector in the local network, and then the collector interacts with the data storage engine in the public network. In this way, each tenant's network security device will not need to communicate with the public network, and thus will not pose a threat to the security of the local network.
[0046] Specifically, when allocating a collector, the management center may send the collector component of the collector configured for the tenant to the tenant, so that the tenant can install the received collector component locally.
[0047] When the management center allocates a collector to a tenant, it will send the collector component of the collector to the tenant. Different collectors have different serial numbers for the collector components to distinguish them. In this way, after the tenant receives the collector component, it can install it in the local network to which the tenant belongs, so as to achieve the purpose of deploying the collector allocated by the management center in the local network. For details, please refer to Figure 2 As shown, Figure 2 The example in which the management center allocates two collectors to two tenants respectively is given. Tenant A and Tenant B are independent of each other. The management center allocates collector A to tenant A and collector B to tenant B. Collector A is deployed in the local network where tenant A is located, and collector B is deployed in the local network where tenant B is located. Figure 2It shows that the IP addresses (intranet IP addresses) of network security device 1 and network security device 2 deployed in the local network where tenant A is located are IP1 and IP2 respectively, and the IP addresses (intranet IP addresses) of network security device 1 and network security device 2 deployed in the local network where tenant B is located are IP1 and IP2 respectively. Since the collector identifiers of collector A and collector B are different, it can be achieved that the log data collected by network security device 1 of tenant A with intranet IP address IP1 and network security device 1 of tenant B can be stored in the log collection system. Similarly, it can be achieved that the log data collected by network security device 2 of tenant A with intranet IP address IP2 and network security device 2 of tenant B can be stored in the log collection system.
[0048] Optionally, based on any of the above embodiments, the management center in this embodiment may also perform the following process to facilitate the management of each tenant: after receiving the application request, the management center configures a tenant identifier for the tenant; on this basis, the management center receives the device binding request from the tenant; according to the binding request, binds the tenant identifier of the tenant, the intranet IP address of the tenant's network security device and the collector identifier of the collector configured for the tenant.
[0049] Specifically, after receiving the tenant's application request, the management center will configure a tenant ID for the tenant, generate the tenant management interface for the tenant and display it to the tenant. The tenant can trigger a device binding request on the tenant management interface, and then the management center can display a relationship binding interface to the tenant, so that the tenant can bind the tenant ID, the intranet IP address of the tenant's network security device and the collector ID on the relationship binding interface, so that the management center can better maintain the tenant and the collector.
[0050] Also Figure 2For example, tenant A can configure the network security device in the local network where tenant A is located in the tenant management interface displayed to it by the management center, and bind the collector to the network security device, thereby establishing a corresponding relationship between tenant A, the network security device and the collector. Specifically, tenant A can bind the intranet IP addresses of network security device 1 and network security device 2 in the local network to the collector identifier of collector A, thereby completing the establishment of a corresponding relationship between the tenant identifier, the intranet IP address of tenant A's network security device and the collector identifier; the binding method of the corresponding relationship between tenant B, the intranet IP address of the network security device and the collector identifier of the collector is similar, and will not be described in detail here. Therefore, since the network security devices configured by tenant A and tenant B are independent of each other and do not affect each other, the log data of the network security devices with the same intranet IP address can be stored normally in the log data collection system, and the isolation of the log data of the network security devices with the same IP address can be achieved only through the corresponding collector and tenant identifier.
[0051] It should be noted that when a tenant (tenant A or tenant B) needs multiple collectors, a drop-down selection box about the collector ID will be provided on the tenant management interface of the tenant when binding the above-mentioned corresponding relationship, so that the tenant can assign corresponding network security devices to each collector, so that all network security devices in the local network of the tenant are evenly distributed on each collector, thereby achieving load balancing of log data collection.
[0052] On this basis, when the collector is encapsulated as an independent collector component, when the management center receives the tenant's application request, the tenant management interface displayed to the tenant will provide a download method for the collector component, so that the tenant can download the collector component assigned to the tenant by the management center based on the tenant management interface. When the tenant downloads the collector component based on the tenant management interface of the tenant, the management center will generate a unique collector identifier for the collector component and then maintain it in the collector management list. After the tenant downloads the collector component, the collector component can be installed and deployed on the server in the local network where the tenant is located, and the registration of the tenant in the management center has been completed. It should be noted that tenants can also apply for multiple collectors according to the actual amount of log data that needs to be collected, and the management center has also realized the flexible expansion of the collector, thereby realizing the distributed collection of log data and greatly improving the collection performance of log data.
[0053] It is worth noting that the management center can monitor and maintain the online status of the collector through heartbeat.
[0054] Optionally, based on any of the above embodiments, in order to facilitate the storage of log data, the log data collection system provided in this embodiment may also include a data cache engine, please also refer to Figure 1 shown.
[0055] The collector is used to send the collected logs of the tenant's network security device to the data cache engine. The log data includes the collector identifier of the collector and the intranet IP address of the network security device.
[0056] The data cache engine is used to parse the collector identifier and the intranet IP address from the log data after receiving the log data; determine the corresponding tenant identifier according to the corresponding relationship; and isolate and cache the log data according to the determined tenant identifier and the parsed intranet IP address;
[0057] The data storage engine is specifically used to store the log data corresponding to each tenant in the data cache engine.
[0058] Specifically, since the log data collection system may maintain multiple tenants, the collector corresponding to each tenant will upload the acquired log data. In order to facilitate the unified management of log data and the efficiency of log storage, a data cache engine will be configured in the log data collection system to cache the log data uploaded by the collector into the data cache engine first, so as to temporarily reduce the storage pressure of the data storage engine.
[0059] In addition, the data cache engine can be implemented using Redis clusters, and the number of clusters can be expanded according to the amount of data, so as to better cache the log data sent by the collectors of each tenant. In addition, in order to isolate log data by tenant, the collector will configure the intranet IP address and collector ID of the corresponding network security device in the received log data. In this way, when the data cache engine receives the log data, it can parse the IP address and collector ID from the log data, and then call the corresponding relationship list from the management center to obtain the tenant ID corresponding to the parsed IP address and collector ID, so that when caching the log data, the log data is isolated and cached according to the tenant ID and IP address.
[0060] On this basis, the log data collection system provided in this embodiment may also include a data normalization processing engine, which is used to normalize the log data corresponding to each tenant in the data cache engine; and write the processed log data of each tenant into the data storage engine.
[0061] Specifically, in order to achieve unified management of log data, the data normalization engine is located between the data cache engine and the data storage engine. Figure 1As shown, the data normalization engine can read log data from the data cache engine, and then determine the business type of the tenant based on the tenant identifier and the intranet IP address, and then normalize the log data according to the business type, and process the log data into log data with a unified data structure, i.e., the processed log data. In order to facilitate the subsequent search of log data, the use of tenants and the isolation of log data, the data normalization processing engine will configure the tenant identifier in the processed log data, and then store it in the data storage engine. Furthermore, when the tenant queries the required log data based on the log data collection system, the log data can be quickly searched based on the tenant identifier.
[0062] It should be noted that the above data storage engine can use Clickhouse cluster, which has the advantages of high availability and good query performance. When the subsequent data analysis or third-party analysis system analyzes the log data of the tenant, it can quickly obtain the log data of the corresponding tenant from Clickhouse according to the tenant identifier, and then analyze the obtained log data, thereby realizing the isolation of the entire log data collection system data based on tenants.
[0063] By deploying the log data collection system provided by any embodiment of the present application, the problem of collecting and storing log data of devices with the same IP address under multiple tenants in the cloud security system is solved. In addition, the log data collection system provided by the present application can configure the collector according to the needs of the tenant, which can improve the collection performance of log data, and the present application can flexibly deploy the collector based on the needs of the tenant, which is easy to expand.
[0064] Based on the same inventive concept, this embodiment also provides a log data collection method, referring to Figure 3 As shown, it is a flow chart of the log data collection method provided in this embodiment, and the method applied in the management center may include the following steps:
[0065] S301. After receiving an application request from a tenant, configure at least one required collector for the tenant.
[0066] S302: Send the configured collector identifier to the tenant, so that the tenant's collector collects log data of the tenant's network security device, and sends the collected log data to the data storage engine.
[0067] Different tenants are configured with different collectors.
[0068] Optionally, step S301 may be performed according to the following process: sending a collector component of a collector configured for the tenant to the tenant, so that the tenant locally installs the received collector component.
[0069] Optionally, based on any of the above embodiments, the log data collection method provided in this embodiment may also include the following processes: after receiving the application request, configuring a tenant identifier for the tenant; receiving the tenant's device binding request; and according to the binding request, binding the tenant identifier of the tenant, the intranet IP address of the tenant's network security device, and the collector identifier of the collector configured for the tenant.
[0070] It should be noted that the implementation of any of the above embodiments may refer to the corresponding description of the management center in the log collection system, which will not be described in detail here.
[0071] By implementing the log data collection method provided by any embodiment of the present application, the problem of collecting and storing log data of devices with the same IP address under multiple tenants in the cloud security system is solved. In addition, the log data collection system provided by the present application can configure the collector according to the needs of the tenant, which can improve the collection performance of log data, and the present application can flexibly deploy the collector based on the needs of the tenant, which is easy to expand.
[0072] Based on the same inventive concept, the present application also provides a log data collection device corresponding to the above log data collection method. The implementation of the log data collection device can refer to the above description of the log data collection method, which will not be discussed here one by one.
[0073] See also Figure 4 , Figure 4 A log data collection device provided by an exemplary embodiment of the present application is provided in a management center. The device includes:
[0074] Configuration module 401, configured to configure at least one collector required for the tenant after receiving an application request from the tenant;
[0075] The sending module 402 is used to send the configured collector identifier to the tenant, so that the tenant's collector collects the log data of the tenant's network security device and sends the collected log data to the data storage engine;
[0076] Different tenants are configured with different collectors.
[0077] Optionally, the sending module 402 is specifically configured to send a collector component of a collector configured for the tenant to the tenant, so that the tenant can locally install the received collector component.
[0078] Optionally, based on any of the above embodiments, in this embodiment, the configuration module 401 is further used to configure a tenant identifier for the tenant after receiving the application request;
[0079] On this basis, the log data collection device provided in this embodiment also includes:
[0080] A receiving module (not shown in the figure), configured to receive a device binding request from the tenant;
[0081] A binding module (not shown in the figure) is used to bind the tenant identifier of the tenant, the intranet IP address of the tenant's network security device and the collector identifier of the collector configured for the tenant according to the binding request.
[0082] Based on the same inventive concept, an embodiment of the present application provides an electronic device that can integrate any of the following functions: the function of the management center, the function of the data cache engine, the function of the data normalization processing engine, and the function of the data storage engine. Figure 5 As shown, the electronic device includes a processor 501 and a machine-readable storage medium 502, wherein the machine-readable storage medium 502 stores a computer program that can be executed by the processor 501, and the processor 501 is prompted by the computer program to execute the message acquisition method provided in any embodiment of the present application. In addition, the electronic device also includes a communication interface 503 and a communication bus 504, wherein the processor 501, the communication interface 503, and the machine-readable storage medium 502 communicate with each other through the communication bus 504.
[0083] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0084] The communication interface is used for communication between the above electronic device and other devices.
[0085] The machine-readable storage medium 502 may be a memory, which may include a random access memory (RAM), a DDR SRAM (Double Data Rate Synchronous Dynamic Random Access Memory), or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the processor.
[0086] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0087] As for the electronic device and machine-readable storage medium embodiments, since the method contents involved are basically similar to the aforementioned method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0088] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0089] The implementation process of the functions and effects of each unit / module in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.
[0090] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can refer to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units / modules described as separate components may or may not be physically separated, and the components displayed as units / modules may or may not be physical units / modules, that is, they may be located in one place, or they may be distributed on multiple network units / modules. Some or all of the units / modules may be selected according to actual needs to achieve the purpose of the present application. A person of ordinary skill in the art can understand and implement it without creative work.
[0091] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.
Claims
1. A log data collection system, characterized in that: include: Management center, multiple collectors and data storage engines, including: The management center is used to configure at least one required collector for the tenant after receiving the application request of the tenant; and send the collector identifier of the configured collector to the tenant, wherein different tenants are configured with different collectors; The tenant's collector is used to collect log data of the tenant's network security device and send the collected log data to the data storage engine; The data storage engine is used to store the received log data; The management center is further used to configure a tenant identifier for the tenant after receiving the application request; The management center is also used to receive a device binding request from the tenant; and according to the binding request, bind the correspondence between the tenant identifier of the tenant, the intranet IP address of the tenant's network security device, and the collector identifier of the collector configured for the tenant; The system further includes: a data cache engine; The collector is specifically used to send the collected log data of the network security device of the tenant to the data cache engine, wherein the log data includes the collector identifier of the collector and the intranet IP address of the network security device; The data cache engine is used to parse the collector identifier and the intranet IP address from the log data after receiving the log data; determine the corresponding tenant identifier according to the corresponding relationship; and isolate and cache the log data according to the determined tenant identifier and the parsed intranet IP address; The data storage engine is specifically used to store log data corresponding to each tenant in the data cache engine.
2. The system according to claim 1, characterized in that The management center is specifically used to send the collector component of the collector configured for the tenant to the tenant, so that the tenant can install the received collector component locally.
3. The system according to claim 1, characterized in that Also includes: A data normalization processing engine, used to normalize the log data corresponding to each tenant in the data cache engine; And write the processed log data of each tenant into the data storage engine.
4. A log data collection method, characterized in that: Applied in a management center, the method comprises: After receiving the application request from the tenant, configuring at least one collector required for the tenant; Sending the configured collector identifier to the tenant, so that the tenant's collector collects log data of the tenant's network security device, and sends the collected log data to the data storage engine; Different tenants are configured with different collectors; The method further comprises: After receiving the application request, configuring a tenant identifier for the tenant; Receive a device binding request from the tenant; According to the binding request, a correspondence between the tenant identifier of the tenant, the intranet IP address of the network security device of the tenant and the collector identifier of the collector configured for the tenant is bound.
5. The method according to claim 4, characterized in that Sending the configured collector identifier to the tenant includes: The collector component of the collector configured for the tenant is sent to the tenant, so that the tenant can locally install the received collector component.
6. A log data collection device, characterized in that: The device is arranged in the management center, and includes: A configuration module, configured to configure at least one collector required for the tenant after receiving an application request from the tenant; A sending module, used to send the configured collector identifier to the tenant, so that the tenant's collector collects the log data of the tenant's network security device, and sends the collected log data to the data storage engine; Different tenants are configured with different collectors; The configuration module is further used to configure a tenant identifier for the tenant after receiving the application request; The device further comprises: A receiving module, configured to receive a device binding request from the tenant; The binding module is used to bind the tenant identifier of the tenant, the intranet IP address of the tenant's network security device and the collector identifier of the collector configured for the tenant according to the binding request.
7. The device according to claim 6, characterized in that The sending module is specifically used to send the collector component of the collector configured for the tenant to the tenant, so that the tenant can locally install the received collector component.
