A method for intelligently collecting logs and data and quickly forming relationship topology

Abstract

The present invention discloses a method for intelligently collecting logs and data and quickly forming a relational topology. It specifically includes the following steps: an external component is responsible for interacting with the kernel, and is responsible for obtaining effective data to construct logs and actively reporting without affecting the working efficiency of the kernel; a data acquisition module is responsible for indiscriminate real-time collection of various types of logs to obtain original log data blocks; a data aggregation module is responsible for sorting out the relationships of various types of real-time collected original log data according to the real existing association relationships, and constructing real and accurate relational standard data to achieve standardized output; a data request module is responsible for registering and logging in the transmission mode of external data requests and transmitting them to a third-party log analysis platform for log cleaning, log marking and log classification, and the data request after successful registration is responded to, otherwise it is invalid. The beneficial effects of the present invention are: improving data quality and content recognition, and reducing the calculation pressure of the calculation unit.

Description

Technical Field

[0001] The present invention relates to the technical field related to log collection, and in particular to a method for intelligently collecting logs and data and quickly forming a relationship topology. Background Art

[0002] Traditional log collection technology mainly collects data from the large amount of operating data generated by the operating system and application programs. After the collection is completed, the collected data is transmitted to a remote storage device for storage through data transmission methods and channels. The stored data is huge, disordered, and contains a large amount of redundant information, which causes a large storage burden. In the big data application scenario, it invisibly increases the work pressure of computing nodes and intelligent analysis-related operation modules, and there is resistance to the formation of effective data. In the process of building data relationships, there are certain misjudgments and information missing, which is not conducive to the comprehensive audit and analysis of the target devices for log extraction by operators.

[0003] In many cases, the purpose of auditing and analyzing the logs extracted from the target device is to satisfy the operators' need to troubleshoot the target device and track the operation behavior, so as to locate the problem, monitor in real time, and troubleshoot risks. Although system logs and related application logs can provide most of the operation data, they often lack key relational data and the data dimensions have certain limitations, which causes a certain degree of information loss in the monitoring and auditing of nodes. Summary of the invention

[0004] The present invention is to overcome the above-mentioned deficiencies in the prior art and provide a method for intelligently collecting logs and data and quickly forming a relationship topology to improve data quality and content recognition.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions:

[0006] A method for intelligently collecting logs and data and quickly forming a relationship topology. The component management module is business-driven. As a core module, the component management module provides registration and access entry for incremental services and drives, monitors and manages external components. All external components need to comply with the registration standards of the component management module. Specifically, the following steps are included:

[0007] (1) Log processing: The external component is responsible for interacting with the kernel, obtaining valid data to build logs and actively reporting them without affecting the kernel's work efficiency;

[0008] (2) Data collection: The data collection module is responsible for the real-time collection of various logs without distinction, and obtains the original log data blocks;

[0009] (3) Data aggregation: The data aggregation module is responsible for sorting out the relationships of various types of real-time collected raw log data according to the real association relationships, and constructing real and accurate relational standard data to achieve standardized output;

[0010] (4) Data request: The data request module is responsible for registering and logging in the transmission method of external data requests and transmitting them to the third-party log analysis platform for log cleaning, log marking and log classification. The data request will be responded to after successful registration, otherwise it will be invalid.

[0011] Based on diversified log information and other auxiliary information, intelligent data collection and aggregation methods are used to continuously, in real time and efficiently provide data users with comprehensive relational standardized data sources based on user roles as the execution root, including but not limited to operation behaviors, execution results, log records, network access and other multi-dimensional data, to build a real, effective, real-time synchronized, and well-defined data relationship association topology, so as to provide users with the ability to track and trace the ultimate responsible persons for various logs and behaviors, as well as the ability to efficiently access the relevant associated data of the current responsible persons, and provide users with aggregated standardized relational data through a variety of compatible methods, thereby improving data quality and content recognition and reducing the computing pressure of computing units.

[0012] Preferably, the component management module includes registration, component management, component monitoring and network communication, the registration includes component authentication, component testing and component loading, the component management includes enabling components, disabling components, restarting components and uninstalling components, the component monitoring includes checking the running status of components, and the network communication includes parsing and executing remote commands, uploading monitoring data and returning log data.

[0013] As a preferred embodiment, the specific operation method of the component management module is as follows:

[0014] (a) External components are accessed through the registration unit of the component management module. The registration unit of the component management module will perform an automated feasibility test on the external components that apply for access and add the external components to the component execution library. After successful addition, the external components will be copied to the NCA-autoScript component directory, and the components can be authenticated, tested and loaded. The external components are responsible for the collection and upload of logs of specified categories. During the production process of the external components, the applied component's unique electronic authentication certificate must be solidified into the external components, and the transmission method and data format must comply with the standards specified by the component management module.

[0015] (b) The component management module will authenticate the component through network communication during the registration process of the external component. The only authentication condition is the unique electronic authentication certificate solidified into the external component. After successful authentication, the external component will send test data to the communication interface of the component management module for data parsing and wait for receipt. If the content of the received receipt matches the content of the sent data, the external component will be moved from the NCA-autoScript to the CA-autoScript directory and marked as authenticated;

[0016] (c) The authenticated components are transmitted to the target device through network communication for use. The authenticated components loaded on each device are checked in the component list of the current device. The operator enables, disables, restarts, and uninstalls the relevant authenticated components of the specified device;

[0017] (d) The authenticated components in the target device will report heartbeat data to the monitoring module in the device in real time. If an abnormality is found in the authenticated component, the component abnormality information will be sent to the component management module and an attempt will be made to restart the component for repair. The component management module will alarm the authenticated component of the target device and highlight it in red. If the target device restores the normal use of the component through internal self-repair, it will send a service recovery notification to the component management module again. The component management module will stop the target abnormality alarm and cancel the related highlighting behavior.

[0018] Preferably, in step (2), each type of log is collected and supported by external components with different divisions of labor. The collected valid data is uploaded to the local storage for storage through network communication. The stored data is divided into two categories: unprocessed original data collected by the system and standardized data after data aggregation. The unprocessed original data collected by the system is stored in a timely manner. The standardized data after data aggregation is obtained by step (3). The standardized data after data aggregation is obtained by regularizing the cleaning of redundant data after real-time storage of the data. Then, the data model is used to build a relationship between the data reported by each authenticated component. The data model's reading of the data reported by the authenticated component depends on the data driver of each component. In the process of relationship construction, the data content is associated and the log content is classified based on this association relationship. Finally, the constructed relational data is formatted and output in a specified data output format.

[0019] Preferably, the data aggregation module includes data cleaning, data aggregation and data standardization, the data cleaning includes data duplication checking, redundancy cleaning and rule cleaning, the data aggregation includes driven learning, data extraction, relationship building and content reorganization, and the data standardization includes metadata formatting and output data format conversion.

[0020] Preferably, in step (3), the data processed by the data aggregation module and output in a standardized manner can be output to the data request module in real time, or can be temporarily stored and output to the data request module at a later time.

[0021] As a preferred method, the data aggregation module is to build relationships between the collected raw data, and to build unrelated raw data into relational data through real related content. The data aggregation module includes three major contents: data cleaning, data aggregation, and data standardization, as follows:

[0022] (31) The data cleaning function relies on the established data cleaning rules. Such rules are maintained by the component management module and configured in two ways: modularization and custom configuration for different devices. Users define the cleaning rules through the management interface of the component management module, mainly cleaning duplicate data, redundant data and cleaning content specified in the rules;

[0023] (32) The cleaned data is used as the data source for data aggregation. When performing data aggregation, it is necessary to ensure that the data aggregation module has the data drivers of all the external components that are functioning normally. The data aggregation module will input the log content provided by various components through the data drivers of each component. During the input process, the log files provided by the external components that have been loaded into the data model will be converted into identifiable data of the data model to extract valid data. The extracted data content will be analyzed by the data model for relationship association and form a relationship topology. The data content of each part will be defined and classified, and finally reorganized into new relational data and stored.

[0024] (33) After receiving the data request from the user, data standardization will extract the data content in the database according to the time rules based on the data format and maximum length requested by the user, forming standard formatted data with accompanying descriptions. The extracted relational data will eventually be packaged in the format required by the third party and sent to the third party, thereby improving the third party's recognition of the data content and reducing the computational pressure on the third party to sort out data relationships. To prevent data explosion, all extracted data will be deleted from the local database in order and will no longer be stored.

[0025] Preferably, the data request module includes real-time data requests, delayed data requests and metadata requests. The real-time data requests transmit real-time data back through whitelist requests, password authentication requests or certificate requests. The delayed data requests and metadata requests both transmit data back and query data through whitelist requests, password authentication requests or certificate requests.

[0026] As a preferred embodiment, the data request categories in the data request module are divided into three categories: real-time data, delayed data and metadata. Each type of data is accompanied by size data required to detect data integrity during transmission. The receiver can parse the size data and measure the data body to ensure data integrity and availability. The specific operation method is as follows:

[0027] (41) The third-party platform makes a request to obtain data from the operating device through identity authentication. The third-party device making such a request must add data request authorization to the target operating device through the management interface of the component management module. Otherwise, the target operating device will reject the data request service of the unauthorized third-party service and perform security control on the unauthorized addresses that are detected to make frequent requests;

[0028] (42) The storage node address for log data transmission is configured by the component management module for each device. It supports network transmission modes of both TCP and UDP protocols. For routine, unimportant, and easily outdated data, UDP active transmission is provided for automatic transmission. For high-value, highly structured important data, TCP is used to open the data acquisition request interface, and authorized and authenticated third parties can directly obtain data through the request interface.

[0029] The beneficial effects of the present invention are: constructing a real, effective, real-time synchronized, and clearly related data relationship association topology, thereby providing users with the ability to track and trace the ultimate responsible persons for various logs and behaviors, as well as the ability to efficiently access the relevant associated data of the current responsible person, and providing users with aggregated standardized relational data through a variety of compatible methods, thereby improving data quality and content recognition and reducing the computing pressure of the computing unit. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of the method of the present invention;

[0031] Figure 2 It is a structural block diagram of the component management module in the present invention;

[0032] Figure 3 It is a structural block diagram of the data aggregation module in the present invention;

[0033] Figure 4 It is a structural block diagram of the data request module in the present invention. DETAILED DESCRIPTION

[0034] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments.

[0035] like Figure 1In the described embodiment, a method for intelligently collecting logs and data and quickly forming a relationship topology is driven by a component management module. The component management module is a core module that provides registration and access portals for incremental services and drives, monitors and manages external components. All external components need to comply with the registration standards of the component management module. Unauthenticated external components synchronously collect source data, but such source data will not be added to the log collection of authorized components for data aggregation. The component management module includes registration, component management, component monitoring and network communication. Registration includes component authentication, component testing and component loading. Component management includes enabling components, disabling components, restarting components and uninstalling components. Component monitoring includes checking the running status of components. Network communication includes parsing and executing remote commands, uploading monitoring data and returning log data. Specifically, the following steps are included:

[0036] (1) Log processing: The external component is responsible for interacting with the kernel, obtaining valid data to build logs and actively reporting them without affecting the kernel's work efficiency;

[0037] (2) Data collection: The data collection module is responsible for real-time collection of various types of logs without distinction, and obtains raw log data blocks. Raw log data blocks refer to the general term for raw data sets that have not been standardized, and are used to provide raw data that can be analyzed to third-party log analysis platforms.

[0038] Each type of log is collected and supported by components with different divisions of labor. The collected valid data is uploaded to the local storage through internal communication services for storage. The stored data is divided into two categories: unprocessed raw data collected by the system and standardized data after data aggregation for use by third parties.

[0039] The original data is stored as it is collected without any secondary editing. However, due to component inconsistency, there are some differences in the content and format of each part of the data.

[0040] Standardized data is the process of cleaning redundant data in a regular manner after real-time storage of data. Then, the data reported by each component is constructed based on the data model. The model's reading of component reported data depends on the data driver of each component. In the process of relation construction, the data including but not limited to user IP, user name, user behavior, process tree, PID, PPID, quintuple, etc. are associated and the log content is classified based on this association. Finally, the constructed relational data is formatted and output in the specified data output format, which is obtained by step (3).

[0041] (3) Data aggregation: The data aggregation module is responsible for sorting out the relationships of various types of raw log data collected in real time according to the real existing association relationships, and constructing real and accurate relational standard data to achieve standardized output; the data aggregation module includes data cleaning, data aggregation and data standardization. Data cleaning includes data duplication checking, redundancy cleaning and rule cleaning. Data aggregation includes driven learning, data extraction, relationship construction and content reorganization. Data standardization includes metadata formatting and output data format conversion. The standardized output data processed by the data aggregation module can be output to the data request module in real time, or it can be temporarily stored and output to the data request module later;

[0042] The main task of the data aggregation module is to build relationships between the collected raw data, and to build unrelated raw data into easy-to-understand, intuitive and obvious relational data through real related content. Through relational data, we can clearly understand the relationship between content including but not limited to users, processes, ports, traffic, logs, etc., which is convenient for analyzing the causes of log generation and other related results. The data aggregation module has three major functions: data cleaning, data aggregation, and data standardization, as follows:

[0043] (31) The data cleaning function relies on the established data cleaning rules. Such rules are maintained by the component management module and can be configured in two ways: modularization and custom configuration for different devices. Users can define the cleaning rules through the management interface of the component management module, mainly cleaning duplicate data, redundant data and the cleaning content specified in the rules.

[0044] (32) The cleaned data is used as the data source for data aggregation. When performing data aggregation, it is necessary to ensure that the intelligent data aggregation module has the data drivers of all external components that are functioning normally. The data aggregation function will enter the log content provided by various components through the data drivers of each component. During the entry process, the log files provided by the external components that have been loaded into the data model will be converted into recognizable data of the data model for effective data extraction. The extracted data content will be analyzed by the data model for relationship association and form a relationship topology. The data content of each part will be defined and classified, and finally reorganized into new relational data and stored. This type of data is stored in a combination of noSql and mySql.

[0045] (33) After receiving the data request from the user, the data standardization function will extract the data content in the database according to the time rules based on the data format and maximum length requested by the user, forming standard formatted data with attached descriptions. The extracted relational data will eventually be packaged in the format required by the third party and sent to the third party, thereby improving the third party's recognition of the data content and reducing the computational pressure on the third party to sort out data relationships.

[0046] (34) To prevent data explosion, all extracted data will be deleted from the local database in order and will no longer be stored.

[0047] (4) Data request: The data request module is responsible for registering and logging in to the transmission method of external data requests and transmitting them to the third-party log analysis platform for log cleaning, log marking and log classification. The data request will be responded to after successful registration, otherwise it will be invalid; the data request module includes real-time data request, delayed data request and metadata request. Real-time data request transmits real-time data back through whitelist request, password authentication request or certificate request. Delayed data request and metadata request both transmit data back and query data through whitelist request, password authentication request or certificate request. The specific operation method is as follows:

[0048] (41) The external platform makes a request to obtain data from the operating equipment through identity authentication. The content of the request includes but is not limited to the data format, single data size, data transmission frequency, etc. The third-party device making such a request must add data request authorization (white list) to the target operating equipment through the management interface of the component management module. Otherwise, the target operating equipment will reject the data request service of the unauthorized third-party service and perform security control on the unauthorized addresses detected with frequent requests.

[0049] (42) The storage node address for log data transmission is configured by the component management module for each device, and supports network transmission modes of TCP and UDP protocols. For regular, unimportant, and easily outdated data, UDP active transmission is provided for automatic transmission. For high-value, highly structured and important data, TCP is used to open the data acquisition request interface, and authorized and authenticated third parties can directly obtain data through the request interface (such as FTP).

[0050] Data requests are divided into three categories: real-time data, delayed data, and metadata. Each type of data is accompanied by size data required to detect data integrity during transmission. The receiver can ensure data integrity and availability by parsing the size data and measuring the data body.

[0051] In addition to the above three categories, data requests are divided into two categories based on data format: standardized data requests and original data requests. The difference between the two is that original data is stored immediately after collection without being processed, while standardized data constructs standardized relational data through the association relationship between data. Through the format definition of standardized data, the original data can also be output to the data format specified by the user to increase the recognizability of the data.

[0052] The entire method relies on various log collection modules to collect data. In addition to the built-in kernel log collection, system log collection, and application log collection, it also provides a series of external component access functions to horizontally expand the scope of log collection. The external components accessed must meet the requirements of the component access module, otherwise they cannot be used normally. The specific operation methods of the component management module are as follows:

[0053] (a) External components are connected through the registration unit of the component management module. The registration unit of the component management module will perform automated feasibility tests on the external components that apply for access. External components can be added to the component execution library by entering the storage path of the component through the system function page and uploading the component file. After successful addition, the target component will be copied to the NCA-autoScript component directory. By clicking the "Register" button in the target component operation bar on the page, the component can be authenticated, tested and loaded. The function of the external component is mainly responsible for the collection and upload of logs of specified categories. During the production process of the external component, the applied component's unique electronic authentication certificate must be solidified into the component, and the transmission method and data format must comply with the standards specified by the component management module.

[0054] (b) The component management module will authenticate the component through network communication during the component registration process. The only authentication condition is the unique electronic authentication certificate embedded in the component. After successful authentication, the external component will send test data to the communication interface of the component management module for data parsing and wait for receipt. If the content of the received receipt matches the content of the sent data, the external component will be moved from the NCA-autoScript to the CA-autoScript directory and marked as authenticated. Administrators can enable, disable, and delete authenticated components in the library through the management interface.

[0055] (c) Authenticated components can be transmitted to the target device for use through communication services. The components loaded on each device can be viewed in the component list of the current device. Operators can enable, disable, restart, uninstall, and perform other operations on the relevant components of the specified device through the operation bar on the right side of the target component of the specified device. The operation functions of such components are independent and will only affect the data dimensions of the log collection of the specified device, and will not affect the core collection business.

[0056] (d) The components in the target device will report heartbeat data to the monitoring module in the device in real time. The heartbeat data includes but is not limited to the current timestamp, log collection target, number of log collection entries and other information. If the monitoring module finds that the component has an abnormal situation such as data expiration, it will send component abnormality information to the component management module and try to restart the component for repair. The component management module will alarm the abnormal operation of the component of the target device and highlight it in red. If the target device restores the normal use of the component through internal self-repair, it will send a service recovery notification to the component management module again. The component management module will stop the target abnormality alarm and cancel the related highlighting behavior.

[0057] (e) In addition to enabling, disabling, restarting, and uninstalling related components, the operations on the right side of the device list on the component management module function page also include enabling, disabling, and configuring the behavior of returning logs. The configuration items mainly configure the core business of the target device, including but not limited to the monitoring node address for uploading monitoring data, the storage node address for returning log data, local whitelist settings, and the configuration of parameter parsing and data driving of external components.

[0058] The core of the present invention lies in the log collection capability for various operating devices installed with Linux and Linux-like operating systems, including but not limited to. The specific collection scope of logs is affected by the number of extension components used by the operating device's own services, including but not limited to kernel logs, system logs, application logs, behavior logs and other logs not described.

[0059] Based on diversified log information and other auxiliary information, intelligent data collection and aggregation methods are used to continuously, in real time and efficiently provide data users with comprehensive relational standardized data sources based on user roles as the execution root, including but not limited to operation behaviors, execution results, log records, network access and other multi-dimensional data, to build a real, effective, real-time synchronized, and well-defined data relationship association topology, so as to provide users with the ability to track and trace the ultimate responsible persons for various logs and behaviors, as well as the ability to efficiently access the relevant associated data of the current responsible persons, and provide users with aggregated standardized relational data through a variety of compatible methods, thereby improving data quality and content recognition and reducing the computing pressure of computing units.

Claims

1. A method for intelligently collecting logs and data and quickly forming a relationship topology, characterized in that: The component management module drives the business. As the core module, the component management module provides registration and access entry for incremental services and drives, monitors and manages external components. All external components must comply with the registration standards of the component management module, which includes the following steps: (1) Log processing: The external component is responsible for interacting with the kernel, obtaining valid data to build logs and actively reporting without affecting the kernel's work efficiency; (2) Data collection: The data collection module is responsible for the real-time collection of various logs without distinction, and obtains the original log data blocks; (3) Data aggregation: The data aggregation module is responsible for sorting out the relationships of various types of real-time collected raw log data according to the real association relationships, and constructing real and accurate relational standard data to achieve standardized output; The data aggregation module builds relationships between the collected raw data and constructs unrelated raw data into relational data through real related content. The data aggregation module includes three major contents: data cleaning, data aggregation, and data standardization, as follows: (31) The data cleaning function relies on the established data cleaning rules. Such rules are maintained by the component management module and configured in two ways: modularization and custom configuration for different devices. Users define the cleaning rules through the management interface of the component management module, mainly cleaning duplicate data, redundant data and cleaning content specified in the rules; (32) The cleaned data is used as the data source for data aggregation. When performing data aggregation, it is necessary to ensure that the data aggregation module has the data drivers of all the external components that are functioning normally. The data aggregation module will input the log content provided by various components through the data drivers of each component. During the input process, the log files provided by the external components that have been loaded into the data model will be converted into identifiable data of the data model to extract valid data. The extracted data content will be analyzed by the data model for relationship association and form a relationship topology. The data content of each part will be defined and classified, and finally reorganized into new relational data and stored. (33) After receiving the data request from the user, data standardization will extract the data content in the database according to the time rules based on the data format and maximum length requested by the user, forming standard formatted data with accompanying descriptions. The extracted relational data will eventually be packaged in the format required by the third party and sent to the third party, thereby improving the third party's recognition of the data content and reducing the third party's computing pressure in sorting out data relationships. To prevent data explosion, all extracted data will be deleted from the local database in order and will no longer be stored. (4) Data request: The data request module is responsible for registering and logging in the transmission method of external data requests and transmitting them to the third-party log analysis platform for log cleaning, log marking and log classification. The data request will be responded to after successful registration, otherwise it will be invalid; The specific operation method of the component management module is as follows: (a) External components are connected through the registration unit of the component management module. The registration unit of the component management module will perform an automated feasibility test on the external components that apply for access and add the external components to the component execution library. After successful addition, the external components will be copied to the NCA-autoScript component directory, and the components can be authenticated, tested and loaded. The external components are responsible for the collection and upload of logs of specified categories. During the production process of the external components, the applied component's unique electronic authentication certificate must be solidified into the external components, and the transmission method and data format must comply with the standards specified by the component management module. (b) The component management module will authenticate the component through network communication during the registration process of the external component. The only authentication condition is the unique electronic authentication certificate that is fixed inside the external component. After successful authentication, the external component will send test data to the communication interface of the component management module for data parsing and wait for receipt. If the content of the received receipt matches the content of the sent data, the external component will be moved from the NCA-autoScript directory to the CA-autoScript directory and marked as authenticated; (c) The authenticated components are transmitted to the target device through network communication for use. The authenticated components loaded on each device are checked in the component list of the current device. The operator enables, disables, restarts, and uninstalls the relevant authenticated components of the designated device; (d) The authenticated components in the target device will report heartbeat data to the monitoring module in the device in real time. If an abnormality is found in the authenticated component, the component abnormality information will be sent to the component management module and an attempt will be made to restart the component for repair. The component management module will alarm the authenticated component of the target device for abnormal operation and highlight it in red. If the target device restores the normal use of the component through internal self-repair, it will send a service recovery notification to the component management module again. The component management module will stop the target abnormality alarm and cancel the related highlighting behavior.

2. According to claim 1, a method for intelligently collecting logs and data and quickly forming a relationship topology is characterized in that: The component management module includes registration, component management, component monitoring and network communication. The registration includes component authentication, component testing and component loading. The component management includes enabling components, disabling components, restarting components and uninstalling components. The component monitoring includes checking the running status of components. The network communication includes parsing and executing remote commands, uploading monitoring data and returning log data.

3. A method for intelligently collecting logs and data and quickly forming a relationship topology according to claim 1 or 2, characterized in that: In step (2), each type of log is collected and supported by external components with different divisions of labor. The collected valid data is uploaded to the local storage through network communication for storage. The stored data is divided into two categories: raw data collected by the system without processing and standardized data after data aggregation. The raw data collected by the system without processing is stored in a timely manner. The standardized data after data aggregation is obtained by step (3). The standardized data after data aggregation is obtained by regularizing the redundant data after real-time storage of the data, and then building a relationship between the data reported by each authenticated component based on the data model. The data model's reading of the data reported by the authenticated component depends on the data driver of each component. In the process of relationship construction, the data content is associated and the log content is classified based on this association relationship. Finally, the constructed relational data is formatted and output through the specified data output format.

4. According to claim 1, a method for intelligently collecting logs and data and quickly forming a relationship topology is characterized in that: The data aggregation module includes data cleaning, data aggregation and data standardization. The data cleaning includes data duplication checking, redundancy cleaning and rule cleaning. The data aggregation includes driven learning, data extraction, relationship building and content reorganization. The data standardization includes metadata formatting and output data format conversion.

5. The method for intelligently collecting logs and data and quickly forming a relationship topology according to claim 4 is characterized in that: In step (3), the data processed by the data aggregation module and output in a standardized manner can be output to the data request module in real time, or can be temporarily stored and output to the data request module at a later time.

6. The method for intelligently collecting logs and data and quickly forming a relationship topology according to claim 1, characterized in that: The data request module includes real-time data request, delayed data request and metadata request. The real-time data request transmits real-time data back through whitelist request, password authentication request or certificate request. The delayed data request and metadata request both transmit data back and query data through whitelist request, password authentication request or certificate request.

7. The method for intelligently collecting logs and data and quickly forming a relationship topology according to claim 6, characterized in that: The data request categories in the data request module are divided into three categories: real-time data, delayed data, and metadata. Each type of data is accompanied by size data required to detect data integrity during transmission. The receiver can parse the size data and measure the data body to ensure data integrity and availability. The specific operation method is as follows: (41) The third-party platform makes a request to obtain data from the operating device through identity authentication. The third-party device making such a request must add data request authorization to the target operating device through the management interface of the component management module. Otherwise, the target operating device will reject the data request service of the unauthorized third-party service and implement security control on the unauthorized addresses that are detected to make frequent requests; (42) The storage node address for log data transmission is configured by the component management module for each device. It supports network transmission modes of both TCP and UDP protocols. For routine, unimportant, and easily outdated data, UDP active transmission is provided for automatic transmission. For high-value, highly structured important data, TCP is used to open the data acquisition request interface, and authorized and authenticated third parties can directly obtain data through the request interface.

Images