A log data auditing system, method, device, and medium

Abstract

The application discloses a log data auditing system, method, device and medium. The system comprises: a data collection layer, configured to acquire log data of each application system, perform standardized processing, obtain classified log data, and send the classified log data to a data sedimentation layer; the data sedimentation layer is configured to classify and process the classified log data to obtain to-be-processed log data and send the to-be-processed log data to a data processing layer; the data processing layer is configured to determine to-be-converted log data corresponding to a target field from the to-be-processed log data, call a corresponding conversion rule to perform conversion processing on the to-be-converted log data, obtain to-be-analyzed log data, and send the to-be-analyzed log data to a data analysis layer; the data analysis layer is configured to determine to-be-processed log data associated with the to-be-analyzed log data, perform trajectory analysis, and determine an analysis result as a target audit result; and an application display layer is configured to determine a target display diagram based on the target audit result. The application realizes centralized auditing of log data of each application system.

Description

Technical Field

[0001] This invention relates to the field of log data auditing technology, and in particular to a log data auditing system, method, device and medium. Background Technology

[0002] To strengthen enterprise information security and ensure the safety of enterprise information systems and data, a certain industry attaches great importance to information security work. It uses a log auditing system to audit the enterprise's logs. The log auditing system is a system used to comprehensively collect logs (including operation, alarm, operation, message, status, etc.) generated by common security devices, network devices, databases, servers, application systems, hosts and other devices in the enterprise's IT system, and to store, monitor, audit, analyze, alarm, respond and report them.

[0003] Existing auditing systems, such as system auditing and application auditing, only audit logs of a specific application system within an enterprise's business support system. Their auditing methods are relatively simple and their auditing strategies are insufficient, resulting in low auditing efficiency. Summary of the Invention

[0004] This invention provides a log data auditing system, method, device, and medium to achieve centralized auditing of log data from all application systems on an enterprise's business system.

[0005] According to one aspect of the present invention, a log data auditing system is provided, the system comprising: a data acquisition layer, a data storage layer, a data processing layer, a data analysis layer, and an application presentation layer; wherein,

[0006] The data acquisition layer is used to acquire log data from various application systems, standardize the log data to obtain log data to be classified, and send the log data to be classified to the data sedimentation layer; wherein, the log data of the application systems includes: host log data, database log data, application log data, network log data, middleware log data, or security log data;

[0007] The data sedimentation layer is used to receive the log data to be classified, classify the log data to be classified based on preset classification rules to obtain log data to be processed, and send the log data to be processed to the data processing layer.

[0008] The data processing layer is used to receive the log data to be processed, determine the log data to be transformed corresponding to the target field from the log data to be processed, retrieve the transformation rule corresponding to the target field to transform the log data to be transformed to obtain the log data to be analyzed, and send the log data to be analyzed to the data analysis layer.

[0009] The data analysis layer is used to receive the log data to be analyzed, determine the log data to be processed associated with the log data to be analyzed, perform trajectory analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determine the analysis result as the target audit result, and send the target audit result to the application presentation layer.

[0010] The application presentation layer is used to receive the target audit result, determine the target display image corresponding to the target audit result based on the target audit result, and display the target display image.

[0011] According to another aspect of the present invention, a log data auditing method is provided. This method is applied to a log data auditing system, which includes a data acquisition layer, a data storage layer, a data processing layer, a data analysis layer, and an application presentation layer. The log data auditing method includes:

[0012] Log data from various application systems is acquired through the data acquisition layer, and the log data is standardized to obtain log data to be classified. The log data to be classified is then sent to the data sedimentation layer. The log data of the application systems includes: host log data, database log data, application log data, network log data, middleware log data, or security log data.

[0013] The data sedimentation layer receives the log data to be classified, performs classification processing on the log data to be classified based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to the data processing layer.

[0014] The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to the target field from the log data to be processed, retrieves the transformation rule corresponding to the target field to transform the log data to be transformed, so as to obtain the log data to be analyzed, and sends the log data to be analyzed to the data analysis layer.

[0015] The data analysis layer receives the log data to be analyzed, determines the log data to be processed associated with the log data to be analyzed, performs trajectory analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determines the analysis result as the target audit result, and sends the target audit result to the application presentation layer.

[0016] The application presentation layer receives the target audit results, determines the target display image corresponding to the target audit results, and displays the target display image.

[0017] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0018] At least one processor; and

[0019] A memory communicatively connected to the at least one processor; wherein,

[0020] The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the log data auditing method according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the log data auditing method according to any embodiment of the present invention.

[0022] The technical solution of this invention involves acquiring log data from various application systems through a data acquisition layer, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to a data sedimentation layer. The log data from the application systems includes host log data, database log data, application log data, network log data, middleware log data, or security log data. The data sedimentation layer receives the log data to be classified, classifies it based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to a data processing layer. The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to a target field from the log data to be processed, retrieves the transformation rules corresponding to the target field to transform the log data to be transformed to obtain log data to be analyzed, and sends the log data to be analyzed to a data sedimentation layer. The analysis layer receives the log data to be analyzed, identifies the log data to be processed associated with it, performs trajectory analysis based on the log data to be analyzed and the associated log data, and determines the analysis result as the target audit result, which is then sent to the application presentation layer. The application presentation layer receives the target audit result, determines the corresponding target display image based on it, and displays the target display image. This solves the problems of existing auditing methods that can only audit log data from a single application system, have a single audit strategy, and cannot centrally audit data from different sources or heterogeneous data, resulting in low audit efficiency. This new method enables centralized auditing of all application systems on the business support system, and is applicable to centralized auditing of data from different sources and heterogeneous data, improving audit efficiency and increasing audit strategies.

[0023] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0024] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a structural diagram of a log data auditing system provided in Embodiment 1 of the present invention;

[0026] Figure 2 This is a structural diagram of a log data auditing system provided in Embodiment 2 of the present invention;

[0027] Figure 3 This is a log data acquisition model diagram of a log data auditing system provided in Embodiment 2 of the present invention;

[0028] Figure 4 This is a log tag processing model diagram of a log data auditing system provided in Embodiment 2 of the present invention;

[0029] Figure 5 This is a design diagram of a distributed search system for log data auditing provided in Embodiment 2 of the present invention;

[0030] Figure 6 This is a design diagram of a dynamic field parsing system for log data auditing provided in Embodiment 2 of the present invention;

[0031] Figure 7 This is a flowchart of a log data auditing method provided in Embodiment 3 of the present invention;

[0032] Figure 8 This is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Before introducing this technical solution, let's illustrate the application scenario. This technical solution can be applied to situations requiring centralized auditing of log data from various application systems within a business support system. Log data auditing primarily aims to ensure the data security and normal operation of the business support system, which includes multiple application systems, such as application system A and application system B. Based on this, different application systems generate corresponding log data. The solution allows for the acquisition of log data from various application systems, centralized processing and analysis of the acquired log data from different sources, obtaining and displaying the analysis results, thereby achieving auditing of the business support system.

[0036] Example 1

[0037] Figure 1 This is a structural diagram of a log data auditing system provided in Embodiment 1 of the present invention. This embodiment is applicable to situations where centralized auditing of logs from various application systems is required. The log data auditing system can be executed by the log data auditing system, which can be implemented in hardware and / or software. The log data auditing system device can be configured on a PC or a mobile terminal.

[0038] like Figure 1 As shown, the system includes: a data acquisition layer, a data accumulation layer, a data processing layer, a data analysis layer, and an application presentation layer.

[0039] The data acquisition layer is used to acquire log data from various application systems, standardize the log data to obtain log data to be classified, and send the log data to be classified to the data sedimentation layer. The log data from the application systems includes: host log data, database log data, application log data, network log data, middleware log data, or security log data. Application systems can be various application systems on the business support system, such as a customer management system or other application systems. Enterprise users generate corresponding log data when using these application systems. Different application systems can be connected to the log data auditing system, and a cloud platform used to store the log data of various application systems can also be connected to the log data auditing system, thereby enabling the data acquisition layer of the application auditing system to acquire the log data from each application system. Specifically, corresponding ports can be configured between the log data auditing system and each application system to achieve log data transmission.

[0040] It's understandable that, since the application systems connected to the business support system are developed by different developers, the formats and mapping rules of the log data generated by these different application systems are generally different. Therefore, the acquired log data needs to be standardized. Standardization refers to unifying the log data from different application systems. This could involve unifying log data in different formats into a single format, for example, storing all log data in the .log format. The processed log data is then used as unclassified log data and sent to the data settling layer for further processing.

[0041] In this embodiment, each application system includes a corresponding host, database, application, network system, middleware, and security protection system. Correspondingly, it will generate corresponding log data, namely host log data, database log data, application log data, network log data, middleware log data, and security log data. Obtaining the various types of log data contained in the application system can better audit the application system's log data and ensure the comprehensiveness of the log data audit.

[0042] The data settling layer receives the log data to be classified, classifies it according to preset classification rules to obtain processed log data, and then sends the processed log data to the data processing layer. The preset classification rules can be understood as user-defined data classification rules used to classify the log data. After receiving the log data, the data settling layer can classify it according to the preset classification rules, enabling the data processing layer to process it more efficiently.

[0043] The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to the target field from the log data to be processed, retrieves the transformation rule corresponding to the target field to transform the log data to be transformed, and obtains the log data to be analyzed. The log data to be analyzed is then sent to the data analysis layer. The target field refers to a pre-defined field. If log data corresponding to the target field is detected in the log data to be processed, this portion of the log data can be extracted and used as the log data to be transformed. The transformation rule can be a processing rule for the log data to be processed. Different target fields correspond to different transformation rules. The transformation rules can be pre-stored in the log data auditing system. The log data to be transformed is then transformed using the log transformation rules, and the resulting data is the log data to be analyzed. Furthermore, the log data to be analyzed can be sent to the data analysis layer for analysis and processing.

[0044] The data analysis layer receives the log data to be analyzed, identifies log data to be processed associated with the log data to be analyzed, performs trajectory analysis based on the log data to be analyzed and the associated log data to be processed, determines the analysis result as the target audit result, and sends the target audit result to the application presentation layer. Log data to be processed refers to log data associated with the log data to be analyzed. It can be understood that different log data correspond to different user operations, and different operations are also related. When a user performs an operation on the application system, corresponding log data is generated. For example, after performing the first operation step, corresponding log data can be obtained; after performing the second operation step, corresponding log data can be obtained. Therefore, if the log data corresponding to a certain operation step is data to be analyzed, then the log data corresponding to the operation steps associated with that step can be considered data to be processed. In practical applications, backtracking can be performed based on the log data to be analyzed to identify the log data to be processed that is related to the log data to be analyzed. Further trajectory analysis can be performed based on the log data to be analyzed and the log data to be processed. Trajectory analysis can be understood as analyzing and determining the operation trajectory corresponding to the log data. For example, the operation trajectory of the user on the application system can be determined based on the log data to be analyzed and the log data to be processed.

[0045] For example, when a user deletes an item in an application system, corresponding log data is generated. If this log data is determined to be log data to be analyzed, then the associated log data to be processed can be identified. Furthermore, based on the log data to be analyzed and the log data to be processed, trajectory analysis can be performed to determine under what circumstances the user deleted the item and what operations were performed after deletion. In other words, the user's operation trajectory can be determined, and the operation trajectory can be used as the target audit result, which is then sent to the application presentation layer.

[0046] The application presentation layer receives the target audit results, determines a target display chart corresponding to the target audit results, and displays the target display chart. The target display chart can be a pie chart, line chart, or bar chart, which displays the audit results in this way. Backend users can use the target display chart to more clearly understand the frontend user's operations on the application system, determine whether the user has made a mistake, or whether the user is using the application system according to regulations, ensuring the normal operation of the application system.

[0047] Based on the above technical solution, the data acquisition layer is further used to perform log verification processing, automatic mapping processing, log parsing processing, or information completion processing on the log data after acquiring the log data from each application system, and to determine the processed log data as log data to be classified. In practical applications, firewall logs, intrusion detection system (IDS) logs, risk warning logs, server host logs, security gateway logs, application system logs, and database logs from various application systems can be centrally collected and connected to distributed cloud storage or a big data platform. The log data auditing system can then obtain the corresponding log data from the distributed cloud storage and big data platform, and perform the aforementioned log verification processing, automatic mapping processing, log parsing processing, or information completion processing on the log data. The purpose is to unify the log data from different application systems, and then use the processed log data as log data to be classified.

[0048] Based on the above technical solution, the data sedimentation layer is also used to classify the log data to be classified according to preset fields or organizational structure to obtain log data to be processed. The preset fields are field names pre-set by the user. After receiving the log data to be classified, all the log data can be classified according to these fields, such as classifying the log data according to field A and field B. Specifically, all the log data to be classified can be classified according to field A and field B, that is, the log data to be classified is divided into two different categories: one category of log data corresponds to field A, and the other category corresponds to field B. Alternatively, the log data can be analyzed according to the organizational structure of the application system. In the organizational structure, there are corresponding management users and ordinary users. The log data of ordinary users is in one category, and the log data of management users is in another category. Of course, the log data corresponding to management users or ordinary users can be further subdivided, depending on the actual situation.

[0049] Based on the above technical solution, the data processing layer is further configured to: if the field corresponding to the log data to be processed is the same as the target field, then determine that the log data to be processed is log data to be transformed; retrieve the target transformation rule corresponding to the target field, and perform transformation processing on the log data to be transformed based on the target transformation rule to obtain log data to be analyzed.

[0050] For example, the target field can represent some important fields of the business system. The content of the field corresponding to the target field is important data belonging to the business system. If a certain log data to be processed corresponds to the target field, it is treated as log data to be transformed, and the corresponding transformation rules are determined to process it, such as de-identifying important log data to be transformed.

[0051] Based on the above technical solution, the data analysis layer is further configured to: determine the corresponding log trajectory based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; analyze the log trajectory and the preset trajectory, and determine abnormal results, so as to use the abnormal results as the target audit results. Here, the log trajectory can be the user operation trajectory corresponding to the log, and the preset trajectory can be understood as a pre-set user operation trajectory. Specifically, after determining the log trajectory, if the log trajectory is inconsistent with the preset trajectory, it indicates that the user operation trajectory is abnormal, and the analyzed result can be used as an abnormal result, and the abnormal result can be used as the target audit result to execute subsequent display or warning.

[0052] Based on the above technical solution, the data analysis layer is further used to: determine the corresponding log trajectory based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; determine the operation items, operation steps, and operation frequency corresponding to the log data to be analyzed based on the log trajectory; establish an operation profile according to the operation items, operation steps, and operation frequency, and determine the operation profile as the target audit result, so that the application display layer can display the target audit result. Here, the operation item can be an operation item performed by the user in the application system, such as a login operation item; the operation steps and operation frequency can be the specific steps the user takes to perform the operation and the number of times the operation item is performed; the operation profile is used to represent the user's preference attributes in the application system's operation behavior. Specifically, by analyzing the log data to be analyzed and the log data to be processed using some log analysis techniques, the user's operation items, operation steps, and operation frequency corresponding to the log data can be obtained, thereby analyzing the user's operation habits and generating a corresponding profile. For example, after analyzing the log data, if the corresponding operation items include login and data maintenance, and the operation frequency is multiple times, then a profile of the user can be generated, representing that the user is usually a management user performing data maintenance in the application system. After creating the operational profile, it can be displayed through the application presentation layer.

[0053] Based on the above technical solution, the log data auditing system further includes a log search module; wherein, the log search module is used to split the log field to be searched to obtain at least two log fields to be used corresponding to the log field to be searched; and to determine the target log data corresponding to the log field to be searched based on the at least two log fields to be used and at least two distributed indexes.

[0054] The log search module is developed based on Solr technology. This module enables full-text search of logs. The field to be searched can be the field name corresponding to the log the user wants to find. For example, if a user wants to find log A in the log data auditing system, the field name corresponding to log A is the field to be searched. The user can input the field to be searched into the log search module, which will split it into three new fields. These new fields are then input into a distributed index (there can be multiple distributed indexes). The module then retrieves the corresponding results from these indexes and uses them as the target log data corresponding to the field to be searched. This method is a fuzzy full-text search, mainly based on distributed search and dynamic field parsing. When the distributed index is entered, there is only one field. During the query, the field can be temporarily split for searching, achieving a retrieval speed of millions of records and returning data in seconds. This solves the problem of large audit data volumes and low retrieval efficiency.

[0055] The technical solution of this invention involves acquiring log data from various application systems through a data acquisition layer, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to a data sedimentation layer. The log data from the application systems includes host log data, database log data, application log data, network log data, middleware log data, or security log data. The data sedimentation layer receives the log data to be classified, classifies it based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to a data processing layer. The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to a target field from the log data to be processed, retrieves the transformation rules corresponding to the target field to transform the log data to be transformed to obtain log data to be analyzed, and sends the log data to be analyzed to a data sedimentation layer. The analysis layer receives the log data to be analyzed, identifies the log data to be processed associated with it, performs trajectory analysis based on the log data to be analyzed and the associated log data, and determines the analysis result as the target audit result, which is then sent to the application presentation layer. The application presentation layer receives the target audit result, determines the corresponding target display image based on it, and displays the target display image. This solves the problems of existing auditing methods that can only audit log data from a single application system, have a single audit strategy, and cannot centrally audit data from different sources or heterogeneous data, resulting in low audit efficiency. This new method enables centralized auditing of all application systems on an enterprise information system, and is applicable to centralized auditing of data from different sources and heterogeneous data, improving audit efficiency and increasing audit strategies.

[0056] Example 2

[0057] Figure 2 This is a structural diagram of a log data auditing system provided in Embodiment 2 of the present invention. This embodiment is a preferred embodiment of the above embodiments, and its specific implementation can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0058] like Figure 2 As shown, the log data auditing system includes a data acquisition layer, a data storage layer, and a data processing layer.

[0059] Data analysis layer, application presentation layer, and data collection layer: mainly responsible for collecting logs from resources such as hosts, databases, applications, and networks;

[0060] Data sedimentation layer: preprocesses the collected logs;

[0061] Data processing layer: Processes the collected logs using tags and indexes;

[0062] Data analysis layer: analyzes and processes logs through risk modeling, trajectory analysis, and anomaly detection; Application presentation layer: a portal accessed by users to display audit analysis results.

[0063] Centralized Log Design

[0064] Construct a centralized management system for the unified collection, distributed processing, and storage of heterogeneous data, realizing the transformation from the current independent collection by each system to a centralized big data architecture, forming a distributed, reliable, and highly available massive log aggregation platform system.

[0065] This platform not only supports customizing various data senders within the system for data collection, but also has the ability to customize various data receivers. It also features log standardization (log verification, automatic mapping, parsing, and information completion), data filtering, and log preprocessing.

[0066] Its detailed acquisition model is as follows: Figure 3 As shown:

[0067] Log tagging design

[0068] Based on data tagging, a flexible and customizable thematic audit model is realized. It supports the combination and superposition of log types and log tags to form audit log decision tree mining and analysis strategies, which can be applied to scenarios such as front-end and back-end personnel operation profiling and user behavior trajectory analysis. This not only fulfills the information security audit and audit requirements for actual business support scenarios, but also greatly improves the coverage and efficiency of security audits.

[0069] Log tag processing model such as Figure 4 As shown:

[0070] User behavior data modeling design

[0071] By comprehensively analyzing the characteristics of the system, we planned and implemented an industry-leading user operation behavior data model. By integrating data from multiple sources, we established a complete human behavior chain model, which has the ability to fully describe events such as "human", "source terminal", "access channel", "accessed resources" and "what kind of operation".

[0072] The system centrally processes logs across six categories within the business support system: host logs, application logs, database logs, middleware logs, network logs, and security logs. Based on the fact that "abnormal behavior is sparse among all operational behaviors," the research constructs an algorithm. Utilizing a "behavioral relationship network" data structure, it constructs a behavior flow distribution graph in space based on all offline logs from HDFS. This analysis identifies abnormal flow behaviors with sparse flow between behavioral units, thereby filtering out all abnormal user behavior logs related to system operations.

[0073] Fuzzy full-text search

[0074] Full-text search of security logs is achieved based on Solr technology, with a search performance of millions of records and data return in seconds, solving the problems of large audit data volume and low search efficiency.

[0075] The core of fuzzy full-text search is based on distributed search and dynamic field parsing, where the design of distributed search is as follows: Figure 5 As shown: The design of dynamic field parsing is as follows Figure 6 As shown:

[0076] The technical solution of this invention involves acquiring log data from various application systems through a data acquisition layer, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to a data sedimentation layer. The log data from the application systems includes host log data, database log data, application log data, network log data, middleware log data, or security log data. The data sedimentation layer receives the log data to be classified, classifies it based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to a data processing layer. The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to a target field from the log data to be processed, retrieves the transformation rules corresponding to the target field to transform the log data to be transformed to obtain log data to be analyzed, and sends the log data to be analyzed to a data sedimentation layer. The analysis layer receives the log data to be analyzed, identifies the log data to be processed associated with it, performs trajectory analysis based on the log data to be analyzed and the associated log data, and determines the analysis result as the target audit result, which is then sent to the application presentation layer. The application presentation layer receives the target audit result, determines the corresponding target display image based on it, and displays the target display image. This solves the problems of existing auditing methods that can only audit log data from a single application system, have a single audit strategy, and cannot centrally audit data from different sources or heterogeneous data, resulting in low audit efficiency. This new method enables centralized auditing of all application systems on an enterprise information system, and is applicable to centralized auditing of data from different sources and heterogeneous data, improving audit efficiency and increasing audit strategies.

[0077] Example 3

[0078] Figure 7 This is a flowchart of a log data auditing method provided in Embodiment 3 of the present invention. This embodiment is applicable to situations requiring centralized auditing of logs from various application systems. It can be executed by a log data auditing system, which can be implemented in hardware and / or software. The log data auditing system device can be configured on a PC or a mobile terminal. Figure 7 As shown, the method includes:

[0079] S310. Obtain log data from each application system through the data acquisition layer, standardize the log data to obtain log data to be classified, and send the log data to be classified to the data sedimentation layer.

[0080] The log data from application systems includes: host log data, database log data, application log data, network log data, middleware log data, and security log data. Application systems can be various application systems within a business support system, such as a customer management system or other application systems. Enterprise users generate corresponding log data when using these application systems. Different application systems can be connected to the log data auditing system. Additionally, a cloud platform used to store the log data of various application systems can be connected to the log data auditing system, thereby enabling the data acquisition layer of the application auditing system to obtain the log data from each application system.

[0081] Specifically, the log data auditing system can be configured with corresponding ports to transmit log data to various application systems. The acquired log data undergoes standardization processing, which means unifying log data from different application systems. This can involve unifying log data in different formats into a single format, such as storing all log data in .log format. The processed log data is then used as unclassified log data and sent to the data settling layer for further processing.

[0082] S320. Receive log data to be classified through the data sedimentation layer, classify the log data to be classified according to the preset classification rules to obtain log data to be processed, and send the log data to be processed to the data processing layer.

[0083] Specifically, the preset classification rules can be understood as the data classification rules set by the user in advance. These rules are used to classify the log data to be classified. After the data sedimentation layer receives the log data to be classified, it can classify the log data according to the preset classification rules so that the data processing layer can process it more efficiently afterward.

[0084] S330. Receive log data to be processed through the data processing layer, determine the log data to be transformed corresponding to the target field from the log data to be processed, retrieve the transformation rule corresponding to the target field to transform the log data to be transformed, so as to obtain the log data to be analyzed, and send the log data to be analyzed to the data analysis layer.

[0085] Here, the target field refers to the pre-set field, and the transformation rule can be the processing rule for the log data to be processed. Different target fields correspond to different transformation rules.

[0086] Specifically, if log data corresponding to a target field is detected within the log data to be processed, this portion of log data can be extracted and used as log data to be transformed. The transformation rules can be pre-stored in the log data auditing system. The log data to be transformed is then processed according to these rules, resulting in the log data to be analyzed. Furthermore, the log data to be analyzed can be sent to the data analysis layer for further analysis and processing.

[0087] S340. Receive the log data to be analyzed through the data analysis layer, determine the log data to be processed associated with the log data to be analyzed, perform trajectory analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determine the analysis result as the target audit result, and send the target audit result to the application presentation layer.

[0088] The log data to be processed refers to the log data associated with the log data to be analyzed.

[0089] Specifically, different log data correspond to different user operations, and these operations are also correlated. When a user performs an operation on the application system, corresponding log data is generated. For example, after the user performs the first operation step, corresponding log data can be obtained; after performing the second operation step, corresponding log data can be obtained. If the log data corresponding to a certain operation step is data to be analyzed, then the log data corresponding to the operation steps associated with that step can be considered data to be processed. In practical applications, backtracking can be performed based on the log data to be analyzed to determine the log data to be processed related to it. Further trajectory analysis can be performed based on the log data to be analyzed and the log data to be processed. Trajectory analysis can be understood as analyzing and determining the operation trajectory corresponding to the log data. For example, based on the log data to be analyzed and the log data to be processed, the user's operation trajectory on the application system can be determined.

[0090] S350. Receive the target audit results through the application presentation layer, determine the target display image corresponding to the target audit results based on the target audit results, and display the target display image.

[0091] The target display chart can be a pie chart, line chart, or bar chart.

[0092] Specifically, the audit results can be displayed using the target visualization diagram mentioned above. Backend users can gain a clearer understanding of the frontend users' operations on the application system based on the target visualization diagram, determine whether the user has made a mistake, or whether the user is using the application system in accordance with the regulations, and ensure the normal operation of the application system.

[0093] The technical solution of this invention involves acquiring log data from various application systems through a data acquisition layer, standardizing the log data to obtain log data to be classified, and sending the log data to be classified to a data sedimentation layer. The log data from the application systems includes host log data, database log data, application log data, network log data, middleware log data, or security log data. The data sedimentation layer receives the log data to be classified, classifies it based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to a data processing layer. The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to a target field from the log data to be processed, retrieves the transformation rules corresponding to the target field to transform the log data to be transformed to obtain log data to be analyzed, and sends the log data to be analyzed to a data sedimentation layer. The analysis layer receives the log data to be analyzed, identifies the log data to be processed associated with it, performs trajectory analysis based on the log data to be analyzed and the associated log data, and determines the analysis result as the target audit result, which is then sent to the application presentation layer. The application presentation layer receives the target audit result, determines the corresponding target display image based on it, and displays the target display image. This solves the problems of existing auditing methods that can only audit log data from a single application system, have a single audit strategy, and cannot centrally audit data from different sources or heterogeneous data, resulting in low audit efficiency. This new method enables centralized auditing of all application systems on an enterprise information system, and is applicable to centralized auditing of data from different sources and heterogeneous data, improving audit efficiency and increasing audit strategies.

[0094] Example 4

[0095] Figure 8 This is a schematic diagram of an electronic device according to Embodiment 4 of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0096] like Figure 8As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded into the RAM 43 from storage unit 48. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0097] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0098] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as log data auditing methods.

[0099] In some embodiments, the log data auditing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the log data auditing method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the log data auditing method by any other suitable means (e.g., by means of firmware).

[0100] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0101] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0102] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0103] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0105] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0106] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A log data auditing system, characterized in that, include: The system comprises a data acquisition layer, a data accumulation layer, a data processing layer, a data analysis layer, an application presentation layer, and a log search module; among these, The data acquisition layer is used to acquire log data from various application systems, standardize the log data to obtain log data to be classified, and send the log data to be classified to the data sedimentation layer; wherein, the log data of the application systems includes: host log data, database log data, application log data, network log data, middleware log data, or security log data; The data sedimentation layer is used to receive the log data to be classified, classify the log data to be classified based on preset classification rules to obtain log data to be processed, and send the log data to be processed to the data processing layer. The data processing layer is used to receive the log data to be processed, determine the log data to be transformed corresponding to the target field from the log data to be processed, retrieve the transformation rule corresponding to the target field to transform the log data to be transformed to obtain the log data to be analyzed, and send the log data to be analyzed to the data analysis layer. The data analysis layer is used to receive the log data to be analyzed, determine the log data to be processed associated with the log data to be analyzed, perform trajectory analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determine the analysis result as the target audit result, and send the target audit result to the application presentation layer. The application presentation layer is used to receive the target audit result, determine the target display image corresponding to the target audit result based on the target audit result, and display the target display image; The data processing layer is further configured to: if the field corresponding to the log data to be processed is the same as the target field, then determine that the log data to be processed is log data to be transformed; retrieve the target transformation rule corresponding to the target field, and perform transformation processing on the log data to be transformed based on the target transformation rule to obtain log data to be analyzed; The log search module is used to split the log field to be searched to obtain at least two log fields to be used corresponding to the log field to be searched; and to determine the target log data corresponding to the log field to be searched based on the at least two log fields to be used and at least two distributed indexes; wherein, the log search module implements fuzzy full-text search based on distributed search and dynamic field parsing, and only stores a single field when entering the distributed index, and temporarily decomposes the field for querying during the query.

2. The system according to claim 1, characterized in that, The data acquisition layer is also used for: After obtaining the log data from each of the application systems, the log data is processed by log verification, automatic mapping, log parsing, or information completion, and the processed log data is determined as log data to be classified.

3. The system according to claim 1, characterized in that, The data sedimentation layer is also used for: The log data to be classified is classified based on preset fields or organizational structure to obtain log data to be processed.

4. The system according to claim 1, characterized in that, The data analysis layer is also used for: The corresponding log trajectory is determined based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; The log trajectory and preset trajectory are analyzed to identify abnormal results, which are then used as the target audit results.

5. The system according to claim 1, characterized in that, The data analysis layer is also used for: The corresponding log trajectory is determined based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed; Based on the log trajectory, determine the operation items, operation steps, and operation frequency corresponding to the log data to be segmented; An operation profile is created based on the operation items, operation steps, and operation frequency, and the operation profile is determined as the target audit result so that the application presentation layer can display the target audit result.

6. A log data auditing method, characterized in that, This is applied to a log data auditing system, which includes a data acquisition layer, a data storage layer, a data processing layer, a data analysis layer, an application presentation layer, and a log search module. The log data auditing method includes: Log data from various application systems is acquired through the data acquisition layer, and the log data is standardized to obtain log data to be classified. The log data to be classified is then sent to the data sedimentation layer. The log data of the application systems includes: host log data, database log data, application log data, network log data, middleware log data, or security log data. The data sedimentation layer receives the log data to be classified, performs classification processing on the log data to be classified based on preset classification rules to obtain log data to be processed, and sends the log data to be processed to the data processing layer. The data processing layer receives the log data to be processed, determines the log data to be transformed corresponding to the target field from the log data to be processed, retrieves the transformation rule corresponding to the target field to transform the log data to be transformed, so as to obtain the log data to be analyzed, and sends the log data to be analyzed to the data analysis layer. The data analysis layer receives the log data to be analyzed, determines the log data to be processed associated with the log data to be analyzed, performs trajectory analysis based on the log data to be analyzed and the log data to be processed associated with the log data to be analyzed, determines the analysis result as the target audit result, and sends the target audit result to the application presentation layer. The application presentation layer receives the target audit results, determines the target display image corresponding to the target audit results, and displays the target display image. The method further includes: The data processing layer determines whether the field corresponding to the log data to be processed is the same as the target field. If the field corresponding to the log data to be processed is the same as the target field, the log data to be processed is determined to be log data to be transformed. The target transformation rule corresponding to the target field is retrieved, and the log data to be transformed is transformed based on the target transformation rule to obtain the log data to be analyzed. The log search module splits the log field to be searched to obtain at least two log fields to be used corresponding to the log field to be searched; based on the at least two log fields to be used and at least two distributed indexes, the target log data corresponding to the log field to be searched is determined; wherein, the log search module implements fuzzy full-text search based on distributed search and dynamic field parsing, and only stores a single field when entering the distributed index, and temporarily decomposes the field for querying during the query.

7. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory that is communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the log data auditing method of claim 6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that are used to cause a processor to execute the log data auditing method of claim 6.

Images