Audit log processing system and method, computer equipment and storage medium
By adopting a functional hierarchical governance architecture and multiple access methods in the audit log processing system, the problems of poor concurrency capabilities and single data storage in traditional systems are solved, and high concurrency and multi-store compatibility is achieved, suitable for complex and diversified purposes business environment.
Patent Information
- Application Number
- CN202411253454.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-09
- Publication Date
- 2024-12-31
AI Technical Summary
The traditional audit log processing system is closely coupled with the business system, which has problems such as poor concurrency capabilities and single data storage backend, and cannot be applied to complex and diverse business environments.
An audit log processing system was designed, adopting a functional layered governance architecture, including the access layer, the service layer and the storage layer. The access layer provides a variety of access means, the service layer handles high concurrent requests and dynamic configuration management, and the storage layer Manage structured data storage and support multiple structured data storage backends.
It realizes high concurrent processing of audit logs and compatibility of multiple structured data storage, provides flexible access methods to decouple from the business system, and improves the system's concurrency capabilities and adaptability.
Smart Images
Figure CN119226079A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of computer technology, and specifically relates to an audit log processing system, method, computer equipment and storage medium. Background Art
[0002] In an era of rapid development of computer technology, system security has received increasing attention. Audit logs provide a means of operation tracing and review to ensure system security.
[0003] Traditional audit log processing systems are often tightly coupled with business systems, and have defects such as poor concurrency capabilities and a single data storage backend, making them unsuitable for complex environments with diverse business systems.
[0004] Application Contents
[0005] In order to overcome the deficiencies of the prior art, the present application provides an audit log processing system, method, computer device and storage medium, which can run independently as a service, provide multiple access methods, be decoupled from the business system, adopt a functional hierarchical governance architecture, support high concurrency of audit logs, and be compatible with multiple structured data storage backends.
[0006] In order to achieve the above objectives, this application adopts the following technical solutions:
[0007] An audit log processing system, comprising:
[0008] Access layer, used to couple various external business systems;
[0009] A service layer, coupled to the access layer, for processing high-concurrency audit log requests, dynamic configuration management, and log dumping; and
[0010] The storage layer is coupled to the service layer and is used for reading and writing audit log data and managing structured data storage, wherein the management of structured data storage includes calling a driver program of structured storage and managing a storage connection pool.
[0011] Furthermore, the access layer includes a data access module, a filtering and limiting module, and a data preprocessing module; wherein,
[0012] The data access module is used to process the access of each external business system, perform identity authentication on the accessed business system, deny access to illegal users, and forward legal requests to the filtering and limiting module;
[0013] The filtering and limiting module is used to filter and limit the audit log requests, reject illegal parameter requests, and forward legal requests to the data preprocessing module; when the concurrent audit log requests exceed the preset flow threshold, the filtering and limiting module rejects the new requests until the flow drops below the preset flow threshold;
[0014] The data preprocessing module is used to preprocess the audit log request, and the preprocessing includes extracting audit log elements, repackaging them into private protocol data and forwarding them to the service layer.
[0015] Furthermore, the service layer includes a protocol parsing module, a log cache management module, a log recording and dumping module and a configuration management module; wherein,
[0016] The protocol parsing module is used to process the private protocol data, including writing a request to the audit log, extracting and structuring audit log elements, and storing them in a log buffer pool;
[0017] The log cache module is used to manage the log cache pool in real time. When the number of audit log cache entries exceeds a preset cache threshold, the log cache module batch extracts audit logs whose number is the preset cache threshold in the order of timestamps, and transfers them to the log recording and dumping module, while clearing the transferred audit logs in the log cache pool.
[0018] The log record and dump module is used to store and structure the transferred audit logs, and then transfer them to the storage layer after encapsulation;
[0019] The configuration management module is used to dynamically monitor configuration changes and make them effective in real time.
[0020] Furthermore, the protocol parsing module is also used to process a read request for the audit log. The protocol parsing module first reads from the log buffer pool. If a hit is found, the data is forwarded to the log recording and dumping module. If a hit is found, the data read is directly returned.
[0021] The log recording and dumping module is also used to start a scheduled dumping task according to a preset dumping condition, and perform scheduled dumping inspection and processing on the audit logs written into the structured storage.
[0022] Further, the storage layer includes a storage management module;
[0023] The storage management module is used to call the driver of the structured storage to read and write the audit log in the structured storage and manage the storage connection pool to prevent the connection from exceeding the limit or leaking.
[0024] In addition, the present application also provides an audit log processing method using the above-mentioned audit log processing system, including:
[0025] External business systems request access;
[0026] Perform identity authentication on the service system that requests access, deny access to illegal users, and allow access to legitimate requests;
[0027] After the external business system is connected, an audit log request is initiated;
[0028] Filter and limit the audit log request, reject illegal parameter requests, and pre-process the audit log request for legal requests, wherein the pre-processing includes extracting audit log elements and re-packaging them into private protocol data;
[0029] Processing the private protocol data, including requesting to write audit logs, extracting and structuring audit log elements, and storing them in a log buffer pool;
[0030] The log buffer pool is managed in real time. When the number of audit log cache entries exceeds a preset threshold value of the cache, the audit logs whose number is the preset threshold value of the cache are extracted in batches in the order of timestamps, and are stored in a structured package. At the same time, the transferred audit logs in the log buffer pool are cleared, and the transferred audit logs are stored in a structured package. After the storage structured package, the driver program of the structured storage is called to read and write the audit logs.
[0031] Furthermore, it also includes:
[0032] When the number of concurrent audit log requests exceeds the preset flow threshold, new requests will be rejected until the flow drops below the preset flow threshold.
[0033] Furthermore, it also includes:
[0034] Processes read requests for the audit log, including:
[0035] Reading from the log buffer pool, if there is a miss, the read request is stored in a structured package and forwarded for storage; if there is a hit, the read data is directly returned.
[0036] In addition, the present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-described method when executing the computer program.
[0037] In addition, the present application also provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method as described above is implemented.
[0038] Compared with the prior art, this application has the following advantages:
[0039] The present application provides an audit log processing system, method, computer equipment and storage medium. The front end can be connected to a variety of business systems, and the back end can be connected to a variety of structured data storage. The internal functional layered governance architecture is divided into an access layer, a service layer and a storage layer. Among them, the access layer provides a variety of access methods to connect to various types of business systems, including data access, filtering and limiting, and data preprocessing modules. The service layer handles audit log requests, dynamic configuration management and log dump tasks, including protocol parsing, log cache management, log recording and dumping, and configuration management modules. The storage layer is responsible for reading and writing audit log data. At the same time, it manages the structured data storage connection pool and is compatible with a variety of structured data storage backends. It can run independently as a service, provide a variety of access methods, be decoupled from the business system, adopt a functional layered governance architecture, support high concurrency of audit logs, and be compatible with a variety of structured data storage backends. BRIEF DESCRIPTION OF THE DRAWINGS
[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application but do not constitute an improper limitation on the present application. In the drawings:
[0041] Figure 1 This is the system structure diagram of this application;
[0042] Figure 2 A flow chart of the method of this application;
[0043] Figure 3 A schematic diagram of the structure of the computer device of the present application;
[0044] Figure 4 This is a schematic diagram of the structure of another computer device of the present application. DETAILED DESCRIPTION
[0045] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.
[0046] In the description of the present application, it should be understood that the orientation or positional relationship indicated by the terms and the like is based on the orientation or positional relationship shown in the accompanying drawings, which is only for the convenience of describing the present application and simplifying the description, rather than indicating or implying that the device, element, module, system, platform or device referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application. The following description of the present application is only understood as a description of individual embodiments of the technical solution of the present application. Other embodiments are not reflected in the following description, but it does not mean that the present application excludes these other embodiments, and the technical solution of the present application is not limited to the specific implementation methods described below, and the protection scope of the present application is not limited to only the specific implementation methods described below. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work should fall within the scope of protection of the present application.
[0047] It should be noted that if the terms "first", "second", etc. appear in the specification and claims of the present application and the above-mentioned drawings, the description is only used to distinguish similar objects, and is not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0048] In some embodiments, Figure 1 As shown, the present application provides an audit log processing system, comprising:
[0049] Access layer, used to couple various external business systems;
[0050] A service layer, coupled to the access layer, for processing high-concurrency audit log requests, dynamic configuration management, and log dumping;
[0051] The storage layer is coupled to the service layer and is used for reading and writing audit log data and managing structured data storage, wherein the management of structured data storage includes calling a driver program of structured storage and managing a storage connection pool.
[0052] In some embodiments, Figure 1 As shown, the access layer includes a data access module, a filtering and limiting module, and a data preprocessing module; wherein,
[0053] The data access module is used to process the access of each external business system, perform identity authentication on the accessed business system, deny access to illegal users, and forward legal requests to the filtering and limiting module;
[0054] The filtering and limiting module is used to filter and limit the audit log requests, reject illegal parameter requests, and forward legal requests to the data preprocessing module; when the concurrent audit log requests exceed the preset flow threshold, the filtering and limiting module rejects the new requests until the flow drops below the preset flow threshold;
[0055] The data preprocessing module is used to preprocess the audit log request, and the preprocessing includes extracting audit log elements, repackaging them into private protocol data and forwarding them to the service layer.
[0056] In some embodiments, Figure 1 As shown, the service layer includes a protocol parsing module, a log cache management module, a log recording module, a dump module and a configuration management module; wherein,
[0057] The protocol parsing module is used to process the private protocol data, including writing a request to the audit log, extracting and structuring audit log elements, and storing them in a log buffer pool;
[0058] The log cache module is used to manage the log cache pool in real time. When the number of audit log cache entries exceeds a preset cache threshold, the log cache module batch extracts audit logs whose number is the preset cache threshold in the order of timestamps, and transfers them to the log recording and dumping module, while clearing the transferred audit logs in the log cache pool.
[0059] The log record and dump module is used to store and structure the transferred audit log write data or read request, and then transfer it to the storage layer after encapsulation;
[0060] The configuration management module is used to dynamically monitor configuration changes and make them effective in real time.
[0061] In some embodiments, the protocol parsing module is also used to process a read request for the audit log. The protocol parsing module first reads from the log buffer pool. If a hit is found, the data is forwarded to the log record and dump module. If a hit is found, the data read is directly returned.
[0062] The log recording and dumping module is also used to start a scheduled dumping task according to a preset dumping condition, and perform scheduled dumping inspection and processing on the audit logs written into the structured storage.
[0063] Specifically, the log recording and dump module is also used to start a scheduled dump task according to preset dump conditions, regularly check the audit logs written to the structured storage, and when the dump conditions are met, transfer the audit logs that meet the dump conditions to files or other storage spaces of the structured storage in order of timestamps.
[0064] In some embodiments, Figure 1 As shown, the storage layer includes a storage management module;
[0065] The storage management module is used to call the driver of the structured storage to read and write the audit log in the structured storage, and manage the storage connection pool to prevent the connection from exceeding the limit or leaking.
[0066] In some embodiments, the present application further provides an audit log processing method using the audit log processing system as described above, comprising:
[0067] External business systems request access;
[0068] Perform identity authentication on the service system that requests access, deny access to illegal users, and allow access to legitimate requests;
[0069] After the external business system is connected, an audit log request is initiated;
[0070] Filter and limit the audit log request, reject illegal parameter requests, and pre-process the audit log request for legal requests, wherein the pre-processing includes extracting audit log elements and re-packaging them into private protocol data;
[0071] Processing the private protocol data, including requesting to write audit logs, extracting and structuring audit log elements, and storing them in a log buffer pool;
[0072] The log buffer pool is managed in real time. When the number of audit log cache entries exceeds a preset threshold value of the cache, the audit logs whose number is the preset threshold value of the cache are extracted in batches in the order of timestamps, and are stored in a structured package. At the same time, the transferred audit logs in the log buffer pool are cleared, and the transferred audit logs are stored in a structured package. After the storage structured package, the driver program of the structured storage is called to read and write the audit logs.
[0073] In some embodiments, it also includes:
[0074] When the number of concurrent audit log requests exceeds the preset flow threshold, new requests will be rejected until the flow drops below the preset flow threshold.
[0075] In some embodiments, it also includes:
[0076] Processes read requests for the audit log, including:
[0077] Reading from the log buffer pool, if there is a miss, the read request is stored in a structured package and forwarded for storage; if there is a hit, the read data is directly returned.
[0078] Specifically, Figure 2 As shown, the present application can be implemented in the following specific steps.
[0079] 1) Apply for access to the business system;
[0080] 2) The data access module authenticates the access request, directly rejects illegal users and allows access to legal users;
[0081] 3) The business system initiates an audit log request;
[0082] 4) The filtering and limiting module filters and limits the audit log requests. Requests with illegal parameters are directly rejected. When the concurrent requests are within the limit, requests with legal parameters are forwarded to the data preprocessing module.
[0083] 5) The data preprocessing module preprocesses the audit log request, extracts the audit log elements, repackages them into private protocol data and forwards them to the protocol parsing module;
[0084] 6) The protocol parsing module parses the private protocol data: for audit log write requests, the audit log elements are extracted and structured, and stored in the log buffer pool; for audit log read requests, the audit log is first read from the log buffer pool, and if a hit is found, the read audit log is directly returned, and if a hit is not found, the audit log read request is forwarded to the log recording and dumping module;
[0085] 7) The log cache module manages the log cache pool in real time. When the number of audit log cache entries exceeds the preset threshold, the audit logs of the preset threshold number are extracted in batches in the order of timestamps and transferred to the log recording and dump module for processing. At the same time, the batch of audit logs is cleared in the log cache pool.
[0086] 8) The log recording and dumping module stores and structures the audit log write data / read request and transfers it to the storage management module;
[0087] 9) The storage management module calls the driver of the structured storage to read and write the audit log data after the storage structure is encapsulated. If the execution is successful, the response data is returned, and if the execution fails, the failure is returned.
[0088] Each module in the above audit log processing system can be implemented in whole or in part by software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.
[0089] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 3 As shown. The computer device includes a processor, a memory, an input / output interface (I nput / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an audit log processing method is implemented.
[0090] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface, the display unit and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WI FI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, an audit log processing method is implemented. The display unit of the computer device is used to form a visually visible image, and can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device of the computer device can be a touch layer covered on the display screen, or a button, trackball or touchpad set on the computer device casing, or an external keyboard, touchpad or mouse, etc.
[0091] Those skilled in the art will understand that Figure 3 and Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0092] In one embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above method embodiments when executing the computer program.
[0093] In one embodiment, a computer-readable storage medium is provided, storing a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0094] In one embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.
[0095] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0096] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in any type of volatile or non-volatile storage device, or a combination thereof. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory, a magnetic surface memory, an optical disk, or a compact disk read-only memory (CD-ROM); the magnetic surface memory can be a disk memory or a tape memory. Volatile memory can be random access memory (RAM), which acts as external cache.By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDR AM), enhanced synchronous dynamic random access memory (ESDRAM Enhanced Synchronous Dynamic Random Access Memory), synchronous linked dynamic random access memory (SLDRAM), direct RAM bus random access memory (DRRAM). The storage media described in the embodiments of the present invention are intended to include, but are not limited to, these and any other suitable types of memory.
[0097] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0098] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. An audit log processing system, characterized in that: include: Access layer, used to couple various external business systems; A service layer, coupled to the access layer, for processing high-concurrency audit log requests, dynamic configuration management, and log dumping; as well as The storage layer is coupled to the service layer and is used for reading and writing audit log data and managing structured data storage, wherein the management of structured data storage includes calling a driver program of structured storage and managing a storage connection pool.
2. The audit log processing system according to claim 1, characterized in that: The access layer includes a data access module, a filtering and limiting module, and a data preprocessing module; wherein, The data access module is used to process the access of each external business system, perform identity authentication on the accessed business system, deny access to illegal users, and forward legal requests to the filtering and limiting module; The filtering and limiting module is used to filter and limit the audit log requests, reject illegal parameter requests, and forward legal requests to the data preprocessing module; when the concurrent audit log requests exceed the preset flow threshold, the filtering and limiting module rejects the new requests until the flow drops below the preset flow threshold; The data preprocessing module is used to preprocess the audit log request, and the preprocessing includes extracting audit log elements, repackaging them into private protocol data and forwarding them to the service layer.
3. The audit log processing system according to claim 2, characterized in that: The service layer includes a protocol parsing module, a log cache management module, a log recording and dumping module and a configuration management module; wherein, The protocol parsing module is used to process the private protocol data, including writing a request to the audit log, extracting and structuring audit log elements, and storing them in a log buffer pool; The log cache module is used to manage the log cache pool in real time. When the number of audit log cache entries exceeds a preset cache threshold, the log cache module batch extracts audit logs whose number is the preset cache threshold in the order of timestamps, and transfers them to the log recording and dumping module, while clearing the transferred audit logs in the log cache pool. The log record and dump module is used to store and structure the transferred audit logs, and then transfer them to the storage layer after encapsulation; The configuration management module is used to dynamically monitor configuration changes and make them effective in real time.
4. The audit log processing system according to claim 3, characterized in that: The protocol parsing module is also used to process the read request for the audit log. The protocol parsing module first reads from the log buffer pool. If there is no hit, it is transferred to the log recording and dumping module; if there is a hit, the read data is directly returned; The log recording and dumping module is also used to start a scheduled dumping task according to a preset dumping condition, and perform scheduled dumping inspection and processing on the audit logs written into the structured storage.
5. The audit log processing system according to claim 4, characterized in that: The storage layer includes a storage management module; The storage management module is used to call the driver of the structured storage to read and write the audit log in the structured storage and manage the storage connection pool to prevent the connection from exceeding the limit or leaking.
6. An audit log processing method using the audit log processing system according to any one of claims 1 to 5, characterized in that: include: External business systems request access; Perform identity authentication on the service system that requests access, deny access to illegal users, and allow access to legitimate requests; After the external business system is connected, an audit log request is initiated; Filter and limit the audit log request, reject illegal parameter requests, and pre-process the audit log request for legal requests, wherein the pre-processing includes extracting audit log elements and re-packaging them into private protocol data; Processing the private protocol data, including requesting to write audit logs, extracting and structuring audit log elements, and storing them in a log buffer pool; The log buffer pool is managed in real time. When the number of audit log cache entries exceeds a preset threshold value of the cache, the audit logs whose number is the preset threshold value of the cache are extracted in batches in the order of timestamps, and are stored in a structured package. At the same time, the transferred audit logs in the log buffer pool are cleared, and the transferred audit logs are stored in a structured package. After the storage structured package, the driver program of the structured storage is called to read and write the audit logs.
7. The audit log processing method according to claim 6, characterized in that: Also includes: When the number of concurrent audit log requests exceeds the preset flow threshold, new requests will be rejected until the flow drops below the preset flow threshold.
8. The audit log processing method according to claim 6, characterized in that: Also includes: Processes read requests for the audit log, including: Reading from the log buffer pool, if there is a miss, storing the read request in a structured package and forwarding it for storage; If a hit occurs, the read data is returned directly.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 5 is implemented.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.