A collaborative management method for system logs and operation logs

Abstract

This application relates to a collaborative management method for system logs and operation logs. It involves setting up and deploying a log collection container and a business container; configuring index configuration conditions for each container to collect and acquire log data; collecting the log data into the configured container according to the index configuration conditions; parsing and transforming the log data in the container to obtain log data, and storing the log data in a caching system. By inputting business data into the log module, the data collection process is seamless and automated for the user system, achieving a collaborative management solution for system logs and operation logs. This provides more granular operation and higher readability, not only collecting and converting system logs but also providing operation logs, thus solving the problems of poor readability and inability to analyze user behavior in system logs.

Description

Technical Field

[0001] This disclosure relates to the field of log processing technology, and in particular to a method, apparatus and control system for collaborative management of system logs and operation logs. Background Technology

[0002] In data management, system processing events and operation records are all recorded in corresponding logs, which facilitates the recording and management of runtime sequences.

[0003] Logs are mainly divided into system logs and operation logs. System logs record administrative operations performed by the system to modify or configure the system, such as management permissions, like the permissions for processing functional modules. Operation logs, on the other hand, record the daily system operations performed by users or administrators, reflecting what work they have done, such as modifying certain system event information or setting processing times.

[0004] Text log collectors form the foundation of log management, providing features such as format conversion plugins, data filtering, and data analysis and classification. However, managing logs using text log collectors is a system-level, unified solution, suitable only for system log management. For operation logs, text log collectors are unsuitable, as they cannot manage both system and operation logs simultaneously. Therefore, their data readability is poor, and they cannot analyze user behavior, requiring upgrades and improvements. Summary of the Invention

[0005] To address the aforementioned issues, this application proposes a method, apparatus, and control system for the collaborative management of system logs and operation logs.

[0006] This application proposes a collaborative management method for system logs and operation logs, comprising the following steps:

[0007] Set up and deploy the log collection container and the business container;

[0008] Set index configuration conditions, and configure the index configuration conditions for the log collection container and the business container respectively;

[0009] Collect and acquire log data, and collect the log data into the configured log collection container and the business container respectively according to the index configuration conditions;

[0010] The log data in the log collection container and the business container is parsed and transformed to obtain log data, which is then stored in the cache system.

[0011] As an optional implementation of this application, optionally, a log collection container and a business container are set up and deployed, including:

[0012] Preset container ratio;

[0013] Configure the log collection container and the business container according to the container setting ratio;

[0014] The default deployment environment is to deploy the configured log collection container and the business container in the default deployment environment.

[0015] As an optional implementation of this application, the deployment environment may be a Pod deployment unit in a k8s cluster system.

[0016] As an optional implementation of this application, optionally, index configuration conditions are set, and the index configuration conditions are configured for the log collection container and the business container respectively, including:

[0017] Based on the business attributes of the business system, set the log attributes of the log data output by the business system;

[0018] Based on the log attributes of the log data, set the index configuration conditions corresponding to the log attributes;

[0019] Based on the log attributes, configure the index configuration conditions that match the direction of the log collection container and the business container respectively.

[0020] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, includes:

[0021] Obtain log data output by the business system;

[0022] Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions:

[0023] If the log attributes of the log data match the index configuration conditions of the log collection container, then the log data will be collected into the configured log collection container.

[0024] If the log attributes of the log data match the index configuration conditions of the business container, then the log data will be collected into the configured business container.

[0025] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, further includes:

[0026] Obtain log data output by the business system;

[0027] Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions:

[0028] If the log attributes of the log data do not match the index configuration conditions of the log collection container or the business container, then the log data is recorded and collected into the cache system.

[0029] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, further includes:

[0030] Several log templates are preset, including: several system log templates and several operation log templates;

[0031] The log template is configured on different log pipelines in the log collection container and the business container according to its attributes;

[0032] The log data is collected and saved to the caching system through the log pipelines of the log collection container and the business container.

[0033] As an optional implementation of this application, optionally, parsing and transforming the log data in the log collection container and the business container to obtain log data and storing the log data in the caching system includes:

[0034] The dynamic log data obtained from the log collection container or the business container is obtained from the caching system.

[0035] According to the log parsing rules, the dynamic log data is parsed to obtain log parsing data;

[0036] The parsed log data is converted into a complete system log or operation log, and the system log or operation log is stored in a log file and displayed.

[0037] In another aspect, this application proposes an apparatus for implementing the aforementioned collaborative management method of system logs and operation logs, comprising:

[0038] The deployment module is used to set up and deploy the log collection container and the business container;

[0039] The configuration module is used to set index configuration conditions, and to configure the index configuration conditions for the log collection container and the business container respectively;

[0040] The log collection module is used to collect and obtain log data, and collect the log data into the configured log collection container and the business container respectively according to the index configuration conditions;

[0041] The log parsing module is used to parse and transform the log data in the log collection container and the business container to obtain log data and store the log data in the cache system.

[0042] In another aspect, this application also proposes a control system, comprising:

[0043] processor;

[0044] Memory used to store processor-executable instructions;

[0045] The processor is configured to implement the aforementioned collaborative management method for system logs and operation logs when executing the executable instructions.

[0046] Technical effects of the present invention:

[0047] Based on the implementation scheme of this application, this application sets up and deploys a log collection container and a business container; sets index configuration conditions, and configures the index configuration conditions for the log collection container and the business container respectively to collect and obtain log data; collects the log data into the configured log collection container and the business container according to the index configuration conditions; parses and transforms the log data in the log collection container and the business container to obtain log data, and stores the log data in a caching system. By inputting business data into the log module, the collection work is seamless and automated for the user system, realizing a collaborative management scheme for system logs and operation logs. This provides more granular operation, higher readability, and business analysis capabilities. In addition to the functions of collecting, formatting, and configuring system logs, it also provides operation logs, solving the problems of poor readability and inability to analyze user behavior in system logs.

[0048] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0049] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.

[0050] Figure 1 The diagram illustrates the implementation flow of the collaborative management method for system logs and operation logs of the present invention.

[0051] Figure 2 The diagram shown is a schematic representation of the deployment system of the present invention. Detailed Implementation

[0052] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0053] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0054] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0055] Example 1

[0056] This application uses a log module to input business data, enabling seamless and automated data collection from the user system. It achieves a collaborative management solution for system logs and operation logs, providing more granular operation, higher readability, and enabling business analysis. In addition to the specific functions of system log collection, format conversion, and device management, it also provides operation logs, solving the problems of poor readability and inability to analyze user behavior in system logs.

[0057] like Figure 1 As shown, this application proposes a collaborative management method for system logs and operation logs, including the following steps:

[0058] S1. Set up and deploy the log collection container and the business container;

[0059] This technology first establishes a collaborative management architecture for system logs and operation logs, so that business systems only need to output system logs and operation logs, which are collected according to the log channels of different collection devices. Business data is input through the corresponding log templates, and the log data is processed separately.

[0060] like Figure 2 As shown, as an optional implementation of this application, optionally, a log collection container and a business container are set up and deployed, including:

[0061] Preset container ratio;

[0062] Configure the log collection container and the business container according to the container setting ratio;

[0063] The default deployment environment is to deploy the configured log collection container and the business container in the default deployment environment.

[0064] As an optional implementation of this application, the deployment environment may be a Pod deployment unit in a k8s cluster system.

[0065] The log collection container and the business container collect system logs and operation logs respectively. During collection, the log data is collected and stored in a formatted manner using the corresponding system log template and operation log template.

[0066] First, this embodiment provides a Pod deployment unit in the Kubernetes cluster system, which deploys the log collection container and the business container in the same Pod.

[0067] The application is deployed on a Kubernetes cluster. The SideCar pattern is used to deploy the log collection container and the application containers within the same Pod, collecting logs only from the corresponding container. This approach can be configured when writing the application pod YAML file. This method offers loose coupling, high scalability, and ease of maintenance and upgrades. Even if the SideCar consumes more resources when many applications are running, this resource consumption is within an acceptable range considering the stability of large-scale log collection in the future.

[0068] Secondly, the number of log collection containers and business containers should each be set to at least one. The specific deployment quantity can be set by the user based on the daily data of system logs and operation logs. At the same time, the number of containers should be deployed according to the size of the selected containers. This embodiment does not make a specific limit on the container setting ratio.

[0069] When the business system outputs log data, the log data needs to be categorized and collected into different log collection containers and business containers to facilitate separate management of system logs and operation logs. In this embodiment, each log collection container and business container is configured with corresponding index conditions for collecting logs. When collecting log data, the log collection container and the business container will index and collect the corresponding log data according to the index configuration conditions. For example, if the system determines that a log data (log attribute) matches the index configuration conditions of the log collection container, it will allocate the log channel of the log collection container to collect that log data. Each log in the log data stream undergoes attribute judgment until the end.

[0070] S2. Set the index configuration conditions, and configure the index configuration conditions for the log collection container and the business container respectively;

[0071] The index configuration conditions will be set according to the attributes or types of the business. If it is business operation data, it will be classified as operation log; otherwise, it will be classified as system log (system logs can also be set according to user settings or system customization).

[0072] As an optional implementation of this application, optionally, index configuration conditions are set, and the index configuration conditions are configured for the log collection container and the business container respectively, including:

[0073] Based on the business attributes of the business system, set the log attributes of the log data output by the business system;

[0074] Based on the log attributes of the log data, set the index configuration conditions corresponding to the log attributes;

[0075] Based on the log attributes, configure the index configuration conditions that match the direction of the log collection container and the business container respectively.

[0076] Specifically, the settings should follow the order of business attributes --- log attributes --- index configuration conditions, setting index configuration conditions that conform to system log attributes and operation log attributes respectively. When implementing the forward and backward operations in the architecture, a corresponding index module can be set up. This module identifies and judges the attributes of the logs output by the business system, and collects log data based on the identified attribute results.

[0077] S3. Collect and obtain log data, and collect the log data into the configured log collection container and the business container respectively according to the index configuration conditions;

[0078] After configuring the corresponding index configuration conditions, the log collection container and the business container can be used to identify and collect log data streams sent by the task system line by line. Specifically:

[0079] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, includes:

[0080] Obtain log data output by the business system;

[0081] Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions:

[0082] If the log attributes of the log data match the index configuration conditions of the log collection container, then the log data will be collected into the configured log collection container.

[0083] If the log attributes of the log data match the index configuration conditions of the business container, then the log data will be collected into the configured business container.

[0084] For example, if the log attributes of a certain log data match the index configuration conditions of the business container, then the index module of the business container will index the log data and transmit and store it in the corresponding database (caching system) through the log channel of the business container.

[0085] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, further includes:

[0086] Obtain log data output by the business system;

[0087] Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions:

[0088] If the log attributes of the log data do not match the index configuration conditions of the log collection container or the business container, then the log data is recorded and collected into the cache system.

[0089] When matching the above attributes, if it is found that the log attributes of the log data do not match the index configuration conditions of the log collection container or the business container, it indicates that the log attributes of the log data are special cases, which may be businesses not included in the system. In this case, the log data is recorded and the log data is collected into the cache system.

[0090] Within a log monitoring period, cached log data can be cleaned up, and mismatched log data can be extracted and processed separately. The business corresponding to the log data can be analyzed, or a separate container can be created to cache special business log data.

[0091] As an optional implementation of this application, optionally, collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, further includes:

[0092] Several log templates are preset, including: several system log templates and several operation log templates;

[0093] The log template is configured on different log pipelines in the log collection container and the business container according to its attributes;

[0094] The log data is collected and saved to the caching system through the log pipelines of the log collection container and the business container.

[0095] Each container that collects log data is configured with several log channels, which are used to receive the identified and collected log data through the log channel index and send it to the buffer system after processing.

[0096] To facilitate quick identification and storage of collected log data, log templates are adapted for different log pipelines. Each pipeline contains two essential elements: input and output. Data is parsed and transformed during the input process.

[0097] Log templates are designed for the elegant and dynamic generation of operation logs. They are implemented using the Spring Expression Language and generated through Java annotations. Business users only need to focus on setting key data.

[0098] During the adaptation phase, the so-called log template only stores key information. If the log data is to be readable by humans, it needs to be parsed. Dynamic data can be quickly obtained from the caching system and then finally converted into complete operation logs.

[0099] Log data contains a significant amount of information. Log templates allow for formatted statistical analysis of this information, facilitating information compression and formatted storage, thus reducing the file size of the log data packets. When retrieving and viewing log data packets, the packets are processed according to the specified format to obtain the parsed log data.

[0100] S4. Parse and transform the log data in the log collection container and the business container to obtain log data and store the log data in the cache system.

[0101] When complete log data is needed, dynamic data can be quickly obtained from the caching system and then finally converted into complete operation logs.

[0102] By parsing the log storage files according to the specifications, the log data can be obtained. The obtained log data can then be sent to the display platform / module for visualization.

[0103] As an optional implementation of this application, optionally, parsing and transforming the log data in the log collection container and the business container to obtain log data and storing the log data in the caching system includes:

[0104] The dynamic log data obtained from the log collection container or the business container is obtained from the caching system.

[0105] According to the log parsing rules, the dynamic log data is parsed to obtain log parsing data;

[0106] The parsed log data is converted into a complete system log or operation log, and the system log or operation log is stored in a log file and displayed.

[0107] Users can log in to the platform, obtain dynamic log data from the log collection container or the business container in the caching system, parse it on the platform to obtain log parsing data, and then transform and display the log parsing data through the display module.

[0108] The visualization component offers classic functionalities such as bar charts, line charts, and pie charts. It also provides time series analysis capabilities, allowing users to use powerful yet easy-to-learn expressions to describe queries, transformations, and visualizations. This enables the analysis of user behavior through log-based graph displays.

[0109] The above-described analysis and display are performed in the log pipeline after data collection, where data is parsed and transformed. Once the transformation is complete, the transformed log data is sent to the front end for display via the log pipeline.

[0110] It should be noted that although the above deployment environment is illustrated using Kubernetes as an example, those skilled in the art will understand that this disclosure is not limited to this. In fact, users can flexibly configure the deployment environment according to their actual application scenarios, as long as the technical functions of this application can be achieved by following the above technical methods.

[0111] Example 2

[0112] Based on the implementation principle of Embodiment 1, this application, in another aspect, proposes an apparatus for implementing the collaborative management method of system logs and operation logs, comprising:

[0113] The deployment module is used to set up and deploy the log collection container and the business container;

[0114] The configuration module is used to set index configuration conditions, and to configure the index configuration conditions for the log collection container and the business container respectively;

[0115] The log collection module is used to collect and obtain log data, and collect the log data into the configured log collection container and the business container respectively according to the index configuration conditions;

[0116] The log parsing module is used to parse and transform the log data in the log collection container and the business container to obtain log data and store the log data in the cache system.

[0117] For details on the functions and specific application principles of each of the above modules, please refer to the description in Example 1. This example will not repeat them.

[0118] Obviously, those skilled in the art should understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device, or fabricating them separately as individual integrated circuit modules, or fabricating multiple modules or steps into a single integrated circuit module. Thus, the present invention is not limited to any specific hardware and software combination.

[0119] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the control methods described above. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium can also include combinations of the above types of memory.

[0120] Example 3

[0121] Furthermore, this application also proposes a control system, comprising:

[0122] processor;

[0123] Memory used to store processor-executable instructions;

[0124] The processor is configured to implement the aforementioned collaborative management method for system logs and operation logs when executing the executable instructions.

[0125] This disclosure discloses an embodiment of a system including a processor and a memory for storing processor-executable instructions. The processor is configured to implement, when executing the executable instructions, a collaborative management method for system logs and operation logs as described above.

[0126] It should be noted here that the number of processors can be one or more. Furthermore, the control system in this embodiment may also include input devices and output devices. The processors, memory, input devices, and output devices can be connected via a bus or other means, without specific limitations herein.

[0127] As a computer-readable storage medium, the memory can be used to store software programs, computer-executable programs, and various modules, such as the program or module corresponding to the collaborative management method of system logs and operation logs in this disclosure embodiment. The processor executes various functional applications and data processing of the control system by running the software programs or modules stored in the memory.

[0128] Input devices can be used to receive input digital numbers or signals. These signals can be key signals related to user settings and function control of the device / terminal / server. Output devices can include display devices such as screens.

[0129] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical applications, or technical improvements to the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A method for collaborative management of system logs and operation logs, characterized in that, Includes the following steps: S1. Set up and deploy log collection containers and business containers, including: preset container setting ratio; configure the log collection containers and business containers according to the container setting ratio; preset deployment environment, and deploy the configured log collection containers and business containers in the Pod deployment unit of the k8s cluster system; The specific implementation is as follows: First, establish a collaborative management architecture for system logs and operation logs, so that business systems only need to output system logs and operation logs, which can be collected according to the log channels of different collection devices. Business data is input through the corresponding log templates, and the log data is processed separately. Specifically, when writing the application pod yaml file, the following settings are configured: use the SideCar mode to deploy the log collection container and the business container in the same pod, and only collect logs from the corresponding container; when the business system outputs log data, the log data needs to be classified, and the classified log data is collected in different log collection containers and business containers, so as to facilitate the separate management of system logs and operation logs in the future. S2. Set index configuration conditions, and configure the index configuration conditions for the log collection container and the business container respectively; specifically: the index configuration conditions will be set according to the attributes or types of the business. If it is business operation data, it will be classified as operation log; otherwise, it will be classified as system log. Each log collection container and the business container is set with corresponding index configuration conditions for collecting logs. When collecting log data, the log collection container and the business container will index and collect the corresponding log data according to the index configuration conditions. If the system determines that the log data meets the index configuration conditions of the log collection container, it will allocate the log channel of the log collection container to collect the log data. Each log in the log data stream will be subject to attribute judgment until the end. S3. Collect and acquire log data, and collect the log data into the configured log collection container and the business container according to the index configuration conditions, including: presetting several log templates, wherein the log templates include several system log templates and several operation log templates; configuring the log templates according to their attributes on different log pipelines in the log collection container and the business container; collecting the log data through the log pipelines of the log collection container and the business container and saving it to the cache system; S4. Parse and transform the log data in the log collection container and the business container to obtain log data and store the log data in the cache system. Specifically, the parsing and transformation are performed in the log pipeline after collection. After the transformation is completed, the transformed log data is sent to the front end for display through the log pipeline.

2. The collaborative management method for system logs and operation logs according to claim 1, characterized in that, Set index configuration conditions, and configure the index configuration conditions for the log collection container and the business container respectively, including: Based on the business attributes of the business system, set the log attributes of the log data output by the business system; Based on the log attributes of the log data, set the index configuration conditions corresponding to the log attributes; Based on the log attributes, configure the index configuration conditions that match the direction of the log collection container and the business container respectively.

3. The collaborative management method for system logs and operation logs according to claim 1, characterized in that, Collecting and acquiring log data, and then collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, including: Obtain log data output by the business system; Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions: If the log attributes of the log data match the index configuration conditions of the log collection container, then the log data will be collected into the configured log collection container. If the log attributes of the log data match the index configuration conditions of the business container, then the log data will be collected into the configured business container.

4. The collaborative management method for system logs and operation logs according to claim 3, characterized in that, Collecting and acquiring log data, and collecting the log data into the configured log collection container and the business container respectively according to the index configuration conditions, further includes: Obtain log data output by the business system; Determine the log attributes of the log data, and based on the log attributes, determine the matching relationship between the log data and the index configuration conditions: If the log attributes of the log data do not match the index configuration conditions of the log collection container or the business container, then the log data is recorded and collected into the cache system.

5. The collaborative management method for system logs and operation logs according to claim 1, characterized in that, Parsing and transforming the log data in the log collection container and the business container to obtain log data and storing the log data in the caching system includes: The dynamic log data obtained from the log collection container or the business container is obtained from the caching system. According to the log parsing rules, the dynamic log data is parsed to obtain log parsing data; The parsed log data is converted into a complete system log or operation log, and the system log or operation log is stored in a log file and displayed.

6. An apparatus for implementing the collaborative management method of system logs and operation logs according to any one of claims 1-5, characterized in that, include: The deployment module is used to set up and deploy the log collection container and the business container; The configuration module is used to set index configuration conditions, and to configure the index configuration conditions for the log collection container and the business container respectively; The log collection module is used to collect and obtain log data, and collect the log data into the configured log collection container and the business container respectively according to the index configuration conditions; The log parsing module is used to parse and transform the log data in the log collection container and the business container to obtain log data and store the log data in the cache system.

7. A control system, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to implement, when executing the executable instructions, a collaborative management method for system logs and operation logs as described in any one of claims 1 to 5.

Images