A log management system and log data query method

Abstract

The embodiment of the present invention discloses a log management system and a log data query method, the system includes a log management platform, a data interaction interface and a log collection intermediate agent; the log management platform is used to obtain a log query instruction input by a log query user; wherein the log query instruction is generated according to a custom log information retrieval function; the data interaction interface is used to parse the log query instruction, obtain log retrieval associated parameter information, generate a custom log information retrieval program according to the log retrieval associated parameter information, and send the custom log information retrieval program to a target log collection intermediate agent; the target log collection intermediate agent is used to query the target query log matching the log query instruction according to the custom log information retrieval program. The technical solution of the embodiment of the present invention can realize on-demand query of logs, and improve the efficiency, convenience, maintainability and flexibility of log query retrieval.

Description

Technical Field

[0001] The embodiments of the present invention relate to the technical field of log management, and in particular to a log management system and a log data query method. Background Art

[0002] In the modern digital environment, transaction cooperation between different application systems is completed by exchanging messages. These messages carry all the key information required for transactions, ensuring that transactions can be carried out smoothly between different systems. The message structure is the standardized rule for exchanging information between these systems, which ensures the correct interpretation of information. At the same time, for monitoring and troubleshooting, the application system will generate detailed logs to record every step of the business process and the process of parameter transmission. System administrators or system users often have the need to view system logs. By viewing system logs, they can keep track of the system health at any time and create a system log recording environment according to their own needs. By flexibly using system logs, a more stable computer usage environment can be provided.

[0003] Currently, there are usually two mainstream solutions for querying logs according to user needs: the first method is full-text fuzzy retrieval in the log system. Figure 1 This is a flow chart of the prior art for full-text fuzzy retrieval in a log system. Figure 1 As shown, taking the query of sensitive information in logs as an example, the method specifically uses the log management system to perform full-text fuzzy retrieval. The process is briefly described as follows: a. Provide sensitive information search keywords to generate a comparison benchmark; b. Search the logs within the inspection range from the log storage index database, such as outputting all "electronic banking" logs; c. Use the features of step a to associate and match the data in step b; d. Output the content that successfully matches in step c. For example, if you need to match whether there is mobile phone number information in the log data, you need to enter the fuzzy search information " / 1

[34578] \d{9} / " in the query instruction in the log management system. The second method is to perform regular expression traversal search on system devices (such as servers, etc.). This method uses professional development tools such as Python (a computer programming language) or Shell (a computer programming language) to perform regular expression fuzzy matching and output the hit file name and log content line.

[0004] In the process of implementing the present invention, the inventor found that the prior art has the following defects: the full-text fuzzy search method in the log system needs to query all log data and compare them one by one with the user's log query rule list, which has a very large amount of calculation. The entire implementation process has two time-consuming steps: full-text search and left-join association, which consumes a lot of resources on the log server. In the actual log query process, it often happens that the cluster computing resources are too high, resulting in unstable processing of other log sources. The method of performing regular expression traversal search has poor maintainability. Python or Shell class log information check programs need to be executed separately on each system device. From a global perspective, it is impossible to provide a centralized management page, and it is impossible to grasp which system devices have deployed log information check programs in the first place. At the same time, because it is out of touch with the log management system, the separately deployed log information check program cannot reuse the log collection strategy that the log management system is already running, and cannot reuse the list of log files that need to be checked. Secondly, the log search results obtained by the method of performing regular expression traversal search are stored on each system device, and a global report cannot be output. Summary of the invention

[0005] The embodiments of the present invention provide a log management system and a log data query method, which can realize on-demand log query and improve the efficiency, convenience, maintainability and flexibility of log query retrieval.

[0006] According to one aspect of the present invention, a log management system is provided, including a log management platform, a data interaction interface and a log collection intermediate agent; the log management platform is communicatively connected with the data interaction interface, and the data interaction interface is communicatively connected with the log collection intermediate agent; wherein:

[0007] The log management platform is used to obtain a log query instruction input by a log query user; wherein the log query instruction is generated according to a custom log information retrieval function;

[0008] The data interaction interface is used to parse the log query instruction, obtain log retrieval associated parameter information, generate a custom log information retrieval program according to the log retrieval associated parameter information, and send the custom log information retrieval program to the target log collection intermediate agent;

[0009] The target log collection intermediate agent is used to query the target query log matching the log query instruction according to the custom log information retrieval program.

[0010] According to another aspect of the present invention, a log data query method is provided, the method is applied to a log management system, the log management system includes a log management platform, a data interaction interface and a log collection intermediate agent; the log management platform is communicatively connected with the data interaction interface, the data interaction interface is communicatively connected with the log collection intermediate agent, the method includes:

[0011] Obtaining a log query instruction input by a log query user through the log management platform; wherein the log query instruction is generated according to a custom log information retrieval function;

[0012] Parsing the log query instruction through the data interaction interface to obtain log retrieval associated parameter information, generating a custom log information retrieval program according to the log retrieval associated parameter information, and sending the custom log information retrieval program to the target log collection intermediate agent;

[0013] The target query log matching the log query instruction is queried through the target log collection intermediate agent according to the custom log information retrieval program.

[0014] The embodiment of the present invention constructs a log management system through a log management platform, a data interaction interface and a log collection intermediate agent, and obtains the log query instruction input by the log query user through the log management platform, generates the log query instruction according to the custom log information retrieval function, parses the log query instruction through the data interaction interface, obtains the log retrieval associated parameter information, generates the custom log information retrieval program according to the log retrieval associated parameter information, and sends the custom log information retrieval program to the target log collection intermediate agent, and then queries the target query log matching the log query instruction according to the custom log information retrieval program through the target log collection intermediate agent, solves the problems of low efficiency, high resource consumption, poor maintainability and poor flexibility existing in the existing log query management method, can realize on-demand log query, and improve the efficiency, convenience, maintainability and flexibility of log query retrieval.

[0015] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0017] Figure 1 It is a schematic diagram of the process of performing full-text fuzzy retrieval in a log system in the prior art;

[0018] Figure 2 is a schematic diagram of a log management system provided by Embodiment 1 of the present invention;

[0019] Figure 3 This is a flow chart of performing log query based on a log management system provided by Embodiment 2 of the present invention;

[0020] Figure 4 This is a flow chart of a log management system provided by Embodiment 2 of the present invention collecting log data through a custom log information retrieval script;

[0021] Figure 5 This is a flow chart of a log data query method provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatus.

[0024] Embodiment 1

[0025] Figure 2 is a schematic diagram of a log management system provided by Embodiment 1 of the present invention, such as Figure 2 As shown, the structure of the log management system includes: a log management platform 110, a data interaction interface 120 and a log collection intermediate agent 130; the log management platform 110 is connected to the data interaction interface 120 for communication, and the data interaction interface 120 is connected to the log collection intermediate agent 130 for communication; wherein:

[0026] The log management platform 110 is used to obtain a log query instruction input by a log query user; wherein the log query instruction is generated according to a custom log information retrieval function;

[0027] The data interaction interface 120 is used to parse the log query instruction, obtain log retrieval associated parameter information, generate a custom log information retrieval program according to the log retrieval associated parameter information, and send the custom log information retrieval program to the target log collection intermediate agent;

[0028] The target log collection intermediate agent is used to query the target query log matching the log query instruction according to the custom log information retrieval program.

[0029] Among them, the log management platform 110 can be a functional platform that integrates log query and log management. The log management platform 110 can provide users with functions such as log query and log management, and can also be connected to introduce a customized log information retrieval program to achieve on-demand query management of logs. Log query users can be users who query relevant log data through the log management platform 110. Log query instructions can be query instructions input by log query users to the log management platform 110. Customized log information retrieval functions can be retrieval functions customized according to log query requirements, which can be used to configure query rules for certain types of logs according to log query requirements, etc., to query log data that meets certain specific conditions.

[0030] Among them, the data interaction interface 120 can be an interface for connecting the log management platform 110 and the log collection intermediate agent 130, which is used as an intermediate bridge to realize data transmission and communication between the log management platform 110 and the log collection intermediate agent 130. The log retrieval associated parameter information can be the relevant parameter information required for log retrieval of related devices storing log data, for example, it can include but is not limited to the information of the log collection intermediate agent responsible for collecting log data in the related devices storing log data, and the relevant information of the log files stored in the related devices that meet the user's log query requirements, etc. The custom log information retrieval program can be an external log information retrieval program. The so-called external log information retrieval program can be understood as the log information retrieval program that does not belong to the program configured by the log management platform itself.

[0031] The log collection intermediate agent 130 may be an intermediate agent component for collecting logs for system devices. Optionally, one system device may be configured with one log collection intermediate agent 130. The target log collection intermediate agent may be part or all of the log collection intermediate agents selected from the log collection intermediate agents 130 by the data interaction interface.

[0032] In an embodiment of the present invention, when a log query user has personalized log query requirements, the log query user can generate a log query instruction that meets his or her own log query requirements based on a custom log information retrieval function, and input the log query instruction into the log query interface of the log management platform 110 to indicate that the log management platform 110 can respond to the log query instruction and quickly query the required log data.

[0033] In a specific example, assuming that the log query user needs to query the log data containing sensitive information, before using the log management platform to query the sensitive log information, a custom log information retrieval script can be first developed and configured using a third-party development tool according to the query requirements of the sensitive log information. The program in the custom log information retrieval script can be used to retrieve and find sensitive log information. Accordingly, after the sensitive log information development and configuration is completed, the log query user can create a custom log information retrieval function based on the custom log information retrieval script, and the custom log information retrieval function can also be called a custom log information retrieval instruction. For example, the log query user can configure the name and instruction type of the custom log information retrieval function. Accordingly, when the custom log information retrieval function is configured, the log query user can generate a log query instruction using the custom log information retrieval function, for example, the custom log information retrieval function can be configured with program parameters to generate a log query instruction, and the log query instruction can be input into the log management platform 110.

[0034] Optionally, the log query user's log query requirements can be any type of legitimate query requirements, such as but not limited to sensitive information log query requirements, log query requirements for specific types of service calls, or log query requirements for specific systems or applications. Correspondingly, the custom log information retrieval function can also be developed and configured separately according to the user's personalized log query requirements. In other words, different log query requirements may result in different functions and types of the corresponding custom log information retrieval functions developed and configured.

[0035] Correspondingly, when the log management platform 110 detects that the log query user triggers a log query instruction, the data interaction interface 120 can be called through the log query instruction to parse the log query instruction through the data interaction interface 120 to obtain log retrieval associated parameter information. Exemplarily, the log query instruction can be parsed through the data interaction interface 120 to obtain the target log collection intermediate agent identifier for log query and the list of related logs stored in the target log collection intermediate agent as log retrieval associated parameter information. After obtaining the log retrieval associated parameter information, the log retrieval associated parameter information can be used as a program parameter to generate a custom log information retrieval program, and the generated custom log information retrieval program is sent to the corresponding target log collection intermediate agent.

[0036] Optionally, in order to implement direct calling of a custom log information retrieval program, the data interaction interface 120 can be any type of interface type that has the function of calling each other without programming, for example, it may include but is not limited to REST (Representational State Transfer) API (Application Programming Interface) interface, etc. The embodiment of the present invention does not limit the interface type of the data interaction interface 120.

[0037] Correspondingly, after receiving the custom log information retrieval program, the target log collection intermediate agent can start the custom log information retrieval program to perform targeted queries on the log data stored in the system devices it is responsible for, and obtain the target query log that meets the personalized log retrieval requirements.

[0038] It is understandable that not all log data stored in system devices meet the personalized log retrieval requirements. Therefore, the data interaction interface 120 sends the customized log information retrieval program to the target log collection intermediate agent instead of sending it to all log collection intermediate agents, which can omit unnecessary log retrieval processes and do not need to perform full-text retrieval, thereby greatly improving the efficiency of log retrieval.

[0039] At the same time, the log management system directly matches local log files based on a custom log information retrieval program. The flexibility in developing log matching rules and the available function libraries far exceed the log query language of the log management system itself, and it has more technical advantages in overall efficiency.

[0040] In addition, the log management system uses a data interaction interface to realize the mutual transmission of results. Without changing user usage habits, it realizes the call of third-party programs and the use of results, thereby improving its convenience, maintainability and flexibility.

[0041] The embodiment of the present invention constructs a log management system through a log management platform, a data interaction interface and a log collection intermediate agent, and obtains the log query instruction input by the log query user through the log management platform, generates the log query instruction according to the custom log information retrieval function, parses the log query instruction through the data interaction interface, obtains the log retrieval associated parameter information, generates the custom log information retrieval program according to the log retrieval associated parameter information, and sends the custom log information retrieval program to the target log collection intermediate agent, and then queries the target query log matching the log query instruction according to the custom log information retrieval program through the target log collection intermediate agent, solves the problems of low efficiency, high resource consumption, poor maintainability and poor flexibility existing in the existing log query management method, can realize on-demand log query, and improve the efficiency, convenience, maintainability and flexibility of log query retrieval.

[0042] Embodiment 2

[0043] This embodiment is specific based on the above embodiment. In this embodiment, the specific implementation functions of the log management platform, the data interaction interface and the log collection intermediate agent are introduced in detail.

[0044] Figure 3 is a flow chart of performing log query based on a log management system provided by Embodiment 2 of the present invention. In a specific example, Figure 3 As shown, the Restful API (also known as REST API) interface is used as the data interaction interface, and the log collection Agent is used as the log collection intermediate agent. The specific application scenario of sensitive log information is used as an example to illustrate the specific application scenario. Accordingly, the process of log query based on the log management system can include the following operations:

[0045] The log management platform in the log management system provides users with a way to customize configuration functions. For example, when sensitive information in the log needs to be retrieved, a "sensitive information retrieval function" can be provided to perform the start action of sensitive log information retrieval. The log management platform can send the executable script of the "sensitive information retrieval function" to the corresponding log collection agent through the RESTful API interface.

[0046] If the log collection agent in the log management system receives the executable script in the "sensitive information retrieval function" issued by the log management platform driver, it can use the executable script in the "sensitive information retrieval function" to complete the retrieval and reporting of sensitive log information on the local machine.

[0047] The log collection agent's log retrieval results are reported to the log management platform through the RESTful API interface. The log management platform displays the log retrieval results on the log search Web (World Wide Web, Global Wide Area Network) page.

[0048] In an optional embodiment of the present invention, the log management platform can also be used to: configure the global variables of the custom log information retrieval function; configure the interface resource list of the data interaction interface; receive a custom log information retrieval script configured by a third-party development tool based on the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface; and create the custom log information retrieval program based on the custom log information retrieval script.

[0049] The interface resource list may be used to record information about resources that the data interaction interface can control and process.

[0050] Before using a custom log information retrieval function to generate a log query instruction, you first need to create a custom log information retrieval program that can respond to the log query instruction to perform a log query. Specifically, you can first agree on the global variables of the custom log information retrieval program through the log management platform. Exemplarily, the global variables of the custom log information retrieval function may include but are not limited to Agent_ip, Appname, Source, and ApiKey, etc. Among them, Agent_ip can be the host IP where the log collection agent runs. Appname can be the data range for log information retrieval, for example, "Appname = ebank" represents checking the log of the electronic bank application. Source can be a file list list, which can save the log file path and file name, etc. Exemplarily, source = / data / ebank / logs / 20240407.log, which means that the file name of the saved log file is "20240407.log". ApiKey can be an API interface key provided by the log management platform, such as a user name and password, which is used to verify the user's operating authority on the log management platform.

[0051] In a specific example, such as the command: "http: / / {IP}:8090 / api / v2 / agent / config / ?ip={Agent_ip}&source={Source}&type=Input", the data interaction interface address and version number, as well as the log collection Agent, can be specified to obtain the configuration information of the specified log collection Agent through the specified data interaction interface.

[0052] In an optional embodiment of the present invention, the log management platform can also be used to: determine the general call execution rules of the data interaction interface; configure basic rule elements for the general call execution rules of the data interaction interface; wherein the basic rule elements may include the execution actions, interface resource symbols and filtering conditions of the data interaction interface.

[0053] The general call execution rule may be a general rule for calling a data interaction interface, and the basic rule element may be each basic element constituting the general call execution rule.

[0054] In a specific example, the general call execution rule of the data interaction interface may be: execution action + IP of the data interaction interface + interface resource symbol + filtering condition. Among them, the IP of the data interaction interface in the general call execution rule of the data interaction interface may be defaulted.

[0055] Among them, taking the RestfulAPI interface as an example, the types of execution actions may include but are not limited to POST, PUT and GET. Among them, the execution action "POST" means creating a resource object, and the execution action "PUT" means updating a resource object, such as executing a custom sensitive information search function or a function that comes with the log management platform; the execution action "GET" means requesting or / downloading a resource object, for example, you can obtain the configuration of the log collection agent. The types of interface resource symbols may include but are not limited to customcommands, customscripts, agent, agent / config, search, and search / submit. Among them, customcommands means custom functions, customscripts means custom programs, agent means log collection agent, agent / config means the configuration of log collection agent, search means search, and search / submit means search submission. The filtering conditions can be configured according to the fields of the resource objects supported by the data interaction interface.

[0056] In an optional embodiment of the present invention, the custom log information retrieval script may include input parameters, script body and output results; the input parameters can be determined based on the global variables of the custom log information retrieval function; wherein the input parameters include a log collection intermediate agent identifier and a log file list: the script body is used to match and retrieve log file association information corresponding to the target query log based on the input parameters; the output parameters are used to determine the output method of the target query log based on the interface resource list of the data interaction interface, and output and display the target query log according to the output method of the target query log.

[0057] The log collection intermediate proxy identifier can be used to identify the log collection intermediate proxy. Exemplarily, the log collection intermediate proxy identifier can be the IP address or device unique identifier of the log collection intermediate proxy. The log file association information can be the relevant location information of the target query log. Exemplarily, the log file association information can include but is not limited to the log file name to which the target query log belongs and the line number of the stored log data.

[0058] Figure 4 1 is a flow chart of a log management system provided by the second embodiment of the present invention that collects log data through a custom log information retrieval script. Figure 4 As shown, users can use third-party development tools such as Shell or Python to write custom log information retrieval scripts, such as scripts for sensitive information retrieval. The content of the custom log information retrieval script may include three types: input parameters, script body, and output results. Among them, the input parameters can be determined according to the global variables of the custom log information retrieval function. Exemplarily, the input parameters may include a log collection intermediate agent identifier and a list of log files. The script body can be used to match the log file association information corresponding to the target query log according to the input parameter matching. Exemplarily, the script body can match the log file name and line number of the target query log according to the input parameter. The output parameter can be used to determine the output mode of the target query log according to the interface resource list of the data interaction interface, and output the target query log according to the output mode of the target query log. Exemplarily, the output parameter can be fed back to the RestfulAPI interface through the post method, and the target query log is delivered to the display interface of the log management platform by the RestfulAPI interface. Optionally, the content of the output parameter may include but is not limited to the log collection intermediate agent identifier corresponding to the target query log, the log file association information, and the content of the target query log. For example, the content of the output parameters may include, but is not limited to, the IP address of the log collection agent that collects sensitive log data, the log file name and line number corresponding to the sensitive log data, and the content of the sensitive log information.

[0059] Correspondingly, after the user completes the custom log information retrieval script using the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface through a third-party development tool, the custom log information retrieval script can be sent to the log management platform. After the log management platform receives the custom log information retrieval script, it can create a platform-adapted custom log information retrieval program based on the custom log information retrieval script.

[0060] Optionally, after the user configures the custom log information retrieval script according to the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface through a third-party development tool, it is also possible to store the custom log information retrieval script in the external storage path corresponding to the log management platform without sending the custom log information retrieval script to the log management platform for storage and use. When the log management platform needs to call the custom log information retrieval script, it can call the custom log information retrieval script in real time according to the storage path of the custom log information retrieval script, and generate the corresponding custom log information retrieval program in real time according to the custom log information retrieval script called in real time.

[0061] In an optional embodiment of the present invention, the log management platform can also be used to: upload and bind the custom log information retrieval script to create the custom log information retrieval program; configure the custom log information retrieval script and the name of the corresponding custom log information retrieval program.

[0062] Optionally, if the custom log information retrieval script is sent to the log management platform for storage and use, the user can trigger the custom program function of the log management platform to upload and bind the custom log information retrieval script to the log management platform, and create a custom log information retrieval program for the uploaded custom log information retrieval script, and configure the name of the custom log information retrieval script and the corresponding custom log information retrieval program. Optionally, the name of the custom log information retrieval script and the corresponding custom log information retrieval program can be the same or different, and the embodiment of the present invention does not limit this.

[0063] In a specific example, taking the "sensitive log information retrieval" executable script as an example, the "custom program" function of the log management platform can be used on the log management platform to upload the user-defined "sensitive log information retrieval" executable script and create it as a "sensitive log information search" program that can be managed by the log management platform. Optionally, the names of the "sensitive log information retrieval" executable script and the "sensitive log information search" program can both be configured as "sensitive_search".

[0064] Accordingly, after the user completes the custom log information retrieval script, or after the user uploads the custom log information retrieval script to the log management platform to create a custom log information retrieval program adapted to the platform, the user can create an adapted custom log information retrieval function for the custom log information retrieval script or the custom log information retrieval program on the log management platform, and configure the relevant parameters of the custom log information retrieval function. The custom log information retrieval function can be a function that can be directly executed by the log management platform.

[0065] In a specific example, the custom log information retrieval function can also be called a custom log information retrieval instruction. The user can trigger the function of creating a new instruction on the log management platform so that the log management platform pops up an interface for creating a new instruction. In this interface, the user can configure the instruction name of the corresponding custom log information retrieval function, the name of the corresponding custom log information retrieval program, resource tags, program parameters, and instruction type according to the custom log information retrieval script or the custom log information retrieval program. Exemplarily, the instruction name can be "sensitive_search", the name of the corresponding custom log information retrieval program can be "sensitive_search.tgz", and the program parameters can be "appname, [agentip]", where the parameter "appname" can be used to specify the application name to which the search log belongs, and the parameter "agentip" can be used to specify the target log collection intermediate agent for collecting log data. Optionally, the instruction type can be set to a distributed processing command. Optionally, it can also be set whether the custom log information retrieval function is executed in batches, etc.

[0066] In an optional embodiment of the present invention, the data interaction interface can also be used to: obtain the storage path of the custom log information retrieval script that matches the custom log information retrieval function; determine the original custom log information retrieval program that matches the custom log information retrieval function based on the storage path of the custom log information retrieval script and the associated information of the custom log information retrieval function; and perform parameter processing on the original custom log information retrieval program based on the log retrieval associated parameter information to obtain the custom log information retrieval program.

[0067] The associated information of the custom log information retrieval function may be information used to determine a matching custom log information retrieval program according to the custom log information retrieval function, and may include, but is not limited to, an identifier or name of the custom log information retrieval program. The original custom log information retrieval program may be a custom log information retrieval program or a custom log information retrieval script that does not assign values ​​to program parameters.

[0068] Optionally, when the upload of the custom log information retrieval script is bound to the log management platform, the storage path of the custom log information retrieval script matched by the custom log information retrieval function may be the storage path of the custom log information retrieval script within the log management platform, or may also be the storage path of the custom log information retrieval script outside the log management platform. The embodiment of the present invention does not limit this.

[0069] The user can generate the corresponding log query instruction according to the configuration of the custom log information retrieval function, and input the log query instruction into the search box of the log management platform, so as to call the custom log information retrieval script or the internally integrated custom log information retrieval program through the log management platform to execute the log query instruction for log search. After the user inputs the log query instruction into the search box of the log management platform, the log management platform can call the data interaction interface to parse the log query instruction, obtain the log retrieval associated parameter information and the storage path of the custom log information retrieval script matched by the custom log information retrieval function through the parsing result of the log query instruction, and thus determine the original custom log information retrieval program matched by the custom log information retrieval function in combination with the storage path of the custom log information retrieval script and the corresponding program name in the custom log information retrieval function. Further, the data interaction interface can perform parameter transfer processing on the original custom log information retrieval program according to the log retrieval associated parameter information obtained by parsing, such as determining the target log collection intermediate agent for collecting logs, that is, determining the value of "agentip", etc., and obtain a custom log information retrieval program that can be sent to the target log collection intermediate agent to automatically perform log retrieval.

[0070] In an optional embodiment of the present invention, the data interaction interface can also be used to: parse the log query instruction to obtain a log query object; match the proxy configuration information of the target log collection intermediate agent according to the log query object to obtain the log retrieval associated parameter information; wherein the log retrieval associated parameter information may include the target log collection intermediate agent identifier and a list of log files of the log query object.

[0071] Among them, the log query object can be a functional module or application object used to query logs, etc. Exemplarily, the log query object can be an application program, or a functional submodule in a device system, etc. As long as there is a log query demand, the embodiment of the present invention does not limit the specific object type of the log query object. The target log collection intermediate agent identifier can be used to uniquely identify the target log collection intermediate agent, for example, it can include but is not limited to the IP address or device unique identifier of the target log collection intermediate agent, etc. The log file list list can be a file list list that stores the required retrieved log information.

[0072] Specifically, the data interaction interface can parse the log query instruction to determine the log query object, such as determining the application name or function module name specified by the user for query, etc. At the same time, the data interaction interface can further match the identification of the target log collection intermediate agent and the internal log file and other agent configuration information according to the log query object determined by the analysis, so as to determine the log retrieval associated parameter information.

[0073] In a specific example, assuming that the user-configured custom log information retrieval script is used to retrieve sensitive log information, if the log query instruction entered by the user is: "|sensitive_search appname=ebank", the RestfulAPI interface parses the log query instruction and determines the parameter part "appname=ebank" in the instruction, and then determines that the log query object specified by the user is the application "ebank". Further, the data interaction interface triggers the sequential execution based on the parameter part of the log query object to obtain the following data:

[0074] A. List of Agent IP addresses corresponding to appname=ebank. That is, it is necessary to determine which log collection Agents correspond to the collection system devices (such as business system servers) that store the log information corresponding to the ebank application.

[0075] B. According to the Agent IP address list, obtain the collection log file name corresponding to the ebank application, and build a log file list according to each collection log file. It is understandable that the content of the log file list generated by different log collection agents may be different. The log collection agent can collect the corresponding log files through its own log file list to obtain the sensitive log information of the ebank application.

[0076] C. Concatenate the data results obtained in steps A and B into the URL of the RestfulAPI interface of the "Sensitive Log Information Search" executable program, parse the URL through the RestfulAPI interface, and execute the Agent collection action on the corresponding log collection Agent based on the parsing result to drive the "Sensitive Log Information Search" program to work in the corresponding Agent.

[0077] For example, for step A, the RestfulAPI interface can execute the get method to obtain the AgentIP address list corresponding to appname=ebank, and the method content is as follows:

[0078] get'Authorization:apikey username:laXXtaBKuQxZpjpiIsUEFpfcATVXu zyX' / agent / ? appname=ebank

[0079] get'Authorization:apikey username:laXXtaBKuQxZpjpiIsUEFpfcATVXu zyX' / agent / config / ? ip=xxx

[0080] In the above get method for obtaining the AgentIP address list corresponding to appname=ebank, "Authorization:apikey username:laXXtaBKuQxZpjpiIsUEFpfcATVXuzyX" is used to authenticate the user information to determine whether the current user has log query permissions. The "agent / ?appname=ebank" part is used to determine whether there is an ebank resource object on the agent, and the "agent / config / ?ip=xxx" part is used to obtain the AgentIP and other information corresponding to ebank.

[0081] For example, for step B, the RestfulAPI interface can execute the put method to obtain the list of log files, and the method content is as follows:

[0082] put'Authorization:apikey username:laXXtaBKuQxZpjpiIsUEFpfcATVXu zyX' / agent / config / ? ip={Agent_ip}&source={Source}&type=Input

[0083] In the get method for obtaining the log file list, "Authorization:apikey username:laXXtaBKuQxZpjpiIsUEFpfcATVXuzyX" is used to authenticate the user information to determine whether the current user has log query permissions. "agent / config / ?ip={Agent_ip}&source={Source}&type=Input" is used to specify the ebank log search file list for the determined AgentIP.

[0084] In the above example, when the log query instruction entered by the user is: "|sensitive_search appname=ebank", the call execution instruction generated by the RestfulAPI interface (ie, the URL of the RestfulAPI interface) can be:

[0085] PUT http: / / {IP}:8090 / api / v2 / search / submit / / appname=ebank / directory={custom log information retrieval script save path} / name=sensitive_search}

[0086] In the above method, "http: / / {IP}:8090 / api / v2 / " represents the RestfulAPI interface and its corresponding version number. "appname=ebank" specifies that the log query object is an electronic banking application. "directory={custom log information retrieval script saving path}" specifically specifies the saving path of the custom log information retrieval script. Since the custom log information retrieval program is generated according to the custom log information retrieval script, it can also be used as the saving path of the custom log information retrieval program. It should be noted that if the log management platform has created a custom log information retrieval program corresponding to the custom log information retrieval script, this part of the path information can be defaulted. "name=sensitive_search" specifies the program name of the custom log information retrieval program corresponding to the custom log information retrieval script. "appname=ebank / directory={custom log information retrieval script saving path} / name=sensitive_search}" is the filtering condition of the RestfulAPI interface.

[0087] The RestfulAPI interface parses the log query instruction "|sensitive_search appname=ebank" input by the user, and can determine the value of appname, so as to obtain the corresponding AgentIP and interface resource information of the log file list according to the value of appname, and further fill the values ​​of AgentIP and the log file list into the "Sensitive Log Information Search Program", and send the completed "Sensitive Log Information Search Program" to the corresponding log collection Agent according to the AgentIP, so as to execute the completed "Sensitive Log Information Search Program" through the corresponding log collection Agent to start searching for sensitive log information of electronic banking.

[0088] In an optional embodiment of the present invention, the target log collection intermediate agent can also be used to: query the target device according to the custom log information retrieval program to obtain the target query log, generate log query result data in the target data format according to the target query log, and feed back the log query result data to the data interaction interface; the data interaction interface can also be used to: adopt an interface feedback data display method to return the log query result data in the target data format to the search results page of the log management platform for display.

[0089] The target device may be a device that the target log collection intermediate agent can collect logs from, such as but not limited to a business system server or a personal computer terminal, etc. The target data format may be any type of compliant data format, such as but not limited to Json (a lightweight data exchange format) format, etc.

[0090] In an embodiment of the present invention, the target log collection intermediate agent can obtain the target query log after querying the log file corresponding to the target device according to the received custom log information retrieval program, and further organize the target query log to generate log query result data in a unified target data format, thereby feeding back the log query result data to the data interaction interface. After the data interaction interface receives the log query result data fed back by the target log collection intermediate agent, the log query result data can be fed back to the search result page of the log management platform using the interface feedback data display method for display.

[0091] In a specific example, the log collection agent directly executes the "Sensitive Log Information Search" program and returns the log query result data in Json format. After the RestfulAPI interface receives the log query result data in Json format, it can use the POST method to return it to the search result page of the log management platform. Correspondingly, the log management platform receives and parses the result segment fed back by the RestfulAPI interface, and displays the result in a table by default.

[0092] In an optional embodiment of the present invention, the log management platform can also be used to: determine the target built-in instructions of the log management platform; splice the log query instructions as instruction input parameters to the target built-in instructions of the log management platform; and perform target data operations on the log query data in the target data format through the target built-in instructions of the log management platform.

[0093] The target built-in instruction of the log management platform may be an instruction provided by the log management platform. The target data operation may be an operation that can be performed by the target built-in instruction of the log management platform, for example, including but not limited to the screening or statistics of log data.

[0094] In order to further improve the flexibility and extensibility of the custom log information retrieval function, the embodiment of the present invention can also splice the log query instructions configured by the user based on the custom log information retrieval function as instruction input parameters into the log management platform target built-in instructions, thereby performing corresponding target data operations on the queried log query data in the target data format through the log management platform target built-in instructions containing the log query instructions.

[0095] Exemplarily, the log query instruction can be spliced ​​as an input parameter into the analysis function provided by the platform, so as to query the required logs on demand through the log query instruction, and perform global statistical analysis on the queried logs.

[0096] The above technical solution realizes the registration and calling of the third-party execution program by defining an external execution program in the log management platform of the log management system. At the same time, in the log management platform, a custom function is created, and the custom function is associated with the third-party execution program, so that the flexible expansion of the log management analysis function can be realized. In the log management platform, through the parameter information in the custom function, the data interaction interface is automatically called to obtain other attribute information corresponding to the parameter as the log retrieval associated parameter information. The log collection intermediate agent executes the custom program function, agrees on the unified data format of the program result, and returns the result directly to the search result page of the log management platform through the data interaction interface. That is, the results between the log management platform and the log collection intermediate agent can be transmitted to each other through the data interaction interface, and the calling and result utilization of the third-party program are realized without changing the user's usage habits. Since the embodiment of the present invention can be executed within the scope of the log collection intermediate agent specified by the parameter, and the custom script developed by the general programming language directly matches the local log file, the development flexibility of the matching rule, the available function library, etc. far exceed the SPL (Search Processing Language, the search processing language of the log easy platform) language of the log platform itself, it has a better technical advantage in comprehensive efficiency. At the same time, the log management platform can also support the combined use of custom functions and built-in functions, so that relevant log data processing and analysis can be performed directly on the output results of the custom functions, further improving the flexibility and scalability of log retrieval processing.

[0097] Embodiment 3

[0098] Figure 5 is a flow chart of a log data query method provided by the third embodiment of the present invention. This embodiment is applicable to the case where the log management system described in the above embodiment is used to perform log data query. The method can be executed by the log management system. Figure 5 As shown, the method includes the following operations:

[0099] S110. Obtaining, through the log management platform, a log query instruction input by a log query user; wherein the log query instruction is generated according to a custom log information retrieval function.

[0100] S120: parsing the log query instruction through the data interaction interface to obtain log retrieval associated parameter information, generating a custom log information retrieval program according to the log retrieval associated parameter information, and sending the custom log information retrieval program to a target log collection intermediate agent.

[0101] S130: querying the target query log matching the log query instruction according to the custom log information retrieval program through the target log collection intermediate agent.

[0102] Optionally, a custom log information retrieval program is generated according to the log retrieval associated parameter information, including: obtaining the storage path of the custom log information retrieval script that matches the custom log information retrieval function through the data interaction interface; determining the original custom log information retrieval program that matches the custom log information retrieval function according to the storage path of the custom log information retrieval script and the associated information of the custom log information retrieval function through the data interaction interface; and performing parameter processing on the original custom log information retrieval program according to the log retrieval associated parameter information through the data interaction interface to obtain the custom log information retrieval program.

[0103] Optionally, the log query instruction is parsed through the data interaction interface to obtain log retrieval associated parameter information, including: parsing the log query instruction through the data interaction interface to obtain a log query object; matching the proxy configuration information of the target log collection intermediate agent according to the log query object through the data interaction interface to obtain the log retrieval associated parameter information; wherein the log retrieval associated parameter information includes the target log collection intermediate agent identifier and a list of log files of the log query object.

[0104] Optionally, before obtaining the log query instruction input by the log query user through the log management platform, it also includes: configuring the global variables of the custom log information retrieval function through the log management platform; configuring the interface resource list of the data interaction interface through the log management platform; receiving through the log management platform a custom log information retrieval script configured according to the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface through a third-party development tool; and creating the custom log information retrieval program through the log management platform according to the custom log information retrieval script.

[0105] Optionally, after creating the custom log information retrieval program according to the custom log information retrieval script through the log management platform, it also includes: determining the general call execution rules of the data interaction interface through the log management platform; configuring basic rule elements for the general call execution rules of the data interaction interface through the log management platform; wherein the basic rule elements include the execution actions, interface resource symbols and filtering conditions of the data interaction interface.

[0106] Optionally, the custom log information retrieval script includes input parameters, script body and output results; the input parameters are determined according to the global variables of the custom log information retrieval function; wherein the input parameters include a log collection intermediate agent identifier and a log file list: the script body is used to match and retrieve log file association information corresponding to the target query log according to the input parameters; the output parameters are used to determine the output method of the target query log according to the interface resource list of the data interaction interface, and output and display the target query log according to the output method of the target query log.

[0107] Optionally, after configuring basic rule elements through the general call execution rule configuration of the data interaction interface through the log management platform, it also includes: uploading and binding the custom log information retrieval script through the log management platform to create the custom log information retrieval program; configuring the custom log information retrieval script and the corresponding custom log information retrieval program name through the log management platform.

[0108] Optionally, after querying the target query log matching the log query instruction through the target log collection intermediate agent according to the custom log information retrieval program, the method further includes:

[0109] The target log collection intermediate agent queries the target device according to the custom log information retrieval program to obtain the target query log, generates log query result data in a target data format according to the target query log, and feeds back the log query result data to the data interaction interface; and uses the interface feedback data display method through the data interaction interface to return the log query result data in the target data format to the search result page of the log management platform for display.

[0110] Optionally, after returning the log query result data in the target data format to the search results page of the log management platform for display through the data interaction interface using the interface feedback data display method, it also includes: determining the log management platform target built-in instruction through the log management platform; splicing the log query instruction as an instruction input parameter to the log management platform target built-in instruction through the log management platform; and performing target data operations on the log query data in the target data format through the log management platform target built-in instruction.

[0111] The embodiment of the present invention constructs a log management system through a log management platform, a data interaction interface and a log collection intermediate agent, and obtains the log query instruction input by the log query user through the log management platform, generates the log query instruction according to the custom log information retrieval function, parses the log query instruction through the data interaction interface, obtains the log retrieval associated parameter information, generates the custom log information retrieval program according to the log retrieval associated parameter information, and sends the custom log information retrieval program to the target log collection intermediate agent, and then queries the target query log matching the log query instruction according to the custom log information retrieval program through the target log collection intermediate agent, solves the problems of low efficiency, high resource consumption, poor maintainability and poor flexibility existing in the existing log query management method, can realize on-demand log query, and improve the efficiency, convenience, maintainability and flexibility of log query retrieval.

[0112] The above log data query method is a method executable by the log management system provided by any embodiment of the present invention. For technical details not described in detail in this embodiment, please refer to the method executable by the log management system provided by any embodiment of the present invention.

[0113] Since the log data query method introduced above is a method executable by the log management system in the embodiment of the present invention, based on the method executable by the log management system introduced in the embodiment of the present invention, the technical personnel in the field can understand the specific implementation method of the log data query method of the present embodiment and its various variations, so how the log data query method implements the log data query process is not introduced in detail here. As long as the technical personnel in the field implement the process of the method executable by the log management system in the embodiment of the present invention, it belongs to the scope of protection of this application.

[0114] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information (such as account information, password information and sensitive information involving the user, etc.) are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0115] It should be noted that any arrangement and combination of the technical features in the above embodiments also falls within the protection scope of the present invention.

Claims

1. A log management system, characterized in that: It includes a log management platform, a data interaction interface and a log collection intermediate agent; the log management platform is communicatively connected with the data interaction interface, and the data interaction interface is communicatively connected with the log collection intermediate agent; wherein: The log management platform is used to obtain a log query instruction input by a log query user; wherein the log query instruction is generated according to a custom log information retrieval function, and the custom log information retrieval function is used to configure a query rule according to a log query requirement; The data interaction interface is used to parse the log query instruction, obtain log retrieval associated parameter information, generate a custom log information retrieval program according to the log retrieval associated parameter information, and send the custom log information retrieval program to the target log collection intermediate agent; The target log collection intermediate agent is used to query the target query log matching the log query instruction according to the custom log information retrieval program; The log management platform is also used for: Configure the global variables of the custom log information retrieval function; Configure the interface resource list of the data interaction interface; Receiving a custom log information retrieval script configured by a third-party development tool according to the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface; The custom log information retrieval program is created according to the custom log information retrieval script.

2. The system according to claim 1, characterized in that The data interaction interface is also used for: Obtaining the storage path of the custom log information retrieval script matched by the custom log information retrieval function; Determining an original custom log information retrieval program that matches the custom log information retrieval function according to the storage path of the custom log information retrieval script and the associated information of the custom log information retrieval function; The original custom log information retrieval program is processed by parameter transfer according to the log retrieval associated parameter information to obtain the custom log information retrieval program.

3. The system according to claim 1, characterized in that The data interaction interface is also used for: Parsing the log query instruction to obtain a log query object; Matching the proxy configuration information of the target log collection intermediate proxy according to the log query object to obtain the log retrieval associated parameter information; The log retrieval associated parameter information includes a target log collection intermediate agent identifier and a list of log files of the log query object.

4. The system according to claim 1, characterized in that The log management platform is also used for: Determining a general call execution rule of the data interaction interface; Configure basic rule elements for general call execution rules of the data interaction interface; The basic rule elements include the execution action, interface resource symbol and filtering condition of the data interaction interface.

5. The system according to claim 1, characterized in that The custom log information retrieval script includes input parameters, script text and output results; the input parameters are determined according to the global variables of the custom log information retrieval function; The input parameters include the log collection intermediate agent identifier and the log file list: The script body is used to match and retrieve log file association information corresponding to the target query log according to the input parameter; The output parameter is used to determine the output mode of the target query log according to the interface resource list of the data interaction interface, and output and display the target query log according to the output mode of the target query log.

6. The system according to claim 1, characterized in that The log management platform is also used for: Upload and bind the custom log information retrieval script to create the custom log information retrieval program; Configure the custom log information retrieval script and the name of the corresponding custom log information retrieval program.

7. The system according to claim 1, characterized in that The target log collection intermediate agent is also used for: Querying the target device according to the custom log information retrieval program to obtain the target query log, generating log query result data in a target data format according to the target query log, and feeding back the log query result data to the data interaction interface; The data interaction interface is also used to: adopt an interface feedback data display method to return the log query result data in the target data format to the search result page of the log management platform for display.

8. The system according to claim 6, characterized in that The log management platform is also used for: Determine the target built-in instructions of the log management platform; Splicing the log query instruction as an instruction input parameter to the target built-in instruction of the log management platform; Target data operations are performed on log query data in a target data format through the log management platform target built-in instructions.

9. A log data query method, characterized in that: The method is applied to a log management system, which includes a log management platform, a data interaction interface, and a log collection intermediate agent; The log management platform is communicatively connected with the data interaction interface, and the data interaction interface is communicatively connected with the log collection intermediate agent. The method includes: Obtaining a log query instruction input by a log query user through the log management platform; wherein the log query instruction is generated according to a custom log information retrieval function, and the custom log information retrieval function is used to configure a query rule according to a log query requirement; Parsing the log query instruction through the data interaction interface to obtain log retrieval associated parameter information, generating a custom log information retrieval program according to the log retrieval associated parameter information, and sending the custom log information retrieval program to the target log collection intermediate agent; Querying the target query log matching the log query instruction according to the custom log information retrieval program through the target log collection intermediate agent; Before obtaining the log query instruction input by the log query user through the log management platform, the method further includes: The global variables of the custom log information retrieval function are configured through the log management platform; the interface resource list of the data interaction interface is configured through the log management platform; the custom log information retrieval script configured according to the global variables of the custom log information retrieval function and the interface resource list of the data interaction interface by a third-party development tool is received through the log management platform; and the custom log information retrieval program is created according to the custom log information retrieval script through the log management platform.

Images