Method and apparatus for storing server backend logs

Abstract

The application discloses a kind of storage method and device of server backend log, receive the configuration file defined by user in front end, the configuration file includes the log identification of each type log and corresponding warehouse format, based on configuration file, create the command list model based on log identification naming;Create link with device interaction;Call the read operation method of Filebeat read-write operation device, read a line of command of command list model, send command to device, after receiving command, the device reads the log meeting the requirement of command;After receiving the log returned by device, add search attribute for log, call the write operation method of Filebeat read-write operation device, based on the log identification of log and corresponding warehouse format, write log into ES cluster corresponding server.The application can solve the problem of large space occupation and retrieval difficulty caused by too large number of logs and chaotic types in the log storage process.

Description

Technical Field

[0001] This invention relates to the field of distributed technology, and in particular to a method and apparatus for storing server backend logs. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] The storage and management of server backend logs is a crucial aspect of network operations and maintenance. Currently, with the increasing variety of network devices and the growing complexity of log content, simple and direct log recording and storage methods are not only space-consuming but also inconvenient for log retrieval, making them increasingly unsuitable for the current environment. Furthermore, the complex and irregular log formats can affect the efficiency of maintenance personnel in troubleshooting related functions and may even lead to major production incidents.

[0004] Therefore, a server backend log storage solution is needed to address the problems of excessive space consumption and retrieval difficulties caused by the large number and diverse types of logs during log storage. Summary of the Invention

[0005] This invention provides a method for storing server backend logs to solve the problems of large space consumption and difficult retrieval caused by the large number and diverse types of logs during log storage. The method includes:

[0006] Receive the configuration file defined by the user on the front end. The configuration file includes the log identifier for each type of log and the corresponding database entry format. Create a command list model named based on the log identifier based on the configuration file.

[0007] Create links for interaction with the device;

[0008] The Filebeat read / write operation device's read operation method is invoked to read a line of command from the command list model and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command.

[0009] After receiving the logs returned by the device, a retrieval attribute is added to the logs, and the write operation method of the Filebeat read / write operation device is called to write the logs to the corresponding server in the Elasticsearch cluster based on the log identifier and the corresponding input format.

[0010] This invention also provides a storage device for server backend logs to solve the problems of large space consumption and difficulty in retrieval caused by the large number and diverse types of logs during log storage. The device includes:

[0011] The device configuration interaction module is used to receive configuration files defined by the user on the front end. The configuration files include log identifiers for each type of log and their corresponding database formats. Based on the configuration files, a command list model named according to the log identifiers is created; and a link for interaction with the device is created.

[0012] The log collection module is used to call the read operation method of the Filebeat read and write operation device, read a line of command from the command list model, and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command.

[0013] The log storage module is used to add retrieval attributes to the logs after receiving them from the device, call the write operation method of the Filebeat read / write operation device, and write the logs to the corresponding servers in the Elasticsearch cluster based on the log identifier and the corresponding input format.

[0014] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described method for storing server backend logs.

[0015] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for storing server backend logs.

[0016] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for storing server backend logs.

[0017] In this embodiment of the invention, a configuration file defined by the user on the front end is received. The configuration file includes a log identifier for each type of log and its corresponding storage format. A command list model named based on the log identifier is created based on the configuration file. A link for interaction with the device is created. The read operation method of the Filebeat read / write operation device is called to read a line of command from the command list model and send the command to the device. After receiving the command, the device reads the logs that meet the command requirements. After receiving the logs returned by the device, retrieval attributes are added to the logs. The write operation method of the Filebeat read / write operation device is called to write the logs to the corresponding server in the Elasticsearch cluster based on the log identifier and the corresponding storage format. Compared with existing technologies, a command list model can be constructed through a predefined configuration file, thereby obtaining logs on demand. After obtaining the logs, retrieval attributes are added to the logs, and then the logs are written to the corresponding service in the Elasticsearch cluster based on the log identifier and the corresponding storage format. This makes the stored logs easy to retrieve and facilitates unified management based on the storage format. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0019] Figure 1 This is a flowchart of the server backend log storage method in an embodiment of the present invention;

[0020] Figure 2 This is another flowchart of the server backend log storage method in an embodiment of the present invention;

[0021] Figure 3 This is a flowchart illustrating the log display in an embodiment of the present invention;

[0022] Figure 4 This is a flowchart of log query in an embodiment of the present invention;

[0023] Figure 5 This is another flowchart illustrating the storage of server backend logs in an embodiment of the present invention;

[0024] Figure 6 This is a schematic diagram of the storage device for server backend logs in an embodiment of the present invention;

[0025] Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0027] The approach of this invention is to set a unified entry format for logs when they are entered into the backend database, and to give the entered logs field search functionality, so as to improve the efficiency of log entry and the user experience when searching for logs.

[0028] The solutions proposed in the embodiments of the present invention will be described in detail below.

[0029] Figure 1 This is a flowchart of a method for storing server backend logs in an embodiment of the present invention, including:

[0030] Step 101: Receive the configuration file defined by the user on the front end. The configuration file includes the log identifier for each type of log and the corresponding database entry format. Create a command list model named based on the log identifier based on the configuration file. Specifically, fields of the same type adopt a unified database entry format, which can be a standard JSON format, to facilitate management and provide visualized data.

[0031] Step 102: Create a link for interacting with the device; for example, it could be a session link.

[0032] Step 103: Call the read operation method of the Filebeat read / write operation device to read a line of command from the command list model and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command.

[0033] Step 104: After receiving the logs returned by the device, add retrieval attributes to the logs, call the write operation method of the Filebeat read / write operation device, and write the logs to the corresponding servers in the Elasticsearch cluster based on the log identifier and the corresponding input format. Specifically, one or more servers can be selected.

[0034] In this embodiment of the invention, a command list model can be constructed through a predefined configuration file, thereby obtaining logs on demand. After obtaining the logs, retrieval attributes are added to the logs, and the logs are written to the corresponding service of the ES cluster based on the log identifier and the corresponding storage format, making the stored logs easy to retrieve and easy to achieve unified management based on the storage format.

[0035] In one embodiment, see Figure 2 After writing the logs to the corresponding servers in the Elasticsearch cluster, the method further includes:

[0036] Step 201: Receive the write result returned by the server;

[0037] Step 202: After the connection with the device is closed, the write result is returned to Filebeat, where the write result is persisted to the database.

[0038] In one embodiment, the retrieval attributes include one or any combination of log name, log path, log source file, entry time, server IP, device IP, and affiliated organization.

[0039] Figure 3 The flowchart for displaying logs in this embodiment of the invention includes:

[0040] Step 301: Create a result list model named based on log identifiers;

[0041] Step 302: After receiving the logs returned by the device, add the logs to the result list model;

[0042] Step 303: Using log identifiers as keywords, query the command list model and the result list model;

[0043] Step 304: Calculate the length of the command list model and the length of the result list model, and calculate the log collection progress;

[0044] Step 305: Display the contents of the command list model and the result list model, as well as the log collection progress.

[0045] In one embodiment, the log collection progress is calculated using the following formula:

[0046] R=round(SIZE2 / SIZE1)×100%

[0047] Where R represents the log collection progress, SIZE1 represents the length of the command list model, and SIZE2 represents the length of the result list model.

[0048] Figure 4 The flowchart for log querying in this embodiment of the invention includes:

[0049] Step 401: Receive a query command input by the user on the front end (e.g., the Kibana interface of the ES server). The query command includes the query conditions input by the user. The query conditions include one or any combination of query type, time range, server IP, and device IP.

[0050] Step 402: Based on the query conditions, match the retrieval attributes of the logs in the server to obtain the retrieval results.

[0051] Figure 5 Another flowchart of server backend log storage in this embodiment of the invention includes:

[0052] Step 501: Receive the log task management signal set by the user on the front end, and cache the log task management signal in the Filebeat log collection model. The log task management signal is a log task pause signal and has a log identifier.

[0053] Step 103 specifically includes: after reading a line of command from the command list model, reading the log task management signal, and determining whether the log task management signal is a log task pause signal based on the log identifier of the command. If not, the command is sent to the device.

[0054] In summary, the method proposed in the embodiments of the present invention has the following beneficial effects and advantages:

[0055] First, it can effectively improve the query speed of backend logs that have been entered into the database, improve the efficiency of troubleshooting when the system is abnormal, and further realize the transformation of manual operation by operation and maintenance personnel to automated functions.

[0056] Secondly, it can display the log storage progress and entry status, providing effective data and judgment basis for operation and maintenance personnel to monitor the log storage process.

[0057] Third, it enhances the controllability of automated tasks, reducing the operational and maintenance burden on frontline production personnel. It has significant application value in improving the automation control capabilities of the IT industry.

[0058] This invention also provides a storage device for server backend logs, as described in the following embodiments. Since the principle by which this device solves the problem is similar to the method for storing server backend logs, the implementation of this device can refer to the implementation of the method for storing server backend logs; repeated details will not be elaborated further.

[0059] Figure 6 This is a schematic diagram of a server backend log storage device in an embodiment of the present invention, comprising:

[0060] The device configuration interaction module 601 is used to receive the configuration file defined by the user on the front end. The configuration file includes the log identifier of each type of log and the corresponding database format. Based on the configuration file, a command list model named according to the log identifier is created; and a link for interaction with the device is created.

[0061] The log collection module 602 is used to call the read operation method of the Filebeat read and write operation device, read a line of command from the command list model, and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command.

[0062] The log storage module 603 is used to add retrieval attributes to the logs after receiving them from the device, call the write operation method of the Filebeat read and write operation device, and write the logs to the corresponding servers in the ES cluster based on the log identifier and the corresponding storage format.

[0063] In one embodiment, the log storage module is further configured to:

[0064] After writing the logs to the corresponding server in the Elasticsearch cluster, receive the write results returned by the server;

[0065] After the connection with the device is closed, the write result is returned to Filebeat, where the write result is persisted to the database.

[0066] In one embodiment, the retrieval attributes include one or any combination of log name, log path, log source file, entry time, server IP, device IP, and affiliated organization.

[0067] In one embodiment, the device configuration interaction module is further configured to: create a result list model named based on log identifiers;

[0068] The log storage module is also used to add logs to the result list model after receiving logs returned by the device;

[0069] The device also includes a display module 604, used for:

[0070] Query the command list model and the result list model using log identifiers as keywords;

[0071] Calculate the length of the command list model and the length of the result list model, and calculate the log collection progress;

[0072] Display the contents of the command list model and result list model, as well as the log collection progress.

[0073] In one embodiment, the display module is specifically used for:

[0074] The log collection progress is calculated using the following formula:

[0075] R=round(SIZE2 / SIZE1)×100%

[0076] Where R represents the log collection progress, SIZE1 represents the length of the command list model, and SIZE2 represents the length of the result list model.

[0077] In one embodiment, a query module 605 is further included, for:

[0078] Receive query instructions input by users on the front end. The query instructions include query conditions input by users. The query conditions include one or any combination of query type, time range, server IP, and device IP.

[0079] Based on the query conditions, the search attributes of the logs in the server are matched to obtain the search results.

[0080] In one embodiment, a log task management module 606 is further included, for:

[0081] Receive the log task management signal set by the user on the front end, and cache the log task management signal in the Filebeat log collection model. The log task management signal is a log task pause signal and has a log identifier.

[0082] The log acquisition module is specifically used to: after reading a line of command from the command list model, read the log task management signal, and determine whether the log task management signal is a log task pause signal based on the log identifier of the command. If not, the command is sent to the device.

[0083] In summary, the device proposed in the embodiments of the present invention has the following beneficial effects and advantages:

[0084] First, it can effectively improve the query speed of backend logs that have been entered into the database, improve the efficiency of troubleshooting when the system is abnormal, and further realize the transformation of manual operation by operation and maintenance personnel to automated functions.

[0085] Secondly, it can display the log storage progress and entry status, providing effective data and judgment basis for operation and maintenance personnel to monitor the log storage process.

[0086] Third, it enhances the controllability of automated tasks, reducing the operational and maintenance burden on frontline production personnel. It has significant application value in improving the automation control capabilities of the IT industry.

[0087] This invention also provides a computer device. Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 700 includes a memory 710, a processor 720, and a computer program 730 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 730, it implements the above-described method for storing server backend logs.

[0088] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described method for storing server backend logs.

[0089] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described method for storing server backend logs.

[0090] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0091] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0092] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0093] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0094] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for storing server backend logs, characterized in that, include: Receive the configuration file defined by the user on the front end. The configuration file includes the log identifier for each type of log and the corresponding database entry format. Create a command list model named based on the log identifier based on the configuration file. Create links for interaction with the device; The Filebeat read / write operation device's read operation method is invoked to read a line of command from the command list model and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command. After receiving the logs returned by the device, add retrieval attributes to the logs, call the write operation method of the Filebeat read and write operation device, and write the logs to the corresponding servers in the ES cluster based on the log identifier and the corresponding database format. Receive the log task management signal set by the user on the front end, and cache the log task management signal in the Filebeat log collection model. The log task management signal is a log task pause signal and has a log identifier. After reading a line of command from the command list model, the log task management signal is read. Based on the log identifier of the command, it is determined whether the log task management signal is a log task pause signal. If not, the command is sent to the device.

2. The method as described in claim 1, characterized in that, After writing the logs to the corresponding servers in the Elasticsearch cluster, the following steps are also included: Receive the write result returned by the server; After the connection with the device is closed, the write result is returned to Filebeat, where the write result is persisted to the database.

3. The method as described in claim 1, characterized in that, The retrieval attributes include one or any combination of log name, log path, log source file, entry time, server IP, device IP, and affiliated organization.

4. The method as described in claim 1, characterized in that, Also includes: Create a result list model named based on log identifiers; After receiving the logs returned by the device, add the logs to the results list model; Query the command list model and the result list model using log identifiers as keywords; Calculate the length of the command list model and the length of the result list model, and calculate the log collection progress; Display the contents of the command list model and result list model, as well as the log collection progress.

5. The method as described in claim 4, characterized in that, The log collection progress is calculated using the following formula: R = round(SIZE2 / SIZE1) × 100% Where R represents the log collection progress, SIZE1 represents the length of the command list model, and SIZE2 represents the length of the result list model.

6. The method as described in claim 1, characterized in that, Also includes: Receive query instructions input by users on the front end. The query instructions include query conditions input by users. The query conditions include one or any combination of query type, time range, server IP, and device IP. Based on the query conditions, the search attributes of the logs in the server are matched to obtain the search results.

7. A storage device for server backend logs, characterized in that, include: The device configuration interaction module is used to receive configuration files defined by the user on the front end. The configuration files include log identifiers for each type of log and their corresponding database formats. Based on the configuration files, a command list model named according to the log identifiers is created; and a link for interaction with the device is created. The log collection module is used to call the read operation method of the Filebeat read and write operation device, read a line of command from the command list model, and send the command to the device. After receiving the command, the device reads the log that meets the requirements of the command. The log storage module is used to add retrieval attributes to the logs after receiving them from the device, call the write operation method of the Filebeat read and write operation device, and write the logs to the corresponding servers in the ES cluster based on the log identifier and the corresponding storage format. The log task management module is used to receive log task management signals set by the user on the front end and cache the log task management signals in the Filebeat log acquisition model. The log task management signals are log task pause signals and have log identifiers. The log acquisition module is specifically used to read the log task management signal after reading a line of command from the command list model, and determine whether the log task management signal is a log task pause signal based on the log identifier of the command. If not, the command is sent to the device.

8. The apparatus as claimed in claim 7, characterized in that, The log storage module is also used for: After writing the logs to the corresponding server in the Elasticsearch cluster, receive the write results returned by the server; After the connection with the device is closed, the write result is returned to Filebeat, where the write result is persisted to the database.

9. The apparatus as claimed in claim 7, characterized in that, The retrieval attributes include one or any combination of log name, log path, log source file, entry time, server IP, device IP, and affiliated organization.

10. The apparatus as claimed in claim 7, characterized in that, The device configuration interaction module is also used to: create a result list model named based on log identifiers; The log storage module is also used to add logs to the result list model after receiving logs returned by the device; The device also includes a display module for: Query the command list model and the result list model using log identifiers as keywords; Calculate the length of the command list model and the length of the result list model, and calculate the log collection progress; Display the contents of the command list model and result list model, as well as the log collection progress.

11. The apparatus as claimed in claim 10, characterized in that, The display module is specifically used for: The log collection progress is calculated using the following formula: R = round(SIZE2 / SIZE1) × 100% Where R represents the log collection progress, SIZE1 represents the length of the command list model, and SIZE2 represents the length of the result list model.

12. The apparatus as claimed in claim 7, characterized in that, It also includes a query module, used for: Receive query instructions input by users on the front end. The query instructions include query conditions input by users. The query conditions include one or any combination of query type, time range, server IP, and device IP. Based on the query conditions, the search attributes of the logs in the server are matched to obtain the search results.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 6.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

15. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 6.

Images