Log management system and method and storage medium

Abstract

The invention provides a log management system and method and a storage medium. The log system comprises a dispatching center platform, a configuration center platform, a log extractor arranged in each external production system, a sorting center platform and a storage center platform, the dispatching center platform obtains corresponding configuration information from the configuration center platform, generates corresponding instructions and sends the instructions to the corresponding nodes respectively, and the corresponding nodes correspondingly process the log data according to the instructions to obtain target log data; and sending the target log data to a specified log node in the storage center platform for storage. According to the method provided by the invention, the log data in each production system is uniformly managed, so that the log management efficiency is improved, and the operation and maintenance cost is reduced.

Description

Technical Field

[0001] This application relates to the field of log management, and particularly to a log management system, method, and storage medium. Background Art

[0002] Logs are generated by software and record information about application programs, servers, or system operations, activities, and usage patterns. Logs include the history of all processes, events, and messages, as well as other descriptive data. Through logs, errors can be identified and troubleshot, operations can be improved, resource allocation efficiency can be enhanced, user behavior can be understood, security can be strengthened, etc. Therefore, logs play an important role in operation and maintenance work.

[0003] In the prior art, an enterprise may have multiple systems, and each system independently constructs its own log system and provides a simple log query function based on the log system, or performs simple secondary analysis on the collected log data to extract some simple statistical information to enable developers to perform tasks such as fault location on the system.

[0004] Based on this, enterprise maintenance personnel need to separately maintain each system and adjust their respective log systems when needed, resulting in increased maintenance costs. Summary of the Invention

[0005] This application provides a log management system, method, and storage medium to solve the technical problem of increased maintenance costs of the log system.

[0006] In a first aspect, this application provides a log management system, including: a scheduling center platform, a configuration center platform, log extractors set in each external production system, a sorting center platform, and a storage center platform; wherein,

[0007] The scheduling center platform is respectively connected to the configuration center platform and multiple log extractors, and is used to obtain pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generate scheduling instructions based on the pre-configured log configuration information, and send the scheduling instructions to each log extractor respectively, and generate sorting instructions based on the pre-configured sorting configuration information, and send the sorting instructions to each memory in the storage center platform;

[0008] The log extractor is used to retrieve raw log data from the corresponding log system according to the received scheduling instructions, and send the raw log data to the specified sorter in the sorting center platform;

[0009] The sorter, connected to the storage center platform, is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the specified log node in the storage center platform for storage.

[0010] Optionally, the above log management system includes:

[0011] The sorter is specifically used to extract report log data from the original log data according to the received sorting instruction, and perform format conversion on the report log data according to the pre-configured log structure in the sorting instruction to obtain the target log data, and send the target log data to the log node specified in the sorting instruction in the storage center platform for storage.

[0012] Optionally, the above log management system further includes: the computing center platform; the storage center platform further includes: the report node;

[0013] Then the scheduling center platform, connected to the computing center platform, is further used to obtain the pre-configured computing configuration information from the configuration center platform, generate a computing instruction based on the pre-configured computing configuration information, and send the computing instruction to each computing node in the computing center platform;

[0014] The computing node is used to read the specified target log data from the specified log node in the storage center platform according to the computing instruction, perform report processing on the specified target log data according to the pre-configured computing task to obtain the report data, and send the report data to the specified report node in the storage center platform for storage.

[0015] Optionally, the storage center platform further includes: the archival node;

[0016] The storage center platform performs archival processing on the target log data already stored in the log node and / or the report data already stored in the report node according to the pre-configured archival information, so as to store the data after archival processing in the archival node.

[0017] Optionally, the above log management system includes:

[0018] The log extractor is used to retrieve the original log data matching the specified log type from the corresponding log system according to the specified log type of the received scheduling instruction, and send the original log data to the specified sorter in the sorting center platform;

[0019] Wherein, the specified log type includes one or several combinations of the following: specified log identifier, specified log structure, specified log storage, and specified log performance.

[0020] Optionally, the dispatching center platform includes: a dispatching service unit, a perception service unit, and an analysis service unit; where

[0021] The perception service unit is used to obtain the running status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node respectively;

[0022] The analysis service unit is used to analyze and process the running status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node, so as to generate or update the pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

[0023] Optionally, the above log management system further includes:

[0024] The application center platform is connected to the report node in the storage center platform, and is used to obtain specified report data from the specified report node according to the obtained display request, and perform corresponding display processing on the specified report data.

[0025] In a second aspect, the present application provides a log management method, which is applied to a log management system. The log system includes: a dispatching center platform, a configuration center platform, log extractors set in each external production system, a sorting center platform, and a storage center platform; then the method includes:

[0026] The dispatching center platform obtains the pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generates a dispatching instruction based on the pre-configured log configuration information, and sends the dispatching instruction to each log extractor respectively, and generates a sorting instruction based on the pre-configured sorting configuration information, and sends the sorting instruction to each storage device in the storage center platform;

[0027] The log extractor retrieves the original log data from the corresponding log system according to the received dispatching instruction, and sends the original log data to the specified sorter in the sorting center platform;

[0028] The sorter is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the specified log node in the storage center platform for storage.

[0029] Optionally, the scheduling center platform in the log management system as described above includes: a scheduling service unit, a perception service unit, and an analysis service unit; wherein, the perception service unit obtains the operation status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node respectively;

[0030] The analysis service unit analyzes and processes the operation status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node to generate or update the pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

[0031] In a third aspect, the present application provides an electronic device, including: a processor, and a memory communicatively connected to the processor;

[0032] The memory stores computer-executable instructions;

[0033] The processor executes the computer-executable instructions stored in the memory to implement the log management method as described above.

[0034] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer-executable instructions are stored, and when the computer-executable instructions are executed by a processor, they are used to implement the log management method as described above.

[0035] A log management system, method, and storage medium provided by this application. The system includes a scheduling center platform, a configuration center platform, log extractors set in each external production system, a sorting center platform, and a storage center platform. Among them, the scheduling center platform obtains pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generates scheduling instructions based on the pre-configured log configuration information, sends the scheduling instructions to each log extractor respectively, and generates sorting instructions based on the pre-configured sorting configuration information, and sends the sorting instructions to each storage device in the storage center platform; the log extractor retrieves the original log data from the corresponding log system according to the received scheduling instructions, and sends the original log data to the specified sorter in the sorting center platform; the sorter is used to perform corresponding sorting processing on the original log data according to the received sorting instructions to obtain the target log data, and send the target log data to the specified log node in the storage center platform for storage. Compared with the relatively high maintenance cost caused by each external production system independently building its own log system and performing separate maintenance in the prior art, this application proposes a centralized log management system, that is, a scheduling center platform, a configuration center platform, a sorting center platform, and a storage center platform are constructed, and log extractors are constructed in each external production system, so as to centrally manage the log data extracted from each external production system based on the scheduling center platform, the configuration center platform, the sorting center platform, and the storage center platform, thereby realizing the efficient collection and storage of massive logs, reducing the operations of maintenance personnel, and reducing the maintenance cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The accompanying drawings here are incorporated into the specification and form a part of this specification, showing embodiments consistent with this application, and are used together with the specification to explain the principles of this application.

[0037] Figure 1 It is a schematic structural diagram of a log management system provided by this application;

[0038] Figure 2 It is a schematic structural diagram of a log management system provided by this application;

[0039] Figure 3 It is a schematic structural diagram of a log management system provided by this application;

[0040] Figure 4 It is a schematic structural diagram of another log management system provided by this application;

[0041] Figure 5 It is a schematic flowchart of a log management method provided by this application;

[0042] Figure 6Schematic structural diagram of the electronic device provided by this application.

[0043] Through the above-mentioned drawings, specific embodiments of this application have been shown, and there will be more detailed descriptions hereinafter. These drawings and textual descriptions are not intended to limit the scope of the concept of this application in any way, but to illustrate the concept of this application to those skilled in the art by referring to specific embodiments. Detailed implementation manners

[0044] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description refers to the drawings, unless otherwise indicated, the same numerals in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all implementation manners consistent with this application. On the contrary, they are merely examples of devices and methods consistent with some aspects of this application as detailed in the appended claims.

[0045] It should be noted that the data involved in this application (including but not limited to data for analysis, stored data, displayed data, etc.) are all information and data that have been fully authorized by all parties, and the collection, use, and processing of relevant data need to comply with relevant laws, regulations, and standards, and corresponding operation entrances are provided for users to choose to authorize or reject.

[0046] In the prior art, in view of the continuous increase in the number of logs in each enterprise system, log management has become an important part of the operation and maintenance work, so as to improve the efficiency of the operation and maintenance work.

[0047] However, in an enterprise, multiple systems may be set up, and each system independently constructs its own log system. Enterprise maintenance personnel need to separately maintain each system and adjust their respective systems when needed, thus increasing the operation and maintenance costs.

[0048] Based on this, in the prior art, each system independently constructs its own log system and conducts separate maintenance, which will lead to an increase in operation and maintenance costs.

[0049] To solve the above problems, the inventive concept of this application lies in: how to set configuration information according to the current information, generate instructions, and centralize the logs of each system into one system, so as to uniformly and efficiently manage the logs of each system, thereby solving the problem of increased operation and maintenance costs caused by each system independently constructing its own log system and conducting separate maintenance in the prior art.

[0050] The following uses specific embodiments to elaborate in detail on the technical solution of this application and how the technical solution of this application solves the above technical problems. The following several specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of this application will be described below in conjunction with the accompanying drawings.

[0051] Figure 1 It is a schematic structural diagram of a log management system provided for the first embodiment of this application. As Figure 1 shown, the system may include: a scheduling center platform 11, a configuration center platform 12, a log extractor 13 provided in each external production system, a sorting center platform 14, and a storage center platform 15.

[0052] Specifically, the scheduling center platform 11 is respectively connected to the configuration center platform 12 and multiple log extractors 13, and is used to obtain pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform 12, generate a scheduling instruction based on the pre-configured log configuration information, and send the scheduling instruction to each log extractor 13 respectively, and generate a sorting instruction based on the pre-configured sorting configuration information, and send the sorting instruction to each memory 151 in the storage center platform 15. The log extractor 13 is used to retrieve the original log data from the corresponding log system according to the received scheduling instruction, and send the original log data to the specified sorter 141 in the sorting center platform 14. The sorter 141 is connected to the storage center platform 15, and is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the specified log node 151 in the storage center platform 15 for storage.

[0053] In this embodiment, for example, the pre-configured log configuration information in the configuration center platform 12 may be an error log type identifier, the log structure is in the form of a script programming language object notation (JavaScript Object Notation, JSON) format, and the storage address for log storage. For example, the storage address for log storage may be from log node 1 to log node 5. In addition, the pre-configured sorting configuration information may be that the sorter 141 may specifically be the 5th sorter to the 10th sorter to sort the above-obtained log data.

[0054] Furthermore, the dispatching center platform 11 can generate a dispatching instruction according to the pre-configured log configuration information and the pre-configured sorting configuration information, and send the dispatching instruction to each log extractor 13, so that the log extractor 13 extracts the original log data in each production system based on the dispatching instruction, and sends the extracted original log data to the 5th sorter to the 10th sorter respectively. The 5th sorter to the 10th sorter have already received the sorting instruction in advance. Thus, according to the sorting instruction, the 5th sorter to the 10th sorter can respectively perform corresponding sorting processing on the original log data, and after obtaining the corresponding target log data respectively, they can respectively send the target log data to the specified log nodes in the storage center platform for storage. For example: the received original log data includes the first original log data, the second original log data, and the third original log data; according to the dispatching instruction generated according to the pre-configured log configuration information and the pre-configured sorting configuration information, the original log data is sent to the 5th sorter to the 10th sorter; the sorter performs corresponding sorting processing according to the sorting instruction generated according to the pre-configured sorting configuration information, and obtains the first target log data, the second target log data, and the third target log data, stores the first target log data in log node 1, stores the second target log data in log nodes 2 to 3, and stores the third target log data in log nodes 4 to 5.

[0055] In this embodiment, the system includes a scheduling center platform, a configuration center platform, a log extractor set in each external production system, a sorting center platform, and a storage center platform. Among them, the scheduling center platform obtains pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generates a scheduling instruction based on the pre-configured log configuration information, sends the scheduling instruction to each log extractor respectively, and generates a sorting instruction based on the pre-configured sorting configuration information, and sends the sorting instruction to each memory in the storage center platform; the log extractor retrieves the original log data from the corresponding log system according to the received scheduling instruction, and sends the original log data to the designated sorter in the sorting center platform; the sorter is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the designated log node in the storage center platform for storage. Compared with the relatively high maintenance cost caused by each external production system independently building its own log system and performing separate maintenance in the prior art, the present application proposes a centralized log management system, that is, a scheduling center platform, a configuration center platform, a sorting center platform, and a storage center platform are constructed, and a log extractor is constructed in each external production system, so as to centrally manage the log data extracted from each external production system based on the scheduling center platform, the configuration center platform, the sorting center platform, and the storage center platform, thereby realizing the efficient collection and storage of massive logs, reducing the operations of maintenance personnel, and reducing the maintenance cost.

[0056] Figure 2 It is a schematic structural diagram of a log management system provided in the second embodiment of the present application. As Figure 2 shown, on the basis of the above embodiment, the log extractor 13 is used to retrieve the original log data matching the specified log type from the corresponding log system according to the specified log type of the received scheduling instruction, and send the original log data to the designated sorter in the sorting center platform.

[0057] Among them, the specified log type includes one or several combinations of the following: specified log identifier, specified log structure, specified log storage, and specified log performance.

[0058] In this embodiment, for example, the specified log identifiers include but are not limited to diagnostic logs, information logs, warning logs, error logs, critical error logs, login logs, and operation logs; the specified log structures include but are not limited to JSON format, binary format, and Extensible Markup Language (XML) format; the specified log storage can refer to the storage area of the sorted log data. For example, error logs are stored in log node 3 of the storage center platform, and diagnostic logs are stored in log node 5 of the storage center platform; the specified log performance can refer to the archiving policy and / or compression algorithm of the logs stored in the storage center platform.

[0059] In this embodiment, the log extractor 13 extracts the raw log data that matches the specified log type from each production system according to the scheduling instruction, without extracting all the raw log data. It can specifically extract the raw log data that matches the specified log type for management, reducing the load of the log extractor 13 and improving the efficiency of log management.

[0060] Furthermore, the sorter 141 is specifically configured to extract the report log data from the raw log data according to the received sorting instruction, and perform format conversion on the report log data according to the pre-configured log structure in the sorting instruction to obtain the target log data, and send the target log data to the log node specified in the sorting instruction in the storage center platform for storage.

[0061] In this embodiment, for example, since the raw log data comes from different production systems, there is a situation where the formats of the raw log data are inconsistent. The raw log data is subjected to format conversion according to the pre-configured log structure in the sorting instruction. For example, the raw log data in XML format is converted into the target log data in JSON format.

[0062] In this embodiment, the sorter 141 extracts the report log data from the raw log data according to the received sorting instruction, which can effectively reduce invalid data and save storage space for subsequent storage. And performing format conversion on the report log data according to the pre-configured log structure in the sorting instruction can unify the log format and facilitate subsequent calculation operations on the log data.

[0063] Furthermore, the log management system may further include: a computing center platform 16; where the computing center platform 16 may include multiple computing nodes 161. In addition, the storage center platform 15 may further include: a report node 152.

[0064] Specifically, the scheduling center platform 11 is connected to the computing center 16 platform, and is also used to obtain pre-configured computing configuration information from the configuration center platform 12, generate a computing instruction based on the pre-configured computing configuration information, and send the computing instruction to each computing node 161 in the computing center platform 16. For each computing node 161, it is used to read specified target log data from the specified log node 151 in the storage center platform 15 according to the computing instruction, perform report processing on the specified target log data according to the pre-configured computing task to obtain report data, and send the report data to the specified report node 152 in the storage center platform 15 for storage.

[0065] In this embodiment, for example, if the pre-configured computing configuration information is to count the number of login logs of each system in this quarter, then generate to read the login log data within the time period of this quarter from the log node storing the login logs in the storage center platform, perform computing processing on the login log data within the time period of this quarter through the pre-configured script code for report generation, obtain the corresponding report instruction, and send the instruction to the computing node 161 for computing, and store the obtained report data in the specified report node 152 in the storage center platform 15.

[0066] In this embodiment, by calling each computing node 161 to perform computing operations, the computing nodes 161 are enabled to execute computing instructions in a multi-threaded and high-concurrency manner, thereby improving the computing efficiency. And through computing, report data is generated, which facilitates users to retrieve data and improves the user experience. Further, the storage center platform 15 may further include: an archiving node 153.

[0067] Specifically, the storage center platform performs archiving processing on the target log data already stored in the log node and / or the report data already stored in the report node according to the pre-configured archiving information, so as to store the data after archiving processing in the archiving node.

[0068] In this embodiment, for example, the archiving policy may be to compress the logs stored in the storage center platform for more than 30 days, where the compression algorithms include but are not limited to the extreme compression algorithm (ZIP) and the standard compression algorithm (RAR).

[0069] In this embodiment, by performing archiving processing on the target log data already stored in the log node and / or the report data already stored in the report node according to the pre-configured archiving information, the occupation of storage space by data with low usage rate can be effectively reduced.

[0070] Figure 3 It is a schematic structural diagram of a log management system provided for Embodiment 3 of the present application. As Figure 3As shown, based on the above respective embodiments, the dispatching center platform 11 specifically includes: a dispatching service unit 111, a perception service unit 112, and an analysis service unit 113.

[0071] Specifically, the perception service unit is used to obtain the running state information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node respectively; the analysis service unit is used to analyze and process the running state information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node to generate or update the pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

[0072] In this embodiment, the running state information includes a normal state and an abnormal state. The load information may refer to data throughput. For example, if the load information of log extractor A exceeds the threshold that log extractor A can bear, the running state information of log extractor A is in an abnormal state; if the load information of log extractor A does not exceed the threshold that log extractor A can bear, the running state information of log extractor A is in a normal state. If the running state information of log extractor A is in an abnormal state, it is necessary to update the pre-configured log configuration information and adjust the load information of log extractor A to make the running state information of log extractor A return to the normal state.

[0073] Furthermore, if the current running state information of log extractor A is in a normal state, but based on the historical load information of log extractor A, it can be analyzed that the load information of log extractor A shows an upward trend and is about to exceed the threshold that log extractor A can bear, then it is necessary to update the pre-configured log configuration information and adjust the load information of log extractor A to keep the running state information of log extractor A in a normal state. In this embodiment, by adjusting the pre-configured information, the load balance of the log management system is achieved, and prediction and analysis are performed based on historical load information, avoiding the occurrence of abnormal states, reasonably allocating tasks, and reducing the waste of resources.

[0074] Optionally, the log management system may further include: an application center platform 17.

[0075] Specifically, the application center platform 17 is connected to the report node in the storage center platform and is used to obtain specified report data from the specified report node according to the obtained display request and perform corresponding display processing on the specified report data.

[0076] In this embodiment, the application center platform 17 can provide the display of reports, facilitating query by operation and maintenance personnel, and can also provide external interfaces to support the external system to call report data.

[0077] Figure 4 Another structural schematic diagram of the log management system provided in the fourth embodiment of the present application is shown in Figure 4 As shown, the log management system includes: a log extractor 11 provided in each external production system i , a scheduling center platform 12, a configuration center platform 13, a sorting center platform 14, a storage center platform 15, a computing center platform 16, and an application center platform 17; the scheduling center platform 12 includes a scheduling service unit 121, a perception service unit 122, and an analysis service unit 123; the configuration center platform 13 includes a log configuration unit 131 and a report configuration unit 132; the sorting center platform 14 includes a sorter 14 i ; the storage center platform 15 includes a log node 151 i , a report node 152 i , and an archiving node 153 i ; the computing center platform 16 includes a computing node 16 i ; the application center platform 17 includes a report display unit 171 and an external interface unit 172. Wherein, i is an integer greater than 1.

[0078] In this embodiment, the scheduling center platform 12 is connected to the computing center platform 16, obtains pre-configured computing configuration information from the configuration center platform 13, generates a computing instruction based on the pre-configured computing configuration information, and sends the computing instruction to each computing node 16 in the computing center platform i .

[0079] In this embodiment, the perception service unit 122 of the scheduling center platform 12 respectively obtains the running status information and / or load information of each log extractor 11 i , each sorter 14 i , each log node 151 i , each report node 152 i , each archiving node 153 i and / or each computing node 16 i .

[0080] In this embodiment, the analysis service unit 123 of the scheduling center platform 12 is connected to the corresponding computing node 16 i , and obtains a computing result according to the running status information and / or load information obtained by the perception service unit 122 and the computing instruction in the computing node.

[0081] In this embodiment, the calculation result is input into the scheduling service unit 121 in the scheduling center platform 12, and is calculated through the corresponding computing node 16 i to obtain pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

[0082] In this embodiment, the scheduling center platform 12 generates a scheduling instruction according to the pre-configured log configuration information, and sends the scheduling instruction to each log extractor 11 i , so that the log extractor 11 i extracts the original log data of the corresponding generation system to the corresponding sorter 14 i .

[0083] In this embodiment, the scheduling center platform 12 generates a sorting instruction according to the pre-configured sorting configuration information, and sends the sorting instruction to each memory in the storage center platform 15, so that the sorter of the sorting center platform performs corresponding sorting processing on the original log data to obtain target log data, and sends the target log data to the specified log node 151 in the storage center platform i for storage.

[0084] In this embodiment, the computing node in the computing center platform 16 calculates the target log data in the specified log node 151 in the storage center platform 15 i according to the computing instruction, obtains a log report, and stores the log report in the specified report node 152 in the storage center platform 15 i for storage.

[0085] In this embodiment, the storage center platform 15 archives the target log data stored in the log node 151 i and / or the report data stored in the report node 152 i according to the pre-configured archiving information, so as to store the data after the archiving process in the archiving node 153 i for storage.

[0086] Figure 5 is a schematic flowchart of a log management method provided in the third embodiment of this application. Based on the log management system described in the above embodiment, as Figure 5 shown, the method includes:

[0087] S501. The scheduling center platform obtains the pre-configured log configuration information and the pre-configured sorting configuration information from the configuration center platform, generates a scheduling instruction based on the pre-configured log configuration information, and sends the scheduling instruction to each log extractor respectively, and generates a sorting instruction based on the pre-configured sorting configuration information, and sends the sorting instruction to each memory in the storage center platform.

[0088] S502. According to the received scheduling instruction, the log extractor retrieves the original log data from the corresponding log system and sends the original log data to the specified sorter in the sorting center platform.

[0089] S503. The sorter is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the specified log node in the storage center platform for storage.

[0090] In this embodiment, compared with the prior art in which each system separately constructs a log system and manages them separately, the log management method proposed in this embodiment realizes centralized management of the log data in each system, effectively realizes the overall planning of resources, and reduces the operation and maintenance costs.

[0091] Figure 6 It is a schematic structural diagram of an electronic device provided in an embodiment of the present application. As Figure 6 shown, the electronic device 60 includes:

[0092] The electronic device 60 may include a processor 601 with one or more processing cores, a memory 602 with one or more computer-readable storage media, a communication component 603 and other components. Among them, the processor 601, the memory 602 and the communication component 603 are connected through a bus 604.

[0093] In a specific implementation process, at least one processor 601 executes the computer execution instructions stored in the memory 602, so that at least one processor 601 executes the above log management method.

[0094] For the specific implementation process of the processor 601, reference may be made to the above method embodiment, and its implementation principle and technical effects are similar, so they will not be elaborated here in this embodiment.

[0095] In the above Figure 6In the illustrated embodiments, it should be understood that the processor may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in combination with the invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules in the processor.

[0096] The memory may include random access memory (RAM), and may also include non-volatile memory (NVM), such as at least one disk memory.

[0097] The bus may be an industry standard architecture (ISA) bus, a peripheral component interconnect (PCI) bus, an extended industry standard architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of convenience in representation, the buses in the drawings of this application are not limited to only one bus or one type of bus.

[0098] In some embodiments, a computer program product is also proposed, including a computer program or instruction, which when executed by a processor implements the steps in any of the above-mentioned log management methods.

[0099] For the specific implementation of each of the above operations, reference may be made to the previous embodiments and will not be elaborated here.

[0100] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application, which follow the general principles of the present application and include known common knowledge or conventional technical means in the technical field not disclosed in the present application. The specification and embodiments are only regarded as exemplary, and the true scope and spirit of the present application are pointed out by the following claims.

Claims

1. A log management system, characterized in that, the log system includes: a scheduling center platform, a configuration center platform, log extractors provided in each external production system, a sorting center platform, and a storage center platform; wherein, the scheduling center platform is respectively connected to the configuration center platform and multiple log extractors, and is used to obtain pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generate a scheduling instruction based on the pre-configured log configuration information, and send the scheduling instruction to each log extractor respectively, and generate a sorting instruction based on the pre-configured sorting configuration information, and send the sorting instruction to each memory in the storage center platform; the log extractor is used to retrieve the original log data from the corresponding log system according to the received scheduling instruction, and send the original log data to the designated sorter in the sorting center platform; the sorter is connected to the storage center platform, and is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain target log data, and send the target log data to the designated log node in the storage center platform for storage.

2. The log management system according to claim 1, characterized in that, it includes: the sorter is specifically used to extract report log data from the original log data according to the received sorting instruction, and perform format conversion on the report log data according to the pre-configured log structure in the sorting instruction to obtain target log data, and send the target log data to the log node designated in the sorting instruction in the storage center platform for storage.

3. The log management system according to claim 1, characterized in that, the log management system further includes: a computing center platform; the storage center platform further includes: a report node; then the scheduling center platform is connected to the computing center platform, and is further used to obtain pre-configured computing configuration information from the configuration center platform, generate a computing instruction based on the pre-configured computing configuration information, and send the computing instruction to each computing node in the computing center platform; the computing node is used to read the designated target log data from the designated log node in the storage center platform according to the computing instruction, and perform report processing on the designated target log data according to the pre-configured computing task to obtain report data, and send the report data to the designated report node in the storage center platform for storage.

4. The log management system according to claim 3, characterized in that, the storage center platform further includes: an archiving node; the storage center platform performs archiving processing on the target log data already stored in the log node and / or the report data already stored in the report node according to the pre-configured archiving information, so as to store the data after the archiving processing in the archiving node.

5. The log management system according to claim 1, characterized in that: The log extractor is used to retrieve the original log data matching the specified log type from the corresponding log system according to the specified log type of the received scheduling instruction, and send the original log data to the specified sorter in the sorting center platform; Among them, the specified log type includes one or several combinations of the following: specified log identifier, specified log structure, specified log storage, and specified log performance.

6. The log management system according to any one of claims 1 to 5, characterized in that, The scheduling center platform includes: a scheduling service unit, a perception service unit, and an analysis service unit; among them, The perception service unit is used to obtain the running status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node respectively; The analysis service unit is used to analyze and process the running status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node, so as to generate or update the pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

7. The log management system according to claim 3, characterized in that, The log management system further includes: an application center platform; among them, The application center platform is connected to the report node in the storage center platform, and is used to obtain the specified report data from the specified report node according to the obtained display request, and perform corresponding display processing on the specified report data.

8. A log management method, characterized in that, The method is applied to the log management system, and the log system includes: a scheduling center platform, a configuration center platform, log extractors set in each external production system, a sorting center platform, and a storage center platform; then the method includes: The scheduling center platform obtains the pre-configured log configuration information and pre-configured sorting configuration information from the configuration center platform, generates a scheduling instruction based on the pre-configured log configuration information, and sends the scheduling instruction to each log extractor respectively, and generates a sorting instruction based on the pre-configured sorting configuration information, and sends the sorting instruction to each memory in the storage center platform; The log extractor retrieves the original log data from the corresponding log system according to the received scheduling instruction, and sends the original log data to the specified sorter in the sorting center platform; The sorter is used to perform corresponding sorting processing on the original log data according to the received sorting instruction to obtain the target log data, and send the target log data to the specified log node in the storage center platform for storage.

9. The method according to claim 8, characterized in that, The scheduling center platform in the log management system includes: a scheduling service unit, a perception service unit, and an analysis service unit; among them, The perception service unit acquires the operation status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node respectively; The analysis service unit analyzes and processes the operation status information and / or load information of each log extractor, each sorter, each log node, each report node, each archiving node, and / or each computing node to generate or update the pre-configured log configuration information, pre-configured sorting configuration information, pre-configured computing configuration information, and / or pre-configured archiving information.

10. An electronic device, Characterized in that, It includes: A processor and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to claims 8 to 9.

11. A computer-readable storage medium, Characterized in that, The computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method according to claim 8 or 9.

Images