Log collection method, device and system
By using sub-control centers and log acquisition robots in complex systems for distributed real-time log acquisition, the problems of long log acquisition cycles and difficult real-time query in the existing technology are solved, and efficient and real-time log acquisition and storage are achieved.
Patent Information
- Application Number
- CN202311753150.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-19
- Publication Date
- 2025-06-20
AI Technical Summary
The existing log collection technology has problems such as scattered logs, long collection cycles, and difficulty in real-time data query in complex systems.
Through multiple sub-control centers, distribution instructions and script execution instructions are sent to the corresponding equipment clusters of each sub-control center, and the log collection robot is used for real-time collection and processing, and finally the target log is stored in the timing database.
Distributed real-time log collection and extraction are realized, which improves the efficiency and effectiveness of log collection and simplifies the manual deployment process.
Smart Images
Figure CN120179623A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of computer technology, specifically to the field of log processing technology, and particularly to a log collection method and device. Background Art
[0002] In the Internet field, with the continuous development of various log collection and monitoring systems, technical frameworks such as graylog and Flume have become the current mainstream log collection technologies.
[0003] However, with the increase in system complexity, the complex functions of Internet distributed business systems and the characteristics of many microservice modules make the logs of existing log collection technologies scattered, the log collection, data cleaning and calculation cycles long, and the problem of difficult real-time query of target data increasingly prominent. Summary of the Invention
[0004] Embodiments of this application provide a log collection method, device, system, device, and storage medium.
[0005] According to the first aspect, embodiments of this application provide a log collection method, which includes: sending a distribution instruction to the device cluster corresponding to each sub-control center via multiple sub-control centers; sending a script execution instruction for a script file to the device cluster corresponding to each sub-control center via multiple sub-control centers; and storing the target log into a time series database in response to obtaining the target log via the sub-control center.
[0006] According to the second aspect, embodiments of this application provide a log collection device, which includes: a distribution module configured to send a distribution instruction to the device cluster corresponding to each sub-control center via multiple sub-control centers; a collection module configured to send a script execution instruction for a script file to the device cluster corresponding to each sub-control center via multiple sub-control centers; and a storage module configured to store the target log into a time series database in response to obtaining the target log via the sub-control center.
[0007] According to the third aspect, embodiments of this application provide a log collection system, which includes: a main control center for executing the log collection method of any embodiment of the first aspect; multiple sub-control centers for receiving the distribution instruction and the script execution instruction, and distributing the distribution instruction and the script execution instruction to the corresponding device clusters, and the devices in the device clusters corresponding to each sub-control center are used to receive the distribution instruction and the script execution instruction, and in response to obtaining the script execution instruction, start a log collection robot and control the log collection robot to process the collected logs to obtain the target log, and return the target log to the sub-control center.
[0008] According to a sixth aspect, an embodiment of the present application provides an electronic device, which includes one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the log collection method according to any embodiment of the first aspect.
[0009] According to a seventh aspect, an embodiment of the present application provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, it implements the log collection method according to any embodiment of the first aspect.
[0010] In the present application, distribution instructions are sent to the device clusters corresponding to each sub-control center via multiple sub-control centers; script execution instructions for a script file are sent to the device clusters corresponding to each sub-control center via multiple sub-control centers; in response to obtaining target logs via a sub-control center, the target logs are stored in a time series database, that is, log collection is performed by remotely controlling a log collection robot, avoiding manual deployment of a collection Agent program, realizing distributed real-time collection and extraction of logs, and effectively improving the efficiency and effectiveness of log collection for complex systems.
[0011] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it used to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. Description of the Drawings
[0012] Figure 1 is an exemplary system architecture diagram to which the present application can be applied;
[0013] Figure 2 is a flowchart of an embodiment of the log collection method according to the present application;
[0014] Figure 3 is a schematic diagram of an embodiment of the log collection method according to the present application;
[0015] Figure 4 is a flowchart of another embodiment of the log collection method according to the present application;
[0016] Figure 5 is a flowchart of an embodiment of the log collection device according to the present application;
[0017] Figure 6 is a flowchart of an embodiment of the log collection system according to the present application;
[0018] Figure 7 is a schematic structural diagram of a computer system of a server suitable for implementing the embodiments of the present application. Detailed Embodiments
[0019] The following describes exemplary embodiments of the present application with reference to the accompanying drawings. Various details of the embodiments of the present application are included to facilitate understanding, and they should be considered merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, descriptions of well-known functions and structures are omitted in the following description for clarity and conciseness.
[0020] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The present application will be described in detail below with reference to the drawings and in combination with the embodiments.
[0021] Figure 1 An exemplary system architecture 100 is shown in which the log collection method of the present application can be applied.
[0022] As Figure 1 shown, the system architecture 100 may include a general control center 101, sub-control centers 102, 103, a network 104, and device clusters 105, 106. The network 104 is used to provide a medium for communication links between the sub-control centers 102, 103 and the device clusters 105, 106. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, etc.
[0023] The general control center 101 interacts with the device clusters 105, 106 via the sub-control centers 102, 103 through the network 104 to send distribution instructions, script execution instructions, and obtain the target logs returned by the device clusters, and store the target logs into the time series database.
[0024] Among them, the general control center 101 may be a server that provides various services. For example, it issues distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers; issues script execution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers; and stores the target logs into the time series database in response to obtaining the target logs via the sub-control center.
[0025] The sub-control centers 102, 103 may be functional modules deployed in the servers corresponding to the general control center, or may be separate terminal devices or servers, and the present application does not make any limitations in this regard.
[0026] The devices included in the device clusters 105, 106 may be any terminal devices or servers deployed with log collection robots. The terminal devices may be various electronic devices, including but not limited to mobile phones and laptop computers.
[0027] It should be understood that Figure 1The numbers of the total control center, sub-control centers, networks, and device clusters therein are merely illustrative. According to the implementation requirements, there can be any number of total control centers, sub-control centers, networks, and device clusters.
[0028] Figure 2 FIG. 200 shows a flowchart of an embodiment of a log collection method that can be applied to the present application. In this embodiment, the log collection method is applied to the total control center and specifically includes the following steps:
[0029] Step 201: Send distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0030] In this embodiment, the execution entity (such as Figure 1 the total control center 101 shown in
[0031] can send distribution instructions to multiple sub-control centers, so that the sub-control centers send the distribution instructions to each device in the corresponding device cluster.
[0032] Here, usually the network environments and / or service types corresponding to each device cluster are different from each other. For example, device cluster A corresponds to network environment 1, and device cluster B corresponds to network environment 2; network cluster A corresponds to service type 1, and device cluster B corresponds to service type 2, and so on.
[0033] Here, the sub-control center can be a functional module, or a separate terminal device or server, and the present application does not limit this.
[0034] Step 202: Send script execution instructions for the script file to the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0035] In this embodiment, after sending the distribution instructions, the execution entity can send script execution instructions for the above script file to each device in the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0036] Among them, the script execution instruction is used to control the devices in the device cluster to start a pre-deployed log collection robot, control the log collection robot to process the collected logs to obtain target logs, and return the target logs to the corresponding sub-control center.
[0037] Here, the method of collecting logs can include various types. For example, directly collect all log data; use data collectors such as telegraf and categraf for collection, etc. The content of the collected logs can include various types. For example, collect the CPU usage rate, memory usage, disk usage, kernel exception information, network traffic, user behavior, etc. during system operation.
[0038] Among them, there are various ways to process the collected logs. For example, cleaning the collected logs to obtain target logs; screening the collected logs according to preset screening rules (such as calculation models and matching rules) to obtain target logs; cleaning, classifying, and performing correlation analysis on the collected logs to obtain target logs, etc.
[0039] Here, the target logs may include the business modules and device addresses associated with the target logs. In addition, the target logs may also include the identifiers of the sub-control centers corresponding to the target logs.
[0040] In some optional ways, processing the collected logs to obtain target logs includes: processing the collected logs based on the screening rules to obtain target logs.
[0041] In this implementation, the distribution instruction may also include screening rules, which can be set according to experience and actual needs and may include information such as matching rules and calculation models. The script execution instruction can be used to control the log collection robot to process the collected logs based on the screening rules to obtain target logs and return the target logs to the sub-control center.
[0042] Specifically, after each device in the device cluster responds to obtaining the script execution instruction, it can start the log collection robot to collect logs in real time, read the screening rules, which may include matching rules and calculation models, perform calculations and extraction of target logs according to the screening rules, and return the extracted target logs to the sub-control center in a preset format.
[0043] This implementation obtains target logs by processing the collected logs based on the screening rules, improving the effectiveness and pertinence of the collected target logs.
[0044] In some optional ways, the method further includes: in response to determining that the distribution instruction has been updated, updating the distribution instructions of the devices in each device cluster based on the updated distribution instruction.
[0045] In this implementation, the execution entity can detect in real time or regularly whether the distribution instruction has been updated. If it has been updated (such as the matching rule and / or calculation model in the screening rules have been updated), the updated distribution instruction can be sent to the device cluster corresponding to each sub-control center via each sub-control center to update the distribution instructions of the devices in each device cluster.
[0046] This implementation realizes the real-time update of the distribution instruction by, in response to determining that the distribution instruction has been updated, updating the distribution instructions of the devices in each device cluster based on the updated distribution instruction.
[0047] In some alternative ways, the updated distribution instructions are obtained as follows: Based on the target log, determine the running status of each device cluster; based on the status changes of each device cluster, update the distribution instructions to obtain the updated distribution instructions.
[0048] In this implementation, after obtaining the target log, the execution entity can directly input the real-time obtained target log into a preset classification model to obtain the running status of each device cluster, or can first preprocess the target log and input the preprocessed target log into the preset classification model to obtain the running status of each cluster. This application does not make any restrictions on this. Among them, the running status of the device cluster is a normal running status or an abnormal running status.
[0049] Further, based on the status changes of each device cluster, update the distribution instructions to obtain the updated distribution instructions.
[0050] This implementation method determines the running status of each device cluster based on the target log; updates the distribution instructions based on the status changes of each device cluster to obtain the updated distribution instructions, realizing the timely automatic update of the distribution instructions, and further improving the accuracy and effectiveness of the collected target log.
[0051] In some alternative ways, determining the running status of each device cluster based on the target log includes: via the sub-control center, determining the running status of each device cluster based on the target log.
[0052] In this implementation, the execution entity can control each sub-control center to determine the running status of the device cluster corresponding to each sub-control center according to the target log corresponding to each sub-control center.
[0053] Specifically, for each sub-control center, the execution entity can control the sub-control center to input the target log returned by the corresponding device cluster into a preset classification model to obtain the running status of the corresponding device cluster.
[0054] This implementation method determines the running status of each device cluster via the sub-control center based on the target log, that is, each sub-control center processes the respective corresponding target log simultaneously to obtain the running status of the device cluster corresponding to each sub-control center, effectively improving the processing efficiency of the target log.
[0055] In some alternative ways, determining the running status of each device cluster based on the target log includes: preprocessing and feature extraction of the target log to obtain feature data; inputting the feature data into a preset classification model to obtain the running status of each device cluster.
[0056] In this implementation, the execution entity can first preprocess the target log, such as cleaning, deduplication, normalization, etc., to obtain the preprocessed data. Further, feature extraction is performed on the preprocessed data, that is, features that can reflect the running state and performance of the software system are extracted. For example, statistical features such as the mean, variance, and maximum value of the CPU usage rate, the change trend of the memory usage, the change trend of the abnormal log, etc., are obtained to get the feature data.
[0057] Further, the feature data is input into a preset classification model to obtain the running states of each device cluster.
[0058] Among them, the preset classification model can be trained based on the feature data samples marked with the running states.
[0059] In addition, it should be noted that if the running state is an abnormal running state, an alarm message can be directly output, or the feature data corresponding to the abnormal running state can be deeply analyzed to determine the change pattern and trend of the feature data corresponding to the abnormal running state and output it to help developers quickly locate the problem and repair it.
[0060] Further, the feature data and model parameters can be automatically adjusted according to the change of the running state.
[0061] This implementation method preprocesses and extracts features from the target log to obtain feature data; inputs the feature data into a preset classification model to obtain the running states of each device cluster, improving the accuracy and reliability of the obtained running states.
[0062] In some alternative ways, the method further includes: obtaining a fault warning message based on the target log and a preset fault warning model, and presenting the fault warning message and the corresponding fault handling suggestions.
[0063] In this implementation, the execution entity can input the target log into a preset fault prediction model to obtain a fault warning message, and present the fault warning message and the corresponding fault handling suggestions for the user to quickly understand the system state and fault information, and query more information about the fault and historical fault records, etc. through the interaction interface.
[0064] Among them, the fault prediction model can be trained based on the target log samples marked with the fault warning information.
[0065] This implementation method obtains a fault warning message based on the target log and a preset fault warning model, and presents the fault warning message and the corresponding fault handling suggestions, which helps users take timely measures to avoid or cope with faults.
[0066] Step 203, in response to obtaining the target log via the sub-control center, store the target log into the time series database.
[0067] In this embodiment, in response to obtaining the target log via the sub-control center, the execution entity may store the target log into the time series database for subsequent processing, such as retrieval, fault warning, fault location, etc.
[0068] Among them, the target log stored in the time series database supports time series query.
[0069] In some alternative ways, the method further includes: in response to detecting an instruction to add or delete a specified device in the device cluster, adding or deleting the specified device via the sub-control center corresponding to the specified device.
[0070] In this implementation, the execution entity may detect in real time or periodically an instruction to add or delete a specified device in the device cluster. In response to detecting the instruction to add or delete a specified device in the device cluster, the specified device may be added or deleted via the sub-control center corresponding to the specified device.
[0071] This implementation realizes flexible control of each device in the device cluster by adding or deleting the specified device via the sub-control center corresponding to the specified device in response to detecting the instruction to add or delete the specified device.
[0072] In some alternative ways, the method further includes: in response to obtaining a query instruction from the user, performing a retrieval in the time series database based on the query instruction.
[0073] In this implementation, in response to obtaining a query instruction from the user, the execution entity may convert the query instruction into a target instruction to be executed in the time series database, such as an SQL (Structured Query Language) statement; and perform a retrieval in the time series database based on the target instruction.
[0074] Among them, the query instruction from the user may include information such as keywords, sub-control center identifier, device address, service module, etc.
[0075] This method improves the efficiency and effectiveness of log retrieval by performing a retrieval in the time series database based on the query instruction in response to obtaining the query instruction from the user.
[0076] In some alternative ways, the script execution instruction is further used to dynamically configure and adjust the log collection robot to adapt to operating systems and log file formats with different architectures.
[0077] In this implementation, the execution entity can dynamically configure and adjust the log collection robot by sending script execution instructions to adapt to operating systems with different architectures, such as x86 32 / 64-bit, arm64, amd64, domestic CPU system architectures, Windows, Linux, MacOS, Linux-like, etc., and log file formats.
[0078] This implementation supports heterogeneous systems by remotely dynamically configuring and adjusting the log collection robot to adapt to operating systems with different architectures and log file formats.
[0079] Continue to refer to Figure 3 , Figure 3 is a schematic diagram of the application scenario of the log collection method according to this embodiment.
[0080] In Figure 3 's application scenario, the execution entity 301 can send distribution instructions to the device clusters 302, 303 corresponding to each sub-control center via multiple sub-control centers. The distribution instructions include script files; send script execution instructions for the above script files to the device clusters 302, 303 corresponding to each sub-control center via multiple sub-control centers. The script execution instructions are used to control the devices in the device cluster to start the pre-deployed log collection robot, control the log collection robot to process the collected logs to obtain target logs, and return the target logs to the corresponding sub-control center; in response to obtaining the target logs via the sub-control center, store the target logs in the time series database for subsequent processing, such as retrieval, fault warning, fault location, etc.
[0081] The log collection method of the present disclosure sends distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers; sends script execution instructions for the script files to the device clusters corresponding to each sub-control center via multiple sub-control centers; in response to obtaining the target logs via the sub-control center, stores the target logs in the time series database, that is, collects logs by remotely controlling the log collection robot, avoiding manual deployment of the collection Agent program, realizing distributed real-time collection and extraction of logs, and effectively improving the efficiency and effectiveness of log collection.
[0082] Figure 4 Shows the flow 400 of the embodiment of the log collection method that can be applied to the present application. In this embodiment, the log collection method specifically includes the following steps:
[0083] Step 401, send distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0084] In this embodiment, for the implementation details and technical effects of step 401, reference can be made to the description of step 201, which will not be elaborated here.
[0085] Step 402: Send a script execution instruction for the script file to the device cluster corresponding to each sub-control center via multiple sub-control centers.
[0086] In this embodiment, for the implementation details and technical effects of step 402, reference can be made to the description of step 202, which will not be elaborated here.
[0087] Step 403: In response to obtaining the target log via the sub-control center, store the target log into the time series database.
[0088] In this embodiment, for the implementation details and technical effects of step 403, reference can be made to the description of step 203, which will not be elaborated here.
[0089] Step 404: Query and calculate the target log in the time series database according to the preset monitoring service rules to obtain a calculation result.
[0090] In this embodiment, the execution entity can pre-obtain the monitoring service rules configured by the user, and query and calculate the logs in the time series database according to the monitoring service rules to obtain a calculation result.
[0091] Step 405: In response to determining that the calculation result meets the preset alarm policy, generate an alarm message.
[0092] In this embodiment, after the execution entity obtains the calculation result, it can determine whether the calculation result meets the preset alarm policy. If it meets, it outputs an alarm message to notify the alarm handling personnel, and the alarm handling personnel can perform fault troubleshooting and positioning according to the alarm message.
[0093] Specifically, the monitoring service rules configured by the user are to query and count the preset keywords in (all target logs in the time series database, target logs corresponding to the specified sub-control center in the time series database, or target logs corresponding to specific devices), and the preset alarm policy is: if the number of preset keywords is greater than or equal to 3, output an alarm; the execution entity queries and counts the corresponding logs. If the number of preset keywords is greater than or equal to 3, it can directly output an alarm message or output an alarm maintenance work order via the work order system to instruct the alarm handling personnel to perform problem troubleshooting and positioning.
[0094] From Figure 4 it can be seen that compared with Figure 2Compared with the corresponding embodiments, the process 400 of the video generation method in this embodiment reflects querying and calculating the target logs in the time series database according to the preset monitoring service rules to obtain the calculation results. In response to determining that the calculation results meet the preset alarm policy, alarm information is generated. In this method, the monitoring service rules and the alarm policy can be flexibly configured to meet the monitoring and alarm requirements of various complex rules.
[0095] Further referring to Figure 5 , as an implementation of the methods shown in the above figures, the present application provides an embodiment of a log collection device. This device embodiment corresponds to Figure 2 the method embodiment shown, and this device can be specifically applied to the general control center.
[0096] As Figure 5 shown, the log collection device 500 in this embodiment includes: a distribution module 501, a collection module 502, and a storage module 503.
[0097] Among them, the distribution module 501 can be configured to issue distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0098] The collection module 502 can be configured to issue script execution instructions for the script files to the device clusters corresponding to each sub-control center via multiple sub-control centers.
[0099] The storage module 503 can be configured to store the target logs into the time series database in response to obtaining the target logs via the sub-control center.
[0100] In some optional ways of this embodiment, the device further includes a monitoring module, configured to query and calculate the target logs in the time series database according to the preset monitoring service rules to obtain the calculation results; and an alarm module, configured to generate alarm information in response to determining that the calculation results meet the preset alarm policy.
[0101] In some optional ways of this embodiment, processing the collected logs to obtain the target logs includes: processing the collected logs based on the screening rules to obtain the target logs.
[0102] In some optional ways of this embodiment, the device further includes an update module, configured to update the distribution instructions of the devices in each device cluster based on the updated distribution instructions in response to determining that the distribution instructions have changed.
[0103] In some optional ways of this embodiment, the updated distribution instructions are obtained by the following method: determining the running status of each device cluster based on the target logs; and updating the distribution instructions based on the changes in the running status of each device cluster to obtain the updated distribution instructions.
[0104] In some alternative embodiments of the present embodiment, based on the target log, the operating status of each device cluster is determined, including: via the sub-control center, based on the target log, the operating status of each device cluster is determined.
[0105] In some alternative embodiments of the present embodiment, based on the target log, the operating status of each device cluster is determined, including: preprocessing the target log and extracting features to obtain feature data; inputting the feature data into a preset classification model to obtain the operating status of each device cluster.
[0106] In some alternative embodiments of the present embodiment, the apparatus further includes: a detection module configured to, in response to detecting an instruction to add or delete a specified device in the device cluster, add or delete the specified device via the sub-control center corresponding to the specified device.
[0107] In some alternative embodiments of the present embodiment, the apparatus further includes: a retrieval module configured to, in response to obtaining a query instruction from a user, perform a retrieval in the time series database based on the query instruction.
[0108] In some alternative embodiments of the present embodiment, the script execution instruction is further used to dynamically configure and adjust the log collection robot to adapt to operating systems and log file formats of different architectures.
[0109] Further referring to Figure 6 , an embodiment of a log collection system is provided in the present application.
[0110] In the present embodiment, the system includes a main control center 601, sub-control centers 602, 603, 604, and device clusters 605, 606, 607 corresponding to each sub-control center.
[0111] Among them, the main control center 601 is used to execute the log collection method described in Embodiment 2 above. The sub-control centers 602, 603, 604 are used to receive distribution instructions and script execution instructions, and distribute the distribution instructions and script execution instructions to the corresponding device clusters. The devices in the device clusters 605, 606, 607 corresponding to each sub-control center are used to receive the distribution instructions and script execution instructions, and in response to obtaining the script execution instruction, start the log collection robot and control the log collection robot to process the collected logs to obtain the target log, and return the target log to the sub-control center.
[0112] It should be understood that Figure 6The numbers of the total control center 601, the sub-control centers 602, 603, 604 and the device clusters 605, 606, 607 corresponding to the respective sub-control centers in [description] are merely illustrative. According to the implementation requirements, there can be any number of total control centers, sub-control centers and device clusters corresponding to the respective sub-control centers.
[0113] In the technical solution of the present disclosure, the acquisition, storage and application of the user's personal information involved all comply with the provisions of relevant laws and regulations and do not violate public order and good customs.
[0114] According to an embodiment of the present application, the present application also provides an electronic device and a readable storage medium.
[0115] As Figure 7 shown, it is a block diagram of an electronic device for a log collection method according to an embodiment of the present application.
[0116] 700 is a block diagram of an electronic device for a log collection method according to an embodiment of the present application. The electronic device is intended to represent various forms of digital computers, such as, a laptop computer, a desktop computer, a workbench, a personal digital assistant, a server, a blade server, a mainframe computer, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as, a personal digital processor, a cellular phone, a smart phone, a wearable device, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described herein and / or claimed.
[0117] As Figure 7 shown, the electronic device includes: one or more processors 701, a memory 702, and an interface for connecting the components, including a high-speed interface and a low-speed interface. The various components are interconnected using different buses and can be installed on a common motherboard or in other ways as needed. The processor can process instructions executed within the electronic device, including instructions stored in the memory or on the memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, if needed, multiple processors and / or multiple buses can be used with multiple memories and multiple memories. Similarly, multiple electronic devices can be connected, and each device provides some necessary operations (such as, as a server array, a set of blade servers, or a multi-processor system). Figure 7 In [description], one processor 701 is taken as an example.
[0118] The memory 702 is the non-transitory computer-readable storage medium provided by this application. Among them, the memory stores instructions executable by at least one processor, so that the at least one processor executes the log collection method provided by this application. The non-transitory computer-readable storage medium of this application stores computer instructions, and these computer instructions are used to make a computer execute the log collection method provided by this application.
[0119] As a non-transitory computer-readable storage medium, the memory 702 can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the log collection method in the embodiments of this application (for example, the Figure 5 distribution module 501, collection module 502, and storage module 503 shown in the appendix). The processor 701 executes various functional applications and data processing of the server by running the non-transitory software programs, instructions, and modules stored in the memory 702, that is, implements the log collection method in the above method embodiments.
[0120] The memory 702 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system and application programs required for at least one function; the data storage area may store data created by the use of the electronic device for log collection, etc. In addition, the memory 702 may include a high-speed random access memory, and may also include non-transitory memories, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 702 may optionally include memories remotely provided with respect to the processor 701, and these remote memories may be connected to the electronic device for log collection through a network. Examples of the above networks include, but are not limited to, the Internet, enterprise intranets, local area networks, mobile communication networks, and combinations thereof.
[0121] The electronic device for the log collection method may further include: an input device 703 and an output device 704. The processor 701, the memory 702, the input device 703, and the output device 704 may be connected through a bus or other means, Figure 7 taking the connection through the bus as an example.
[0122] The input device 703 can receive input digital or character information, such as input devices such as touch screens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, and joysticks. The output device 704 may include a display device, an auxiliary lighting device (for example, an LED), and a tactile feedback device (for example, a vibration motor), etc. The display device may include, but is not limited to, a liquid crystal display (LCD), a light-emitting diode (LED) display, and a plasma display. In some embodiments, the display device may be a touch screen.
[0123] The various embodiments of the systems and techniques described herein can be implemented in digital electronic circuitry, integrated circuit systems, application specific ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include: being implemented in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which may be a special-purpose or general-purpose programmable processor that receives data and instructions from, and transmits data and instructions to, a storage system, at least one input device, and at least one output device.
[0124] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, apparatus, and / or device (e.g., a disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.
[0125] In order to provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic, speech, or tactile input).
[0126] The systems and techniques described herein can be implemented in a computing system including backend components (e.g., as a data server), or a computing system including middleware components (e.g., an application server), or a computing system including frontend components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described herein), or a computing system including any combination of such backend components, middleware components, or frontend components. The components of the system can be interconnected to each other by digital data communication in any form or medium (e.g., a communication network). Examples of communication networks include: local area network (LAN), wide area network (WAN), and the Internet.
[0127] A computer system can include a client and a server. The client and the server are generally far from each other and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other.
[0128] According to the technical solution of the embodiment of the present application, the efficiency and effectiveness of log collection are improved.
[0129] It should be understood that various forms of the processes shown above can be used, with steps reordered, added, or deleted. For example, the steps recited in the present application can be executed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in the present application can be achieved. No limitation is made herein.
[0130] The above specific embodiments do not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present application shall be included within the protection scope of the present application.
Claims
1. A log collection method, applied to a total control center, the method comprising: Send distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers, where the distribution instructions include script files; Send script execution instructions for the script file to the device clusters corresponding to each sub-control center via the multiple sub-control centers. The script execution instructions are used to control the devices in the device clusters to start log collection robots, control the log collection robots to process the collected logs to obtain target logs, and return the target logs to the corresponding sub-control centers; In response to obtaining the target logs via the sub-control center, store the target logs in the time series database.
2. The method according to claim 1, the method further comprising: Query and calculate the target logs in the time series database according to preset monitoring service rules to obtain calculation results; In response to determining that the calculation results meet the preset alarm policy, generate alarm information.
3. The method according to claim 1, wherein, The distribution instructions further include: screening rules, and the processing of the collected logs to obtain target logs includes: Process the collected logs based on the screening rules to obtain target logs.
4. The method according to claim 3, the method further comprising: In response to determining that the distribution instructions are updated, update the distribution instructions of the devices in each device cluster based on the updated distribution instructions.
5. The method according to claim 4, wherein, The updated distribution instructions are obtained by the following method: Based on the target logs, determine the running states of each device cluster; Based on the changes in the running states of each device cluster, update the distribution instructions to obtain the updated distribution instructions.
6. The method according to claim 5, wherein, The determining the running states of each device cluster based on the target logs includes: Via the sub-control center, determine the running states of each device cluster based on the target logs.
7. The method according to any one of claims 5 or 6, wherein, The determining the running states of each device cluster based on the target logs includes: Preprocess and extract features from the target logs to obtain feature data; Input the feature data into a preset classification model to obtain the running states of each device cluster. The preset classification model is trained based on feature data samples labeled with running states.
8. The method according to any one of claims 1 - 6, the method further comprising: Based on the target logs and a preset fault warning model, obtain fault warning information, and present the fault warning information and corresponding fault handling suggestions. The fault warning model is trained based on target log samples labeled with fault warning information.
9. The method according to claims 1 - 6, the method further comprising: In response to detecting an instruction to add or delete a specified device in the device cluster, add or delete the specified device via the sub-control center corresponding to the specified device.
10. The method according to claims 1 - 6, the method further comprising: In response to obtaining a query instruction from a user, perform a search in the time series database based on the query instruction.
11. The method according to claims 1 - 6, the script execution instruction is further used to dynamically configure and adjust the log collection robot to adapt to operating systems and log file formats of different architectures.
12. A log collection device, the device comprising: A distribution module, configured to send distribution instructions to the device clusters corresponding to each sub-control center via multiple sub-control centers, where the distribution instructions include script files; An acquisition module, configured to send script execution instructions for the script file to the device clusters corresponding to each sub-control center via the multiple sub-control centers. The script execution instructions are used to control the devices in the device clusters to start log collection robots, control the log collection robots to process the collected logs to obtain target logs, and return the target logs to the corresponding sub-control centers; A storage module, configured to store the target log into a time series database in response to obtaining the target log via a sub-control center.
13. A log collection system, the system comprising: A main control center, multiple sub-control centers, and device clusters corresponding to each sub-control center The main control center is configured to execute the method according to any one of claims 1-11; The sub-control center is configured to receive a distribution instruction and a script execution instruction, and distribute the distribution instruction and the script execution instruction to the corresponding device cluster; Devices in the device cluster are configured to receive the distribution instruction and the script execution instruction, and in response to obtaining the script execution instruction, start a log collection robot and control the log collection robot to process the collected logs to obtain a target log, and return the target log to the sub-control center.
14. An electronic device, characterized in that Comprising: At least one processor; And A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, enabling the at least one processor to execute the method according to any one of claims 1-11.
15. A non-transitory computer-readable storage medium storing computer instructions, characterized in that The computer instructions are used to cause the computer to execute the method according to any one of claims 1-11.