A log management method and device, electronic equipment and readable storage medium

Abstract

This invention relates to the field of log collection technology, and discloses a log management method, apparatus, electronic device, and readable storage medium. The method includes: obtaining log collection configuration information through a visual page, wherein the visual page includes a log registration component, a collection rule configuration component, and a host association component; the collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component; deploying a log collection service based on the target host information; configuring log collection tasks based on the log registration information and collection rules; and controlling the log collection service to execute the log collection tasks, converting the collected log data into quantifiable data. By implementing this invention, the technical threshold for operations and maintenance personnel is lowered, and the efficiency and accuracy of log collection are effectively improved.

Description

Technical Field

[0001] This invention relates to the field of log acquisition technology, and more specifically to a log management method, apparatus, electronic device, and readable storage medium. Background Technology

[0002] Typically, log file collection and quantification analysis are implemented using open-source technologies like Flume and Logstash. The main approach involves deploying Flume-agent on the target server to crawl and collect log file content, then using Logstash to quantify and analyze the collected log data before finally storing it in Elasticsearch. However, this method involves multiple operational components and is relatively complex to configure, making it difficult for less technically skilled operations personnel to configure successfully. This results in low log collection efficiency and high operational costs. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a log management method, apparatus, electronic device, and readable storage medium to solve the problems of high operation and maintenance costs and low log collection efficiency caused by the complexity of the log collection configuration process.

[0004] According to a first aspect, embodiments of the present invention provide a log management method, which is executed by a log management platform and includes the following steps: obtaining log collection configuration information through a visual page; wherein the visual page includes a log registration component, a collection rule configuration component, and a host association component, and the collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component; deploying a log collection service according to the target host information, and configuring a log collection task according to the log registration information and the collection rules; controlling the log collection service to execute the log collection task, and converting the collected log data into quantifiable data.

[0005] The log management method provided in this invention obtains log collection configuration information through a visual page. This configuration information includes log registration information, collection rules, and target host information. Then, based on the target host information, a log collection service is deployed, and log collection tasks are configured according to the log registration information and collection rules. This controls the log collection service to execute log collection tasks and transforms the collected log data into quantifiable data. This method provides effective configuration process guidance for log collection configuration information through a visual page, allowing operations and maintenance personnel to customize the log collection configuration information, thereby lowering the technical threshold for operations and maintenance personnel. Furthermore, by automatically collecting the corresponding log data through the log collection configuration information, the efficiency and accuracy of log collection are effectively improved.

[0006] In conjunction with the first aspect, in the first embodiment of the first aspect, at least one of the log registration component, the collection rule configuration component, and the host association component is pre-configured with verification rules; the verification rules are used to restrict the input collection configuration information, and / or to verify the input collection configuration information, and display a prompt message if the verification fails, the prompt message indicating a modification prompt for the collection configuration information.

[0007] The log management method provided in this embodiment of the invention restricts the input collection configuration information through verification rules and / or verifies the input collection configuration information so that a prompt message for modifying the collection configuration information is generated when the verification fails. This prompts the user to adjust the collection configuration information according to the prompt message to avoid configuration errors, effectively improves configuration efficiency, and achieves maximum optimization of collection configuration.

[0008] In conjunction with the first aspect, in the second embodiment of the first aspect, at least one of the log registration component, the collection rule configuration component, and the host association component has a pre-set parameter generation rule; obtaining the log collection configuration information through the visualization page includes: generating second collection configuration information based on the parameter generation rule and the first collection configuration information input through the visualization page.

[0009] The log management method provided in this embodiment of the invention generates second collection configuration information for subsequent log collection by using preset parameter generation rules and first collection configuration information. Thus, the log collection configuration information can be customized through a visual page.

[0010] In conjunction with the first aspect, in the third embodiment of the first aspect, the step of deploying the log collection service based on the target host information includes: gradually releasing the log collection service to the target host based on the installation package of the log collection service.

[0011] The log management method provided in this embodiment of the invention implements log collection services in the target host based on canary release, thereby enabling the deployment and maintenance of log collection in batches, effectively improving security.

[0012] In conjunction with the first aspect, in the fourth embodiment of the first aspect, the step of converting the collected log data into quantifiable data includes: collecting text data of the original log; asynchronously processing the text data to obtain processed text data; and performing word segmentation on the processed text data based on regular expression rules to obtain quantifiable data.

[0013] The log management method provided in this embodiment of the invention asynchronously processes the text data of the collected raw logs and performs word segmentation on the processed text data based on regular expression rules to obtain quantifiable data, thereby improving the concurrency of log collection and thus improving the efficiency of log collection.

[0014] In conjunction with the first aspect, in the fifth embodiment of the first aspect, the visualization page further includes an operation and maintenance configuration component, the collected configuration information further includes operation and maintenance configuration information obtained through the operation and maintenance configuration component, and the method further includes: based on the operation and maintenance configuration information, determining whether log monitoring needs to be performed; if log monitoring needs to be performed, monitoring whether the quantifiable data triggers an alarm condition; if the quantifiable data triggers an alarm condition, issuing an alarm message.

[0015] The log management method provided in this embodiment of the invention determines whether log monitoring is required based on operation and maintenance configuration information. When target logs need to be monitored, it further monitors whether quantifiable data triggers alarm conditions. When quantifiable data triggers alarm conditions, it issues alarm information, thereby avoiding log collection errors and ensuring the security and accuracy of log collection.

[0016] In conjunction with the first aspect or any one of the first to fifth embodiments of the first aspect, in the sixth embodiment of the first aspect, the method further includes: sending the obtained quantifiable data to a log search engine for storage.

[0017] The log management method provided in this embodiment of the invention sends the obtained quantifiable data to a log search engine for storage, thereby facilitating subsequent log queries.

[0018] According to a second aspect, embodiments of the present invention provide a log management device applied to a log management platform for executing the log management method described in the first aspect or any embodiment of the first aspect.

[0019] According to a third aspect, embodiments of the present invention provide an electronic device, including: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the log management method described in the first aspect or any embodiment of the first aspect.

[0020] According to a fourth aspect, embodiments of the present invention provide a computer-readable storage medium storing computer instructions for causing a computer to perform the log management method described in the first aspect or any embodiment of the first aspect.

[0021] It should be noted that the beneficial effects of the log management device, electronic device, and computer-readable storage medium provided in the embodiments of the present invention can be found in the description of the corresponding content in the log management method, and will not be repeated here. Attached Figure Description

[0022] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0023] Figure 1 This is a flowchart of a log management method according to an embodiment of the present invention;

[0024] Figure 2 This is another flowchart of the log management method according to an embodiment of the present invention;

[0025] Figure 3 This is another flowchart of the log management method according to an embodiment of the present invention;

[0026] Figure 4 This is a structural block diagram of a log management device according to an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0029] Log file collection and quantification can be achieved using open-source technologies like Flume and Logstash. The main approach involves deploying Flume-agent on the target server to crawl and collect log file content, then using Logstash to quantify and parse the collected log data before finally storing it in Elasticsearch. However, this method involves multiple components and is relatively complex to configure, making it difficult for less technically skilled operations personnel to configure successfully. This results in low log collection efficiency and high operational costs.

[0030] Based on this, the technical solution of the present invention provides configuration process guidance for log collection configuration information through a visual page, and automatically collects the corresponding target logs through the log collection configuration information, thereby reducing the technical threshold for operation and maintenance personnel and improving the efficiency and accuracy of log collection.

[0031] According to an embodiment of the present invention, an embodiment of a log management method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0032] This embodiment provides a log management method that can be used in log management platforms composed of electronic devices such as mobile phones, tablets, and computers. Figure 1 This is a flowchart of a log management method according to an embodiment of the present invention, such as... Figure 1 As shown, the process includes the following steps:

[0033] S11 allows you to obtain log collection configuration information through a visual page.

[0034] The visualization page includes a log registration component, a collection rule configuration component, and a host association component. The collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component.

[0035] The log collection visualization page is based on Python and utilizes technologies such as Django REST Framework and Vue to generate log registration components, collection rule configuration components, and host association components. Collection configuration information includes log registration information, collection rule configuration information, and target host information for associated log collection. Technical personnel can customize log collection configurations through the visualization page, including log registration configuration, collection rule configuration, and associating target hosts for log collection. Combined with monitoring and push notifications, it enables more convenient and comprehensive interactive management.

[0036] Specifically, technicians input log registration information through the log registration component on the visual page, input collection rules through the collection rule configuration component on the visual page, and input target host information through the host association component on the visual page. Correspondingly, the electronic device can obtain log collection configuration information such as log registration information, collection rules, and target host information, realizing intelligent, navigation-style collection configuration input. Each step in the log collection process can be displayed in a linked manner using icons; for example, blue, yellow, and gray icons can be used to represent the configured, configuring, and unconfigured states, respectively. Therefore, this embodiment of the invention implements a customizable log collection configuration workflow, facilitating use by operations and maintenance personnel.

[0037] S12, deploy the log collection service based on the target host information, and configure the log collection task based on the log registration information and collection rules.

[0038] After the log collection configuration is completed, the log management platform can determine the target host for log collection based on the collection configuration information, and deploy the corresponding log collection service on the target host to perform log collection. At the same time, it can configure the corresponding log collection task on the target host according to the log registration information and collection rules in the collection configuration information.

[0039] S13 controls the log collection service to execute log collection tasks and converts the collected log data into quantifiable data.

[0040] Execute log collection tasks on the target host. These tasks may include deploying the log collection service, updating the log collection configuration, starting and stopping the log collection service, and viewing details of the log collection task execution.

[0041] The log management platform controls the target host to start the log collection service and execute log collection tasks. Specifically, the target host captures and collects log data, and uses Logstash to quantify and parse the captured and collected log data to obtain the quantifiable data corresponding to the logs.

[0042] The log management method provided in this embodiment obtains log collection configuration information through a visual page. This configuration information includes log registration information, collection rules, and target host information. Then, based on the target host information, a log collection service is deployed, and log collection tasks are configured according to the log registration information and collection rules. This controls the log collection service to execute log collection tasks and transforms the collected log data into quantifiable data. This method provides effective configuration process guidance for log collection configuration information through a visual page, allowing operations and maintenance personnel to customize the log collection configuration information, thereby lowering the technical threshold for operations and maintenance personnel. By automatically collecting the corresponding log data through the log collection configuration information, the efficiency and accuracy of log collection are effectively improved.

[0043] This embodiment provides a log management method that can be used in log management platforms composed of electronic devices such as mobile phones, tablets, and computers. Figure 2 This is a flowchart of a log management method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:

[0044] S21. Obtain log collection configuration information through a visual page. This visual page includes a log registration component, a collection rule configuration component, and a host association component. The collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component.

[0045] Specifically, at least one of the log registration component, the collection rule configuration component, and the host association component has pre-defined validation rules. These validation rules are used to restrict the input collection configuration information and / or to validate the input collection configuration information. If the validation fails, a prompt message is displayed, indicating a suggestion to modify the collection configuration information.

[0046] The pre-defined validation rules are used for log configuration checks and analysis. These rules restrict the data collection configuration information input by technical personnel. If the currently input data collection configuration information does not match the pre-defined validation rules, it indicates that the current configuration does not comply with the log collection configuration rules and needs to be modified or optimized. The log management platform can use these validation rules to assist in the analysis of the data collection configuration information input by technical personnel, performing real-time configuration checks and analysis on configured and in-process data collection configuration information, and displaying modification or optimization prompts through a visual interface. When the log management platform detects configuration problems or optimization items, it immediately displays modification or optimization suggestions at the corresponding icon location. Technical personnel can click on these suggestions to confirm the changes, at which point the log management platform can automatically adjust the current data collection configuration information.

[0047] The verification rules may include: consistency checks during the configuration process, configuration security scans, and recommendations for optimizing configuration parameters.

[0048] Specifically, the consistency check includes checking the consistency of Flume and Logstash configurations, whether the Kafka clusters used are consistent, and whether the topics used are consistent. If any inconsistencies are found, an immediate prompt will be issued. After the technician clicks the prompt button, the system will adjust the configuration parameters based on the user's operation.

[0049] The configuration security scan performs a security check on the data before saving at each configuration stage to determine if there is any sensitive data, including user passwords, tokens, private keys, SSL certificate information, etc. If such data is found and is not encrypted, a warning will be issued and the next step will not be allowed.

[0050] For Flume, the system recommends optimizing configuration parameters by setting the ratio of sources, channels, and sinks to the collected files. The number of sources, channels, and sinks is set according to this ratio, and the channel data level and growth rate are monitored in real time. When the level exceeds 60% and the growth rate is greater than the weighting factor set by the system than the current sink-to-collected-file ratio, an optimization prompt is immediately issued. Technical personnel can click the optimization button in the prompt box, and the system will automatically perform batch canary configuration optimization. For Logstash, the system automatically recommends setting the number of consumer threads for Logstash according to a fixed ratio based on the number of partitions in the top layer of different Kafka servers. Simultaneously, when Kafka data backlog exceeds 30%, the system automatically suggests scaling up the collection and incrementally scaling up the Logstash service (scaling up instance by instance based on load test data). Scaling up stops once the backlog data returns to normal levels.

[0051] Specifically, at least one of the log registration component, the collection rule configuration component, and the host association component has a pre-set parameter generation rule; step S21 may include: generating second collection configuration information based on the parameter generation rule and the first collection configuration information input through the visualization page.

[0052] The parameter generation rules are pre-defined rules within the log management platform used to generate collection configuration information. The log registration component, collection rule configuration component, and host association component each have their own corresponding parameter generation rules. The first collection configuration information consists of the configuration data for log collection input by technical personnel through the visual interface; the second collection configuration information consists of the configuration data that the log management platform can recognize.

[0053] The log management platform can respond to the first collection configuration information entered by technicians through a visual page, and generate the second collection configuration information for log collection according to the parameter generation rules.

[0054] S22, deploy the log collection service based on the target host information, and configure the log collection task based on the log registration information and collection rules.

[0055] Specifically, step S22 may include: deploying the log collection service in a canary manner on the target host based on the installation package of the log collection service.

[0056] The log management platform parses the installation package of the log collection service to determine the target host for executing the service. The platform can control the target host to execute the log collection service using a canary release method via a communication interface. Specifically, the canary release method employs a canary release approach: after the canary release begins, a newly configured collection agent is deployed and started, but the collected data is not directly stored in the backend database; instead, it is temporarily stored in a cache for testers to verify the new version online. If the new version works correctly, the cached data is sent to the backend database for storage. The running status of the new version is then observed, and various runtime data are collected for comparison with the old version. Once the new version is confirmed to be running well, more agents with the new configuration are gradually started on more servers until 100% of the agents are switched to the new version, completing the canary release. Based on the updated log collection configuration information, the log collection service is started according to the updated configuration, log collection is performed, and the execution details of the log collection process are viewed in real time.

[0057] S23, control the log collection service to execute the log collection task, and convert the collected log data into quantifiable data. For detailed explanations, please refer to the relevant descriptions of the corresponding steps in the above embodiments, which will not be repeated here.

[0058] The log management method provided in this embodiment restricts the input collection configuration information through validation rules and / or validates the input collection configuration information. When validation fails, a prompt message is generated to modify the collection configuration information, thus prompting the user to adjust the collection configuration information according to the prompt message to avoid configuration errors, effectively improving configuration efficiency and achieving maximum optimization of collection configuration. Second collection configuration information is generated using preset parameter generation rules and the first collection configuration information for subsequent log collection, thereby enabling customization of log collection configuration information through a visual page. By implementing the log collection service on the target host using a canary release approach, log collection deployment and maintenance can be carried out in batches, effectively improving security.

[0059] This embodiment provides a log management method that can be used in log management platforms composed of electronic devices such as mobile phones, tablets, and computers. Figure 3 This is a flowchart of a log management method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:

[0060] S31 allows users to obtain log collection configuration information through a visual interface.

[0061] The visualization page includes a log registration component, a collection rule configuration component, and a host association component. The collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component.

[0062] For detailed explanations, please refer to the relevant descriptions of the corresponding steps in the above embodiments, which will not be repeated here.

[0063] As an optional implementation, the aforementioned visualization page may include an operation and maintenance configuration component, and the collected configuration information may further include operation and maintenance configuration information obtained through the operation and maintenance configuration component. Step S31 may also include:

[0064] S310 determines whether log monitoring needs to be performed based on operation and maintenance configuration information.

[0065] If the target log needs to be monitored, its corresponding collection configuration information contains the log's operation and maintenance configuration information. This allows the log collection source to determine whether the target log needs to be monitored, what type of monitoring should be performed, the monitoring cycle, and whether alarms should be triggered. After obtaining the log collection configuration information, the log management platform can parse it to determine whether it contains operation and maintenance configuration information, thereby determining whether the target log needs to be monitored. When log monitoring is required, step S320 is executed; otherwise, no monitoring is needed, and step S32 is executed directly.

[0066] It should be noted that electronic devices can configure log operations and maintenance through the operation and maintenance configuration component in the visual page, and inject this operation and maintenance configuration information into the log collection configuration information.

[0067] If log monitoring is required for S320, the monitoring can quantify whether the data triggers alarm conditions.

[0068] When log monitoring is required, the corresponding operation and maintenance configuration information is enabled, and the quantifiable data corresponding to the logs is periodically monitored based on the operation and maintenance configuration information. The log management platform can monitor the log collection process in real time to obtain real-time monitoring results. When the real-time monitoring results of the quantifiable data are inconsistent with the expected results, the log management platform can trigger alarm conditions.

[0069] It should be noted that before collecting logs, logs can be registered and their basic information, such as capacity, traffic, and concurrency, can be automatically verified. While collecting target logs, the basic information of the target logs can be monitored to prevent them from exceeding the resource capacity of the platform on which the electronic device is located.

[0070] S330: If quantifiable data triggers the alarm condition, an alarm message will be issued.

[0071] When quantifiable data triggers an alarm condition, it indicates an anomaly in the log collection process. At this point, the log management platform can issue an alarm message so that technical personnel can handle the anomaly promptly. For example, if a monitored item triggers an alarm, the log management platform can send the alarm message via SMS, email, or WeChat.

[0072] The system determines whether log monitoring is needed by using operation and maintenance configuration information. When target logs need to be monitored, it further monitors whether quantifiable data triggers alarm conditions. When quantifiable data triggers alarm conditions, it issues alarm information, thereby avoiding log collection errors and ensuring the security and accuracy of log collection.

[0073] S32, deploy the log collection service based on the target host information, and configure the log collection task based on the log registration information and collection rules. For detailed explanations, please refer to the relevant descriptions of the corresponding steps in the above embodiments; they will not be repeated here.

[0074] S33 controls the log collection service to execute log collection tasks and convert the collected log data into quantifiable data.

[0075] Specifically, step S33 above may include the following steps:

[0076] S331, collect text data from the raw log.

[0077] Log management platforms can use flume-agent to capture and parse the text data corresponding to the collected raw logs.

[0078] S332 performs asynchronous processing on the text data to obtain the processed text data.

[0079] The text data corresponding to the original logs is sent to the distributed publish-subscribe messaging system Kafka for asynchronous processing, resulting in Kafka-processed text data. Asynchronous processing of the text data improves response efficiency for handling large volumes of text data requests.

[0080] S333 performs word segmentation on the processed text data based on regular expression rules to obtain quantifiable data.

[0081] Once Kafka has finished processing the text data, Logstash, configured in the log management platform, can retrieve the processed text data from Kafka and perform word segmentation on the text data based on regular expression rules to obtain quantifiable data corresponding to the text data (i.e., quantifiable data corresponding to the log).

[0082] S34. The obtained quantifiable data is sent to the log search engine for storage.

[0083] After obtaining the quantifiable data corresponding to the logs, the log management platform can transmit the quantifiable data to a log search engine (such as Elasticsearch) for storage, so as to facilitate subsequent querying of the log data.

[0084] The log management method provided in this embodiment asynchronously processes the text data of the collected raw logs and performs word segmentation on the processed text data based on regular expression rules to obtain quantifiable data, thereby improving the concurrency of log collection and thus improving the efficiency of log collection.

[0085] This embodiment also provides a log management device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation or a combination of software and hardware is also possible and contemplated.

[0086] This embodiment provides a log management device, such as... Figure 4 As shown, it includes:

[0087] Module 41 is used to obtain log collection configuration information through a visual page. The visual page includes a log registration component, a collection rule configuration component, and a host association component. The collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0088] The deployment configuration module 42 is used to deploy the log collection service based on the target host information, and to configure the log collection task based on the log registration information and collection rules. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0089] The conversion module 43 is used to control the log collection service to execute log collection tasks and convert the collected log data into quantifiable data. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0090] The log management device provided in this embodiment offers effective configuration process guidance for log collection configuration information through a visual page. Operation and maintenance personnel can customize the log collection configuration information through the visual page, thereby reducing the technical threshold for operation and maintenance personnel. By automatically collecting the corresponding log data through the log collection configuration information, the efficiency and accuracy of log collection are effectively improved.

[0091] Optionally, at least one of the components in module 41—the log registration component, the collection rule configuration component, and the host association component—has pre-defined verification rules. These verification rules are used to restrict the input collection configuration information and / or to verify the input collection configuration information. If the verification fails, a prompt message is displayed, indicating a suggestion to modify the collection configuration information.

[0092] Optionally, at least one of the components—the log registration component, the collection rule configuration component, and the host association component—has pre-defined parameter generation rules. The aforementioned acquisition module 41 may include:

[0093] The generation submodule is used to generate second collection configuration information based on the parameter generation rules and the first collection configuration information input through the visualization page. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0094] Optionally, the deployment configuration module 41 may include:

[0095] The deployment submodule is used to distribute the log collection service installation package to the target host in a canary deployment phase. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0096] Optionally, the conversion module 43 described above may include:

[0097] The data collection submodule is used to collect text data from the raw logs. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0098] The asynchronous processing submodule is used to asynchronously process text data to obtain processed text data. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0099] The word segmentation submodule is used to segment the processed text data based on regular expression rules to obtain quantifiable data. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0100] Optionally, the above-mentioned log management device may further include:

[0101] The judgment module is used to determine whether log monitoring needs to be performed based on the operation and maintenance configuration information. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0102] The triggering module is used to monitor whether quantifiable data triggers alarm conditions if log monitoring is required. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0103] The alarm module is used to issue an alarm message if quantifiable data triggers an alarm condition. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0104] The storage module is used to send the obtained quantifiable data to the log search engine for storage. For detailed explanations, please refer to the relevant descriptions in the above method embodiments; they will not be repeated here.

[0105] In this embodiment, the log management device is presented in the form of a functional unit. Here, a unit refers to an ASIC circuit, a processor and memory that execute one or more software or fixed programs, and / or other devices that can provide the above functions.

[0106] This invention also provides an electronic device having the above-described features. Figure 4 The log management device shown.

[0107] Please see Figure 5 , Figure 5 This is a schematic diagram of the structure of an electronic device provided in an optional embodiment of the present invention, such as... Figure 5 As shown, the electronic device may include: at least one processor 501, such as a central processing unit (CPU), at least one communication interface 503, memory 504, and at least one communication bus 502. The communication bus 502 is used to enable communication between these components. The communication interface 503 may include a display screen or a keyboard; optionally, the communication interface 503 may also include a standard wired interface or a wireless interface. The memory 504 may be high-speed volatile random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory 504 may also be at least one storage device located remotely from the aforementioned processor 501. The processor 501 may be combined with... Figure 4 The described apparatus has an application program stored in memory 504, and a processor 501 calls the program code stored in memory 504 to perform any of the above method steps.

[0108] The communication bus 502 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication bus 502 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0109] The memory 504 may include volatile memory, such as random-access memory (RAM); the memory may also include non-volatile memory, such as flash memory, hard disk drive (HDD) or solid-state drive (SSD); the memory 504 may also include a combination of the above types of memory.

[0110] The processor 501 can be a central processing unit (CPU), a network processor (NP), or a combination of a CPU and an NP.

[0111] The processor 501 may further include a hardware chip. This hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD), or a combination thereof. The PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), or any combination thereof.

[0112] Optionally, memory 504 is also used to store program instructions. Processor 501 can call the program instructions to implement the functions described in this application. Figures 1 to 3 The log management method shown in the embodiment.

[0113] This invention also provides a non-transitory computer storage medium storing computer-executable instructions that can execute the log management method processing method in any of the above method embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0114] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and such modifications and variations all fall within the scope defined by the appended claims.

Claims

1. A log management method, characterized in that, The method is executed by the log management platform and includes: Log collection configuration information can be obtained through a visual page; wherein, the visual page includes a log registration component, a collection rule configuration component, and a host association component, and the collection configuration information includes log registration information obtained through the log registration component, collection rules obtained through the collection rule configuration component, and target host information obtained through the host association component; Deploy log collection services based on the target host information, and configure log collection tasks based on the log registration information and the collection rules; Control the log collection service to execute the log collection task and convert the collected log data into quantifiable data.

2. The method according to claim 1, characterized in that, At least one of the log registration component, the collection rule configuration component, and the host association component has a pre-defined verification rule; The verification rules are used to restrict the input collection configuration information and / or to verify the input collection configuration information. If the verification fails, a prompt message is displayed, which indicates a prompt to modify the collection configuration information.

3. The method according to claim 1, characterized in that, At least one of the log registration component, the collection rule configuration component, and the host association component has pre-defined parameter generation rules; The process of obtaining log collection configuration information through a visual page includes: Based on the parameter generation rules and the first acquisition configuration information input through the visualization page, the second acquisition configuration information is generated.

4. The method according to claim 1, characterized in that, The deployment of the log collection service based on the target host information includes: Based on the installation package of the log collection service, the log collection service is deployed in a canary manner on the target host.

5. The method according to claim 1, characterized in that, The process of converting the collected log data into quantifiable data includes: Collect text data from raw logs; The text data is processed asynchronously to obtain the processed text data; The processed text data is segmented using regular expression rules to obtain quantifiable data.

6. The method according to claim 1, characterized in that, The visualization page also includes an operations and maintenance configuration component, and the collected configuration information also includes operations and maintenance configuration information obtained through the operations and maintenance configuration component. The method further includes: Based on the aforementioned operation and maintenance configuration information, determine whether log monitoring needs to be performed; If log monitoring is required, monitor whether the quantifiable data triggers alarm conditions; If the quantifiable data triggers an alarm condition, an alarm message will be issued.

7. The method according to any one of claims 1-6, characterized in that, Also includes: The obtained quantifiable data is sent to a log search engine for storage.

8. A log management device, characterized in that, Applications in log management platforms include: The acquisition module is used to acquire log collection configuration information through a visual page; wherein, the visual page includes a log registration component, a collection rule configuration component, and a host association component, and the collection configuration information includes log registration information acquired through the log registration component, collection rules acquired through the collection rule configuration component, and target host information acquired through the host association component; The deployment configuration module is used to deploy the log collection service according to the target host information, and to configure the log collection task according to the log registration information and the collection rules. The conversion module is used to control the log collection service to execute the log collection task and convert the collected log data into quantifiable data.

9. An electronic device, characterized in that, include: A memory and a processor are communicatively connected, the memory stores computer instructions, and the processor executes the log management method according to any one of claims 1-7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the log management method according to any one of claims 1-7.

Images