Configurable log progress analysis method, device and system

Abstract

One aspect of the present invention relates to a configurable log progress analysis method, device and system. A configurable log progress analysis method is disclosed, comprising: generating a first configuration file corresponding to a first deployment tool based on the use of the first deployment tool, the first configuration file specifies one or more log matchers, and the one or more log matchers are used to search for one or more log files corresponding thereto by matching; starting the one or more log matchers based on the first configuration file to extract the one or more log files corresponding thereto and determine the progress of each of the one or more log files; and calculating the total progress of the first deployment tool based on the progress of each of the one or more log files.

Description

Technical Field

[0001] The present invention generally relates to the field of cloud computing. In particular, the present invention relates to the field of system deployment progress analysis. Background Art

[0002] With the popularization of cloud computing, deployment tools can be used to install systems on multiple devices at the same time. Deployment tools are tools that solidify the original manual deployment process and provide simplified and unified configuration items for automatic deployment by product, environment, and module. Existing open source deployment tools include cobbler, ansible, chef, etc., as well as some customized tools.

[0003] However, these deployment tools do not provide interfaces and scripts for querying the deployment progress. If the operator wants to understand the deployment progress, he needs to manually analyze the deployment logs, such as manually writing program code to analyze the logs. Since the deployment log formats and contents of different tools are different, the current method is to develop different log analysis programs for different tools. If different tools are implemented separately, the system program code needs to be frequently modified and maintained in the face of many deployment tools. Such log analysis methods have a low degree of automation and are not universal and extensible. Summary of the invention

[0004] A brief overview of the invention is given below in order to provide a basic understanding of some aspects of the invention. However, it should be understood that this overview is not an exhaustive overview of the invention. It is not intended to identify the key or important parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to give some concepts of the invention in a simplified form as a prelude to a more detailed description that will be given later.

[0005] In order to solve the problem that the current log progress analysis method has a low degree of automation and lacks universality and scalability, the present invention proposes a configurable log progress analysis method, device and system. The method, device and system of the present invention realize universal and configurable log progress analysis for deployment tools. By automatically generating configuration files for different deployment tools, accurate log progress analysis and aggregation can be automatically performed without modifying the system program code, thereby improving the efficiency of system development and maintenance.

[0006] According to one aspect of the present invention, a configurable log progress analysis method is provided, comprising: generating a first configuration file corresponding to a first deployment tool based on the use of the first deployment tool, the first configuration file specifying one or more log matchers, the one or more log matchers being used to search for one or more log files corresponding thereto through matching; starting the one or more log matchers based on the first configuration file to extract the one or more log files corresponding thereto and determine the progress of each of the one or more log files; and calculating the total progress of the first deployment tool based on the progress of each of the one or more log files.

[0007] According to another aspect of the present invention, a configurable log progress analysis device is provided, comprising: a memory on which instructions are stored; and a processor configured to execute the instructions stored in the memory to perform the above method.

[0008] According to another aspect of the present invention, a configurable log progress analysis system is provided, comprising: a log matcher group, including one or more log matchers, the one or more log matchers being used to extract the one or more log files that match them and determine the progress of each of the one or more log files; and the log progress analysis device as described above.

[0009] According to yet another aspect of the present invention, a computer program product is provided, which comprises computer executable instructions. When the computer executable instructions are executed by one or more processors, the one or more processors execute the method according to the above aspect of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings, which constitute a part of the specification, illustrate embodiments of the present invention and, together with the description, serve to explain the principles of the present invention.

[0011] The present invention may be more clearly understood from the following detailed description with reference to the accompanying drawings, in which:

[0012] Figure 1 A block diagram of a configurable log progress analysis system according to an embodiment of the present invention is shown;

[0013] Figure 2 A flow chart of a configurable log progress analysis method according to an embodiment of the present invention is shown;

[0014] Figure 3 A flowchart showing a method for determining the progress of a single log file by analyzing each log file line by line according to an embodiment of the present invention; and

[0015] Figure 4 An exemplary hardware configuration diagram of a configurable log progress analysis device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0016] The following detailed description is made with reference to the accompanying drawings, and the following detailed description is provided to assist in a comprehensive understanding of various exemplary embodiments of the present invention. The following description includes various details to assist in understanding, but these details are considered to be examples only and are not intended to limit the present invention, which is defined by the appended claims and their equivalents. The words and phrases used in the following description are only used to enable a clear and consistent understanding of the present invention. In addition, for the sake of clarity and brevity, descriptions of well-known structures, functions, and configurations may be omitted. Those of ordinary skill in the art will recognize that various changes and modifications may be made to the examples described herein without departing from the spirit and scope of the present invention.

[0017] Refer to the following Figure 1 A configurable log progress analysis system according to an embodiment of the present invention is described.

[0018] Figure 1 The configurable log progress analysis system 100 according to an embodiment of the present invention is shown, which includes two main components: a log analysis device 101 and a log matcher group 102. The log matcher group 102 includes one or more log matchers 102a, 102b...102n.

[0019] The log analysis device 102 may be configured to identify the deployment tool used, generate a configuration file based on the deployment tool used, and call a corresponding log matcher in the log matcher group 102 based on the configuration file to analyze and summarize the progress of the log file. According to one embodiment, the generated configuration file may be stored in a memory.

[0020] The log matcher group 102 includes a log matcher corresponding to each log file of each deployment tool in the existing deployment tools. Assuming that the system is mainly developed to detect the progress of three deployment tools, cobbler, ansible and chef, where the log of cobbler includes three files a, b and c, the log of ansible includes two files d and e, and the log of chef includes three files f, g and h, then the log matcher group 102 will include log matchers 102a, 102b, 102c, 102d, 102e, 102f, 102g and 102h corresponding to each file respectively.

[0021] The log matcher is used to search for log files corresponding to the log matcher through matching.

[0022] According to one embodiment, log matcher set 102 may be extended to include new log matchers corresponding to log files of new deployment tools or new versions of existing deployment tools.

[0023] Those skilled in the art should understand that the above are only examples, and the present invention is not limited to these three deployment tools, and each deployment tool is not limited to the number of log files listed.

[0024] The system of the present invention can be applied to an automated deployment platform for network clouds, for example, as a subsystem that provides log progress analysis functions, and can also be transplanted to other systems to provide installation and deployment log analysis services or used as a separate system.

[0025] Refer to the following Figure 2 A flow chart describing a configurable log progress analysis method according to an embodiment of the present invention is provided. The configurable log progress analysis method may be Figure 1 The log progress analysis device 101 shown is executed.

[0026] In step 201, a first configuration file corresponding to a first deployment tool is generated based on the use of the first deployment tool, wherein the first configuration file specifies one or more log matchers, and the one or more log matchers are used to search for one or more log files corresponding thereto through matching.

[0027] The use of the first deployment tool may be detected by any means known to those skilled in the art. According to one embodiment, the use of the first deployment tool may be identified by identifying an initialization or startup process of the first deployment tool.

[0028] According to one embodiment, the first configuration file is generated by selecting one or more log matchers corresponding to the first deployment tool from the preset log matcher group 102 .

[0029] In step 202, the one or more log matchers are started based on the first configuration file to extract the one or more log files corresponding thereto and determine the progress of each of the one or more log files.

[0030] The progress of each log file may be analyzed by a log progress analysis method known to those skilled in the art.

[0031] According to one embodiment, the progress of the log file may be determined by line-by-line analysis.

[0032] In step 203, the total progress of the first deployment tool is calculated based on the progress of each log file in the one or more log files.

[0033] After obtaining the progress of each log file, those skilled in the art can summarize it into the total progress of the first deployment tool in an appropriate manner according to the deployment characteristics of the first deployment tool. The summary method can be selected by those skilled in the art according to specific circumstances and will not be described in detail here.

[0034] The log progress analysis method realizes a general, automated and configurable log analysis method for system installation and deployment by automatically generating a configuration file specific to a deployment tool.

[0035] When the above method is adopted, for different deployment tools, only the configuration file needs to be modified to complete the log analysis. The following embodiment describes the situation when different deployment tools are used for deployment.

[0036] In this embodiment, a second configuration file corresponding to the second deployment tool is generated based on the use of the second deployment tool, and the second configuration file specifies one or more log matchers, wherein at least one of the one or more log matchers specified by the second configuration file is different from the one or more log matchers specified by the first configuration file.

[0037] The second deployment tool is a deployment tool different from the first deployment tool. Figure 1 In the described example, assuming that the first deployment tool is cobbler and the second deployment tool is ansible, the first configuration file specifies log matchers 102a, 102b, and 102c, and the second configuration file specifies log matchers 102d and 102e.

[0038] According to one embodiment, the second deployment tool may also be a different version of the first deployment tool. In this case, only some log matchers in the first configuration file and the second configuration file may be different.

[0039] In this way, different configuration files are used to analyze different log files for different deployment tools. The log analysis of deployment tools in the past is not universal, nor is it universal between tools. In this way, log progress detection of the new version of the deployment tool can be achieved by only modifying part of the configuration file, such as adding / deleting / modifying a small number of log matchers.

[0040] According to one embodiment, the generated first configuration file can be stored in the memory for future use when the first deployment tool is used again for deployment. When the first deployment tool is used again, the memory can be searched for whether there is an existing appropriate configuration file. If there is, the stored configuration file is directly used, which can avoid the step of repeatedly generating the configuration file and improve efficiency.

[0041] In the above embodiment, different configuration files for different deployment tools may be stored in the memory for future reuse.

[0042] Refer to the following Figure 3 A flowchart is provided to describe a method for determining the progress of a single log file by analyzing each log file line by line according to an embodiment of the present invention. The method is executed by a log progress analysis device by calling a log matcher.

[0043] In this embodiment, each log matcher is associated with one or more line matchers, and the line matchers are used to search for corresponding log lines in the log file by matching. Here, the log line refers to a line of text obtained by performing line break segmentation on the log file.

[0044] In step 301, a log matcher is started to extract corresponding log files.

[0045] The storage path of the log file is based on the rules determined when the tool is deployed. In one embodiment, the log analysis device can calculate the path of a specific log according to the rules.

[0046] Taking the installation of the operating system by cobbler as an example, the basic path of the installation log pulled by cobbler from the target machine is / var / log / cobbler / anamon, and the subdirectory is determined by the hostname of the target machine. Therefore, the system can automatically calculate the full path of anaconda.log of the machine with hostname host-1 as / var / log / cobbler / anamon / host-1 / anaconda.log.

[0047] In this way, when extracting log files, each device can be accurately found.

[0048] In step 302, the log file is segmented by line breaks to obtain each line of the log file.

[0049] In step 303, one or more line matchers associated with the log matcher search for corresponding log lines by matching line by line, and submit the matching results to the log analysis device for aggregation.

[0050] According to one embodiment, in addition to the specified file matcher, the configuration file also includes: a matching regular expression of the line matcher and the progress of the corresponding log line of the matcher. The progress of the log line can be preset by a person skilled in the art based on experience or actual conditions. Therefore, while obtaining the matching result, the log progress analyzer can also obtain the progress of the matched log line according to the configuration file.

[0051] According to one embodiment, the configuration file may further include return information and a success mark. The return information may be used to display the successfully matched log lines to the user in user-understandable text. The success mark is used to indicate whether the matching process is successful, or whether a failure occurs and a rematch is required.

[0052] In step 304, corresponding aggregation modes are selected for different log files.

[0053] According to one embodiment, the configuration file may further include parameters for determining the aggregation mode. Based on these parameters, the log progress analysis device may determine different aggregation modes for different log files. As described above, log files of different deployment tools, or different log files of the same deployment tool may have different characteristics. Therefore, compared with using the same aggregation mode for all log files, selecting different aggregation methods according to the characteristics of different log files can make the aggregation results more accurate.

[0054] In the above embodiment, the log progress analysis device determines the aggregation mode according to the parameters in the configuration file. According to the determined aggregation mode, the process will proceed to one or more of steps 306-308, so as to aggregate different log files in different aggregation modes.

[0055] In step 305, the maximum value of the progress of all rows is taken. This is usually applicable to the internal summary of a single file.

[0056] In step 306, the minimum value of the progress of all rows is taken. This is usually applicable to the situation where the progress of multiple devices is located in a single log file.

[0057] In step 307, the progress values ​​of all the lines are added together. This is usually applicable to the case where the lines in the log file appear repeatedly for multiple times.

[0058] In step 308, the progress value of the last line searched is obtained (overwrite mode), which is usually used for internal summary of a single file.

[0059] In step 309, the progress of each log file aggregated in different ways is obtained.

[0060] In the above single file summary, there is a special case, that is, a line matcher of the log matcher is cumulative. In this case, the progress of the lines matched by the line matcher can be added as the progress of the line matcher, and then summarized with the progress of the lines identified by other line matchers according to the pattern in steps 305-308.

[0061] Those skilled in the art should understand that the aggregation methods in the above steps 305-308 are only examples and not limitations. Those skilled in the art may also select other more suitable aggregation methods according to the characteristics of the deployment tool.

[0062] After obtaining the progress of a single file, the log progress analysis device can further summarize the progress of multiple files. The way to summarize multiple files can be selected by those skilled in the art as needed. For example, according to one embodiment, the maximum value of the progress of multiple log files can be taken as the final progress of the deployment tool.

[0063] In actual applications, deployment tools usually deploy multiple devices at the same time. Therefore, the log progress analysis device can also extract log files from multiple devices for analysis at the same time. According to one embodiment, different aggregation methods can be used for different situations according to the number of devices and log files included in a single statistics. For example, the aggregation modes of multiple files are divided into 4 categories: single log file single device statistics, single log multiple device statistics, multiple log file single device statistics, and multiple log file multiple device statistics. Each deployment tool can select the corresponding statistical method to automatically aggregate the results.

[0064] The following takes the cobbler deployment tool as an example to introduce an example of a log progress analysis method according to an embodiment of the present invention.

[0065] A possible cobbler.progress.yml configuration file is shown below.

[0066]

[0067]

[0068] The above configuration file specifies three log files: sys.log, anaconda.log, and packaging.log. Each log file corresponds to a log matcher (fileMatcher), and each log matcher has a set of line matchers (LineMatcher).

[0069] According to the log file storage rules of the cobbler deployment tool, the three log matchers can obtain the three log files under the cobbler log directory (for example, / var / log / cobbler / anamon). After that, a set of line matchers under each log matcher matches the corresponding line logs line by line in the log file.

[0070] Assume the matching results are as follows:

[0071]

[0072] Each log file is summarized below according to its respective summary mode.

[0073] The type of the log matcher "sys.log" is specified as "INCREMENT" in the configuration file. Therefore, the progress of the three matched log lines will be added to get 0.3 as the progress of the log file sys.log.

[0074] For anaconda.log and packaging.log, the maximum value is used for aggregation. Therefore, the progress of the log file anaconda.log is 0.3, and the progress of packaging.log is 0.5.

[0075] After that, multiple files are summarized. For example, the maximum value summary method can be adopted, that is, the maximum value of the progress in the three files is taken as the total progress. Therefore, the progress counted by the cobbler deployment tool this time is 0.5.

[0076] The log progress analysis method, device and system of the present invention realize a universal and configurable log analysis system for deployment tools. Different deployment tools only need to configure different configuration files to automatically perform accurate log progress analysis and summary without modifying the system program code. In addition, the log analysis can be accurate to the deployed device, and only the matching parameters of the device name need to be specified when matching rows.

[0077] Compared with the previous log progress analysis method, the log progress analysis method, device and system of the present invention are universal, configurable, highly automated and have accurate results.

[0078] Refer to the following Figure 4 An embodiment of hardware implementation of a configurable log progress analysis device is described.

[0079] The device 400 is an example of a hardware implementation of a device capable of applying the configurable log progress analysis method of the above aspects of the present invention. The device 400 may be any machine configured to perform processing and / or calculation. The device 400 may be, but is not limited to, a workstation, a server, a desktop computer, a laptop computer, a tablet computer, a personal data assistant (PDA), a smart phone, a vehicle-mounted computer, or a combination thereof.

[0080] like Figure 4As shown, the device 400 may include one or more elements that can be connected or communicated with the bus 402 via one or more interfaces. The bus 402 may include, but is not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MCA) bus, an Enhanced ISA (EISA) bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus. The device 400 may include, for example, one or more processors 404, one or more input devices 406, and one or more output devices 408. The one or more processors 404 may be any kind of processor, and may include, but is not limited to, one or more general-purpose processors or special-purpose processors (such as special-purpose processing chips). The processor 404 is configured, for example, to implement the cloud-edge collaborative implementation method for remote multi-person VR interaction in the present invention. The input device 408 may be any type of input device capable of inputting information to a computing device, and may include, but is not limited to, a mouse, a keyboard, a touch screen, a microphone, and / or a remote controller. The output device 408 may be any type of device capable of presenting information, and may include, but is not limited to, a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer.

[0081] The device 400 may also include or be connected to a non-transitory storage device 414, which may be any non-transitory storage device that can implement data storage, and may include but is not limited to a disk drive, an optical storage device, a solid-state memory, a floppy disk, a flexible disk, a hard disk, a tape or any other magnetic medium, a compressed disk or any other optical medium, a cache memory and / or any other storage chip or module, and / or any other medium from which a computer can read data, instructions and / or code. The device 400 may also include a random access memory (RAM) 410 and a read-only memory (ROM) 412. The ROM 412 may store programs, utilities or processes to be executed in a non-volatile manner. The RAM 410 may provide volatile data storage and store instructions related to the operation of the device 400. The device 400 may also include a network / bus interface 416 coupled to a data link 418. The network / bus interface 416 may be any type of device or system capable of enabling communication with an external device and / or a network, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication device, and / or a chipset (such as a Bluetooth TM equipment, 802.11 equipment, WiFi equipment, WiMax equipment, cellular communication facilities, etc.).

[0082] The present invention may be implemented as any combination of a device, a system, an integrated circuit, and a computer program on a non-transitory computer-readable medium. One or more processors may be implemented as an integrated circuit (IC), an application specific integrated circuit (ASIC), or a large scale integrated circuit (LSI), a system LSI, a super LSI, or a super LSI component that performs some or all of the functions described in the present invention.

[0083] The present invention includes the use of software, applications, computer programs or algorithms. The software, applications, computer programs or algorithms may be stored on a non-transitory computer-readable medium to enable a computer such as one or more processors to perform the above steps and the steps described in the accompanying drawings. For example, one or more memories store the software or algorithms in executable instructions, and one or more processors may associate a set of instructions to execute the software or algorithm to provide various functions according to the embodiments described in the present invention.

[0084] Software and computer programs (which may also be referred to as programs, software applications, applications, components or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural language, an object-oriented programming language, a functional programming language, a logic programming language, or an assembly language or a machine language. The term "computer-readable medium" refers to any computer program product, apparatus or device for providing machine instructions or data to a programmable data processor, such as magnetic disks, optical disks, solid-state storage devices, memories and programmable logic devices (PLDs), including computer-readable media that receive machine instructions as computer-readable signals.

[0085] The subject matter of the present invention is provided as an example of an apparatus, system, method and program for performing the features described in the present invention. However, in addition to the above-mentioned features, other features or variations may also be expected. It is expected that the implementation of the components and functions of the present invention may be accomplished with any emerging technology that may replace any of the above-mentioned implementation technologies.

[0086] In addition, the above description provides examples without limiting the scope, applicability or configuration set forth in the claims. Without departing from the spirit and scope of the present invention, the functions and arrangements of the elements discussed may be changed. Various embodiments may appropriately omit, replace or add various processes or components. For example, features described with respect to certain embodiments may be combined in other embodiments.

[0087] Similarly, while operations are depicted in a particular order in the drawings, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed to achieve desired results. In certain circumstances, multitasking and parallel processing can be advantageous.

Claims

1. A configurable log progress analysis method, comprising: Generate a first configuration file corresponding to the first deployment tool based on the use of the first deployment tool, wherein the first configuration file specifies one or more log matchers, and the one or more log matchers are used to search for one or more log files corresponding thereto through matching; Starting the one or more log matchers based on the first configuration file to extract the one or more log files corresponding thereto and determining the progress of each of the one or more log files; calculating a total progress of the first deployment tool based on the progress of each of the one or more log files; The method further comprises: generating a second configuration file corresponding to the second deployment tool based on the use of the second deployment tool, wherein the second configuration file specifies one or more log matchers, Among them, at least one log matcher among the one or more log matchers specified by the second configuration file is different from any log matcher among the one or more log matchers specified by the first configuration file.

2. The method according to claim 1, wherein: The first configuration file is generated by selecting one or more log matchers corresponding to the first deployment tool from a preset log matcher group.

3. The method according to claim 2, wherein: The log matcher set can be extended to include log matchers corresponding to new deployment tools.

4. The method according to claim 1, wherein: Each of the log matchers is associated with one or more line matchers, and each of the line matchers is used to search for a corresponding log line in a log file through matching.

5. The method according to claim 4, further comprising: One or more line matchers associated with the log matcher are initiated to determine the progress of the log file line by line.

6. The method according to claim 5, further comprising: Starting the one or more line matchers to search for corresponding log lines in the log file line by line; Determines the aggregation mode based on the configuration parameters in the configuration file; The progress of the corresponding log lines found are aggregated according to the determined aggregation mode to determine the progress of the log file.

7. The method according to claim 6, wherein: The configuration file also includes: the progress of the corresponding log line of the matcher and parameters for determining the aggregation mode.

8. The method according to claim 6, wherein: The summary mode at least includes: taking the maximum value among the progress of all rows, taking the minimum value among the progress of all rows, accumulating the progress values ​​of some or all rows, and taking the progress value of the last row searched.

9. A configurable log progress analysis device, comprising: a memory having instructions stored therein; as well as A processor configured to execute instructions stored on the memory to perform the method according to any one of claims 1 to 8.

10. A configurable log progress analysis system, comprising: a log matcher group, comprising one or more log matchers, the one or more log matchers being used to extract the one or more log files corresponding thereto and determine a progress of each of the one or more log files; as well as The log progress analysis device as claimed in claim 9.

11. A computer program product comprising computer executable instructions which, when executed by one or more processors, implement the method of any one of claims 1 to 8.

Images