Log extraction method and log extraction system

Abstract

To more appropriately perform mapping for identifiers across a wide variety of logs.SOLUTION: In a log extraction method, logs relating to requests of the same type are grasped and a same type request related identifier list for managing a correspondence relation between identifier keys for identifying identifiers included in the logs and identifier values representing values that the identifiers take in the logs is generated. Further, in the log extraction method, correlation degrees of the identifier values among the requests of different types are calculated based on the identifier values of the same identifiers associated with the requests of the different types in the same type request related identifier list. And, in the log extraction method, the identifier keys are set as nodes, a relation between the identifier keys and the identifiers is set as a link, and the correlation degrees are set as link metrics and a related identifier graph obtained by grouping the identifier keys based on the link metrics is generated.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a log extraction method and a log extraction system. [Background technology]

[0002] In an information processing system that runs microservices (subsystems) in multiple containers, a huge amount of logs are output as each microservice is executed in each container. For example, when identifying the cause of a failure in an information processing system or identifying users affected by the failure, a huge amount of logs spanning multiple microservices are analyzed. In this case, it is not realistic to cover all possible log combination patterns. Therefore, in order to narrow down the target logs in advance and perform log analysis efficiently, a technology has been disclosed that extracts highly relevant logs based on identifiers and context attributes (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] U.S. Patent No. 11,354,301 Summary of the Invention [Problem to be solved by the invention]

[0004] However, the above-mentioned conventional technology assumes that the names of identifiers and context attributes are known and fixed, and that the values ​​of the identifiers and context attributes match. Therefore, when there is a wide variety of logs, such as in an information processing system built by combining open software and existing applications, there is a problem in that it is difficult to associate related identifiers, even though the identifier names and values ​​are different. If the identifier association is not appropriate, it is impossible to extract, for example, highly related logs.

[0005] The present invention has been made in consideration of the above circumstances, and has as its object to more appropriately associate identifiers of a wide variety of logs. [Means for solving the problem]

[0006] In one aspect of the present invention, there is provided a log extraction method executed by a log extraction system that extracts logs output from an information processing system configured to include a plurality of subsystems, the log extraction system having a processor and a memory unit, the memory unit including an event log storage unit that stores an event log relating to an event that occurs in the information processing system in response to a request to the information processing system, a log storage unit that stores the logs output from the plurality of subsystems in response to the event, and a system configuration information storage unit that stores system configuration information that manages the correspondence between information that can identify the type of the request and information that can identify the subsystem, and the processor extracts logs of the same type of request based on the system configuration information and the event log. the log relating to the request of the same type is grasped, and based on the log relating to the request of the same type, a list of similar-request-related identifiers is generated that manages the correspondence between identifier keys that identify identifiers used in the information processing system and included in the log and identifier values ​​that represent the values ​​that the identifiers take in the log; based on the identifier values ​​of the same identifiers that are associated with the requests of the same type in the list of similar-request-related identifiers, a degree of correlation of the identifier values ​​between the requests of the same type is calculated; and, with the identifier keys as nodes, the relationships between the identifier keys and the identifiers as links, and the degrees of correlation as link metrics, a related identifier graph is generated in which the identifier keys are grouped based on the link metrics. [Effects of the Invention]

[0007] According to the present invention, it is possible to more appropriately associate identifiers of a wide variety of logs. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram showing the configuration of an entire system including a log extraction system according to a first embodiment. [Figure 2] FIG. 2 is a diagram showing the configuration of system configuration information according to the first embodiment. [Figure 3] 10 is a flowchart showing related identifier extraction processing according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing the configuration of a similar request related identifier list according to the first embodiment. [Figure 5] FIG. 4 is a diagram for explaining a method for calculating a correlation between different types of requests according to the first embodiment. [Figure 6] FIG. 2 is a diagram showing the configuration of an associated identifier graph according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing the configuration of an entire system including a log extraction system according to a second embodiment. [Figure 8] 10 is a flowchart showing related identifier extraction processing according to the second embodiment. [Figure 9] FIG. 10 is a diagram for explaining the process of extracting an identifier key / identifier value related to an identifier according to the second embodiment. [Figure 10] A diagram showing the hardware configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. Note that the following description and drawings are examples for explaining the present invention, and appropriate omissions and simplifications have been made for clarity of explanation. Furthermore, not all combinations of features described in the embodiments are necessarily essential to the solution of the invention. The present invention is not limited to the embodiments, and all application examples consistent with the concept of the present invention are included within the technical scope of the present invention. Those skilled in the art can make various additions and modifications to the present invention within the scope of the present invention. The present invention can also be implemented in various other forms. Unless otherwise specified, each component may be plural or singular.

[0010] In the following description, a "CPU (Central Processing Unit)" is an example of one or more processor devices. The at least one processor device is typically not limited to a CPU, but may be another type of processor device such as a GPU (Graphics Processing Unit). The at least one processor device may be a single-core or multi-core. The at least one processor device may be a processor core.

[0011] At least one processor device may be a circuit that is a collection of gate arrays written in a hardware description language that performs some or all of the processing. The circuit is a processor device in the broad sense, such as a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), or an application-specific integrated circuit (ASIC).

[0012] In the following description, a program is executed by a CPU to realize a processing function called "XXX unit" and become the executing entity of the process. The processing function may be realized by one or more computer programs being executed by a processor, or may be realized by one or more hardware circuits (e.g., FPGA or ASIC), or may be realized by a combination of these.

[0013] When a function is realized by executing a program by a processor, the specified processing is performed using a storage device and / or an interface device, etc., so the function may be considered to be at least a part of the processor. Processing described using a functional unit as the subject may also be processing performed by a processor or a device having that processor.

[0014] The program may be installed from a program source. The program source may be, for example, a program distribution computer or a computer-readable recording medium (e.g., a non-transitory recording medium). The description of each function is an example, and multiple functions may be combined into one function, or one function may be divided into multiple functions.

[0015] In the following explanation, various information may be explained in table format. A "YYY table" may be called "YYY information." The data format of the information may be a format other than a table format (for example, CSV (Comma Separated Values) format). Furthermore, various information may be stored in a memory unit as a table, or may be embedded as logic in a program.

[0016] In addition, in the following description, when describing elements of the same type without distinguishing between them, common reference symbols will be used, and when describing elements of the same type with distinction between them, reference symbols will be used.

[0017] [Embodiment 1] The first embodiment is premised on an environment (for example, a system test environment) in which the output periods of logs from subsystems for different requests do not overlap, that is, an environment in which the output timing of logs from subsystems for requests can be controlled.

[0018] In the first embodiment, logs related to similar requests are grouped based on the log output timing and API gateway logs. Then, the correlation between key items (hereinafter referred to as "identifier keys") of identifiers related to different types of requests is calculated based on the relationship between the item values ​​(hereinafter referred to as "identifier values") of the identifiers within each group. The identifier keys are then clustered based on the correlation between the identifier keys, and a related identifier graph is generated, which provides information about the identifier keys that should be focused on when analyzing the logs.

[0019] (Configuration of the overall system S including the log extraction system 1 according to the first embodiment) 1 is a diagram showing the configuration of an overall system S including a log extraction system 1 according to embodiment 1. The overall system S includes the log extraction system 1, an information processing system 2, and a parsing processing unit 4. The information processing system 2 includes a plurality of subsystems 2a and an API (Application Programming Interface) gateway 3. The overall system S also includes a terminal (not shown) that is used by a user to operate the log extraction system 1, the subsystems 2a, the API gateway 3, and the parsing processing unit 4.

[0020] The multiple subsystems 2a that make up the information processing system 2 are, for example, microservices and DB (Data Base) servers running in multiple containers. Each subsystem 2a outputs a log as it executes processing. The API gateway 3 receives requests from users, instructs the subsystem 2a, and receives a notification from the subsystem 2a that the request processing has been completed. The API gateway 3 outputs an event log that includes the time the request was received and the time the response was made.

[0021] The parsing processing unit 4 parses the general log output from the subsystem 2a and the event log output from the API gateway 3. By parsing the general log and the event log, the parsing processing unit 4 removes variable portions in which variable assignment values ​​are assigned to variables in the logs.

[0022] The log extraction system 1 has system configuration information 11a, a similar request related identifier list 11b, a related identifier graph 11c, a general log queue 12a, and an event log queue 12b. The log extraction system 1 also has a request related log extraction unit 13, a similar request identifier information calculation unit 14, a correlation degree calculation unit 15 between different request identifiers, and a related identifier graph generation unit 16.

[0023] As shown in FIG. 2, the system configuration information 11a has columns for "API path" and "subsystem ID." The "API path" indicates the path where the API that receives a request from a user and outputs the request to the subsystem 2a identified by the "subsystem ID" is stored. The system configuration information 11a manages the correspondence between information that can identify the type of request and information that can identify the subsystem 2a. The system configuration information 11a is stored, for example, in a system configuration information storage unit.

[0024] Returning to the explanation of Figure 1, the general log queue 12a is a queue configured on a storage device such as a memory, and is an example of a log storage unit that stores general logs output from the subsystem 2a and parsed by the parsing processing unit 4. The event log queue 12b is a queue configured on a storage device such as a memory, and is an example of an event log storage unit that stores event logs output from the API gateway 3 and parsed by the parsing processing unit 4.

[0025] The request-related log extractor 13 reads general logs from the general log queue 12a and event logs from the event log queue 12b, and identifies general logs related to the same type of request based on the system configuration information 11a and the event logs.

[0026] The similar request identifier information calculation unit 14 performs the following process based on the general log related to similar requests grasped by the request related log extraction unit 13. That is, it generates a similar request related identifier list 11b (FIG. 4) that manages the correspondence between identifier keys that identify identifiers used in the information processing system 2 and are included in the general log, and identifier values ​​that represent the values ​​that the identifiers take in the general log.

[0027] The correlation degree calculation unit 15 for different request identifiers calculates the correlation degree between identifiers of different types of requests based on the identifier values ​​of the same identifiers associated with different types of requests in the same type of request related identifier list 11b. The metrics (distance) of the link is the correlation degree between the identifiers.

[0028] The related identifier graph generation unit 16 generates a related identifier graph 11c (FIG. 6) in which identifiers are represented as nodes, relationships between identifiers are represented as links, and degrees of correlation are represented as link metrics, and the identifier keys are grouped based on the link metrics. In this embodiment, clustering such as the K-means method is used as the grouping method.

[0029] (Related Identifier Extraction Process According to the First Embodiment) FIG. 3 is a flowchart showing the related identifier extraction process according to the first embodiment.

[0030] First, in step S11, the request-related log extractor 13 identifies the components (subsystems 2a that are output sources of general logs) related to each API from the system configuration information 11a.

[0031] Next, in step S12, the request-related log extractor 13, based on the event log stored in the event log queue 12b, sets the time at which the request was received at the API gateway 3 as the start time tis and the time at which the request was responded to as the end time tie, and then calculates the request processing time timei=tie-tis for each request i, where i is a natural number representing the identification number of the request.

[0032] Now, reference will be made to Fig. 4. Fig. 4 is a diagram showing the configuration of similar request related identifier list 11b according to embodiment 1. In the example of Fig. 4, the request processing time timei is "time1:t1s to t1e" for request 11b1-1 and "time2:t2s to t2e" for request 11b1-2. Request related log extraction unit 13 stores request 11b1, which includes the request processing time timei, method (request type), and path (API path), in similar request related identifier list 11b.

[0033] Next, in step S13, the request-related log extraction unit 13 associates the general log extracted from the general log queue 12a with one of the requests 11b1 based on the method, path, whether the timestamp of the general log is included in the request processing time timei of the request 11b1, etc.

[0034] Whether two requests 11b1 are of the same type is determined based on information that can identify the request type, such as the method, path, and URL. In the example of FIG. 4, requests 11b1-1 and 11b1-2 are determined to be of the same type because they have the same "method:post" and "path: / api / v1 / userinfo / xx." If the request processing times timei related to requests of the same type overlap, it is not necessary to exclude associations with general logs in the overlapping time period. On the other hand, if the request processing times timei related to different requests overlap, associations with general logs in the overlapping time period are excluded to improve accuracy.

[0035] Next, in step S14, the similar request identifier information calculation unit 14 performs processing to save a node in the linked list based on the general log extracted in step S13. Specifically, in step S13, the similar request identifier information calculation unit 14 calculates a hash value of a character string obtained by removing the identifier and variable assignment values ​​(such as a timestamp) that differ for each individual log from each general log. The similar request identifier information calculation unit 14 converts the timestamp of the general log in which the character string has been converted into a hash value into a relative time starting from the time the original request was received. The similar request identifier information calculation unit 14 saves the general log with the relative time added as node 11b2 in the similar request related identifier list 11b (FIG. 4).

[0036] Next, in step S15, the similar request identifier information calculation unit 14 recognizes the hash value of node 11b2, which has the same hash value in similar requests 11b1-1 and 11b1-2, as an identifier key related to the similar requests, and recognizes all identifier values ​​corresponding to the identifier keys.The similar request identifier information calculation unit 14 then stores the recognized identifier values ​​as identifier information 11b3 in the similar request related identifier list 11b (FIG. 4).By steps S12 to S15, the similar request related identifier list 11b is completed.

[0037] In the example shown in Figure 4, the "identifiers" are "userID", "clientID", "clientIP", "realmID", and "serviced". The "identifier values" are "1" and "2" for "userID", and "1" and "2" for "clientID". The "identifier values" are "1", "2", "3", "4", and "5" for "clientIP". The "identifier keys" are "key:0x3is9ab", "key:0x1sc9ao", "key:0x8sc9qr", "key:0x08cbzp", "key:0x42s9ac", "key:0x2ec9n0", "key:0xs9qeer", and "key:0xzp08cb".

[0038] Next, in step S16, the different request identifier correlation calculation unit 15 refers to the same type of request related identifier list 11b (FIG. 4) and identifies the identifier key (node ​​11b2) / identifier value (identifier information 11b3) detected between different types of requests.

[0039] Next, in step S17, the correlation degree calculation unit 15 between different request identifiers calculates the ratio of the combination patterns of identifier values ​​that actually appeared between different requests to the combination patterns of all identifier values ​​(identifier information 11b3) related to the different requests identified in step S16, and saves the ratio as the correlation degree.

[0040] Step S17 will be described in detail with reference to Fig. 5. Fig. 5 is a diagram for explaining a method for calculating the degree of correlation between different types of requests according to the first embodiment. First, the different type of request correlation degree calculation unit 15 calculates the total number of combinations of the total number of unique identifier values ​​of the same identifier observed in general logs for different types of requests in general logs within a certain period of time in the past (first total number). In the example of Fig. 6, the total number of identifier values ​​of the identifier "userID" for requests of "Type 1" is "1", "2", "3", and "4", which is four, and the total number of identifier values ​​of the identifier "userID" for requests of "Type 2" is three, which is "2", "3", and "4", so (first total number) is the total number of values ​​of "userID" of "Type 1" × the total number of values ​​of "userID" of "Type 2" = 4 × 3 = 12.

[0041] Next, the different request identifier correlation calculation unit 15 counts the total number (second total) of combinations of identifier values ​​of the same identifier observed in general logs for different requests. In the example of Fig. 5, the combinations of identifier values ​​of the same identifier "userID" observed in general logs for different requests of "Type1" and "Type2" are "1,2,3,4" and "2,3,4", "1,2,3" and "3", "1,3" and "2", "1,3" and "4", and "1" and "3", and the second total is "5".

[0042] Next, the different request identifier correlation degree calculation unit 15 calculates the correlation degree (second total number) / (first total number). In the example of Fig. 5, (second total number) / (first total number)=5 / 12.

[0043] Next, in step S18, the related identifier graph generation unit 16 defines identifier keys as nodes, connections between identifiers as links, and correlations as link metrics.Then, the different request identifier correlation calculation unit 15 creates the related identifier graph 11c (FIG. 6) by grouping the identifier keys based on the link metrics.

[0044] (Effects of the First Embodiment) The effects of the first embodiment will be described with reference to FIG. 6. FIG. 6 is a diagram showing the configuration of an associated identifier graph 11c according to the first embodiment. As shown in FIG. 6, the associated identifier graph generation unit 16 groups identifiers and identifier keys that change simultaneously into the same group 11c1, thereby making it possible to determine which identifier to focus on depending on the type of failure that has occurred in the information processing system 2. For example, in the case of a node failure, a large number of errors may be generated that are biased toward a specific "nodeID," "serviced," or "serverID," whereas there may be no bias toward a specific "clientID" or "userID," and error logs may be detected for various "clientIDs" and "userIDs." In this case, effective failure analysis can be performed by focusing on "nodeID," "serviced," and "serverID" (by partitioning by these identifier keys) rather than by analyzing with a focus on "clientID" or "userID."

[0045] Furthermore, even in information processing systems where logs contain various identifiers and proper associations are unclear, the system can automatically and appropriately associate identifiers. For example, it can appropriately identify the cause of a failure and affected tenants in a cloud environment.

[0046] It also becomes possible to dynamically partition logs according to the type of error.

[0047] [Embodiment 2] Unlike the first embodiment, which is premised on an environment where the timing of outputting the log of the subsystem 2a in response to a request can be controlled (for example, a system test environment), the second embodiment assumes the time when an error log is received in a production environment.

[0048] (Configuration of the overall system SB including the log extraction system 1B according to the second embodiment) Fig. 7 is a diagram showing the configuration of an entire system SB including a log extraction system 1B according to embodiment 2. The log extraction system 1B has the configuration shown in Fig. 7 in addition to or in place of the log extraction system 1 according to embodiment 1. In the log extraction system 1B, the same components as those in the log extraction system 1 are given the same names and symbols.

[0049] The log extraction system 1B includes a similar request related identifier list 11b, a related identifier graph 11c, a general log queue 12a, an event log queue 12b, an error related identifier specifying unit 21, a related identifier specifying unit 22, a partition configuration unit 23, and one or more queues 24.

[0050] When an error is detected from a general log extracted from the general log queue 12a, the error-related identifier identifying unit 21 calculates the error distribution for each identifier value of each cluster in the related identifier graph 11c. Specifically, the error-related identifier identifying unit 21 refers to the related identifier graph 11c and identifies identifiers whose identifier values ​​have a bias towards errors.

[0051] The related identifier specifying unit 22 refers to the similar request related identifier list 11b and extracts the identifier key / identifier value related to the identifier specified by the error-related identifier specifying unit 21. This is because there is a high possibility of a failure occurring in the configuration requirement indicated by the identifier specified by the error-related identifier specifying unit 21 (a configuration requirement of the information processing system 2, such as a service if the identifier is "serviceID" or a node if the identifier is "nodeID").

[0052] The process of extracting an identifier key / identifier value associated with an identifier will be described with reference to Fig. 9. Fig. 9 is a diagram for explaining the process of extracting an identifier key / identifier value associated with an identifier according to the second embodiment.

[0053] 9, among general logs (general logs with the same identifier key) that are determined to be the same request as the request 11b11 in which an error occurred based on the related identifier graph 11c, pairs of identifier keys / identifier values ​​corresponding to general logs whose timestamps are included between the initial reception time t1s (ts_min) of the request 11b11 and the final response time tne (te_max) are extracted as being highly likely to be related to the error. In the example of FIG. 9, the pairs extracted are "key:0x3is9ab" and "userID:1" and "clientID:1", "key:0x1sc9ao" and "userID:1" and "clientID:1", "key:0x8sc9qr" and "userID:1" and "clientID:2", and "key:0x08cbzp" and "userID:1" and "clientID:2".

[0054] The partition configuration unit 23 generates a queue 24 for each identifier key extracted by the related identifier specification unit 22. Then, the related identifier specification unit 22 stores the general log extracted from the general log queue 12a by the related identifier specification unit 22 in the queue 24 for each identifier key generated by the partition configuration unit 23.

[0055] The partition configuration unit 23 generates a queue 24 for each identifier key each time a new identifier is specified. When all general logs are extracted from the generated queue 24 for each identifier key and the queue 24 becomes empty, the partition configuration unit 23 deletes the empty queue 24. The partition configuration unit 23 may delete the queue 24 when the time to life (Time To Life) set at the time of generation has elapsed. In this way, the partition configuration unit 23 dynamically configures the queue 24 for each identifier key.

[0056] (Related Identifier Extraction Process According to the Second Embodiment) 8 is a flowchart showing the related identifier extraction process according to embodiment 2. The related identifier extraction process is executed when an error log is received in the general log in the production environment.

[0057] First, in step S21, the error-related identifier identifying unit 21 identifies an identifier related to an error. That is, the error-related identifier identifying unit 21 refers to the related identifier graph 11c and identifies an identifier whose identifier value has a biased occurrence of errors. The identifier identified in step S21 is an example of a "specific identifier."

[0058] Next, in step S22, the related identifier specification unit 22 specifies the identifier key / identifier value related to the identifier specified in step S21, and extracts the corresponding general log from the general log queue 12a. Next, in step S23, the partition configuration unit 23 generates a queue 24 for each identifier key specified in step S22.

[0059] Next, in step S24, the related identifier specifying unit 22 stores the general log extracted from the general log queue 12a in the queue 24 for each identifier key generated by the partition configuring unit 23.

[0060] (Effects of the second embodiment) According to the second embodiment, by analyzing logs stored in a queue for each identifier key, i.e., logs partitioned by error type, the efficiency of log analysis when an error occurs can be improved, and the cause of the error and the scope of its impact can be identified quickly and accurately.

[0061] Furthermore, by dynamically configuring a queue for each identifier key, it is possible to avoid straining resources due to an increase in the storage area for storing logs, thereby promoting efficient resource utilization.

[0062] (Hardware configuration of computer 1000) 23 is a diagram showing an example of the hardware configuration of the computer 1000. The computer 1000 implements the log extraction systems 1 and 1B and the various parts of the information processing system 2 (FIGS. 1 and 7) by executing a predetermined program.

[0063] The computer 1000 comprises a processor 1001 including a CPU, a main memory device 1002, an auxiliary memory device 1003, a network interface 1004, an input device 1005, and an output device 1006, all of which are interconnected via an internal communication line 1007 such as a bus.

[0064] The processor 1001 controls the overall operation of the computer 1000. The main memory device 1002 is composed of, for example, a volatile semiconductor memory, and is used as a work memory for the processor 1001. The auxiliary memory device 1003 is composed of a large-capacity nonvolatile memory device such as a hard disk device, an SSD (Solid State Drive), or a flash memory, and is used to store various programs and data for a long period of time.

[0065] An executable program 1003a stored in the auxiliary storage device 1003 is loaded into the main storage device 1002 when the computer 1000 is started up or when needed, and is executed by the processor 1001. This realizes the log extraction systems 1 and 1B, input devices, output devices, various terminals, etc.

[0066] The executable program 1003a may be recorded on a non-transitory recording medium, read from the non-transitory recording medium by a media reading device, and loaded into the main memory device 1002. Alternatively, the executable program 1003a may be obtained from an external computer via a network and loaded into the main memory device 1002.

[0067] The auxiliary storage device 1003 stores an executable program 1003a that realizes the log extraction systems 1 and 1B.

[0068] The network interface 1004 is an interface device for connecting the computer 1000 to each network within the system or for communicating with other computers. The network interface 1004 is configured, for example, by a network interface card (NIC) for a wired local area network (LAN) or a wireless LAN.

[0069] The input device 1005 is composed of a keyboard, a pointing device such as a mouse, and the like, and is used by the user to input various instructions and information to the computer 1000. The output device 1006 is composed of a display device such as a liquid crystal display or an organic EL (Electro Luminescence) display, and an audio output device such as a speaker, and is used to present necessary information to the user when necessary.

[0070] Although the embodiments of the present disclosure have been described above in detail, the present disclosure is not limited to the above-described embodiments and can be modified in various ways without departing from the spirit of the present disclosure. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present disclosure is not necessarily limited to those having all of the described configurations. Furthermore, some of the configurations of the above-described embodiments can be added to, deleted from, or replaced with other configurations.

[0071] Furthermore, the above-described configurations, functional units, processing units, etc. may be partially or entirely implemented in hardware, for example, by designing them as integrated circuits. The above-described configurations, functions, etc. may also be implemented in software by a processor interpreting and executing a program that implements each function. Information such as the programs, tables, and files that implement each function can be stored in memory, storage devices such as HDDs and SSDs, or recording media such as IC cards, SD cards, and DVDs.

[0072] In addition, in the above-mentioned drawings, the control lines and information lines shown are those that are considered necessary for explanation, and do not necessarily show all the control lines and information lines that are actually implemented. For example, it may be considered that almost all components are actually connected to each other.

[0073] The above-described arrangement of the processing functions and data is merely an example, and the arrangement of the processing functions and data can be changed to an optimal arrangement in terms of the performance of the hardware and software, processing efficiency, communication efficiency, etc. [Explanation of symbols]

[0074] 1, 1B: log extraction system, 2: information processing system, 2a: subsystem, 3: API gateway, 4: parsing processing unit, 11a: system configuration information, 11b: list of similar request related identifiers, 11c: related identifier graph, 12a: general log queue, 12b: event log queue, 13: request related log extraction unit, 14: similar request identifier information calculation unit, 15: correlation degree calculation unit between different request identifiers, 16: related identifier graph generation unit, 21: error related identifier identification unit, 22: related identifier identification unit, 23: partition configuration unit, 24: queue, 1000: computer, 1001: processor, 1002: main memory device, S, SB: entire system.

Claims

1. A log extraction method executed by a log extraction system that extracts logs output from an information processing system configured to include a plurality of subsystems, comprising: The log extraction system includes a processor and a storage unit, The storage unit an event log storage unit that stores an event log relating to an event that occurs in the information processing system in response to a request to the information processing system; a log storage unit for storing the logs output from the plurality of subsystems in response to the events; a system configuration information storage unit that stores system configuration information that manages the correspondence between the information that can identify the type of the request and the information that can identify the subsystem, the processor: Based on the system configuration information and the event log, the logs related to the same type of request are identified; generating a list of similar request related identifiers based on the logs related to the requests of the same type, the list managing the correspondence between identifier keys for identifying identifiers used in the information processing system and included in the logs, and identifier values ​​representing values ​​taken by the identifiers in the logs; calculating a correlation degree of the identifier values ​​between the different types of requests based on the identifier values ​​of the same identifiers associated with the different types of requests in the same type of request related identifier list; A related identifier graph is generated in which the identifier keys are nodes, the relationships between the identifier keys and the identifiers are links, and the correlations are link metrics, and the identifier keys are grouped based on the link metrics. A log extraction method comprising the steps of:

2. The log extraction method according to claim 1, the processor: determining a processing time for each request from the event log, based on the reception time at which the request was received by the information processing system and the end time at which the processing of the request by the information processing system was completed; identifying the logs relating to the same type of request based on the system configuration information and the fact that the timestamp of the log corresponds to the processing time; excluding the identifier and variable portion from the log relating to the request of the same type, and calculating a hash value of the remaining portion of the log excluding the identifier and variable portion; The hash value is used as the identifier key, and a correspondence relationship between the identifier key and the identifier value is grasped; generating a list of similar request-related identifiers that manages the correspondence between the requests of the same type, the identifier keys, and the identifier values; calculating a first total number by multiplying the number of values ​​of the identifier value taken by each of the same identifiers associated with the different types of requests in the same type request related identifier list; determining a second total number, which is the total number of combination patterns of all the identifier values ​​for the same identifier associated with the different types of requests in the same type of request related identifier list; The correlation degree is calculated by dividing the second total by the first total. A log extraction method comprising the steps of:

3. The log extraction method according to claim 1, the processor: referring to the associated identifier graph, identifying the identifier associated with the identifier value in the log indicating that an error has occurred in the information processing system obtained from the log storage unit, and setting the identified identifier as a specific identifier; referring to the list of similar request related identifiers, and identifying a combination of the identifier based on the request of the same type as the specific identifier, and the identifier key and the identifier value corresponding to the identifier; The logs are grouped for each identifier key included in the identified combination of the identifier key and the identifier value. A log extraction method comprising the steps of:

4. The log extraction method according to claim 3, the processor: The logs grouped by the identifier key are stored in a queue for each identifier key. A log extraction method comprising:

5. The log extraction method according to claim 4, the processor: The queue for each identifier key is generated each time the specific identifier is newly identified. A log extraction method comprising the steps of:

6. The log extraction method according to claim 5, the processor: When all the logs are extracted from the queue for each identifier key, the queue is deleted. A log extraction method comprising the steps of:

7. The log extraction method according to claim 5, the processor: When the lifetime of the queue for each identifier key has elapsed, the queue is deleted. A log extraction method comprising the steps of:

8. A log extraction system that extracts logs output from an information processing system configured to include a plurality of subsystems, The log extraction system includes a processor and a storage unit, The storage unit an event log storage unit that stores an event log relating to an event that occurs in the information processing system in response to a request to the information processing system; a log storage unit for storing the logs output from the plurality of subsystems in response to the events; a system configuration information storage unit that stores system configuration information that manages the correspondence between the information that can identify the type of the request and the information that can identify the subsystem, The processor: Based on the system configuration information and the event log, the logs related to the same type of request are identified; generating a list of similar request related identifiers based on the logs related to the requests of the same type, the list managing the correspondence between identifier keys for identifying identifiers used in the information processing system and included in the logs, and identifier values ​​representing values ​​taken by the identifiers in the logs; calculating a correlation degree of the identifier values ​​between the different types of requests based on the identifier values ​​of the same identifiers associated with the different types of requests in the same type of request related identifier list; A related identifier graph is generated in which the identifier keys are nodes, the relationships between the identifier keys and the identifiers are links, and the correlations are link metrics, and the identifier keys are grouped based on the link metrics. A log extraction system comprising:

9. The log extraction system according to claim 8, The processor: determining a processing time for each request from the event log, based on the reception time at which the request was received by the information processing system and the end time at which the processing of the request by the information processing system was completed; identifying the logs relating to the same type of request based on the system configuration information and the fact that the timestamp of the log corresponds to the processing time; excluding the identifier and variable portion from the log relating to the request of the same type, and calculating a hash value of the remaining portion of the log excluding the identifier and variable portion; The hash value is used as the identifier key, and a correspondence relationship between the identifier key and the identifier value is grasped; generating a list of similar request-related identifiers that manages the correspondence between the requests of the same type, the identifier keys, and the identifier values; calculating a first total number by multiplying the number of values ​​of the identifier value taken by each of the same identifiers associated with the different types of requests in the same type request related identifier list; determining a second total number, which is the total number of combination patterns of all the identifier values ​​for the same identifier associated with the different types of requests in the same type of request related identifier list; The correlation degree is calculated by dividing the second total by the first total. A log extraction system comprising:

10. The log extraction system according to claim 8, The processor: referring to the associated identifier graph, identifying the identifier associated with the identifier value in the log indicating that an error has occurred in the information processing system obtained from the log storage unit, and setting the identified identifier as a specific identifier; referring to the list of similar request related identifiers, and identifying a combination of the identifier based on the request of the same type as the specific identifier, and the identifier key and the identifier value corresponding to the identifier; The logs are grouped for each identifier key included in the identified combination of the identifier key and the identifier value. A log extraction system comprising:

11. The log extraction system according to claim 10, The processor: The logs grouped by the identifier key are stored in a queue for each identifier key. A log extraction system comprising:

12. The log extraction system according to claim 11, The processor: The queue for each identifier key is generated each time the specific identifier is newly identified. A log extraction system comprising:

13. The log extraction system according to claim 12, The processor: When all the logs are extracted from the queue for each identifier key, the queue is deleted. A log extraction system comprising:

14. The log extraction system according to claim 12, The processor: When the lifetime of the queue for each identifier key has elapsed, the queue is deleted. A log extraction system comprising:

Images