Management device and management method

Abstract

To provide a technology for enabling appropriate re-learning of a machine learning model.SOLUTION: A management device includes operation influence information obtained by defining influences which a management operation to a management object exerts on the management object, a management operation log obtained by recording management operation executed to the management object, and a management operation schedule showing management operations planned or inferred to be executed to the management object, determines whether a difference between actually measured monitoring data being monitoring data acquired from the management object and predicted monitoring data being a result of inference processing for predicting monitoring data exceeds a prescribed threshold, determines whether a significant difference is temporary or continuous on the basis of the operation influence information, the management operation log and the management operation schedule with a difference exceeding the threshold as the significant difference, and determines that re-learning of a machine learning model should be executed when the significant difference is determined to be continuous.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present disclosure relates to a technique for operating and managing a system that utilizes machine learning. [Background technology]

[0002] In recent years, there has been a demand for mechanisms and services that continuously maintain or improve the quality of machine learning (ML) models through the operation of systems that utilize ML. A related technology is disclosed in Patent Document 1.

[0003] Patent Document 1 discloses a method for determining the presence or absence of concept drift based on the results of continuous evaluation of the accuracy of a model, and for re-learning the model when it is determined that concept drift has occurred. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] U.S. Patent Publication US20160371601A1 Summary of the Invention [Problem to be solved by the invention]

[0005] The method of re-learning an ML model disclosed in Patent Document 1 does not take into account the cause of concept drift, and therefore may not necessarily be an appropriate re-learning method. For example, there are possibilities that (1) unnecessary re-learning will result in unnecessary costs, (2) re-learning may actually decrease accuracy, and (3) the timing of re-learning may not be appropriate, resulting in unnecessary costs.

[0006] Here, the problem will be explained using the management of IT infrastructure using ML as a concrete example. For example, consider a configuration in which a data storage device has a storage area (hereinafter referred to as a pool), a storage area (hereinafter referred to as a volume) cut out from the pool is assigned to a computer, and the computer performs data I / O (Input / Output) to the volume. In this case, assume that there is an ML model that learns the past performance data of the pool (e.g., Busy Rate, etc.) using ML and predicts its future value. By applying the re-learning method of Patent Document 1 to this ML model, the model can be re-learned when quality degradation of the ML model is detected.

[0007] However, the method of re-learning an ML model disclosed in Patent Document 1 cannot distinguish whether concept drift is caused by a trend change in I / O from a computer or by management operations on volumes or pools.

[0008] For example, there may be a case where concept drift is detected because there is no change in I / O from a computer to a volume, but the performance load of the pool increases due to a data copy operation performed on the volume. If the data copy operation is a one-off operation performed manually, the increase in performance load is not continuous, so the ML model should not be retrained. However, the ML model retraining method disclosed in Patent Document 1 cannot distinguish between these two cases, so retraining is performed, resulting in the above-mentioned events (1) and (2).

[0009] The problem will be described using another example. Here, for example, assume that a management operation is performed to extract a second volume from the aforementioned pool and assign it to a second computer. As a result, I / O from the second computer will newly flow into the pool, and a case where concept drift is detected is considered. At this time, in the ML model re-learning method disclosed in Patent Document 1, the ML model is re-learned. On the other hand, when a deviation of a certain magnitude or more (i.e., performance abnormality) is recognized between the performance predicted value by the ML model and the performance measured value observed thereafter, a case where the IT management software or the IT administrator performs a management operation to eliminate the performance abnormality is considered. For example, in order to reduce the performance load of the pool and balance the performance load with other pools, an operation may be performed to migrate some of the volumes extracted from the pool to another pool. When the IT management software or the IT administrator performs this operation, the performance load of the pool decreases, and it is considered that the accuracy of the pool performance prediction by the re-learned ML model will decrease again. As a result, in the ML model re-learning method disclosed in Patent Document 1, the ML model is re-learned again, resulting in the above-mentioned event (3).

[0010] One objective of the present disclosure is to provide a technique that enables proper retraining of a machine learning model. [Means for solving the problem]

[0011] A management device according to one aspect included in the present disclosure includes a processor and a storage device, and is realized by the processor executing a software program stored in the storage device, and includes a machine learning model generation unit that generates a machine learning model for inferring monitoring data acquired from a managed object, an inference processing unit that performs inference processing using the machine learning model, a management unit that manages the managed object using the result of the inference processing, and a relearning necessity determination unit that determines whether or not relearning of the machine learning model is required. The management unit includes operation impact information that defines the impact of a management operation on the managed object on the managed object, a management operation log that records management operations performed on the managed object, and a management operation schedule that indicates management operations that are planned or predicted to be performed on the managed object, and determines whether or not a difference between actual monitoring data, which is monitoring data acquired from the managed object, and predicted monitoring data, which is a result of an inference processing that predicts the monitoring data by the inference processing unit, exceeds a predetermined threshold value, and the relearning necessity determination unit determines whether or not the difference exceeds the threshold value as a significant difference, and determines whether the significant difference is temporary or continuous based on the operation impact information, the management operation log, and the management operation schedule, and determines that the significant difference is continuous, and determines that relearning of the machine learning model should be performed. Effect of the Invention

[0012] One aspect of the present disclosure enables graceful retraining of machine learning models. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a block diagram showing a configuration of a computer system. [Diagram 2] FIG. 2 is a block diagram showing an example of the configuration of a management computer. [Diagram 3] FIG. 13 illustrates an example of a managed object table. [Figure 4] FIG. 13 illustrates an example of a configuration information table. [Diagram 5] FIG. 13 illustrates an example of a management operation table. [Figure 6] FIG. 13 is a diagram illustrating an example of an operation effect table. [Figure 7] 13 is a flowchart illustrating an example of a processing procedure of a monitoring process executed by an IT management program. [Figure 8] 5 is a flowchart showing an example of a procedure for a relearning necessity determination process executed by a relearning necessity determination program. [Figure 9] 10 is a flowchart showing an example of a procedure of a re-learning process executed by a learning program. [Figure 10] FIG. 13 is a conceptual diagram showing how re-learning data is corrected. [Figure 11] FIG. 13 is a diagram showing an example of a screen for displaying a determination result of whether or not re-learning is necessary. [Figure 12] FIG. 13 is a diagram showing another example of a screen for displaying the determination result of whether or not re-learning is necessary. [Figure 13] FIG. 13 is a diagram showing another example of a screen for displaying the determination result of whether or not re-learning is necessary. [Figure 14] FIG. 13 is a diagram showing another example of a screen for displaying the determination result of whether or not re-learning is necessary. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Several embodiments will be described below with reference to the drawings. Note that the embodiments described below do not limit the invention according to the claims, and not all of the elements and combinations thereof described in the embodiments are necessarily essential to the solution of the invention. In these drawings, the same reference numerals indicate the same components throughout the figures. Note that in the following description, the information of the present invention will be described using expressions such as "aaa table", but this information may be expressed in a data structure other than a table. Therefore, "aaa table" etc. may be referred to as "aaa information" to indicate that it does not depend on the data structure. Furthermore, when describing the contents of each piece of information, the expressions "identification information", "identifier", "name", and "ID" are used, but these are mutually interchangeable.

[0015] In the following explanation, the "program" may be used as the subject, but since a program is executed by a processor to perform a defined process using memory and communication ports (communication device, management I / F, data I / F), the explanation may also be made with the processor as the subject. Furthermore, the process disclosed with the program as the subject may be a process performed by a computer such as a server or an information processing device. Furthermore, some or all of the program may be realized by dedicated hardware. Furthermore, various programs may be installed on each computer by a program distribution server or a computer-readable storage medium.

[0016] Hereinafter, a collection of one or more computers that manage a computer system and display the display information of the present invention may be referred to as a management system. When the management computer displays the display information, the management computer is the management system. Also, the combination of the management computer and the display computer is also a management system. Also, to increase the speed and reliability of management processing, multiple computers may be used to achieve processing equivalent to that of a management computer, in which case the multiple computers (including the display computer if the display is performed by the display computer) are the management system.

[0017] <<Embodiment>>

[0018] Fig. 1 is a block diagram showing the configuration of a computer system in this embodiment. The computer system 100 is composed of a plurality of computing environments. Fig. 1 illustrates a computer system 100 composed of a computing environment 1000 and a computing environment 2000. Fig. 1 illustrates a configuration in which there are two computing environments 1000 to be managed, but this is not limiting. There may be one or more computing environments 1000. The computing environments 1000 and 2000 may be installed in different geographical locations.

[0019] The computing environment 1000 is configured to include a computer 3000, a storage device 4000, a data network 7000, and a management network 8000. The storage device 4000 may be a device equipped with dedicated hardware, or may be implemented as a program executed by a processor included in the computer 3000. The computer 3000 and the storage device 4000 may be provided in any form, such as a physical device, a virtual device, a container, or a managed service. Each function included in the computer 3000 and the storage device 4000 may be executed on a different device as a microservice for each function. The computers 3000, the storage devices 4000, and the computer 3000 and the storage device 4000 are connected to each other via a data network 7000. The computer 3000 and the storage device 4000 are connected to the management computer 5000 and the data store 6000 of the computing environment 2000 via a management network 8000. In the example of Fig. 1, the two computing environments 1000 have the same configuration, but this is not limited to this example. As another example, the two computing environments 1000 may each have a different configuration. Furthermore, the computing environment 1000 and the computing environment 2000 may be owned by different owners. In this case, the owner of the computing environment 2000 may allow the owner of the computing environment 1000 to use the computing resources and functions of the computing environment 2000 in a form such as a cloud service.

[0020] The computing environment 2000 is configured to include a management computer 5000, a data store 6000, and a management network 8000. The management computer 5000 and the data store 6000 may be provided in any form, such as a physical device, a virtual device, a container, or a managed service. Each function of the management computer 5000 and the data store 6000 may be executed on a different device as a microservice for each function. The management computer 5000 and the data store 6000 are connected to each other via the management network 8000.

[0021] The computing environment 1000 and the computing environment 2000 are connected to each other via a wide area network 9000. That is, the management network 8000 of the computing environment 1000 and the management network 8000 of the computing environment 2000 can communicate with each other via the wide area network 9000. Therefore, the management computer 5000 of the computing environment 2000 can manage the computer 3000 and the storage device 4000 of the computing environment 1000 via the management network 8000. In addition, the monitoring data (also called metrics data) of the computer 3000 and the storage device 4000 of the computing environment 1000 can be stored in the data store 6000 via the management network 8000. The metrics data includes, for example, performance data of the computer 3000 and the storage device 4000, capacity data of the computing resources, and operating status data, but is not limited thereto. In this embodiment, a program executed by a processor included in the computer 3000 and the storage device 4000 transmits the metrics data of the computer 3000 and the storage device 4000 to the data store 6000, thereby storing the metrics data in the data store 6000, but the present invention is not limited to this. For example, a program that collects metrics data from the computer 3000 and the storage device 4000 and transmits the data to the data store 6000 may be executed by a computer other than the computer 3000 and the storage device 4000 connected to the management network 8000 of the computing environment 1000. In addition, for example, the management computer 5000 may collect the metrics data of the computer 3000 and the storage device 4000 via the management network 8000 and store the data in the data store 6000.

[0022] The computer system 100 may have a configuration in which the computing environment 2000 does not exist, and the management computer 5000 and the data store 6000 exist inside the computing environment 1000. Furthermore, the computer system 100 may have a configuration in which the wide area network 9000 does not exist, and there is only one computing environment 1000, and the management computer 5000 and the data store 6000 exist inside the computing environment 1000. The wide area network 9000 may be, for example, the Internet or a dedicated line. The wide area network 9000 may also be a virtual private network.

[0023] 2 is a block diagram showing an example of the configuration of a management computer 5000 in this embodiment. The management computer 5000 is a computer for managing the computers 3000 and storage device 4000 of the computing environment 1000. The management computer 5000 comprises a processor 5100, a memory 5200, a storage device 5300, a management network interface 5400, and an I / O (Input / Output) device 5500. These components are connected to each other via a bus.

[0024] The management network interface 5400 is a network interface used for connection to the management network 8000 .

[0025] The storage device 5300 is configured with a hard disk drive (HDD), a solid state drive (SSD), etc. In this embodiment, the storage device 5300 stores learning data 5310, an ML model 5320, and inference result data 5330. In addition, the IT management program 5210, the learning program 5260, the inference program 5270, and the relearning necessity determination program 5280 are stored in the storage device 5300, and are read out onto the memory 5200 by the processor 5100 and executed.

[0026] The memory 5200 is configured, for example, from a semiconductor memory. In this embodiment, the memory 5200 stores an IT management program 5210, a learning program 5260, an inference program 5270, a relearning necessity determination program 5280, a management target table 5220, a configuration information table 5230, a management operation table 5240, and an operation effect table 5250. The management target table 5220, the configuration information table 5230, the management operation table 5240, and the operation effect table 5250 may be stored in the storage device 5300 to make the data permanent.

[0027] The learning program 5260 is a program that generates an ML model 5320 that predicts future values ​​of metrics data using learning data 5310 created from metrics data acquired from the computer 3000 and the storage device 4000 stored in the data store 6000. For example, the learning program 5260 can be implemented using a known algorithm such as Random Forest. However, this is not limited to this, and other algorithms may be used for the learning program 5260. Furthermore, prediction of future values ​​of metrics data is generally known as a regression problem, and the ML model generated by the learning program 5260 may be one that handles the regression problem. However, this is not limited to this, and the ML model may be one that handles another problem.

[0028] The inference program 5270 is a program that executes inference using the ML model 5320 and generates inference result data 5330. The inference here refers to prediction of future values ​​of the metrics data of the computer 3000 and the storage device 4000. In this embodiment, the inference program 5270 is executed periodically by the processor 5100. However, the inference program 5270 is not limited to this and may be executed at other timings.

[0029] The IT management program 5210 is a program that manages the computer 3000 and the storage device 4000 of the computing environment 1000. In this embodiment, the IT management program 5210 has a function of determining a performance abnormality of the computer 3000 or the storage device 4000 by comparing the inference result data 5330 generated by the inference program 5270 with the metrics data of the computer 3000 and the storage device 4000 stored in the data store 6000. The performance abnormality here means that the actual measured value shows a value different from the prediction. In this embodiment, the objects to be managed by the IT management program 5210 are the computer 3000 and the storage device 4000, but are not limited to this and other objects to be managed may be managed. Details of the IT management program 5210 will be described later.

[0030] The management target table 5220 is a table for storing information on the computers 3000 and storage devices 4000 that are managed by the IT management program 5210. The details of this table will be described later.

[0031] The configuration information table 5230 is a table for storing the configuration information of the computer 3000 and the storage device 4000. The details of this table will be described later.

[0032] The management operation table 5240 is a table for storing logs of management operations executed by the IT management program 5210 on the computers 3000 and storage apparatus 4000, and information on the schedule of management operations to be executed in the future.

[0033] The information stored in the management operation table 5240 is not limited to logs of management operations executed by the IT administrator using the IT management program 5210 and schedules of registered management operations. Other logs and schedules may be stored in the management operation table 5240. For example, when the IT management program 5210 has execution rules for management operations and executes a specific management operation triggered by changes in the metrics data of the computer 3000 and the storage device 4000 according to the execution rules, regardless of the operation of the IT administrator, the log and / or schedule of the management operation may be registered in the management operation table 5240. Details of this table will be described later.

[0034] The operation impact table 5250 is a table that stores information indicating what impact a management operation executed by the IT management program 5210 on the computer 3000 and the storage system 4000 has on the computer 3000 and the storage system 4000. The table will be described in detail later.

[0035] The re-learning necessity determination program 5280 is a program that determines whether or not it is necessary to re-learn the ML model 5320 when a performance abnormality in the ML model 5320 is detected by the IT management program 5210. The program will be described in detail later.

[0036] The storage device 5300 and the memory 5200 may also store general programs and tables for managing the computer 3000 and the storage device 4000. For example, the memory 5200 may store a table that holds user authentication information (user name, password, access authority, etc.).

[0037] 3 is a diagram showing an example of a management target table 5220 in this embodiment. The management target table 5220 is a table for storing information on the computers 3000 and storage devices 4000 managed by the IT management program 5210. The management target table 5220 is configured with a device ID column 5221, a component ID column 5222, and an ML model ID column 5223.

[0038] The device ID column 5221 stores device IDs which are identifiers assigned to the computer 3000 and the storage device 4000. The component ID column 5222 stores component IDs which are identifiers assigned to the components included in the computer 3000 and the storage device 4000. The ML model ID column 5223 stores ML model IDs which are identifiers assigned to the ML models.

[0039] In the example shown in FIG. 3, for example, the record indicated by 5220-1 indicates that the storage device 4000 indicated by the device ID "Storage-01" includes a component (here, a processor) indicated by the component ID "Processor-01", and that an ML model indicated by the ML model ID "Processor-Model-01" is used to predict future values ​​of metrics data of the processor. In this embodiment, one ML model identified by an ML model ID is assigned to a pair of a device ID and a component ID, but this is not limited to this. As another example, multiple ML models may be assigned to a pair of a device ID and a component ID. Also, the same ML model may be assigned to multiple pairs of a device ID and a component ID.

[0040] 4 is a diagram showing an example of a configuration information table 5230 in this embodiment. The configuration information table 5230 is a table that stores information indicating the configuration of the managed computers 3000 and storage apparatuses 4000. The configuration information table 5230 is configured to include a device ID column 5231, a pool ID column 5232, a volume ID column 5233, a processor ID column 5234, a cache ID column 5235, a port ID column 5236, a host ID column 5237, a copy destination volume ID column 5238, and a copy status 5239.

[0041] The device ID column 5231 stores device IDs which are identifiers assigned to the storage devices 4000. The pool ID column 5232 stores pool IDs which are identifiers assigned to the storage area pools included in the storage device 4000. The volume ID column 5233 stores volume IDs which are identifiers assigned to the volumes extracted from the storage area pools included in the storage device 4000. The processor ID column 5234 stores processor IDs which are identifiers assigned to the processors included in the storage device 4000. The cache ID column 5235 stores cache IDs which are identifiers assigned to the cache areas included in the storage device 4000. The port ID column 5236 stores port IDs which are identifiers assigned to the network ports for data I / O included in the storage device 4000. The host ID column 5237 stores host IDs which are identifiers assigned to the computers 3000 to which the volumes included in the storage device 4000 are assigned. The copy destination volume ID column 5238 stores a copy destination volume ID, which is an identifier given to each data copy destination volume of the volumes in the storage apparatus 4000 .

[0042] In this embodiment, the copy destination volume ID is in a format in which the device ID of the data copy destination and the volume ID are concatenated with a dot symbol. That is, when the device ID portion of the copy destination volume ID 5238 indicates the same storage device 4000 as the device ID 5231, it indicates that data copying within the same storage device, i.e., local copying, is performed. When the device ID portion of the copy destination volume ID 5238 indicates a storage device 4000 other than the device ID 5231, it indicates that data copying between storage devices, i.e., remote copying, is performed.

[0043] The copy status 5239 stores the data copy status of the volume in the storage apparatus 4000 .

[0044] In FIG. 4, two types of states, "split" and "sync", are illustrated as copy state 5239. Here, the "split" state indicates a state in which data copy from the volume indicated by volume ID 5233 to the volume indicated by destination volume ID 5238 is stopped. Therefore, when the copy state is "split", even if data is written from the computer 3000 indicated by host ID 5237 to the volume indicated by volume ID 5233, the data is not copied to the volume indicated by destination volume ID 5238. On the other hand, the "sync" state indicates a state in which data copy from the volume indicated by volume ID 5233 to the volume indicated by destination volume ID 5238 is continuing. Therefore, when the copy state is "sync", when data is written from the computer 3000 indicated by host ID 5237 to the volume indicated by volume ID 5233, the data is copied to the volume indicated by destination volume ID 5238. The copy status stored in the copy status 5239 is not limited to the exemplified “split” and “sync”, and other statuses may be stored in the copy status 5239 .

[0045] In response to data writing from the computer 3000 indicated by the host ID 5237, the data may be copied to the volume indicated by the destination volume ID 5238 synchronously or asynchronously. In the case of synchronous data copying, when data is written from the computer 3000 indicated by the host ID 5237 to the volume indicated by the volume ID 5233, the storage device 4000 responds to the computer 3000 indicated by the host ID 5237 that the writing has been completed after completing the process of copying the data to the volume indicated by the destination volume ID 5238. On the other hand, in the case of asynchronous data copying, when data is written from the computer 3000 indicated by the host ID 5237 to the volume indicated by the volume ID 5233, the storage device 4000 responds to the computer 3000 indicated by the host ID 5237 that the writing has been completed before performing the process of copying the data to the volume indicated by the destination volume ID 5238.

[0046] In the example shown in Figure 4, for example, the record indicated by 5230-1 indicates that the storage device 4000 indicated by the device ID "Storage-01" has a storage area pool indicated by the pool ID "Pool-01", a volume indicated by the volume ID "Vol-01" has been extracted from the pool, I / O processing for the volume is processed by the processor indicated by the processor ID "Processor-01", a cache area indicated by the cache ID "Cache-01" has been assigned to the volume, the volume has been assigned to the computer 3000 indicated by the host ID "Host-01" via the port indicated by the port ID "Port-01", data of the volume has been copied to the volume indicated by the destination volume ID "Storage-01.Vol-02", and the data copy status of the volume is "split".

[0047] 4, other information relating to the configuration of the computer 3000 and the storage device 4000 may be stored in the configuration information table 5230. The configuration information table 5230 may further store, for example, a copy type relating to a data copy of a volume, such as a full copy or a snapshot.

[0048] 5 is a diagram showing an example of a management operation table 5240 in this embodiment. The management operation table 5240 is a table that stores logs of management operations executed by the IT management program 5210 on the computer 3000 and the storage device 4000, and information on schedules of management operations that are to be executed in the future. The management operation logs may be referred to as management operation logs hereinafter. Furthermore, the management operation schedules may be referred to as management operation schedules. The management operation table 5240 is configured to include a management operation ID column 5241, a management operation column 5242, an operation target ID column 5243, an execution status column 5244, and an execution date and time column 5245.

[0049] The management operation ID column 5241 stores management operation IDs, which are identifiers assigned to the types of management operations. The management operation column 5242 stores the names of the management operations. The operation target ID column 5243 stores operation target IDs, which are identifiers assigned to the targets of the management operations. In this embodiment, the operation target ID is in a format in which a device ID and a component ID indicating the device and component that are the targets of the management operations are concatenated with a dot symbol. The execution status column 5244 stores the execution status of the management operations. The execution date and time column 5245 stores the execution date and time of the management operations.

[0050] In this embodiment, the execution status 5244 stores one of the states "Completed", "Scheduled", and "Expected". Here, the "Completed" state indicates that the management operation has been completed at the date and time indicated in the execution date and time 5245. Therefore, a record in which "Completed" is stored in the execution status column 5244 is a management operation log. The "Scheduled" state indicates that the management operation is scheduled to be executed at the date and time indicated in the execution date and time 5245. Therefore, a record in which "Scheduled" is stored in the execution status column 5244 is a management operation schedule. The "Expected" state indicates that the management operation is expected to be executed at the date and time indicated in the execution date and time 5245. This indicates that the schedule of the management operation is not explicitly registered by the IT administrator or the IT management program 5210, but that the management operation is expected to be executed at the date and time indicated in the execution date and time 5245 based on the execution period of past management operations. Records in the "Expected" state are generated by the IT management program 5210. The IT management program 5210 periodically monitors the management operation table 5240, groups management operations whose execution status has become "Completed" based on the management operation 5242 and operation target ID 5243 by the management operation name and operation target ID, determines whether there is periodicity in the execution dates and times 5245 of the grouped management operations, and if there is periodicity, infers that the management operation will be executed according to that periodicity and generates a record with an "Expected" status. In this embodiment, records with an execution status of "Expected" are treated as a type of management operation schedule, just like records with an execution status of "Scheduled".

[0051] 5, for example, in the record indicated by 5240-1, the management operation indicated by management operation ID "Task-01" is a management operation of copying data from the volume indicated by "Storage-01.Vol-01" to the volume indicated by "Storage-01.Vol-02", as indicated by management operation 5242 and operation target ID 5243. In addition, as indicated by execution status 5244 and execution date and time 5245, the execution of the management operation was completed at the date and time of "2021 / 02 / 01 00:00:00".

[0052] 6 is a diagram showing an example of an operation impact table 5250 in this embodiment. The operation impact table 5250 is a table for storing information relating to the impact that a management operation executed by the IT management program 5210 on the computer 3000 and the storage system 4000 has on the computer 3000 and the storage system 4000. The operation impact table 5250 is configured by including a management operation column 5251, a host I / O change column 5252, a host I / O processing load change column 5253, and an I / O occurrence column 5254.

[0053] The management operation column 5251 stores the name of the management operation. The host I / O change column 5252 stores information indicating whether the management operation changes the host I / O. The host I / O here refers to I / O from the computer 3000 to the volume. The change in host I / O here refers to a quantitative change in the host I / O, that is, an increase or decrease. The host I / O processing load change column 5253 stores information indicating whether the management operation changes, that is, increases or decreases, the processing load for processing the host I / O. Hereinafter, the processing load for processing the host I / O may be referred to as the host I / O processing load. The I / O occurrence column 5254 stores information indicating whether the management operation generates I / O.

[0054] In the example shown in Fig. 6, for example, the record indicated by 5250-1 indicates that the management operation of "data copy" does not change the host I / O, changes the host I / O processing load if the copy status is "sync", does not change the host I / O processing load if the copy status is not "sync", and the management operation itself generates I / O. Also, for example, the record indicated by 5250-7 indicates that the management operation of "volume allocation" changes the host I / O, does not change the host I / O processing load, and does not generate I / O.

[0055] In this embodiment, each piece of information stored in the operation impact table 5250 is defined in advance. However, this is not limiting, and each piece of information may be defined by other methods. For example, after the IT management program 5210 executes management operations on the computer 3000 and the storage device 4000, it may monitor changes in I / O of the computer 3000 and the storage device 4000, and record or update information in the operation impact table 5250 based on the observed changes.

[0056] Fig. 7 is a flowchart showing an example of the processing procedure of the monitoring process executed by the IT management program 5210. The IT management program 5210 monitors the computer 3000 and the storage device 4000 according to the procedure shown in Fig. 7. In particular, the IT management program 5210 determines a performance abnormality by comparing the inference result data 5330 generated by the inference program 5270 with the metrics data of the computer 3000 and the storage device 4000 stored in the data store 6000 according to the procedure shown in Fig. 7.

[0057] This process starts in step 10010. It should be noted that this process is started periodically by the IT management program 5210, but is not limited to this and may be started in other ways.

[0058] In step 10020, the IT management program 5210 refers to the management target table 5220 and acquires a list of management targets.

[0059] In step 10030, the IT management program 5210 obtains the metrics data of the management target from the data store 6000.

[0060] In step 10040, the IT management program 5210 refers to the inference result data 5330 and acquires the inference result for the managed object. In this embodiment, the inference result for the managed object is a future predicted value of the metrics data for the managed object.

[0061] In step 10050, the IT management program 5210 compares the metrics data of the managed object acquired in step 10030 with the inference result acquired in step 10040. Here, the relationship between the inference result acquired in step 10040 and the metrics data acquired in step 10030 is the relationship between the predicted value of the metrics data of the managed object and the correct answer (actual measured value) thereto.

[0062] In step 10060, the IT management program 5210 determines whether or not there is a deviation greater than a predetermined threshold between the metrics data of the managed object acquired in step 10030 and the inference result acquired in step 10040. If it is determined that there is a deviation greater than the threshold, the process transitions to step 10070. If it is not determined that there is a deviation greater than the threshold, the process transitions to step 10090. Here, the presence of a deviation greater than the threshold means that an actual measurement value that is significantly different from the predicted value of the metrics data has been observed, indicating the possibility of a performance anomaly. In addition, from the viewpoint of the accuracy of the ML model, the observation of an actual measurement value that is significantly different from the predicted value indicates that the accuracy of the ML model has deteriorated. Therefore, it indicates the possibility of concept drift occurring.

[0063] In this embodiment, the concept drift is detected based on the accuracy of the ML model, but the method of detecting the concept drift is not limited to this, and the concept drift may be detected using other methods. For example, the statistical feature amount of the metrics data used for learning the ML model may be compared with the statistical feature amount of the recent metrics data acquired in step 10030, and it may be determined that the concept drift has occurred if there is a deviation between these statistical feature amounts that is greater than a threshold value.

[0064] In step 10070, the IT management program 5210 plans management operations as measures to recover from the performance abnormality for the relevant managed object. In this embodiment, the IT management program 5210 has execution rules for management operations, and registers in the management operation table 5240 execution schedules for specific management operations to recover from the performance abnormality of the computer 3000 and the storage device 4000.

[0065] Here, the process of step 10030 and the process of step 10070 will be described using an example. The record indicated by 5230-7 in FIG. 4 indicates that the storage device 4000 indicated by "Storage-02" has a storage area pool indicated by "Pool-01", a volume indicated by "Vol-01" has been extracted from the pool, and the volume has been allocated to the computer 3000 indicated by "Host-10". Here, as shown in record 5240-4 in FIG. 5, it is assumed that a management operation has been performed to allocate the volume indicated by "Storage-02.Pool-01.Vol-02" to the computer 3000 indicated by "Host-11". As a result, as shown in record 5230-8 in FIG. 4, a volume indicated by "Vol-02" is extracted from the storage area pool indicated by "Pool-01" included in the storage device 4000 indicated by "Storage-02", and the volume is allocated to the computer 3000 indicated by "Host-11". Before this management operation was performed, only data I / O from the computer 3000 indicated as "Host-10" flowed into the memory pool indicated as "Pool-01" in the storage device 4000 indicated as "Storage-02." However, as a result of performing this management operation, data I / O from the computer 3000 indicated as "Host-11" also newly flows into the pool.

[0066] Assume that the IT management program 5210 executes the management process shown in FIG. 7. Assume also that the storage device 4000 indicated by "Storage-02" is the storage pool indicated by "Pool-01". As described above, in step 10060, the IT management program 5210 determines whether or not there is a deviation greater than a threshold between the metrics data of the storage pool indicated by "Pool-01" of the storage device 4000 indicated by "Storage-02" acquired in step 10030 and the inference result acquired in step 10040. Here, the inference result acquired in step 10040 is an inference result that predicts the future value of the metrics data of the pool by an ML model trained on past data in which only data I / O from the computer 3000 indicated by "Host-10" flowed into the pool. In contrast, the metrics data acquired in step 10030 is metrics data for the current state in which data I / O is flowing into the pool from the computer 3000 indicated by "Host-10" and the computer 3000 indicated by "Host-11." If sufficient data I / O is flowing into the pool from the computer 3000 indicated by "Host-11," the IT management program 5210 will determine that there is a deviation between the inference result and the metrics data that is greater than the threshold value, in other words, that there is a performance abnormality in the pool.

[0067] In response to this determination result, in step 10070 , the IT management program 5210 plans a management operation to be performed to recover from the performance abnormality of the pool, and registers the execution schedule of the management operation in the management operation table 5240 .

[0068] The record indicated by 5240-6 in FIG. 5 indicates a management operation registered by the IT management program 5210 to recover from a performance abnormality in the corresponding pool. This management operation is a management operation for migrating the volume indicated by "Storage-02.Pool-01.Vol-02" to the pool indicated by "Storage-02.Pool-02". By executing this management operation, the source of the storage area of ​​the volume indicated by "Storage-02.Pool-01.Vol-02" is changed from the pool indicated by "Storage-02.Pool-01" to the pool indicated by "Storage-02.Pool-02". As a result, only data I / O from the computer 3000 indicated by "Host-10" flows into the pool indicated by "Storage-02.Pool-01", and data I / O from the computer 3000 indicated by "Host-11" flows into the pool indicated by "Storage-02.Pool-02". This recovers the performance abnormality of the pool indicated by “Storage-02.Pool-01.” The above is an explanation of steps 10070 and 10080 using a specific example.

[0069] 7, in step 10080, the IT management program 5210 calls the relearning necessity determination process of the relearning necessity determination program 5280. The details of this process will be described later.

[0070] In step 10090, the IT management program 5210 determines whether or not there is an unchecked managed object in the list of managed objects acquired in step 10020. If it is determined that there is an unchecked managed object, the process transitions to step 10030. If it is not determined that there is an unchecked managed object, the process transitions to step 10100 and ends.

[0071] Fig. 8 is a flowchart showing the processing steps of the relearning necessity determination process executed by the relearning necessity determination program 5280. The relearning necessity determination program 5280 determines whether or not relearning of the ML model for monitoring a specific managed object is required, according to the steps shown in Fig. 8.

[0072] This process is started in step 20010. Note that this process is started by the IT management program 5210 calling this process in step 10080 when performing the monitoring process shown in Fig. 7, but it may be started in another manner. In this embodiment, the IT management program 5210 passes the identifier of the managed object determined in step 10060 to have a deviation larger than the threshold, such as an identifier such as "Storage-02.Pool-01", to the re-learning necessity determination program 5280 to call this process.

[0073] In step 20020, the relearning necessity determination program 5280 refers to the management operation table 5240, and acquires the management operation log and management operation schedule for the managed object corresponding to the identifier passed when the program was called. Specifically, the relearning necessity determination program 5280 extracts records including the identifier of the managed object in the operation target ID column 5243 from the management operation table 5240. Of the extracted records, records in which "Completed" is stored in the execution status column 5244 are management operation logs. Records in which "Scheduled" or "Expected" is stored in the execution status column 5244 are management operation schedules.

[0074] In step 20030, the relearning necessity determination program 5280 determines whether or not a management operation has been performed on the relevant managed object. Specifically, if one or more management operation logs are extracted in step 20020, it is determined that a management operation has been performed. If it is determined that a management operation has been performed, the process proceeds to step 20040. If it is not determined that a management operation has been performed, the process proceeds to step 20100.

[0075] In step 20040, the relearning necessity determination program 5280 refers to the operation impact table 5250 and acquires information on the impact caused by the relevant management operation. Specifically, the relearning necessity determination program 5280 performs a match determination between the management operation column 5242 of the management operation log extracted in step 20020 and the management operation column 5251 of the operation impact table 5250, and acquires information on the impact caused by the relevant management operation from the record in the operation impact table 5250 that has a match.

[0076] In step 20050, the re-learning necessity determination program 5280 references the information on the impact of the management operation acquired in step 20040, and determines whether the host I / O change column 5252 is "True". If it is determined that the host I / O change column 5252 is "True", the process transitions to step 20100. If it is not determined that the host I / O change column 5252 is "True", the process transitions to step 20060.

[0077] In step 20060, the re-learning necessity determination program 5280 references the information on the impact of the management operation acquired in step 20040, and determines whether the host I / O processing load change column 5253 is "True". If it is determined that the host I / O processing load change column 5253 is "True", the process transitions to step 20100. If it is not determined that the host I / O processing load change column 5253 is "True", the process transitions to step 20070.

[0078] In step 20070, the re-learning necessity determination program 5280 references the information on the impact of the management operation acquired in step 20040, and determines whether the I / O occurrence column 5254 is "True". If it is determined that the I / O occurrence column 5254 is "True", the process transitions to step 20080. If it is not determined that the I / O occurrence column 5254 is "True", the process transitions to step 20090.

[0079] In step 20080, the relearning necessity determination program 5280 determines whether or not the management operation is to be executed as a schedule. Specifically, if there is one or more schedules among the management operation schedules extracted in step 20020 that are the same management operation as the relevant management operation and have the same managed objects, it is determined that the management operation is to be executed as a schedule. If it is determined that the management operation is to be executed as a schedule, the process transitions to step 20100. If it is not determined that the management operation is to be executed as a schedule, the process transitions to step 20090.

[0080] In step 20090, the re-learning necessity determination program 5280 determines that re-learning of the ML model for monitoring the management target is not necessary. After execution of this step, the process transitions to step 20140 and ends.

[0081] In step 20100, the re-learning necessity determination program 5280 determines that re-learning of the ML model for monitoring the management target is necessary.

[0082] In step 20110, the re-learning necessity determination program 5280 determines whether or not a management operation is scheduled to be performed for the management target. Specifically, if there is one or more management operation schedules extracted in step 20020, it is determined that a management operation is scheduled to be performed. If it is determined that a management operation is scheduled to be performed, the process transitions to step 20120. If it is not determined that a management operation is scheduled to be performed, the process transitions to step 20130.

[0083] In step 20120, the re-learning necessity determination program 5280 schedules re-learning of the ML model for monitoring the managed object after the management operation that was scheduled to be performed on the managed object is completed.

[0084] In step 20130, the relearning necessity determination program 5280 calls the relearning process of the learning program 5260. The details of this process will be described later. After executing this step, the process transitions to step 20140 and ends.

[0085] 9 is a flowchart showing an example of a procedure for the re-learning process executed by the learning program 5260. The learning program 5260 performs re-learning of the ML model according to the procedure shown in FIG.

[0086] This process starts in step 30010. Note that this process starts when the relearning necessity determination program 5280 calls this process in step 20130 when performing the relearning necessity determination process shown in Fig. 8. However, this is not limiting, and this process may be started in another manner.

[0087] In this embodiment, the following are passed to the learning program 5260 when calling this process: the identifier of the ML model that the relearning necessity determination program 5280 determined in step 20100 to require relearning; the result of the determination in step 20030 as to whether or not a management operation was performed; information on whether or not relearning was scheduled to be performed after completion of the management operation in step 20120; and the time at which the IT management program 5210 determined in step 10060 of the monitoring process flow shown in Figure 7 that there is a deviation greater than the threshold value.

[0088] In step 30020, the learning program 5260 determines whether or not the relearning necessity determination program 5280 determined in step 20030 that a management operation was performed. If it is determined that a management operation was performed, the process transitions to step 30050. If it is not determined that a management operation was performed, the process transitions to step 30030.

[0089] In step 30030, the learning program 5260 selects, as re-learning data, the metrics data after the time at which the IT management program 5210 determined in step 10060 of the monitoring process flow shown in FIG. 7 that there is a deviation greater than the threshold.

[0090] In step 30040, the learning program 5260 executes re-learning of the ML model using the re-learning data selected in step 30030. After execution of this step, the process transitions to step 30130 and ends.

[0091] In step 30050, the learning program 5260 determines whether or not the relearning necessity determination program 5280 scheduled relearning to be executed after the completion of the management operation in step 20120 of the relearning necessity determination process shown in Fig. 8. If it is determined that relearning is scheduled to be executed after the completion of the management operation, the process transitions to step 30060. If it is not determined that relearning is scheduled to be executed after the completion of the management operation, the process transitions to step 30100.

[0092] In step 30060, the learning program 5260 waits for the scheduled management operation to complete execution.

[0093] In step 30070, the learning program 5260 compares the trend of the metrics data before the time when the IT management program 5210 determined that there was a deviation larger than the threshold in step 10060 of the monitoring process flow shown in Fig. 7 with the trend of the current metrics data. There are no particular limitations on the method for comparing the trends of the metrics data, and for example, the statistical features of both data may be compared.

[0094] In step 30080, the learning program 5260 determines whether or not there is a difference between the trends of the two pieces of data that is greater than the threshold value. If it is determined that there is a difference between the trends of the two pieces of data that is greater than the threshold value, the process transitions to step 30100. If it is not determined that there is a difference between the trends of the two pieces of data that is greater than the threshold value, the process transitions to step 30090.

[0095] In step 30090, the learning program 5260 determines that re-learning of the ML model is not necessary. This is processing that corresponds to a case where a management operation is performed on a managed object that is the monitoring target of the ML model, and as a result, the concept drift that occurred in the ML model is resolved, making re-learning unnecessary. The current trend of the metrics data has returned to a trend close to the trend of the metrics data before the time when the IT management program 5210 determined that there was a deviation larger than the threshold in step 10060 of the monitoring processing flow shown in FIG. 7. After executing this step, the processing transitions to step 30130 and ends.

[0096] In step 30100, the learning program 5260 selects the metrics data subsequent to the time when the management operation whose execution has been awaited to be completed in step 30060 was executed as the re-learning data.

[0097] In step 30110, the learning program 5260 determines whether or not another management operation determined not to require re-learning is included within the period of the relearning data. If it is determined that another management operation determined not to require re-learning is included within the period of the relearning data, the process transitions to step 30120. If it is not determined that another management operation determined not to require re-learning is included within the period of the relearning data, the process transitions to step 30040.

[0098] In step 30120, the learning program 5260 corrects the re-learning data for the execution period of the different management operation.

[0099] Here, the processing of step 30110 and the processing of step 30120 will be described using an example. Fig. 10 is a conceptual diagram showing the state of correction of re-learned data. In the computing environment 200 of Fig. 10, a volume indicated by "Vol-1" is extracted from a storage capacity pool indicated by "Pool-1" and assigned to a computer 3000 indicated by "Host-1". The ML model indicated by "ML model-1" is an ML model that predicts future values ​​of the performance of the pool.

[0100] Graph 210 shows an example of the performance (Busy Rate) of the pool. In addition to the actual measured value of the performance of the pool, graph 210 also shows the future value predicted by the ML model. The solid line graph shows the actual measured value when the predicted / actual difference is small, and the dashed line graph shows the actual measured value when the predicted / actual difference is large. The dotted line graph shows the predicted value when the predicted / actual difference is large.

[0101] Here, assume that at the time indicated by "t0", data copying is started from the volume indicated by "Vol-1" to another volume. Graph 210 shows how this operation caused the observed performance of the pool to deviate significantly from the predicted value, i.e., how concept drift occurred. Note that in this example, the data copying was performed on a one-off basis, and it is determined that the ML model indicated by "ML model-1" should not be re-learned at this time.

[0102] Next, at the time indicated by "t1", the volume indicated by "Vol-2" is extracted from the storage capacity pool indicated by "Pool-1" and assigned to the computer 3000 indicated by "Host-2". As a result, data I / O from the computer 3000 indicated by "Host-2" is continuously input to the storage capacity pool indicated by "Pool-1". This is also one of the factors that causes the predicted value and the actual measured value of the performance of the pool to diverge. Note that in this example, the volume assignment is an operation that changes the host I / O. Therefore, it is determined that the ML model indicated by "ML model-1" should be re-learned at this time.

[0103] Next, at the time indicated by "t2", the data copy that started at the time indicated by "t0" is completed.

[0104] As described above, the learning program 5260 determines whether or not the period of the relearning data includes another management operation that is determined not to require re-learning in step 30110 of the re-learning process shown in Fig. 9. In this example, the period of the relearning data is data from the time indicated by "t1" onwards. However, in this example, another management operation that is determined not to require re-learning, i.e., data copying, is performed between the time indicated by "t1" and the time indicated by "t2". Therefore, the learning program 5260 determines in step 30110 of the re-learning process that the period of the relearning data includes another management operation that is determined not to require re-learning.

[0105] Next, the learning program 5260 corrects the re-learned data for the execution period of the different management operation in step 30120. In this example, the re-learned data from the time indicated by "t1" to the time indicated by "t2" is the subject of correction.

[0106] Chart 220 in Fig. 10 shows an example of a method for correcting re-learned data. In chart 220, time 221 indicates each unit time between the time indicated by "t1" and the time indicated by "t2." Pool-1 Busy Rate Prediction / Actual Difference 222 indicates the difference between the predicted value and the actual value of the Busy Rate, which is the performance of the storage capacity pool indicated by "Pool-1," at each time.

[0107] Pool-1 IOPS (Host origin) 223 indicates the IOPS (Input Output Per Second) issued from the computer 3000 indicated by "Host-1" or "Host-2" out of the IOPS issued to the storage capacity pool indicated by "Pool-1". The average or maximum value of the IOPS issued per unit time may also be used here. Pool-1 IOPS (Copy origin) 224 indicates the IOPS issued by data copy out of the IOPS issued to the storage capacity pool indicated by "Pool-1".

[0108] Pool-1 Busy Rate (correction) 225 indicates the Busy Rate after correcting the Pool-1 Busy Rate difference 222. In this example, data correction is performed by multiplying the value of Pool-1 Busy Rate difference 222 by the ratio of IOPS indicated by Pool-1 IOPS (host origin) 223 to the total value of IOPS indicated by Pool-1 IOPS (host origin) 223 and Pool-1 IOPS (copy origin) 224. For example, in the record indicated by 220-1, the value of the Pool-1 Busy Rate difference at time "t1-1" is 10. This is multiplied by the ratio of the Pool-1 IOPS (host origin) value of 1000 to the total IOPS value of 2000, that is, 1000 / (1000+1000), to calculate the value of Pool-1 Busy Rate (correction) as 5. Note that the method of data correction is not limited to this, and correction may be performed by other methods. For example, the Busy Rate of the storage capacity pool may be estimated from the metrics data of the I / O issued from the host using a performance simulator of the storage device 4000, and the estimated value may be used as the correction data. The metrics data of the I / O issued from the host includes Read IOPS, Write IOPS, Read Transfer Rate, Write Transfer Rate, etc.

[0109] Fig. 11 is a diagram showing an example of a screen for displaying the determination result of whether or not relearning is necessary. Fig. 11 shows an example of a screen showing the determination result of whether or not relearning is necessary, which is displayed after the relearning necessity determination program 5280 executes the relearning necessity determination process shown in Fig. 8.

[0110] An example relearning necessity determination result display screen 5280A includes a managed object display field 5280A1, a performance graph display field 5280A2, an impact due to management operation display field 5280A3, a countermeasure display field 5280A4 for performance abnormalities, a relearning necessity determination result display field 5280A5, an OK button 5280A6, and a relearning execution button 5280A7.

[0111] The managed object display column 5280A1 displays the identifiers of the devices and components managed by the IT management program 5210.

[0112] The performance graph display field 5280A2 displays the performance value of the managed object as a graph. Here, the actual measured value of the performance value of the managed object and the predicted value calculated by the ML model assigned to the managed object are displayed. The data displayed here is the metrics data acquired in step 10030 and the inference result acquired in step 10040 when the IT management program 5210 performed the monitoring process shown in FIG. 7.

[0113] The management operation impact display field 5280A3 displays the management operation that caused the concept drift for the managed object, the impact of the management operation, and whether or not the management operation is scheduled. The information displayed here is the information on the impact of the management operation acquired in step 20040 when the relearning necessity determination program 5280 performed the relearning necessity determination process shown in Fig. 8, and the management operation log and management operation schedule acquired in step 20020.

[0114] The countermeasure display field 5280A4 for performance abnormality displays the management operation scheduled to be executed as a countermeasure for the performance abnormality that occurred in the managed object, and the scheduled execution date and time. The information displayed here is the countermeasure information planned in step 10070 when the IT management program 5210 performed the monitoring process shown in FIG.

[0115] The relearning necessity determination result display field 5280A5 displays an explanation of the determination result of whether relearning is necessary. The information displayed here is a prepared explanation selected according to which branch was passed in each of steps 20030, 20050, 20060, 20070, and 20080 when the relearning necessity determination program 5280 performed the relearning necessity determination process shown in FIG.

[0116] The OK button 5280A6 is a button for closing the display screen of the re-learning necessity determination result.

[0117] The re-learning execution button 5280A7 is a button for executing re-learning of the ML model. When the user presses this button, the learning program 5260 executes the re-learning process shown in Fig. 9. In this case, only steps 30030 and 30040 of the re-learning process may be executed. This allows the user to execute re-learning regardless of the determination result of the re-learning necessity determination program 5280.

[0118] In the example shown in FIG. 11, as shown in the management target display column 5280A1, for the storage capacity pool indicated by "Pool-01" included in the storage device 4000 indicated by "Storage-01", the judgment result of whether or not relearning of the ML model assigned to the pool is required is displayed. In the performance graph display column 5280A2, a state is displayed in which an actual measurement value having a difference larger than a threshold value is observed with respect to a predicted value of Busy Rate indicating the performance of the pool. In addition, in the impact display column 5280A3 due to management operation, a data copy operation for the volume indicated by "Vol-01" is displayed as the management operation that caused the performance abnormality indicated by the difference between the predicted and actual values. It is shown that there was an impact of "I / O occurrence" as a result. It is also shown that the data copy operation is not executed as scheduled. In this case, according to the re-learning necessity determination process shown in Fig. 8, the determination result in step 20030 is "YES", the determination result in step 20050 is "NO", the determination result in step 20060 is "NO", the determination result in step 20070 is "YES", and the determination result in step 20080 is "NO", and it is determined that re-learning is unnecessary. The performance abnormality countermeasure display field 5280A4 indicates that no management operation is planned as a countermeasure for this difference between the predicted and actual performance. The relearning necessity determination result display field 5280A5 indicates the determination result that the relearning necessity determination program 5280 has determined that this difference between the predicted and actual performance has occurred due to a temporary impact on performance caused by a management operation, and therefore re-learning of the ML model will not be performed.

[0119] Figures 12, 13, and 14 are diagrams showing other examples of the screen for displaying the result of the determination of whether or not relearning is necessary. The configuration of the display screen in Figures 12, 13, and 14 is the same as the display screen example 5280A shown in Figure 11. Here, the differences between Figures 12, 13, and 14 and Figure 11 will be mainly described.

[0120] In Fig. 12, the management operation impact display field 5280B3 indicates that the data copy operation is scheduled to be executed. In this case, in the relearning necessity determination process shown in Fig. 8, the determination result in step 20030 is "YES", the determination result in step 20050 is "NO", the determination result in step 20060 is "NO", the determination result in step 20070 is "YES", and the determination result in step 20080 is "YES", and it is determined that relearning is necessary. Therefore, the relearning necessity determination result display field 5280B5 indicates the determination result that the performance abnormality represented in the difference between expected and actual results occurs due to the performance impact of a management operation such as a data copy, and that the management operation is scheduled to be executed, and therefore the relearning necessity determination program 5280 has determined that the ML model should be re-learned.

[0121] 13, as shown in the management target display column 5280C1, for a storage capacity pool indicated by "Pool-01" included in the storage device 4000 indicated by "Storage-02", a judgment result of whether or not relearning of the ML model assigned to the pool is required is shown. The management operation impact display column 5280C3 shows that the cause of the performance abnormality indicated by the difference between expected and actual values ​​observed in the pool is the allocation of the volume indicated by "Vol-02" to the computer 3000 indicated by "Host-11", which has had an impact of "host I / O change". In this case, in the relearning necessity judgment process shown in FIG. 8, the judgment result in step 20030 is "YES", and the judgment result in step 20050 is "YES", and it is judged that relearning is required. The countermeasure display field 5280C4 for performance anomaly indicates that a management operation to migrate the source of the storage capacity of the volume indicated by "Vol-02" to the storage capacity pool indicated by "Pool-02" is scheduled to be executed at the time indicated by "2021 / 02 / 02 00:30:00" as a countermeasure to this difference between the expected and actual values. In this case, the judgment result of step 20110 in the re-learning necessity judgment process of FIG. 8 is "YES", and re-learning is scheduled to be executed after the completion of the management operation. Therefore, the re-learning necessity judgment result display field 5280C5 indicates the judgment result that the re-learning necessity judgment program 5280 has determined that the performance anomaly indicated in the expected and actual values ​​has occurred due to the performance impact caused by the volume allocation, which is a management operation, and that since volume migration is scheduled as a countermeasure to the performance anomaly, it has been determined that the ML model will be re-learned after the countermeasure is executed.

[0122] In FIG. 14, as in FIG. 13, as shown in the management object display column 5280D1, for the storage capacity pool indicated by "Pool-01" included in the storage device 4000 indicated by "Storage-02", the judgment result of whether or not relearning of the ML model assigned to the pool is required is shown. In the influence display column 5280D3 due to management operation, it is shown that there is no management operation that caused the performance abnormality indicated by the difference between the expected and actual performance observed in the pool. In this case, in the relearning necessity judgment process shown in FIG. 8, the judgment result of step 20030 is "NO", and it is judged that relearning is required. In the countermeasure display column 5280D4 for the performance abnormality, it is shown that a management operation to migrate the source of the storage capacity of the volume indicated by "Vol-02" to the storage capacity pool indicated by "Pool-02" is scheduled to be executed at the time indicated by "2021 / 02 / 02 00:30:00" as a countermeasure for the performance abnormality indicated by the difference between the expected and actual performance. 8, the determination result in step 20110 is "YES", and re-learning is scheduled to be executed after the management operation is completed. Therefore, the relearning necessity determination result display field 5280D5 shows the determination result that the relearning necessity determination program 5280 has determined that the performance anomaly shown in the difference between expected and actual occurred due to a change in I / O from the host computer 3000, and that since volume migration is scheduled as a measure to deal with the performance anomaly, it has determined that the ML model will be re-learned after the measure is taken.

[0123] As described above, in the computer system 100 of this embodiment, the management computer 5000 monitors the metrics data of the managed computers 3000 and storage device 4000, and judges whether or not there is a deviation larger than a threshold between the predicted future value of the metrics data calculated by the ML model and the actual measured value of the metrics data, i.e., concept drift. If the management computer 5000 judges that there is concept drift, it judges whether or not re-learning of the ML model is necessary based on a table that defines the influence of management operations on the managed objects, a log of management operations executed on the computers 3000 and the storage device 4000, and a schedule of management operations to be executed in the future. If it is judged that re-learning of the ML model is necessary, the management computer 5000 judges whether or not execution of a management operation is scheduled for the managed objects monitored using the ML model, and if execution of a management operation is scheduled, schedules the re-learning of the ML model after the completion of the management operation. When re-learning the ML model, the management computer 5000 selects appropriate re-learning data and executes the re-learning of the ML model.

[0124] Therefore, according to the management computer 5000 of this embodiment, it is possible to appropriately determine whether or not to re-learn the ML model based on the management operation that caused the concept drift, the impact that management operation has on the managed object, and the management operation that is scheduled to be executed on the managed object, thereby avoiding unnecessary re-learning of the ML model and realizing a method for optimizing the timing of performing the re-learning.

[0125] In this embodiment, an IT operations management system has been described as an example of a system that utilizes ML, but the application of the present invention is not limited to IT operations management systems and may be other application targets.

[0126] This embodiment also includes the following items. However, the items included in this embodiment are not limited to those listed below.

[0127] (Item 1) A management device having a processor and a storage device, the management device being realized by the processor executing a software program stored in the storage device, the management device comprising: a machine learning model generation unit that generates a machine learning model for inferring monitoring data acquired from a managed object; an inference processing unit that performs inference processing using the machine learning model; a management unit that manages the managed object using a result of the inference processing; and a re-learning necessity determination unit that determines whether re-learning of the machine learning model is necessary; The management unit operation impact information defining an impact on the managed object of a management operation performed on the managed object, a management operation log recording management operations performed on the managed object, and a management operation schedule indicating management operations planned or predicted to be performed on the managed object, determining whether a difference between actual monitoring data, which is monitoring data acquired from the managed object, and predicted monitoring data, which is a result of an inference process for predicting the monitoring data by the inference processing unit, exceeds a predetermined threshold; The relearning necessity determination unit If the difference exceeds the threshold, the difference is determined to be a significant difference, and based on the operation influence information, the management operation log, and the management operation schedule, it is determined whether the significant difference is temporary or continuous; If it is determined that the significant difference continues, it is determined that re-learning of the machine learning model should be performed. Management device. According to this, when a significant difference occurs between the actual monitoring data and the predicted monitoring data, it is determined whether the significant difference is due to a temporary management operation or a continuous difference, and if it is a continuous difference, it is determined that the machine learning model should be re-trained, thereby enabling appropriate re-training of the machine learning model.

[0128] (Item 2) Item 1, a management device according to item 1, the managed object is a computer system having one or more computers, the operation effect information includes information defining an effect on data input / output exchanged between the one or more computers; The relearning necessity determination unit determining whether a management operation performed on the managed object increases or decreases data input / output between the one or more computers; if it is determined that the management operation increases or decreases the data input / output, it is determined that the significant difference continues; Management device. According to this, the need for re-learning of the machine learning model is determined by taking into account the impact on data input / output due to management operations in the computer system, thereby enabling appropriate re-learning of the machine learning model for performing inference processing related to the computer system.

[0129] (Item 3) Item 1, a management device according to item 1, the managed object is a computer system having one or more computers, the operation impact information includes information on an impact on a processing load of data input / output exchanged between the one or more computers; The relearning necessity determination unit determining whether a management operation executed on the managed object changes a processing load on the one or more computers due to data input / output between the one or more computers; if it is determined that the management operation changes the processing load, it is determined that the significant difference continues. management device According to this, the need for re-learning of the machine learning model is determined by taking into account the impact on the processing load on the computer due to data input / output through management operations in the computer system, thereby enabling appropriate re-learning of the machine learning model for performing estimation processing related to the computer system.

[0130] (Item 4) Item 1, a management device according to item 1, the managed object is a computer system having one or more computers, the operation impact information includes information associating a type of management operation with whether or not the type of management operation generates data input / output to the one or more computers; The relearning necessity determination unit determining whether or not a management operation executed on the managed device is of a type that generates data input / output to the one or more computers based on the management operation log; If the management operation is of a type that generates the data input / output, determining whether or not the management operation is to be executed continuously based on the schedule; if it is determined that the management operation is to be continuously executed, it is determined that the significant difference is to be continued; Management device. According to this, the need for re-learning of the machine learning model is determined by taking into account the impact on the processing load on the computer due to data input / output through management operations in the computer system, thereby enabling appropriate re-learning of the machine learning model for performing estimation processing related to the computer system.

[0131] (Item 5) Item 1, a management device according to item 1, The relearning necessity determination unit When it is determined that re-learning of the machine learning model should be performed, determining whether or not execution of a management operation on the managed object is scheduled based on the management operation schedule; if execution of the management operation is scheduled, retraining of the machine learning model is not performed until execution of the management operation is completed; Management device. According to this, if a management operation is scheduled when the machine learning model is re-trained, the re-training will be performed after the management operation is completed, thereby reducing the impact of the management operation on the machine learning model after re-training.

[0132] (Item 6) Item 5. A management device according to item 5, The relearning necessity determination unit acquiring actual monitoring data from the managed object after the scheduled management operation is executed as post-management-operation actual monitoring data, determining whether a difference between the post-management-operation actual monitoring data and pre-significant-difference actual monitoring data, which is actual monitoring data acquired before the significant difference occurs, exceeds a predetermined threshold, and if the difference does not exceed the threshold, not executing re-learning of the machine learning model. Management device. According to this, if a management operation is scheduled when the machine learning model is re-learned, the re-learning will be performed after the management operation is completed, thereby suppressing unnecessary re-learning in cases where, for example, a significant difference is eliminated by a management operation to eliminate the significant difference.

[0133] (Item 7) Item 1, a management device according to item 1, The management unit When a management operation is performed on the managed object, monitoring changes that occur in the managed object after the management operation is completed; updating the operation effect information based on the change occurring in the managed object; Management device. This allows the operation effect information to be updated according to the actual situation.

[0134] (Item 8) Item 1, a management device according to item 1, The management unit Identifying a management operation that is periodically executed on the managed object as a periodic management operation based on the management operation log; predicting a future execution time of the periodic management operation based on the management operation log; registering the periodic management operation and the future execution time in the management operation schedule; Management device. According to this, since the management operations that are periodically executed are reflected in the management operation schedule, it becomes possible to more accurately determine whether or not a significant difference is continuous.

[0135] (Item 9) Item 1, a management device according to item 1, When the re-learning necessity determination unit determines that re-learning of the machine learning model should be performed, the management unit uses, among the monitoring data acquired from the managed object, monitoring data acquired after the time when the management operation that is the cause of the continued occurrence of the significant difference was completed, as learning data for re-learning the machine learning model. Management device. According to this, the monitoring data after the management operation that caused the significant difference when it was determined that the machine learning model should be re-trained is completed is used as the training data, making it possible to generate a machine learning model that reflects the new state after the significant difference occurred.

[0136] (Item 10) Item 1, a management device according to item 1, When the re-learning necessity determination unit determines that the machine learning model should be re-learned, if a second management operation different from the first management operation that is the cause of the continuing significant difference has been performed within a time period in which learning data to be used for re-learning the machine learning model was acquired, the re-learning necessity determination unit corrects the learning data to exclude an influence of the second management operation from the learning data and uses the corrected learning data for the re-learning. Management device. This allows the effects of other management operations to be excluded from the learning data, making it possible to generate a machine learning model that reflects continuous conditions.

[0137] (Item 11) Item 1, a management device according to item 1, and a display unit that displays information on a basis for determining that the significant difference will continue to occur based on the operation effect information, the management operation log, and the management operation schedule when the re-learning necessity determination unit determines that the machine learning model should be re-learned. Management device. According to this, since information on the basis for determining that re-learning is required is displayed, the basis for determining that re-learning is required can be easily known.

[0138] (Item 12) A management method executed by a computer having a processor and a storage device, the computer being realized by the processor executing a software program stored in the storage device, the computer being equipped with: a machine learning model generation unit that generates a machine learning model for inferring monitoring data acquired from a managed object; an inference processing unit that performs inference processing using the machine learning model; a management unit that manages the managed object using a result of the inference processing; and a relearning necessity determination unit that determines whether or not relearning of the machine learning model is necessary, The management unit, recording operation impact information that defines an impact on the managed object of a management operation performed on the managed object, a management operation log that records management operations performed on the managed object, and a management operation schedule that indicates management operations that are planned or expected to be performed on the managed object; determining whether a difference between actual monitoring data, which is monitoring data acquired from the managed object, and predicted monitoring data, which is a result of an inference process for predicting the monitoring data by the inference processing unit, exceeds a predetermined threshold; The relearning necessity determination unit, If the difference exceeds the threshold, the difference is determined to be a significant difference, and based on the operation influence information, the management operation log, and the management operation schedule, it is determined whether the significant difference is temporary or continuous; If it is determined that the significant difference continues, it is determined that re-learning of the machine learning model should be performed. Management method. [Industrial Applicability]

[0139] An apparatus and method according to one aspect of the present disclosure are suitable for application to a system operation management apparatus that continuously maintains and improves quality through the operation of a system that utilizes machine learning. [Explanation of symbols]

[0140] 100...computer system, 200...computing environment, 1000...computing environment, 2000...computing environment, 3000...computer, 4000...storage device, 5000...management computer, 5100...processor, 5200...memory, 5210...IT management program, 5220...management target table, 5230...configuration information table, 5240...management operation table, 5250...operation impact table, 5260...learning program, 5270...inference program, 5280...relearning necessity determination program, 5300...storage device, 5310...learning data, 5320...ML model, 5330...inference result data, 5400...management network interface, 5500...I / O device, 6000...data store, 7000...data network, 8000...management network, 9000...wide area network

Claims

1. A management device having a processor and a storage device, the management device being realized by the processor executing a software program stored in the storage device, the management device comprising: a machine learning model generation unit that generates a machine learning model for inferring monitoring data acquired from a managed object; an inference processing unit that performs inference processing using the machine learning model; a management unit that manages the managed object using a result of the inference processing; and a re-learning necessity determination unit that determines whether re-learning of the machine learning model is necessary; The management unit operation impact information defining an impact on the managed object of a management operation performed on the managed object, a management operation log recording management operations performed on the managed object, and a management operation schedule indicating management operations planned or predicted to be performed on the managed object, determining whether or not a difference between actual monitoring data, which is monitoring data acquired from the managed object, and predicted monitoring data, which is a result of an inference process for predicting the monitoring data by the inference processing unit, exceeds a predetermined threshold value; The relearning necessity determination unit If the difference exceeds the threshold, the difference is determined to be a significant difference, and based on the operation influence information, the management operation log, and the management operation schedule, it is determined whether the significant difference is temporary or continuous; If it is determined that the significant difference continues, it is determined that re-learning of the machine learning model should be performed. Management device.

2. The management device according to claim 1 , the managed object is a computer system having one or more computers, the operation effect information includes information defining an effect on data input / output exchanged between the one or more computers, The relearning necessity determination unit determining whether a management operation executed on the managed object increases or decreases data input / output between the one or more computers; if it is determined that the management operation increases or decreases the data input / output, it is determined that the significant difference continues; Management device.

3. The management device according to claim 1 , the managed object is a computer system having one or more computers, the operation impact information includes information on an impact on a processing load of data input / output exchanged between the one or more computers, The relearning necessity determination unit determining whether or not a management operation executed on the managed object changes a processing load on the one or more computers due to data input / output exchanged between the one or more computers; if it is determined that the management operation changes the processing load, it is determined that the significant difference continues. Management device.

4. The management device according to claim 1 , the managed object is a computer system having one or more computers, the operation impact information includes information associating a type of management operation with whether or not the type of management operation generates data input / output to the one or more computers; The relearning necessity determination unit determining whether or not a management operation executed on the managed device is of a type that generates data input / output to the one or more computers based on the management operation log; If the management operation is of a type that generates the data input / output, determining whether or not the management operation is to be executed continuously based on the schedule; if it is determined that the management operation is to be continuously executed, it is determined that the significant difference is to be continued; Management device.

5. The management device according to claim 1 , The relearning necessity determination unit When it is determined that re-learning of the machine learning model should be performed, determining whether or not execution of a management operation on the managed object is scheduled based on the management operation schedule; If the management operation is scheduled to be executed, retraining of the machine learning model is not executed until the execution of the management operation is completed. Management device.

6. The management device according to claim 5 , The relearning necessity determination unit acquiring actual monitoring data from the managed object after the scheduled management operation is executed as post-management-operation actual monitoring data, determining whether a difference between the post-management-operation actual monitoring data and pre-significant-difference actual monitoring data, which is actual monitoring data acquired before the significant difference occurs, exceeds a predetermined threshold, and if the difference does not exceed the threshold, not executing re-learning of the machine learning model. Management device.

7. The management device according to claim 1 , The management unit When a management operation is performed on the managed object, monitoring changes that occur in the managed object after the management operation is completed; updating the operation effect information based on the change occurring in the managed object; Management device.

8. The management device according to claim 1 , The management unit Identifying a management operation that is periodically executed on the managed object as a periodic management operation based on the management operation log; predicting a future execution time of the periodic management operation based on the management operation log; registering the periodic management operation and the future execution time in the management operation schedule; Management device.

9. The management device according to claim 1 , When the re-learning necessity determination unit determines that re-learning of the machine learning model should be performed, the management unit uses, among the monitoring data acquired from the managed object, monitoring data acquired after the time when the management operation that is the cause of the continued occurrence of the significant difference was completed, as learning data for re-learning the machine learning model. Management device.

10. The management device according to claim 1 , When the re-learning necessity determination unit determines that the machine learning model should be re-learned, if a second management operation different from the first management operation that is the cause of the continuing significant difference has been performed within a time period in which learning data to be used for re-learning the machine learning model was acquired, the re-learning necessity determination unit corrects the learning data to exclude an influence of the second management operation from the learning data and uses the corrected data for the re-learning. Management device.

11. The management device according to claim 1 , and a display unit that displays information on a basis for determining that the significant difference will continue to occur based on the operation effect information, the management operation log, and the management operation schedule when the re-learning necessity determination unit determines that the machine learning model should be re-learned. Management device.

12. A management method executed by a computer having a processor and a storage device, the computer being realized by the processor executing a software program stored in the storage device, the computer being equipped with: a machine learning model generation unit that generates a machine learning model for inferring monitoring data acquired from a managed object; an inference processing unit that performs inference processing using the machine learning model; a management unit that manages the managed object using a result of the inference processing; and a relearning necessity determination unit that determines whether or not relearning of the machine learning model is necessary, The management unit, recording operation impact information that defines an impact on the managed object of a management operation performed on the managed object, a management operation log that records management operations performed on the managed object, and a management operation schedule that indicates management operations that are planned or expected to be performed on the managed object; determining whether or not a difference between actual monitoring data, which is monitoring data acquired from the managed object, and predicted monitoring data, which is a result of an inference process for predicting the monitoring data by the inference processing unit, exceeds a predetermined threshold value; The relearning necessity determination unit, If the difference exceeds the threshold, the difference is determined to be a significant difference, and based on the operation influence information, the management operation log, and the management operation schedule, it is determined whether the significant difference is temporary or continuous; If it is determined that the significant difference continues, it is determined that re-learning of the machine learning model should be performed. Management method.

Images