Abnormality determination system, abnormality determination method, and program

Abstract

An index acquisition unit (101) of an abnormality determination system (1) acquires indexes usable for abnormality determination regarding occurrence of an abnormality in a service providing system (2) for providing a predetermined service. A schedule data acquisition unit (103) acquires schedule data regarding schedules of the service. An abnormality determination unit (102) executes abnormality determination based on the indexes and the schedule data.

Description

[Technical field]

[0001] The present disclosure relates to an abnormality determination system, an abnormality determination method, and a program. [Background technology]

[0002] Conventionally, there is known a technique for executing an abnormality judgment regarding the occurrence of an abnormality in a system. For example, Patent Document 1 describes a technique for judging whether a computer is operating abnormally based on the load on the computer in a service providing system that provides a specified service. In the technique of Patent Document 1, when it is judged that a computer is operating abnormally, an administrator is notified of the abnormality based on the response time, which is the time it takes for the computer to respond to a specified command. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2011-154483 A Summary of the Invention [Problem to be solved by the invention]

[0004] Patent Document 1 also describes that anomaly determination criteria are determined based on the tendency of the computer load. However, in a service using a computer, if there is some kind of schedule, the optimal anomaly determination criteria may change according to the schedule. In the technology of Patent Document 1, the anomaly determination criteria are determined according to real-time trends, so it is not possible to make the anomaly determination criteria flexible according to the schedule. This point is the same for other technologies other than the technology of Patent Document 1. For this reason, the conventional technology has not been able to sufficiently increase the flexibility of anomaly determination.

[0005] One objective of the present disclosure is to increase the flexibility of anomaly determination. [Means for solving the problem]

[0006] The abnormality determination system according to the present disclosure includes an index acquisition unit that acquires indexes that can be used in abnormality determination regarding the occurrence of an abnormality in a service providing system for providing a specified service, a schedule data acquisition unit that acquires schedule data regarding a schedule for the service, and an abnormality determination unit that performs the abnormality determination based on the indexes and the schedule data. Effect of the Invention

[0007] According to the present disclosure, the flexibility of anomaly determination is increased. [Brief description of the drawings]

[0008] [Figure 1] 1 is a diagram illustrating an example of an overall configuration of an abnormality determination system. [Diagram 2] FIG. 1 is a diagram illustrating an example of the concept of an abnormality determination system and a service providing system. [Diagram 3] FIG. 13 is a diagram illustrating an example of a performance index used in anomaly determination. [Figure 4] FIG. 2 is a diagram illustrating an example of functions realized by the abnormality determination system of the first embodiment. [Diagram 5] FIG. 2 is a diagram illustrating an example of a performance indicator database. [Figure 6] FIG. 2 is a diagram illustrating an example of a process executed by the abnormality determination system of the first embodiment. [Figure 7] FIG. 13 is a diagram showing an example of performance indicators during a campaign period and during a normal period. [Figure 8] FIG. 11 is a diagram illustrating an example of functions realized by the abnormality determination system according to the second embodiment. [Figure 9] FIG. 4 is a diagram illustrating an example of a schedule database. [Figure 10] FIG. 4 is a diagram illustrating an example of an abnormality determination criteria database. [Figure 11] FIG. 11 is a diagram illustrating an example of a process executed by the abnormality determination system according to the second embodiment. [Figure 12]FIG. 13 is a diagram illustrating an example of anomaly determination based on performance indicators across multiple services. [Figure 13] FIG. 13 is a diagram illustrating an example of anomaly determination based on a relationship between a performance indicator of a certain service and a performance indicator of another service. [Figure 14] FIG. 13 is a diagram showing an example of a function realized in modification example 1-4. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] [1. First embodiment] A first embodiment, which is an example of an embodiment of an abnormality determination system according to the present disclosure, will be described.

[0010] [1-1. Overall configuration of the anomaly detection system] FIG. 1 is a diagram showing an example of the overall configuration of an anomaly determination system. For example, the anomaly determination system 1 can be connected to a network N such as the Internet or a LAN. A service providing system 2 can also be connected to the network N. Other systems or terminals may also be connectable to the network N. FIG. 1 illustrates an example in which the anomaly determination system 1 and the service providing system 2 are connected via the network N, but the anomaly determination system 1 may be included in the service providing system 2. In this case, the administrator authority of the anomaly determination system 1 is included in the administrator authority of the service providing system 2.

[0011] For example, the abnormality determination system 1 includes an abnormality determination server 10 and an administrator terminal 20. The abnormality determination server 10 is a server computer. For example, the abnormality determination server 10 includes a control unit 11, a storage unit 12, and a communication unit 13. The control unit 11 includes at least one processor. The storage unit 12 includes a volatile memory such as a RAM, and a non-volatile memory such as a flash memory. The communication unit 13 includes at least one of a communication interface for wired communication and a communication interface for wireless communication.

[0012] The administrator terminal 20 is a computer of an administrator who manages the anomaly determination system 1. For example, the administrator terminal 20 is a personal computer, a tablet terminal, or a smartphone. For example, the administrator terminal 20 includes a control unit 21, a memory unit 22, a communication unit 23, an operation unit 24, and a display unit 25. The hardware configurations of the control unit 21, the memory unit 22, and the communication unit 23 are similar to those of the control unit 11, the memory unit 12, and the communication unit 13, respectively. The operation unit 24 is an input device such as a touch panel or a mouse. The display unit 25 is a display such as a liquid crystal or organic EL.

[0013] For example, the service providing system 2 includes a first server 30, a second server 40, a third server 50, and a common server 60. The hardware configurations of the first server 30, the second server 40, the third server 50, and the common server 60 are similar to those of the anomaly determination server 10. For example, the hardware configurations of the control units 31, 41, 51, and 61, the storage units 32, 42, 52, and 62, and the communication units 33, 43, 53, and 63 are similar to those of the control unit 11, the storage unit 12, and the communication unit 13, respectively. The service providing system 2 is interpreted as a tenant in a cloud infrastructure or cloud platform, and is configured by a plurality of offering services selected from an offering service group including a software configuration or hardware configuration of the anomaly determination system 1, etc.

[0014] The programs stored in the storage units 12, 22, 32, 42, 52, and 62 may be supplied via the network N. Each computer may include at least one of a reading unit (e.g., a memory card slot) that reads a computer-readable information storage medium and an input / output unit (e.g., a USB port) for inputting and outputting data to and from an external device. For example, a program stored in an information storage medium may be supplied via at least one of the reading unit and the input / output unit.

[0015] Furthermore, the abnormality determination system 1 only needs to include at least one computer, and is not limited to the example in Fig. 1. For example, the abnormality determination system 1 may include only the abnormality determination server 10, without including the administrator terminal 20. The abnormality determination system 1 may include only the administrator terminal 20, without including the abnormality determination server 10. The abnormality determination system 1 may include other computers in addition to the abnormality determination server 10 and the administrator terminal 20.

[0016] [1-2. Overview of the first embodiment] For example, the service providing system 2 provides at least one service to the user. In the first embodiment, three services, a first service, a second service, and a third service, are given as an example, but the service providing system 2 may provide one service, two services, or four or more services to the user. The first server 30 is a server computer for the first service. The second server 40 is a server computer for the second service. The third server 50 is a server computer for the third service.

[0017] In the first embodiment, an example is given in which the first service is an electronic commerce service, the second service is a payment service, and the third service is a financial service, but the service providing system 2 can provide any service to the user. The services provided to the user by the service providing system 2 are not limited to the examples in the first embodiment. For example, the service providing system 2 may provide to the user a communication service, a travel reservation service, a video distribution service, an insurance service, a ticket reservation service, or other services.

[0018] For example, points common to a plurality of services provided by the service providing system 2 may be available. Points acquired by using a certain service may be available not only for that service but also for other services. The service providing system 2 may provide a plurality of services that allow the user to circulate and use the points acquired. The plurality of services may be provided by the same business operator, or may be provided by different business operators.

[0019] For example, instead of multiple services being provided by one service providing system 2, multiple services may be provided by multiple service providing systems 2. Multiple service providing systems 2 may work together to provide one service. The relationship between the service providing system 2 and the services may be one-to-one, one-to-many, many-to-one, or many-to-many. The service providing system 2 only needs to include at least one computer, and is not limited to the example of FIG. 1. The service providing system 2 may include multiple computers. The number of service providing systems 2 may also be any number.

[0020] The anomaly determination system 1 executes anomaly determination in the service providing system 2. The anomaly determination is a determination regarding the occurrence of an anomaly in the service providing system 2. For example, determining whether an anomaly has occurred or not, or calculating an anomaly score indicating the degree of anomaly, corresponds to an anomaly determination. The anomaly score can be expressed as a numerical value, a character, or other symbol. For example, when the anomaly score is a numerical value, the higher the anomaly score, the higher the degree of anomaly. Conversely, the lower the anomaly score, the higher the degree of anomaly.

[0021] An abnormality in the service providing system 2 means an abnormality in at least one of the hardware and software included in the service providing system 2. An abnormality in the service providing system 2 may mean an abnormality in both the hardware and software, or an abnormality in the hardware only, or an abnormality in the software only. An abnormality can also be said to be an abnormal state. For example, the occurrence of a failure, or a state that does not amount to a failure but requires attention, corresponds to an abnormality.

[0022] For example, the abnormality determination system 1 subjects the hardware included in the service providing system 2 to an abnormality determination. The hardware subject to the abnormality determination may be of any type. For example, the hardware subject to the abnormality determination may be a server computer, a personal computer, a tablet, a smartphone, a network device, a network cable, an information storage medium, a battery, a power cord, a sensor, a camera, a card reader, a reader / writer, a display, a printer, other peripheral devices, or a combination of these. For example, an increase in the load of the hardware, a failure of the hardware, an unintended operation of the hardware, a failure of the hardware, an inability of the hardware to communicate with an external device, or an unintended power outage of the hardware correspond to an abnormality.

[0023] For example, the anomaly determination system 1 subjects software included in the service providing system 2 to anomaly determination. The software subject to anomaly determination may be of any type. For example, the software subject to anomaly determination may be an operating system, a program for executing a payment, a program for accepting product orders, a program for accepting facility reservations, a program for accepting ticket reservations, a program for accepting membership registration for a service, other programs executed to provide a service, a program for referencing a database, an API, or a combination of these. For example, software taking longer to operate than normal, software stopping operation, software not terminating, software having other errors, software not starting up, or database not being able to be updated are considered to be anomalies.

[0024] 2 is a diagram showing an example of the concept of the anomaly determination system 1 and the service providing system 2. In reality, each of a plurality of services is provided to a user by various hardware (including virtual hardware) and software included in the anomaly determination system 1 and the service providing system 2, but conceptually, various services are provided to a user by one cloud-based system. Therefore, conceptually, various services are provided to a user by one cloud system including the anomaly determination system 1 and the service providing system 2.

[0025] For example, the service providing system 2 includes an API of a first service realized by a first server 30, an API of a second service realized by a second server 40, and an API of a third service realized by a third server 50. In the first embodiment, the second service, which is a payment service, and the third service, which is a financial service, are the same in the sense of being FinTech services, so a common platform is used. The platform is at least one of hardware and software included in the common server 60.

[0026] For example, the first service, the second service, and the third service use a common API that is common to each other. For example, the common API may be an API for each of the first service, the second service, and the third service to use a common point. This API is realized by the common server 60. For example, the common service provided by the common API may be a service for each of the first service, the second service, and the third service to use a common point. The common service is provided by the common server 60. The service providing system 2 may include a configuration for providing other services, and may include various layers such as a physical layer for controlling communication of the service providing system 2.

[0027] For example, the service providing system 2 includes hardware such as a first server 30, a second server 40, a third server 50, and a common server 60. The hardware may be anything that is used to provide a service, and is not limited to server computers. For example, the hardware may include other hardware such as the network devices described above. The hardware is not limited to physical hardware, and may refer to a virtual configuration. For example, the hardware also includes a container or a virtual machine. These virtual configurations may be classified as software, rather than hardware.

[0028] A conventional technique is known that uses at least one of latency, errors, traffic, and saturation as an index used in determining an abnormality in the service providing system 2. Latency is the processing time (response time) of a request for a service. Errors are the number of errors returned in response to a request. Traffic is the amount of service used per unit time (e.g., the number of requests). Saturation is the degree of resource consumption in the service providing system 2. These four indexes are sometimes called golden signal metrics.

[0029] In the first embodiment, the anomaly determination system 1 acquires performance indicators related to the performance of the service in order to perform anomaly determination regarding the occurrence of an anomaly in the service providing system 2. The performance indicators are indicators for evaluating performance from a business perspective. The performance indicators are also called performance evaluation indicators or key performance indicators. For example, the performance indicators are the number of payments, the payment amount, the number of orders, the number of uses, the number of new users, the number of active users, the number of accesses, the number of logins, or a combination of these. As shown in FIG. 2, the anomaly determination system 1 can also acquire logs of the service and notify an administrator of an alert.

[0030] FIG. 3 is a diagram showing an example of a performance index used in anomaly determination. The vertical axis of FIG. 3 is an axis showing the performance index. The horizontal axis of FIG. 3 is an axis showing time. FIG. 3 shows an example in which the payment amount for the entire service per unit time corresponds to the performance index. In the first embodiment, the unit time is one minute, but the unit time may be any length that is determined in advance and is not limited to the example of the first embodiment. For example, the unit time may be less than one minute, several minutes to one hour, several hours to one day, or another length. The unit time may be specified by an administrator, or may differ depending on the service.

[0031] In the example of Fig. 3, the current time on the horizontal axis means the most recent one minute. The solid line indicates the time series change in the performance index of the first service. The dotted line indicates the time series change in the performance index of the second service. The dashed line indicates the time series change in the performance index of the third service. For example, the anomaly determination server 10 acquires the performance indexes of the first service, the second service, and the third service by communicating with each of the first server 30, the second server 40, and the third server 50 for each unit time.

[0032] In the upper example of FIG. 3, the performance index of the first service suddenly dropped shortly before the present time and has remained at a low level. Meanwhile, the performance index of the second service and the performance index of the third service remain stable and unchanged. In this case, since only the performance index of the first service has suddenly dropped, it may be difficult to accept payments using only the first service. For example, there may be no abnormality occurring in the second server 40 and the third server 50, and only the first server 30 may be abnormal.

[0033] In this case, as shown in the lower part of Fig. 3, an administrator screen SC indicating a possibility that an abnormality has occurred in the first service is displayed on the display unit 25 of the administrator terminal 20. The administrator screen SC displays the current performance indicators and normal performance indicators of each of the first service, the second service, and the third service. The normal performance indicator is a performance indicator that serves as a criterion for determining an abnormality. For example, the normal performance indicator may be the average value of the performance indicators for a certain period in the past, a performance indicator for the same period in the past, or a value manually determined by the administrator.

[0034] For example, if the difference between the current performance index and the normal performance index is equal to or greater than a threshold, it is determined that an abnormality has occurred. The normal performance index may be calculated from past performance, or may be specified by the administrator. In the example of FIG. 3, since only the current performance index of the first service has dropped sharply, a message indicating that an abnormality may have occurred in the first service is displayed on the administrator screen SC.

[0035] In the first embodiment, even if no abnormality is detected in known indices such as golden signal metrics, if an abnormality occurs in a performance index, some kind of notification is given to the administrator. For example, when the administrator selects button B1, the screen transitions to a screen for checking the status of the first server 30. On this screen, the operation of at least one of the hardware and software of the first server 30 can be confirmed. This screen itself may be a known maintenance tool. From this screen, the administrator analyzes the cause of the sudden drop in only the current performance index of the first service. The administrator checks the status of the first server 30, and performs maintenance on the first server 30 as necessary.

[0036] For example, when the difference between the performance index of the second service, the performance index of the third service, or the performance index common to all services and the normal performance index becomes equal to or greater than a threshold value, the administrator screen SC indicating that an abnormality has occurred is displayed on the display unit 25. The administrator selects one of the buttons B2 to B4 to check the status of the second server 40, the third server 50, or the common server 60. For example, in the case of known indices such as golden signal metrics, even if some abnormality has occurred, if no abnormality has occurred in the performance index, no message may be displayed, or a message may be displayed indicating that since no abnormality has occurred in the performance index, the response may be postponed.

[0037] As described above, in the first embodiment, anomaly determination is performed in the service providing system 2 based on the performance indicators of each of the first service, the second service, and the third service. This makes it possible to determine anomalies that cannot be identified using conventional technology, thereby improving the accuracy of anomaly determination. For example, anomalies that cannot be detected using golden signal metrics can be determined from a business perspective, thereby improving the accuracy of anomaly determination. Details of the first embodiment will be described below.

[0038] [1-3. Functions realized by the anomaly determination system of the first embodiment] FIG. 4 is a diagram showing an example of functions realized by the abnormality determination system 1 of the first embodiment.

[0039] [1-3-1. Functions realized by the anomaly judgment server] The abnormality determination server 10 includes a data storage unit 100, an index acquisition unit 101, and an abnormality determination unit 102. The data storage unit 100 is realized by the storage unit 12. The index acquisition unit 101 and the abnormality determination unit 102 are realized by the control unit 11.

[0040] [Data storage section] The data storage unit 100 stores data necessary for anomaly determination. For example, the data storage unit 100 stores a performance index database DB1.

[0041] Fig. 5 is a diagram showing an example of the performance indicator database DB1. The performance indicator database DB1 is a database in which performance indicators of services are stored. For example, the performance indicator database DB1 stores a service ID, an execution date and time of an abnormality determination, a performance indicator acquired at the execution date and time, and an execution result of the abnormality determination. Any information may be stored in the performance indicator database DB1. The information stored in the performance indicator database DB1 is not limited to the example of Fig. 5.

[0042] The service ID is an example of service identification information capable of identifying a service. The service identification information may be any information and is not limited to a service ID. For example, the service identification information may be the name of the service or the name of the company providing the service. The period is a period that is the subject of anomaly determination. In the first embodiment, a case in which the length of the period is one day will be described, but the period may be any length. For example, the length of the period may be about 10 seconds to 30 minutes, about one hour to half a day, about several days to one week, or other lengths.

[0043] The performance indicators are acquired by the performance indicator acquisition unit 101A described below. Specific examples of the performance indicators are as described above. In the first embodiment, since anomaly judgment is performed every minute, a performance indicator is acquired every minute and stored in the performance indicator database DB1. The result of the abnormality judgment is the result of the abnormality judgment performed by the abnormality judgment unit 102 described below. In the example of FIG. 5, the abnormality judgment result indicates either a first value meaning normality or a second value meaning abnormality. The abnormality judgment result may not be these two values, and may include an intermediate value. For example, the abnormality judgment result may be the abnormality score described above.

[0044] The data stored in the data storage unit 100 is not limited to the performance indicator database DB1. The data storage unit 100 is only required to store data necessary for abnormality determination, and can store any data. For example, the data storage unit 100 may store an abnormality determination criterion. In the first embodiment, an abnormality determination criterion that is a criterion to be compared with the performance indicator is set for each service. For example, the data storage unit 100 may store an abnormality determination criterion for a first service, an abnormality determination criterion for a second service, and an abnormality determination criterion for a third service. For example, the abnormality determination criterion includes a threshold value used in abnormality determination.

[0045] [Indicator acquisition section] The index acquisition unit 101 acquires indexes that can be used in anomaly judgment regarding the occurrence of anomalies in the service providing system 1 for providing a predetermined service. These indexes are information that are referenced when making an anomaly judgment. For example, information that is compared with an anomaly judgment criterion corresponds to the index. In the first embodiment, a performance index is described as an example of the index. The index acquisition unit 101 includes a performance index acquisition unit 101A. When no index other than the performance index is used, the index acquisition unit 101 that is a higher conceptualization of the performance index acquisition unit 101A does not need to exist.

[0046] The performance indicator acquiring unit 101A acquires a performance indicator relating to the performance of the service based on the usage status of the service providing system 2 for providing a predetermined service. The usage status is the degree to which the service is used. For example, the results of payment execution (e.g., payment amount, payee, payment date and time, and number of errors), the results of order acceptance (e.g., product price, store that accepted the order, and order date and time), the results of reservation acceptance (e.g., reserved facility, usage amount, and reservation date and time), the results of membership registration by new users, the access history to the service, and the login history to the service correspond to the usage status.

[0047] For example, in the case of a video distribution service, the number of viewers of the streaming distribution corresponds to the usage status. In the case of a service that holds some kind of event, the number of entries to the event corresponds to the usage status. In the case of a service that distributes coupons, the coupon acquisition rate corresponds to the usage status. In the case of a service that allows points to be used, the amount of points granted or the amount of points used corresponds to the usage status. In the case of a call center service, the amount of calls made to operators corresponds to the usage status. In the case of a fraud detection service that is executed when a user uses something, the amount of fraud detected corresponds to the usage status. In the case of a SNS, the number of posts corresponds to the usage status. For other services as well, usage status according to the service may be acquired in a similar manner.

[0048] The service providing system 1 of the first embodiment is a system for providing a plurality of services. Therefore, the performance indicator acquisition unit 101A acquires a performance indicator of each of the plurality of services. For example, the performance indicator acquisition unit 101A acquires a performance indicator of each of a first service, a second service, and a third service. For example, the performance indicator acquisition unit 101A acquires a usage status of the first service from the first server 30. The performance indicator acquisition unit 101A acquires a performance indicator of the first service based on the usage status of the first service.

[0049] For example, the performance indicator acquisition unit 101A acquires the usage status of the second service from the second server 40. The performance indicator acquisition unit 101A acquires the performance indicator of the second service based on the usage status of the second service. For example, the performance indicator acquisition unit 101A acquires the usage status of the third service from the third server 50. The performance indicator acquisition unit 101A acquires the performance indicator of the third service based on the usage status of the third service. In the first embodiment, it is assumed that the performance indicator acquisition unit 101A does not receive the usage status from the common server 60, but the performance indicator acquisition unit 101A may acquire the usage status of each of the first service, the second service, and the third service from the common server 60.

[0050] In the first embodiment, a case will be described in which the performance indicator acquisition unit 101A acquires the performance indicators of a service in operation in real time. "In operation" means that the service is being provided. For example, the first server 30, the second server 40, and the third server 50 are in operation when they communicate with a user terminal operated by a user or execute a process in response to a user request received from the user terminal. In other words, the usage status of each of the first service, the second service, and the third service being updated is equivalent to being in operation.

[0051] Real time means the current time or the most recent period. The most recent period is a period from a time a predetermined time before the current time to the current time. The most recent period can also be said to be the latest period among the periods in which performance indicators are stored in the performance indicator database DB1. In the first embodiment, since the unit time for abnormality determination is one minute, the performance indicator acquisition unit 101A acquires performance indicators every minute from one minute before the current time to the current time. For example, the performance indicator acquisition unit 101A acquires the latest usage status from each of the first server 30, the second server 40, and the third server 50 every minute, and acquires performance indicators of each of the first service, the second service, and the third service.

[0052] The performance indicator acquisition unit 101A may acquire performance indicators from some time in the past, not in real time. For example, the performance indicator acquisition unit 101A may acquire performance indicators from a period earlier than the most recent period. The performance indicator acquisition unit 101A may predict future performance indicators using a simulator created in advance, and acquire future performance indicators. For example, the simulator defines a relationship between a period to which the present time belongs and a future performance indicator based on past performance values ​​for that period. This relationship is defined in a formula, a table, a part of a program code, or a model using a machine learning technique.

[0053] In the first embodiment, the performance indicator acquisition unit 101A acquires a payment indicator related to payments in a service as a performance indicator. A payment indicator is an indicator that indicates the execution status of a payment. For example, a payment indicator is a payment amount, a total payment amount, an average payment amount, a number of payments, a payee, a payment date and time, or a combination of these. In the example of FIG. 3, the total payment amount per minute in the first service, the second service, and the third service corresponds to the payment indicator.

[0054] For example, the performance index acquisition unit 101A acquires the execution status of the payment for the first service from the first server 30. The performance index acquisition unit 101A acquires the payment index for the first service based on the execution status of the payment for the first service. In the example of FIG. 3, the performance index acquisition unit 101A acquires, for each minute, a payment index that is the total payment amount for the payments for the first service executed during that minute. The performance index acquisition unit 101A similarly acquires, for each minute, the payment index for that minute for the second service and the third service.

[0055] The performance indicator may not be the total value as described above, but may be another value such as an average value, a median value, or a mode value. These calculations may be performed on the first server 30, the second server 40, and the third server 50 side. In this case, the usage status and the performance indicator may be the same. The performance indicator acquisition unit 101A may acquire the usage status received from the first server 30, the second server 40, and the third server 50 as the performance indicator. Similarly, for performance indicators other than the payment amount, the performance indicator acquisition unit 101A may acquire the performance indicator by calculating the total value of the usage status, or may acquire the received usage status as it is as the performance indicator.

[0056] Also, each of the first server 30, the second server 40, and the third server 50 stores a usage status database regarding the usage status of the service provided by the server itself. When a certain user uses a service (for example, when a payment is made), each of the first server 30, the second server 40, and the third server 50 stores in the usage status database a usage status indicating that the user has used the service. In response to a request from the anomaly determination server 10, each of the first server 30, the second server 40, and the third server 50 transmits usage status for an arbitrary period (for the last one minute in the first embodiment) from among the usage status stored in the usage status database to the anomaly determination server 10. The performance index acquisition unit 101A may acquire a performance index based on this usage status.

[0057] [Abnormality determination section] The anomaly determination unit 102 executes anomaly determination in the service providing system 1 based on the index acquired by the index acquisition unit 101. In the first embodiment, a performance index is used as this index, so that the anomaly determination unit 102 executes anomaly determination in the service providing system 1 based on the performance index acquired by the performance index acquisition unit 101A. In the first embodiment, an anomaly determination using an anomaly determination criterion is given as an example, but the method of executing the anomaly determination is not limited to the example of the first embodiment. As in a modified example described later, a machine learning technique may be used instead of the anomaly determination criterion.

[0058] For example, the abnormality determination unit 102 executes the abnormality determination by determining whether or not the performance index satisfies a predetermined abnormality determination criterion. The abnormality determination criterion is a criterion defined for abnormality determination. The abnormality determination criterion is defined so that an abnormality occurs when the abnormality determination criterion is satisfied. For example, the abnormality determination criterion includes at least one threshold. In the first embodiment, a case is exemplified in which the performance index falling below the threshold corresponds to the occurrence of an abnormality, but the performance index being equal to or greater than the threshold may also correspond to the occurrence of an abnormality. Alternatively, for example, the abnormality determination may be executed by determining whether or not the amount of change in the performance index is equal to or greater than the threshold, rather than comparing the performance index with the threshold.

[0059] In the first embodiment, the abnormality determination unit 102 performs an abnormality determination for at least one of the multiple services based on the performance indicators of each of the multiple services. For example, the abnormality determination unit 102 performs an abnormality determination for a first service based on the performance indicator of the first service. The abnormality determination unit 102 performs an abnormality determination for a second service based on the performance indicator of the second service. The abnormality determination unit 102 performs an abnormality determination for a third service based on the performance indicator of the third service.

[0060] In the first embodiment, a case will be described in which the service from which the performance indicator is obtained is the same as the service that is the target of anomaly judgment, but they may be different. For example, the abnormality judgment unit 102 may perform an abnormality judgment for the second service or the third service based on the performance indicator of the first service. The abnormality judgment unit 102 performs an abnormality judgment for the first service or the third service based on the performance indicator of the second service. The abnormality judgment unit 102 performs an abnormality judgment for the first service or the second service based on the performance indicator of the third service.

[0061] As another example, the abnormality determination unit 102 may execute an abnormality determination for the first service based on a performance indicator of the first service and a performance indicator of at least one of the second service and the third service. The abnormality determination unit 102 may execute an abnormality determination for the second service based on a performance indicator of the second service and a performance indicator of at least one of the first service and the third service. The abnormality determination unit 102 may execute an abnormality determination for the first service based on a performance indicator of the third service and a performance indicator of at least one of the first service and the second service.

[0062] In the first embodiment, the anomaly determination unit 102 performs anomaly determination in real time based on performance indicators acquired in real time. The meaning of "real time" is as described above. The anomaly determination unit 102 may perform anomaly determination for some time in the past, rather than in real time. For example, the anomaly determination unit 102 may perform anomaly determination for a period earlier than the most recent period. Conversely, the anomaly determination unit 102 may perform future anomaly determination based on future performance indicators acquired by a simulator.

[0063] In the first embodiment, since the settlement index corresponds to the performance index, the abnormality determination unit 102 executes the abnormality determination based on the settlement index. For example, the abnormality determination unit 102 executes the abnormality determination by determining whether or not the settlement index satisfies the abnormality determination criterion. The abnormality determination unit 102 executes the abnormality determination by determining whether or not a predetermined abnormality determination criterion is satisfied based on the settlement index. For example, the abnormality determination unit 102 outputs a determination result indicating that an abnormality has occurred when it is determined that the abnormality determination criterion is satisfied. Conversely, the abnormality determination unit 102 may output a determination result indicating that an abnormality has occurred when it is determined that the abnormality determination criterion is not satisfied. Which process is executed depends on how the abnormality determination criterion is defined.

[0064] In the example of FIG. 3, the abnormality determination unit 102 determines every minute whether the payment indicator of the first service during that minute satisfies the abnormality determination criterion defined for the first service. For example, if the payment indicator is equal to or greater than the threshold indicated by the abnormality determination criterion, the abnormality determination unit 102 outputs a determination result indicating that no abnormality has occurred. If the payment indicator is less than the threshold indicated by the abnormality determination criterion, the abnormality determination unit 102 outputs a determination result indicating that an abnormality has occurred. The abnormality determination unit 102 performs abnormality determination in the same manner for the second service and the third service.

[0065] The abnormality determination unit 102 may output a determination result indicating that no abnormality has occurred if the payment indicator of the first service for each minute is less than a threshold indicated by the abnormality determination criterion defined for the first service. The abnormality determination unit 102 may output a determination result indicating that an abnormality has occurred if the payment indicator is equal to or greater than the threshold indicated by the abnormality determination criterion. The abnormality determination unit 102 may perform abnormality determination for the second service and the third service in the same manner.

[0066] [1-3-2. Functions realized on the administrator terminal] For example, the administrator terminal 20 includes a data storage unit 200, an operation reception unit 201, and a display control unit 202. The data storage unit 200 is realized mainly by the storage unit 22. The operation reception unit 201 and the display control unit 202 are realized mainly by the control unit 21.

[0067] [Data storage section] The data storage unit 200 stores data necessary for the administrator's work. For example, the data storage unit 200 stores data necessary for displaying an administrator screen SC. The data storage unit 200 may store a maintenance tool necessary for the administrator's work. The maintenance tool itself may be a known tool, for example, a tool capable of monitoring the status of at least one of hardware and software.

[0068] [Operation reception section] The operation reception unit 201 receives various operations by an administrator. For example, the operation reception unit 201 receives operations on an administrator screen SC.

[0069] [Display control section] The display control unit 202 causes various screens to be displayed on the display unit 25. For example, the display control unit 202 causes an administrator screen SC to be displayed on the display unit 25. When the administrator selects any of the buttons B1 to B4, the display control unit 202 causes the display unit 25 to display the status of any of the first server 30, the second server 40, the third server 50, and the common server 60.

[0070] [1-4. Processing Executed in the Anomaly Determination System of the First Embodiment] Fig. 6 is a diagram showing an example of processing executed by the abnormality determination system 1 of the first embodiment. The processing of Fig. 6 is executed by the control units 11 and 21 executing programs stored in the storage units 12 and 22, respectively. As shown in Fig. 6, the abnormality determination server 10 judges whether or not one minute, which is a unit time, has elapsed (S100). In S100, the abnormality determination server 10 judges whether or not one minute has elapsed since the previous abnormality determination. If it is not judged that one minute has elapsed (S100:N), the process returns to S100 again.

[0071] If it is determined in S100 that one minute has passed (S100:Y), the anomaly determination server 10 acquires the usage status of each of the first service, the second service, and the third service from each of the first server 30, the second server 40, and the third server 50 (S101). The usage status acquired in S101 includes the payment amounts of all payments made in the last minute. Each of the first server 30, the second server 40, and the third server 50 may calculate the total amount of the payment and transmit it to the anomaly determination server 10 as the usage status.

[0072] The anomaly determination server 10 acquires the performance index of each of the first service, the second service, and the third service based on the usage status of each of the first service, the second service, and the third service (S102). In S102, the anomaly determination server 10 calculates the total amount of payments made in the most recent minute for each service, and acquires it as the performance index of the service.

[0073] The abnormality determination server 10 executes an abnormality determination for each of the first service, the second service, and the third service based on the performance indicators of each of the first service, the second service, and the third service (S103). In S103, the abnormality determination server 10 determines whether or not an abnormality determination criterion associated with the first service is satisfied based on the performance indicator of the first service. The abnormality determination server 10 determines whether or not an abnormality determination criterion associated with the second service is satisfied based on the performance indicator of the second service. The abnormality determination server 10 determines whether or not an abnormality determination criterion associated with the third service is satisfied based on the performance indicator of the third service.

[0074] The abnormality determination server 10 determines whether or not at least one of the first service, the second service, and the third service is determined to be abnormal (S104). In S104, the abnormality determination server 10 determines whether or not an abnormality determination criterion is satisfied in at least one of the first service, the second service, and the third service. When the abnormality determination criterion is satisfied in a certain service, the abnormality determination server 10 determines that an abnormality has occurred in that service. In S104, when at least one of the first service, the second service, and the third service is not determined to be abnormal (S104: N), this process ends.

[0075] In S104, when at least one of the first service, the second service, and the third service is judged to be abnormal (S104: Y), the abnormality determination server 10 transmits an abnormality occurrence notification indicating that an abnormality has occurred in the at least one service to the administrator terminal 20 (S105). The notification may be sent via any medium, for example, email, SMS, SNS, a message app, or a notification on a maintenance tool. When the administrator terminal 20 receives the abnormality occurrence notification from the abnormality determination server 10 (S106), the administrator terminal 20 displays an administrator screen SC on the display unit 25 based on the abnormality occurrence notification (S107), and this process ends.

[0076] The anomaly determination system 1 of the first embodiment acquires a performance index related to the performance of the service based on the usage status of the service providing system 2. The anomaly determination system 1 executes anomaly determination regarding the occurrence of anomalies in the service providing system 2 based on the performance index. This allows anomalies that cannot be determined by golden signal metrics, for example, to be determined from a business perspective, thereby improving the accuracy of anomaly determination. In the example of FIG. 3, even if the golden signal metrics of the first server 30 do not determine an anomaly, there is a possibility that an anomaly has occurred in a part that cannot be determined by the golden signal metrics alone. In this case, the performance index can determine that some anomaly has occurred in the first server 30, thereby improving the accuracy of anomaly determination. For example, even if some anomaly has occurred in the golden signal metrics, if the performance index is not significantly affected, the administrator can determine that there is no need to respond immediately.

[0077] Furthermore, the anomaly determination system 1 performs an anomaly determination for at least one of the multiple services based on the performance index of each of the multiple services. This allows the service providing system 1 to perform an anomaly determination for at least one service even if the service providing system 1 provides multiple services. For example, when performing an anomaly determination for the first service based on the performance index of the first service and the performance index of at least one of the second service and the third service, the anomaly determination can be performed by comprehensively considering not only the performance index of the first service but also the performance indexes of the other services, thereby improving the accuracy of the anomaly determination.

[0078] Furthermore, the anomaly determination system 1 performs anomaly determination in real time based on the performance index acquired in real time, thereby enabling anomalies occurring in the service providing system 1 to be detected promptly.

[0079] Furthermore, the anomaly determination system 1 performs anomaly determination based on the payment index. This allows for rapid detection of any abnormality that occurs in the payment in the service providing system 1. For example, by using a payment index that is directly related to the business, anomalies that have a large impact on the business can be detected.

[0080] [2. Second embodiment] Next, a second embodiment, which is another embodiment of the anomaly determination system 1, will be described. For example, the service providing system 2 may hold a campaign at a specific time. During the campaign, the use of the service increases, and the usage situation may be different from that during normal times when the campaign is not held. Therefore, the anomaly determination criteria may be made different between campaign times and normal times to increase the flexibility of anomaly determination.

[0081] Fig. 7 is a diagram showing an example of performance indicators during a campaign period and during normal periods. The example at the top of Fig. 7 shows the time-series changes in the performance indicators when the first service is on campaign and the second and third services are on normal periods. The example at the bottom of Fig. 7 shows the time-series changes in the performance indicators when the first service, second service, and third service are all on normal periods. As in the example at the top of Fig. 7, the performance indicator of the first service during a campaign period is higher than the performance indicator during normal periods.

[0082] For example, if an abnormality occurs during a campaign, more opportunity losses will occur, resulting in a greater impact on business. For this reason, the abnormality judgment standard is higher during campaign periods. In the example of FIG. 7, the abnormality judgment standard for the first service is shown by a thick solid line. As shown in FIG. 7, the abnormality judgment standard during campaign periods is higher than the abnormality judgment standard during normal periods. For the second service and the third service, no campaigns are held, so the abnormality judgment standard remains unchanged. Details of the second embodiment will be described below. Note that in the second embodiment, explanations of configurations similar to those of the first embodiment will be omitted.

[0083] [2-1. Functions realized by the anomaly determination system of the second embodiment] 8 is a diagram showing an example of functions realized by the abnormality determination system 1 of the second embodiment. For example, the abnormality determination server 10 includes a data storage unit 100, an index acquisition unit 101, an abnormality determination unit 102, and a scheduled data acquisition unit 103. Regarding the data storage unit 100, the index acquisition unit 101, and the abnormality determination unit 102, differences from the first embodiment will be described. The scheduled data acquisition unit 103 is realized by the control unit 11. Note that the functions realized by the administrator terminal 20 may be similar to those of the first embodiment.

[0084] [Data storage section] The data storage unit 100 of the second embodiment stores a schedule database DB2 and an abnormality determination criterion database DB3 in addition to the performance index database DB1 described in the first embodiment.

[0085] 9 is a diagram showing an example of the schedule database DB2. The schedule database DB2 is a database in which information related to schedules in the service providing system 2 is stored. In the second embodiment, a case will be described in which a campaign corresponds to a schedule, but the schedule itself may be of any type and is not limited to a campaign. For example, the schedule may be a schedule for some event other than a campaign, an offline event, the release of a new service, maintenance, or another service.

[0086] For example, the schedule database DB2 stores the date and time of the schedule, the name of the schedule, and the schedule content. In the second embodiment, each record contained in the schedule database DB2 corresponds to schedule data. This information is registered in the schedule database DB2 by an administrator or other person. For example, the schedule content is the service ID of the service to which the schedule relates, the type of schedule, the expected usage status at the date and time of the schedule, or a combination of these. Note that the schedule database DB2 may store any information related to the schedule. The information stored in the schedule database DB2 is not limited to the example of FIG. 9.

[0087] FIG. 10 is a diagram showing an example of the abnormality determination criterion database DB3. The abnormality determination criterion database DB3 is a database in which abnormality determination criteria used in abnormality determination are stored. For example, the abnormality determination criterion is stored for each schedule in the abnormality determination criterion database DB3. For example, the administrator registers the abnormality determination criterion in the abnormality determination criterion database DB3 for each schedule in the service providing system 2. When a campaign is held, the administrator specifies the abnormality determination criterion in consideration of performance indicators in similar past campaigns. The abnormality determination criterion database DB3 stores abnormality determination criteria for campaign periods and abnormality determination criteria for normal periods.

[0088] The abnormality determination criteria may be determined by a method other than the method specified by the administrator. For example, the abnormality determination criteria may be determined based on a learning model in which past trends have been learned using a machine learning technique. In this case, the abnormality determination server 10 inputs the details of upcoming events into the learned learning model. The abnormality determination server 10 stores the abnormality determination criteria output from the learning model in association with the schedule in the abnormality determination criteria database DB3. For example, the abnormality determination criteria may also be determined by other methods such as a rule-based technique.

[0089] [Indicator acquisition section] In the second embodiment, the case where the index acquisition unit 101 acquires the performance index described in the first embodiment will be described, but the index acquisition unit 101 may acquire other indexes besides the performance indexes. The parts described as performance indexes in the second embodiment can be read as other indexes. For example, the index acquisition unit 101 may acquire at least one of hardware indexes, which are indexes related to hardware, and software indexes, which are related to software. The golden signal metrics described in the first embodiment are an example of hardware indexes and software indexes. The hardware indexes and software indexes may be acquired based on a program used in a benchmark test or a maintenance tool.

[0090] The hardware index is an index related to the load of the hardware. The software index is an index related to the operation of the software. The hardware index and the software index themselves may be known indexes. For example, the hardware index may be a CPU usage rate, a memory usage rate, a network usage rate, a power usage rate, or a combination of these. Indexes other than the hardware index may be used. For example, the temperature of the hardware or its surroundings exceeding an abnormality criterion may correspond to a hardware abnormality. For example, the software index is the processing time required to execute a test program, or the processing time required to read specified data.

[0091] [Schedule Data Acquisition Section] The schedule data acquisition unit 103 acquires schedule data related to the schedule of the service. For example, the schedule data acquisition unit 103 acquires the schedule data by referring to the schedule database DB2. When the schedule data is stored in a database other than the schedule database DB2, the schedule data acquisition unit 103 may acquire the schedule data by referring to the other database. When the schedule data is stored in a computer or information storage medium other than the abnormality determination server 10, the schedule data acquisition unit 103 may acquire the schedule data from the other computer or information storage medium.

[0092] In the second embodiment, the schedule data indicates a campaign period during which a campaign related to a service will be held. The campaign period is a period during which a campaign will be held. The campaign period may be of any length. For example, the length of the campaign period may be one minute to one hour, several hours to several days, one week to one month, or other lengths. In the example of FIG. 10, a case is described in which the campaign period is indicated by date and time, but the campaign period may include only the date without including the time.

[0093] [Abnormality determination section] The abnormality determination unit 102 of the second embodiment executes abnormality determination based on the performance indicators and the schedule data. In the second embodiment, since the schedule data indicates a campaign period, the abnormality determination unit 102 executes abnormality determination based on the performance indicators and the campaign period indicated by the schedule data. For example, the abnormality determination unit 102 determines an abnormality determination criterion based on the campaign period. The abnormality determination unit 102 determines whether or not the campaign period has arrived.

[0094] For example, when it is determined that the campaign period has not arrived, the abnormality determination unit 102 performs abnormality determination based on the abnormality determination criteria for the normal period. When it is determined that the campaign period has arrived, the abnormality determination unit 102 refers to the abnormality determination criteria database DB3 and acquires the abnormality determination criteria associated with the campaign. The abnormality determination unit 102 performs abnormality determination based on the business performance evaluation and the acquired abnormality determination criteria. Although the method of acquiring the abnormality determination criteria differs from that of the first embodiment, the abnormality determination process itself after the abnormality determination criteria are acquired is similar to that of the first embodiment.

[0095] The method in which the abnormality determination unit 102 uses the scheduled data for abnormality determination is not limited to the above example. For example, the abnormality determination unit 102 may determine whether or not to execute abnormality determination based on the scheduled data, rather than determining the abnormality determination criteria based on the scheduled data. The abnormality determination unit 102 may determine whether or not it is the campaign period indicated by the scheduled data, and if it is determined that it is not the campaign period, it may not execute abnormality determination, and if it is determined that it is the campaign period, it may execute abnormality determination based on the performance index. Conversely, if it is determined that it is the campaign period, it may not execute abnormality determination, and if it is determined that it is not the campaign period, it may execute abnormality determination based on the performance index.

[0096] [2-2. Processing Executed in the Anomaly Determination System of the Second Embodiment] Fig. 11 is a diagram showing an example of processing executed by the abnormality determination system 1 of the second embodiment. The processing in Fig. 11 is executed by the control units 11 and 21 executing programs stored in the storage units 12 and 22, respectively. As shown in Fig. 11, the processing in S200 to S202 is similar to the processing in S100 to S102, respectively.

[0097] The abnormality determination server 10 refers to the schedule database DB2 and determines whether or not there is a schedule to which the current time belongs (S203). If it is determined in S203 that there is no schedule to which the current time belongs (S203), the abnormality determination server 10 refers to the abnormality determination criterion database DB3 and acquires abnormality determination criteria for normal periods (S204).

[0098] In S203, when it is determined that there is no schedule that includes the current time (S203), the abnormality determination server 10 refers to the abnormality determination criteria database DB3 and acquires the abnormality determination criteria for the campaign period (S205). The processes of S206 to S211 differ from the processes of S102 to S107 in that the abnormality determination criteria acquired in S204 or S205 are used, but are otherwise similar to the processes of S102 to S107.

[0099] The anomaly determination system 1 of the second embodiment executes anomaly determination based on business performance indicators and schedule data. This enables anomaly determination according to the schedule indicated by the schedule data, increasing the flexibility of anomaly determination. For example, when a business performance indicator is expected to rise at a certain time, it becomes easier to prevent opportunity loss during important times such as campaign periods by raising the anomaly determination standard. During normal times, by setting an anomaly determination standard different from that during times when there are plans, it is possible to prevent unnecessary notifications from being sent to the administrator terminal 20. By executing anomaly determination only during campaign periods, anomaly determination can be executed only when necessary.

[0100] Furthermore, the anomaly detection system 1 executes anomaly detection based on the performance indicators and the campaign period indicated by the schedule data. This enables anomaly detection according to important periods such as campaign periods, thereby increasing the flexibility of anomaly detection.

[0101] [3. Modifications] The present disclosure is not limited to the first and second embodiments described above, and may be modified as appropriate without departing from the spirit and scope of the present disclosure.

[0102] [3-1. Modifications of the First Embodiment] First, a modification of the first embodiment will be described.

[0103] [Variation 1-1] For example, as shown in Fig. 2, the service providing system 1 may include a configuration specific to each service and a configuration spanning multiple services. In this case, if only the performance index of a specific service has decreased, an abnormality may have occurred in the configuration specific to this service. On the other hand, if the performance index of the entire multiple services has decreased, an abnormality may have occurred in the configuration spanning multiple services. Therefore, an abnormality determination may be performed taking into account not only the performance index of each service but also the overall performance index of the multiple services.

[0104] The abnormality determination unit 102 of the first modification executes an abnormality determination for each service to be subjected to the abnormality determination based on the relationship between the performance index of the service and the performance index of the entire plurality of services. The performance index of the entire plurality of services is the sum of the performance indexes of each of the plurality of services. For example, the abnormality determination unit 102 calculates the sum of the performance index of the first service, the performance index of the second service, and the performance index of the third service.

[0105] For example, the abnormality determination unit 102 performs an abnormality determination for the first service based on the relationship between the performance index of the first service and the calculated sum. When the performance index of the first service satisfies the abnormality determination criterion for the first service and the sum does not satisfy the overall abnormality determination criterion, the abnormality determination unit 102 determines that an abnormality has occurred in the first service. When the sum satisfies the overall abnormality determination criterion, the abnormality determination unit 102 determines that an abnormality has occurred in the entire service. In this case, it may be determined that an abnormality has occurred in the entire service regardless of whether the performance index of the first service satisfies the abnormality determination criterion for the first service. The abnormality determination unit 102 performs an abnormality determination for the second service and the third service in the same manner.

[0106] Fig. 12 is a diagram showing an example of anomaly determination based on performance indicators of a plurality of services as a whole. In the example of Fig. 12, the performance indicators of all of the first service, the second service, and the third service drop sharply. In this case, it is possible that anomalies have occurred in all of the first server 30, the second server 40, and the third server 50, but it is also possible that anomalies in the common server 60 common to the first service, the second service, and the third service have made it difficult for all of the services to accept payments.

[0107] For example, the abnormality determination unit 102 determines that an abnormality has occurred in the common server 60, not in the first server 30, the second server 40, or the third server 50. In this case, the abnormality determination unit 102 transmits an abnormality occurrence notification indicating that an abnormality has occurred in the common server 60 to the administrator terminal 20. As shown in the administrator screen SC at the bottom of FIG. 12, a message indicating that an abnormality may have occurred in the common server 60 is displayed.

[0108] The anomaly determination system 1 of the modified example 1-1 performs anomaly determination for each service to be subjected to anomaly determination based on the relationship between the performance index of the service and the performance index of the entire plurality of services. This allows the anomaly determination to be performed taking into account the performance index of the entire plurality of services, thereby improving the accuracy of the anomaly determination. For example, it becomes easier to identify that an anomaly has occurred in a configuration that spans multiple services, rather than in a configuration specific to an individual service.

[0109] [Variation 1-2] For example, the abnormality determination unit 102 may perform the abnormality determination for each service to be subjected to the abnormality determination based on the relationship between the performance index of the service and the performance index of other services. That is, the abnormality determination may be performed in consideration of the relationship between the performance indexes of the individual services, instead of the relationship between the performance index of each service and the performance index of the entire plurality of services (the relationship described in the modification 1-1).

[0110] FIG. 13 is a diagram showing an example of anomaly determination based on the relationship between a performance index of a certain service and a performance index of another service. In the example of FIG. 13, the performance index of the second service and the performance index of the third service each drop sharply. The performance index of the first service does not change much from normal times. In this case, an anomaly may have occurred in each of the second server 40 and the third server 50, but it is also possible that an anomaly in a common platform (e.g., part of the configuration of the common server 60) common to the second service and the third service makes it difficult for the second service and the third service to accept payments.

[0111] For example, when the performance indicator of the second service satisfies the abnormality judgment criterion and the performance indicator of the third service also satisfies the abnormality judgment criterion, the abnormality judgment unit 102 judges that an abnormality has occurred in the configuration common to the second service and the third service. The abnormality judgment unit 102 judges that an abnormality has occurred in the common platform realized by the common server 60, not in the second server 40 and the third server 50. In this case, the abnormality judgment unit 102 transmits an abnormality occurrence notification to the administrator terminal 20, indicating that an abnormality has occurred in the common platform of the common server 60. As shown in the administrator screen SC at the bottom of FIG. 13, a message indicating that an abnormality may have occurred in the common platform of the common server 60 is displayed.

[0112] For example, when the performance index of the second service satisfies the abnormality judgment criterion and the performance index of the third service does not satisfy the abnormality judgment criterion, the abnormality judgment unit 102 judges that an abnormality has occurred in the configuration specific to the second service. When the performance index of the second service does not satisfy the abnormality judgment criterion and the performance index of the third service satisfies the abnormality judgment criterion, the abnormality judgment unit 102 judges that an abnormality has occurred in the configuration specific to the third service. The processing when it is judged that an abnormality has occurred in the configuration specific to each service is the same as that in the first embodiment. Note that the relationship between each service and the part of the service providing system 2 where the abnormality has occurred is stored in advance in the data storage unit 100.

[0113] The anomaly determination system 1 of the modified example 1-2 performs anomaly determination for each service that is the target of anomaly determination, based on the relationship between the performance index of the service and the performance index of other services. This allows the anomaly determination to be performed taking into account the relationship between the performance indexes of the individual services, thereby improving the accuracy of anomaly determination. For example, it becomes easier to identify that an anomaly has occurred in a configuration that spans multiple services, rather than in a configuration specific to an individual service.

[0114] [Variation 1-3] For example, as described in the first embodiment, when the service providing system 2 provides each of a plurality of services to a user, the abnormality determination unit 102 may determine whether or not the performance index of each of the plurality of services satisfies the abnormality determination criterion. In this case, the type of the performance index may be useful for identifying a service in which an abnormality has occurred among the plurality of services.

[0115] For example, suppose that the performance indicator acquisition unit 101A acquires two performance indicators: a payment indicator related to payment, and a login indicator related to the number of logins. If the payment indicator does not satisfy the abnormality judgment criterion, an abnormality may have occurred in a FinTech service in the service providing system 2. If the login indicator does not satisfy the abnormality judgment criterion, an abnormality may have occurred in a member management service in the service providing system 2. In this way, the type of performance indicator that satisfies the abnormality judgment criterion is useful for identifying the service in which the abnormality has occurred.

[0116] Therefore, the anomaly determination unit 102 of the modified example 1-3 executes anomaly determination based on the type of performance indicator that meets the anomaly determination criterion. In the modified example 1-3, it is assumed that type data in which the type of performance indicator and the service in which an anomaly may have occurred are associated with each other is stored in the data storage unit 100. The type data may be in any format, and may be, for example, a table, a part of a program code, or a learning model using a machine learning technique. For example, the type data is associated with a payment indicator and a FinTech-related service. The type data is associated with a login indicator and a member management service.

[0117] For example, the anomaly determination unit 102 determines, based on the type data, that an anomaly has occurred in a service among a plurality of services that is associated with a type of performance indicator that satisfies the anomaly determination criterion. When it is determined that the payment indicator satisfies the anomaly determination criterion, the anomaly determination unit 102 determines that an anomaly has occurred in a FinTech-related service associated with the payment indicator. When it is determined that the login indicator satisfies the anomaly determination criterion, the anomaly determination unit determines that an anomaly has occurred in a FinTech-related service associated with the payment indicator.

[0118] Note that the abnormality determination based on the type of performance indicator that satisfies the abnormality determination criterion is not limited to the above example. The type data may be associated with a relationship between the type of performance indicator and the hardware or software in which an abnormality may have occurred. In this case, the abnormality determination unit 102 may determine that an abnormality has occurred in the hardware or software associated with the type of performance indicator that satisfies the abnormality determination criterion. The abnormality determination unit 102 transmits an abnormality occurrence notification indicating the hardware or software to the administrator terminal 20. Information on the hardware or software is displayed on the administrator screen SC.

[0119] The anomaly determination system 1 of the modified example 1-3 executes anomaly determination based on the type of performance indicator that meets the anomaly determination criterion. This allows anomaly determination to be executed according to the type of performance indicator, thereby improving the accuracy of anomaly determination. For example, when a service in which an anomaly may have occurred is identified based on the type of performance indicator, it becomes easier for an administrator to identify the service in which an anomaly has occurred. When hardware or software in which an anomaly may have occurred is identified based on the type of performance indicator, it becomes easier for an administrator to identify the hardware or software in which an anomaly has occurred.

[0120] [Variation 1-4] For example, in the first embodiment, a case has been described in which an abnormality determination is performed based on a performance index. In the abnormality determination, an index other than the performance index may also be taken into consideration. In Modification 1-4, a hardware index is described as an example of the other index. The other index is not limited to a hardware index, and may be the software index described in the first embodiment.

[0121] FIG. 14 is a diagram showing an example of functions realized in Modification 1-4. FIG. 14 shows functions when the first embodiment and the second embodiment are combined, but the functions described in the second embodiment do not have to be included in the abnormality determination system 1. The abnormality determination system 1 of Modification 1-4 includes a hardware index acquisition unit 101B. The hardware index acquisition unit 101B acquires hardware indexes related to hardware included in the service providing system 2. Details of the hardware indexes are as described in the first embodiment. Furthermore, the method of acquiring the hardware index itself can use a publicly known method, as described in the first embodiment.

[0122] The abnormality determination unit 102 executes abnormality determination based on the performance index and the hardware index. For example, the abnormality determination unit 102 determines whether or not the abnormality determination criteria are satisfied not only for the performance index but also for the hardware index. Hereinafter, in Modifications 1-4, the abnormality determination criteria for the performance index are referred to as the performance index criteria, and the abnormality determination criteria for the hardware index are referred to as the hardware index criteria. The process of determining whether or not the performance index criteria are satisfied is as described in the first embodiment.

[0123] For example, the hardware index criterion includes a threshold value related to the hardware index. The abnormality determination unit 102 executes the abnormality determination by determining whether the hardware index is equal to or greater than the threshold value. The abnormality determination unit determines that an abnormality has occurred when at least one of the CPU usage rate, memory usage rate, and network usage rate indicated by the hardware index is equal to or greater than the threshold value. Note that the hardware index criterion does not necessarily need to include a threshold value. For example, the hardware index criterion may be a criterion as to whether or not a value output from a learning model in which the tendency of the hardware index has been learned is a predetermined value.

[0124] For example, when a determination result indicating that an abnormality has occurred is obtained from at least one of the abnormality determination based on the performance index and the abnormality determination based on the hardware index, the abnormality determination unit 102 transmits an abnormality occurrence notification to the administrator terminal 20. When an abnormality has occurred in the abnormality determination based on the hardware index, the abnormality occurrence notification includes information regarding the hardware in which the abnormality has occurred. Based on the abnormality occurrence notification, the display control unit 202 causes the information regarding the hardware in which the abnormality has occurred to be displayed on the administrator screen SC.

[0125] The abnormality determination system 1 of the modification 1-4 executes the abnormality determination based on the performance index and the hardware index. This allows the abnormality determination to be executed taking into account not only the performance index but also the hardware index, thereby improving the accuracy of the abnormality determination.

[0126] [Variation 1-5] For example, as described in Modification 1-4, the abnormality determination unit 102 may determine whether or not the performance index satisfies a predetermined performance index criterion. The abnormality determination unit 102 may determine whether or not the hardware index satisfies a predetermined hardware index criterion. In this case, when it is determined that the hardware index criterion is satisfied and the performance index criterion is not satisfied, the abnormality determination unit 102 may execute the abnormality determination so that the degree of abnormality is lower than when both the hardware index criterion and the performance index criterion are satisfied.

[0127] The degree of abnormality is information indicating the degree of suspicion of an abnormality. In other words, the degree of abnormality is information indicating the seriousness of an abnormality. The degree of abnormality is expressed by information having an intermediate value, not a binary value. In other words, the degree of abnormality can take three or more levels of values. The abnormality score described in the first embodiment corresponds to the degree of abnormality. Therefore, the portion written as the degree of abnormality can be read as the abnormality score. The degree of abnormality may be called by a name other than the abnormality score.

[0128] For example, when it is determined that the hardware index criterion is satisfied and the performance index criterion is not satisfied, the abnormality determination unit 102 may lower the abnormality score compared to when it is determined that both the hardware index criterion and the performance index criterion are satisfied. When it is determined that the hardware index criterion is satisfied and the performance index criterion is not satisfied, the abnormality determination unit 102 may lower the abnormality score compared to when the hardware index criterion is not satisfied and the performance index criterion is satisfied.

[0129] In other words, even if it is determined that an abnormality has occurred when only the hardware index is taken into consideration, when the abnormality determination unit 102 determines that an abnormality has not occurred when the performance index criterion is taken into consideration, the degree of the abnormality is lowered compared to when it is determined that an abnormality has occurred when the performance index criterion is taken into consideration. That is, the abnormality determination based on the performance index may be used with priority over the abnormality determination based on the hardware index. Even if an abnormality has occurred in terms of hardware, when the impact on business is small, the degree of the abnormality is relatively small.

[0130] The anomaly determination system 1 of the modified example 1-5 determines whether the hardware index satisfies a predetermined hardware index criterion, and when it is determined that the hardware index criterion is satisfied and the performance index criterion is not satisfied, executes anomaly determination so that the degree of anomaly is lower than when both the hardware index criterion and the performance index criterion are satisfied. This improves the accuracy of anomaly determination. For example, even if a hardware abnormality is likely, if the impact on business is small, the administrator can postpone response or not respond at all.

[0131] [Variation 1-6] For example, the performance index can be used not only to determine abnormalities at the present time, but also to determine abnormalities in the future. In variant example 1-6, the first service (electronic commerce service) described in the first embodiment is taken as an example. During a campaign for the first service, more users use the first service, so the performance index can become high. When the performance index becomes high, the load on the service providing system 2 can become high. However, the load on the service providing system 2 does not necessarily become high immediately after the performance index becomes high.

[0132] For example, assume that the number of items put into a shopping cart corresponds to the performance index. In this case, it is considered that payments for purchasing items are made intensively after a certain amount of time has passed since the performance index became equal to or greater than a threshold. In this case, a failure may occur in the service providing system 2 after a certain amount of time has passed since the performance index became equal to or greater than the threshold. Therefore, the anomaly determination unit 102 may perform anomaly determination regarding the occurrence of an anomaly in the future based on the performance index.

[0133] For example, when a performance index satisfies an abnormality judgment criterion, the anomaly judgment unit 102 judges that an abnormality will occur not at the time indicated by the performance index (e.g., the present time) but in the future. The future here means a time later than the time indicated by the performance index. The time interval between these times may be of any length. This time interval may be specified by an administrator or may be dynamically determined from past trends. This time interval may be determined for each service. It is assumed that data indicating the time interval is stored in advance in the data storage unit 100.

[0134] For example, if the average time interval between when an item is placed in a shopping cart and when a payment is made is five minutes, the abnormality determination unit 102 determines that an abnormality may occur in the first server 30 five minutes after the time indicated by the performance index. The abnormality determination unit 102 transmits an abnormality determination notice to the administrator terminal 20 indicating that an abnormality may occur in the first server 30 in five minutes. Upon receiving the abnormality determination notice, the display control unit 202 causes the display unit 25 to display an administrator screen SC indicating that an abnormality may occur in the first server 30 in five minutes.

[0135] The anomaly determination system 1 of the modified example 1-6 executes anomaly determination regarding the occurrence of an anomaly in the future based on the performance index. This makes it possible to determine the possibility of an anomaly occurring in the future even if there is a certain amount of time lag between when the performance index meets the anomaly determination criterion and when an anomaly may actually occur. As a result, the occurrence of an anomaly can be prevented in advance by the administrator taking measures manually or by using a load balancer that automatically distributes the load.

[0136] [Variation 1-7] For example, in the first embodiment, an anomaly determination criterion including a threshold value to be compared with a performance index is used in the anomaly determination, but the anomaly determination is not limited to the example of the first embodiment. The anomaly determination unit 102 may execute an anomaly determination based on the performance index and a learning model in which the relationship between the training performance index and the occurrence of an anomaly is learned. The learning model is a model trained using a machine learning technique.

[0137] The machine learning method itself may use various known methods. For example, the machine learning method may be any of supervised learning, semi-supervised learning, and unsupervised learning. In Modification 1-7, a learning model using a neural network is taken as an example. For example, the administrator prepares training data that is a pair of an input part indicating a performance indicator for training and an output part indicating whether an abnormality has occurred. The training data may be created manually by the administrator or by using a known tool.

[0138] The method for learning the training data itself may be a known method used in machine learning. For example, the administrator terminal 20 executes learning of the learning model by adjusting the parameters of the learning model so that the output part of the training data is output when the input part of the training data is input. The administrator terminal 20 transmits the learned learning model to the anomaly determination server 10. The anomaly determination server 10 records the learned learning model received from the administrator terminal 20 in the data storage unit 100.

[0139] For example, the anomaly judgment unit 102 inputs a performance index of a certain service to the learning model. The learning model calculates a feature amount of the performance index, and outputs a judgment result indicating whether or not an anomaly has occurred based on the feature amount. The process performed by the learning model is sometimes called estimation, so the judgment result output from the learning model can also be called an estimation result by the learning model. The judgment result is not limited to two values ​​as described in the first embodiment, and may be information having an intermediate value such as an anomaly score. The anomaly judgment unit 102 executes anomaly judgment by acquiring the judgment result output from the learning model.

[0140] The anomaly detection system 1 of the modified example 1-7 performs anomaly detection based on the performance index and a learning model in which the relationship between the training performance index and the occurrence of anomalies is learned. This allows the anomaly detection to be performed using a machine learning technique, thereby improving the accuracy of the anomaly detection.

[0141] [Variation 1-8] For example, the learning model described in Modification 1-7 may be prepared for each individual service, or the performance indicators of each of the multiple services may be learned comprehensively. The learning model in Modification 1-8 learns the relationship between the training performance indicators of each of the multiple services and the occurrence of anomalies. The data structure of the training data itself is the same as that of Modification 1-7, but the input portion of the training data is not the training performance indicators of one service, but the training performance indicators of each of the multiple services.

[0142] The abnormality determination unit 102 of the modified example 1-8 executes an abnormality determination for at least one of the multiple services based on the performance indicators of each of the multiple services and the learning model. For example, the abnormality determination unit 102 executes an abnormality determination for each service based on the performance indicators of the service and each of the other services and the learning model. The modified example 1-8 differs from the modified example 1-7 in that the performance indicators input to the learning model are not for one service but for multiple services, but is otherwise similar to the modified example 1-7.

[0143] The anomaly determination system 1 of the modification 1-8 performs an anomaly determination for at least one of the multiple services based on the performance indicators of each of the multiple services and the learning model. This allows the anomaly determination to be performed by comprehensively considering the performance indicators of each of the multiple services, thereby improving the accuracy of the anomaly determination.

[0144] [Variation 1-9] For example, the service providing system 2 may be renewed in stages. In this case, an old system, which is a relatively old service providing system 2, and a new system, which is a relatively new service providing system 2, may coexist. The old system and the new system differ from each other in at least one of the hardware and software used to provide the service. In this case, an abnormality determination of the new system may be performed based on the performance indicators of the old system.

[0145] For example, when a user uses a service with a browser, the user can access either the URL of the old system or the URL of the new system. When a user uses an application other than a browser, the user can access either the application for the old system or the application for the new system. In variant 1-9, there are two service providing systems 2, an old system and a new system.

[0146] The performance indicator acquisition unit 101A of the modified example 1-9 acquires an old system indicator, which is a performance indicator based on the old system, and a new system indicator, which is a performance indicator based on the new system. For example, the performance indicator acquisition unit 101A acquires the old system indicator from the old system. The old system indicator differs from the performance indicator described in the first embodiment in that it is a performance indicator based on the old system, but is similar in other respects. Therefore, the old system indicator may be any of the various performance indicators described in the first embodiment.

[0147] For example, the performance index acquisition unit 101A acquires a new system index from a new system. The new system index is different from the performance index described in the first embodiment in that it is a performance index based on the new system, but is similar in other respects. Therefore, the new system index may be any of the various performance indexes described in the first embodiment.

[0148] The anomaly determination unit 102 of the modified example 1-9 performs an anomaly determination for the new system based on an old system index, which is a performance index based on the old system. For example, the anomaly determination unit 102 determines that an anomaly has occurred in the new system when the old system index satisfies an anomaly determination criterion. The anomaly determination criterion here may be that the old system index is equal to or greater than a threshold. If the old system index is equal to or greater than the threshold, it is possible that some kind of anomaly has occurred in the new system because the user is only using the old system and not the new system. For this reason, the anomaly determination unit 102 determines that an anomaly has occurred in the new system when the old system index is equal to or greater than the threshold.

[0149] The anomaly determination system 1 of the modified example 1-9 performs anomaly determination of the new system based on the old system index, which is a performance index based on the old system. This improves the accuracy of anomaly determination of the new system. It is possible to identify anomalies that cannot be identified only by the performance index based on the new system.

[0150] [Variation 1-10] For example, in the first embodiment, a payment index has been described as an example of a performance index. The performance index may be any information that indicates some performance of a service, and is not limited to a payment index. For example, the performance index acquisition unit 101A acquires a transaction index related to a transaction in an e-commerce service as a performance index. A transaction is the use of an e-commerce service. For example, a purchase of a product or content. The transaction index is the number of purchases, the purchase amount, the total purchase amount, products added to a shopping cart, the number of drops from a shopping cart, products added to bookmarks, or a combination of these.

[0151] The abnormality determination unit 102 of the modified example 1-10 executes the abnormality determination based on the transaction index. The modified example 1-10 differs from the first embodiment in that the transaction index is used in the abnormality determination, but the other points are the same as those described in the first embodiment.

[0152] The abnormality determination system 1 of the modification 1-10 performs abnormality determination based on a transaction index, thereby improving the accuracy of abnormality determination in the electronic commerce service.

[0153] [Variation 1-11] For example, the performance indicator acquiring unit 101A may acquire a membership indicator related to the membership of the service as the performance indicator. The membership indicator is an indicator related to the members of the service. For example, the membership indicator is the number of members, the change in the number of members, the number of withdrawals, the change in the number of withdrawals, or a combination thereof.

[0154] The abnormality determination unit 102 executes the abnormality determination based on the membership index. The second embodiment differs from the first embodiment in that the membership index is used in the abnormality determination, but other points are the same as those described in the first embodiment.

[0155] The abnormality determination system 1 of the modification 1-11 executes abnormality determination based on the membership index, thereby improving the accuracy of abnormality determination in services that require member registration.

[0156] [3-2. Modifications of the second embodiment] Next, a modification of the second embodiment will be described.

[0157] [Variation 2-1] For example, in the second embodiment, the schedule data indicates a campaign period, but the schedule data is not limited to a campaign period as long as it indicates some schedule in the service providing system 1. In the modified example 2-1, a case where the schedule data indicates a maintenance period when maintenance related to a service is performed will be taken as an example.

[0158] Maintenance is performed on the hardware and software included in the service providing system 2. For example, inspection, replacement, removal, expansion, or a combination of these of hardware corresponds to maintenance. Inspection, installation, uninstallation, or a combination of these of software corresponds to maintenance. The maintenance period is the period during which maintenance is performed. The maintenance period may be indicated by date and time, as with the campaign period, or may be indicated by just the date.

[0159] The abnormality determination unit 102 of the modified example 2-1 executes the abnormality determination based on the performance index and the maintenance time indicated by the scheduled data. For example, the abnormality determination unit 102 determines whether or not the current time is included in the maintenance time. If the current time is included in the maintenance time, the abnormality determination unit 102 does not execute the abnormality determination based on the performance index, and if the current time is not included in the maintenance time, the abnormality determination unit 102 executes the abnormality determination based on the performance index.

[0160] The abnormality determination system 1 of the modified example 2-1 executes an abnormality determination based on the performance index and the maintenance timing indicated by the scheduled data. This enables an abnormality determination according to the maintenance timing, and increases the flexibility of the abnormality determination. For example, by not determining an abnormality when the performance index drops due to maintenance, it is possible to prevent unnecessary abnormality determination notifications from being sent to the administrator.

[0161] [Variation 2-2] For example, in the modified example 2-1, a case has been described in which it is determined whether or not to perform an abnormality determination based on the maintenance timing, but the abnormality determination using the maintenance timing is not limited to the example of the modified example 2-1. The abnormality determination unit 102 in the modified example 2-2 determines an abnormality determination criterion for the abnormality determination based on the scheduled data. The abnormality determination unit 102 performs an abnormality determination based on the determined abnormality determination criterion and an index.

[0162] For example, the abnormality determination unit 102 may determine an abnormality determination criterion based on a maintenance time indicated by the scheduled data, and execute an abnormality determination based on a performance index and the determined abnormality determination criterion. When the current time is included in the maintenance time, the abnormality determination unit 102 lowers the threshold indicated by the abnormality determination criterion compared to when the current time is not included in the maintenance time. The abnormality determination unit 102 executes an abnormality determination based on the performance index and the lowered threshold.

[0163] In the modified example 2-2, the case where the schedule data indicates the maintenance time is taken as an example, but in the case where the schedule data indicates the campaign time, the abnormality determination unit 102 may determine the abnormality determination criterion based on the campaign time. This process may be the same as the method described in the second embodiment.

[0164] The abnormality determination system 1 of the modification 2-2 determines the abnormality determination criteria for the abnormality determination based on the scheduled data, and executes the abnormality determination based on the determined abnormality determination criteria and the index. This allows the abnormality determination criteria to be based on the schedule, thereby improving the accuracy of the abnormality determination.

[0165] [Variation 2-3] For example, when the abnormality determination criterion is dynamically determined as in the modified example 2-2, the abnormality determination unit 102 may determine the abnormality determination criterion based on the relationship between the condition related to the scheduled data and the abnormality determination criterion when the condition is satisfied. This relationship is defined in the abnormality determination criterion database DB3. The condition related to the scheduled data is a condition that is determined based on information included in the scheduled data.

[0166] For example, the conditions related to the campaign timing, the length of the campaign, the amount of the campaign, the target user group of the campaign, the number of users expected in the campaign, the performance indicators expected in the campaign, or a combination thereof correspond to the conditions related to the scheduled data. For example, the conditions related to the maintenance timing, the length of the maintenance, the services to be maintained, the hardware to be maintained, the software to be maintained, or a combination thereof correspond to the conditions related to the scheduled data.

[0167] For example, in the abnormality determination criterion database DB3 of the modified example 2-3, an abnormality determination criterion is defined for each condition related to the scheduled data, which is applied when the condition is satisfied. The amount of a campaign may be defined as an example of a condition. For example, the abnormality determination criterion database DB3 is created so that the higher the amount of the campaign, the higher the abnormality determination criterion. For example, the abnormality determination unit 102 determines whether or not each condition is satisfied based on the scheduled data. The abnormality determination unit 102 identifies the abnormality determination criterion associated with the condition that is determined to be satisfied. The abnormality determination unit 102 performs an abnormality determination based on the performance indicator and the identified abnormality determination criterion.

[0168] The abnormality determination system 1 of the modification 2-3 determines the abnormality determination criterion based on the relationship between the condition related to the scheduled data and the abnormality determination criterion when the condition is satisfied. This makes it possible to set the optimal abnormality determination criterion according to the scheduled data, thereby improving the accuracy of abnormality determination.

[0169] [Variation 2-4] For example, in the modification 2-3, the abnormality determination criterion is determined using the abnormality determination criterion database DB3, but the method of determining the abnormality determination criterion is not limited to the example described in the modification 2-3. The abnormality determination unit 102 may determine the abnormality determination criterion based on the scheduled data and a learning model in which the relationship between the scheduled data for training and the abnormality determination criterion is learned. This learning model is different from the learning model described in the modification 1-7.

[0170] The machine learning method itself may use various known methods. For example, the machine learning method may be any of supervised learning, semi-supervised learning, and unsupervised learning. In Modification 2-4, a learning model using a neural network is taken as an example. For example, the administrator prepares training data that is a pair of an input part indicating a training schedule and an output part indicating an anomaly judgment criterion that is considered to be optimal. The training data may be created manually by the administrator or may be created using a known tool.

[0171] The training data learning method itself may be a known method used in machine learning. For example, the administrator terminal 20 executes learning of the learning model by adjusting the parameters of the learning model so that the output part of the training data is output when the input part of the training data is input. The administrator terminal 20 transmits the learned learning model to the anomaly determination server 10. The anomaly determination server 10 records the learned learning model received from the administrator terminal 20 in the data storage unit.

[0172] For example, the anomaly judgment unit 102 inputs the schedule data to the learning model. The learning model calculates the feature amount of the schedule indicated by the schedule data, and outputs the anomaly judgment criterion according to the feature amount. The anomaly judgment unit 102 executes the anomaly judgment based on the performance indicator and the anomaly judgment criterion output from the learning model. Although the method of acquiring the anomaly judgment criterion is different from that of the second embodiment, the anomaly judgment using the anomaly judgment criterion may be the same as that of the second embodiment. For example, the learning model outputs the anomaly judgment criterion so that the greater the influence of the campaign on the schedule indicated by the schedule data (for example, the increase in the payment amount), the larger the threshold included in the anomaly judgment criterion becomes. For other elements included in the schedule data that are correlated with the anomaly judgment criterion, the learning model outputs the anomaly judgment criterion according to the other elements.

[0173] The abnormality determination system 1 of the modification 2-4 determines the abnormality determination criterion based on the scheduled data and a learning model in which the relationship between the scheduled data for training and the abnormality determination criterion is learned. This makes it possible to obtain an optimal abnormality determination criterion by utilizing a machine learning technique, thereby improving the accuracy of the abnormality determination.

[0174] [Variation 2-5] For example, the learning model of Modification 2-5 may learn a relationship based on past performance. The relationship based on past performance is the relationship between the actual performance indicator in the past and the anomaly judgment criterion that is considered optimal at that time. For example, the training data of Modification 2-5 is manually created by an administrator based on past performance. The training data of Modification 2-5 may also be automatically created using a known tool.

[0175] The abnormality determination unit 102 of the modified example 2-5 determines the abnormality determination criterion based on a learning model in which a relationship based on past performance has been learned. The modified example differs from the modified example 2-4 in that a relationship based on past performance has been learned in the learning model, but in other respects is similar to the modified example 2-4. For example, the learning model of the modified example 2-5 learns the relationship between elements included in the schedule data of past plans and the abnormality determination criterion that is considered optimal in consideration of the actual performance indicators at that time. When schedule data of future plans is input, the learning model outputs the abnormality determination criterion corresponding to the schedule data.

[0176] The abnormality determination system 1 of the modified example 2-5 determines the abnormality determination criteria based on a learning model that has learned relationships based on past performance. This makes it possible to use a machine learning technique to determine optimal abnormality determination criteria, thereby improving the accuracy of abnormality determination.

[0177] [Variation 2-6] For example, by combining the modified example 1-7 of the first embodiment with the second embodiment, the anomaly determination unit 102 may execute anomaly determination based on the performance index and a learning model in which the relationship between the training index and the anomaly determination result regarding the occurrence of an anomaly is learned. This process is as described in the modified example 1-7.

[0178] The anomaly detection system 1 of the modified example 2-6 executes anomaly detection based on the performance indicators and a learning model that has learned the relationship between the training indicators and the anomaly detection result regarding the occurrence of anomalies. This allows the anomaly detection to be executed using a machine learning technique, thereby improving the accuracy of the anomaly detection.

[0179] [Variation 2-7] For example, the learning model of Modification 2-6 may learn a relationship based on past performance, similar to the learning model of Modification 2-5. The relationship based on past performance is the relationship between actual performance indicators in the past and whether or not an abnormality occurred at that time. For example, the training data of Modification 2-6 is manually created by an administrator based on past performance. The training data of Modification 2-6 may also be automatically created using a known tool.

[0180] The abnormality determination unit 102 executes the abnormality determination based on a learning model in which a relationship based on past performance has been learned. This is different from the modified example 2-6 in that a relationship based on past performance has been learned in a learning model, but other points are the same as the modified example 2-6.

[0181] In the abnormality determination system 1 of the modification 2-7, the abnormality determination unit 102 executes the abnormality determination based on a learning model in which a relationship based on past performance is learned. This allows the abnormality determination to be executed using a machine learning technique, thereby improving the accuracy of the abnormality determination.

[0182] [Variation 2-8] For example, in the second embodiment, the campaign for the first service is taken as an example, but the schedule data acquisition unit 103 may acquire the schedule for each of the multiple services. In the modified example 2-8, it is assumed that the schedule database DB2 stores schedule data indicating the schedule for each of the multiple services. Each piece of schedule data is as described in the second embodiment.

[0183] The abnormality determination unit 102 of the modification 2-8 executes an abnormality determination for at least one of the multiple services based on the performance indicators of each of the multiple services and the schedule of each of the multiple services. For example, the abnormality determination unit 102 executes an abnormality determination for each service based on the performance indicators of the service and the schedule of the service. For example, the abnormality determination unit 102 executes an abnormality determination for each service based on the performance indicators of the service and the other services and the schedule of the service and the other services.

[0184] The abnormality determination system 1 of the modified example 2-8 executes abnormality determination for at least one of the multiple services based on the index of each of the multiple services and the schedule of each of the multiple services. This makes it possible to execute abnormality determination for at least one service even when the service providing system 1 provides multiple services. For example, when executing abnormality determination for the first service based on the performance index of the first service and the schedule of the first service, the abnormality determination can be executed taking into account the schedule of the first service, so that the accuracy of the abnormality determination is improved. When executing abnormality determination for the first service based on the performance index of each of the first service and the second service and the schedule of each of the first service and the second service, the abnormality determination can be executed taking into account not only the first service but also the schedule of the second service, so that the accuracy of the abnormality determination is improved.

[0185] [Variation 2-9] For example, by combining the modified example 1-1 of the first embodiment with the second embodiment, the abnormality determination unit 102 may perform an abnormality determination for each service to be subjected to the abnormality determination based on the relationship between the performance indicators and schedule of the service and the performance indicators and schedule of the entire plurality of services. The performance indicators of the entire plurality of services are as described in the modified example 1-1. The schedule of the entire plurality of services is a schedule that spans the multiple services.

[0186] For example, maintenance of the common server 60 relates to all services, and therefore corresponds to a schedule for all of the multiple services. When maintenance for all of the multiple services occurs, the abnormality determination unit 102 does not perform abnormality determination. A points campaign that can be used for all of the multiple services relates to all services, and therefore corresponds to a schedule for all of the multiple services. When a campaign for all of the multiple services is held, the abnormality determination unit 102 raises the abnormality determination standard overall.

[0187] The anomaly determination system 1 of the modified example 2-9 performs anomaly determination for each service to be subjected to anomaly determination based on the relationship between the performance indicators and schedule of the service and the performance indicators and schedule of the entire plurality of services. This allows the anomaly determination to be performed taking into account the schedule of the entire plurality of services, thereby improving the accuracy of the anomaly determination. For example, it becomes possible to perform anomaly determination taking into account the schedule spanning the plurality of services, rather than the schedule specific to each individual service.

[0188] [Variation 2-10] For example, by combining the modified example 1-2 of the first embodiment with the second embodiment, the anomaly determination unit 102 may perform anomaly determination for each service to be subjected to anomaly determination based on the relationship between the performance indicators and schedule of the service and the performance indicators and schedule of the other services. The performance indicators of the other services are as described in the modified example 1-2. For example, in the example of the embodiment, since the second service and the third service are both FinTech-related services, the anomaly determination unit 102 determines anomaly determination criteria for the second service based on the schedule of the second service and the schedule of the third service. The anomaly determination unit 102 may perform anomaly determination for the second service by determining whether the performance indicators of the second service and the performance indicators of the third service satisfy the determined anomaly determination criteria for the second service.

[0189] In the abnormality determination system 1 of the modification 2-10, the abnormality determination unit 102 performs abnormality determination for each service that is the target of abnormality determination, based on the relationship between the index and schedule of the service and the index and schedule of other services.

[0190] [3-3. Other Modifications Common to the First and Second Embodiments] For example, the above modifications may be combined.

[0191] For example, in the first and second embodiments, a case has been described in which the main processing is executed by the anomaly determination server 10, but the processing described as being executed by the anomaly determination server 10 may be executed by the administrator terminal 20 or another computer, or may be shared among multiple computers.

Claims

1. an index acquisition unit that acquires an index that can be used in an abnormality determination regarding an occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an abnormality determination unit that determines an abnormality determination criterion based on a relationship between a condition related to the scheduled data and an abnormality determination criterion related to the abnormality determination when the condition is satisfied, and performs the abnormality determination based on the determined abnormality determination criterion and the indicator; An abnormality determination system including:

2. An index acquisition unit that acquires an index that can be used in an abnormality determination regarding the occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an abnormality determination unit that determines the abnormality determination criterion based on the scheduled data and a learning model in which a relationship between the scheduled data for training and an abnormality determination criterion related to the abnormality determination is learned, and performs the abnormality determination based on the determined abnormality determination criterion and the index; An abnormality determination system including:

3. An index acquisition unit that acquires an index that can be used in an abnormality determination regarding the occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an anomaly determination unit that executes the anomaly determination based on the index and a learning model in which a relationship between the training index and an anomaly determination result regarding the occurrence of the anomaly is learned; An abnormality determination system including:

4. an index acquisition unit that acquires an index that can be used in abnormality determination regarding occurrence of an abnormality in each of a plurality of services in a service providing system for providing a plurality of services; a schedule data acquisition unit that acquires schedule data relating to each schedule of the plurality of services; an abnormality determination unit that executes the abnormality determination for at least one of the plurality of services based on the index of each of the plurality of services and the schedule data of each of the plurality of services; An abnormality determination system including:

5. the schedule data indicates a campaign time when a campaign related to the service will be held; The abnormality determination unit executes the abnormality determination based on the indicator and the campaign time indicated by the schedule data. The abnormality determination system according to any one of claims 1 to 4.

6. The schedule data indicates a maintenance time when maintenance related to the service will be performed, The abnormality determination unit executes the abnormality determination based on the index and the maintenance time indicated by the schedule data. The abnormality determination system according to any one of claims 1 to 4.

7. The abnormality determination unit determines the abnormality determination criterion based on the learning model in which the relationship based on past performance has been learned. The abnormality determination system according to claim 2 .

8. The abnormality determination unit performs the abnormality determination based on the learning model in which the relationship based on past performance has been learned. The abnormality determination system according to claim 3 .

9. the abnormality determination unit performs the abnormality determination for each of the services to be subjected to the abnormality determination based on a relationship between the index and the schedule of the service and the index and the schedule of all of the plurality of services; The abnormality determination system according to claim 4 .

10. the abnormality determination unit performs the abnormality determination for each of the services to be subjected to the abnormality determination based on a relationship between the index and the schedule of the service and the index and the schedule of another service; The abnormality determination system according to claim 4 .

11. An index acquisition step of acquiring an index that can be used in an abnormality determination regarding an occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition step of acquiring schedule data relating to a schedule of the service; an abnormality determination step of determining an abnormality determination criterion based on a relationship between a condition related to the scheduled data and an abnormality determination criterion related to the abnormality determination when the condition is satisfied, and performing the abnormality determination based on the determined abnormality determination criterion and the indicator; An abnormality determination method comprising:

12. An index acquisition step of acquiring an index that can be used in an abnormality determination regarding an occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition step of acquiring schedule data relating to a schedule of the service; an abnormality determination step of determining the abnormality determination criterion based on the scheduled data and a learning model in which a relationship between the scheduled data for training and an abnormality determination criterion related to the abnormality determination is learned, and performing the abnormality determination based on the determined abnormality determination criterion and the index; An abnormality determination method comprising:

13. An index acquisition step of acquiring an index that can be used in an abnormality determination regarding an occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition step of acquiring schedule data relating to a schedule of the service; an anomaly determination step of executing the anomaly determination based on the index and a learning model in which a relationship between the training index and an anomaly determination result regarding the occurrence of the anomaly has been learned; An abnormality determination method comprising:

14. An index acquisition step of acquiring an index usable for abnormality determination regarding occurrence of an abnormality in each of a plurality of services in a service providing system for providing a plurality of services; a schedule data acquisition step of acquiring schedule data relating to each schedule of the plurality of services; an abnormality determination step of performing the abnormality determination for at least one of the plurality of services based on the indicators of each of the plurality of services and the schedule data of each of the plurality of services; An abnormality determination method comprising:

15. an index acquisition unit that acquires an index that can be used in determining an abnormality regarding the occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an abnormality determination unit that determines an abnormality determination criterion based on a relationship between a condition related to the scheduled data and an abnormality determination criterion related to the abnormality determination when the condition is satisfied, and executes the abnormality determination based on the determined abnormality determination criterion and the indicator; A program that makes a computer function as a

16. An index acquisition unit that acquires an index that can be used in an abnormality determination regarding an occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an abnormality determination unit that determines the abnormality determination criterion based on the scheduled data and a learning model in which a relationship between the scheduled data for training and an abnormality determination criterion related to the abnormality determination is learned, and performs the abnormality determination based on the determined abnormality determination criterion and the index; A program that makes a computer function as a

17. An index acquisition unit that acquires an index that can be used in an abnormality determination regarding the occurrence of an abnormality in a service providing system for providing a predetermined service; a schedule data acquisition unit for acquiring schedule data relating to a schedule of the service; an abnormality determination unit that executes the abnormality determination based on the index and a learning model in which a relationship between the training index and an abnormality determination result regarding the occurrence of the abnormality is learned; A program that makes a computer function as a

18. An index acquisition unit that acquires an index that can be used in abnormality determination regarding occurrence of an abnormality in each of a plurality of services in a service providing system for providing a plurality of services; a schedule data acquisition unit that acquires schedule data relating to each schedule of the plurality of services; an abnormality determination unit that executes the abnormality determination for at least one of the plurality of services based on the index of each of the plurality of services and the schedule data of each of the plurality of services; A program that makes a computer function as a