Learning device, abnormality inspection device, method for producing learning model, abnormality inspection method, and program

Abstract

To solve such a problem that, conventionally, it has been difficult to highly accurately determine the presence / absence of an abnormal operation.SOLUTION: The presence / absence of an abnormal operation can be highly accurately determined by an abnormality detection device 2 that includes: an information acquisition unit 12 that acquires transition information regarding transitions of feature quantity vectors of windows within frames of a frame set to be inspected by using a learning model acquired by a learning device 1 that includes an information acquisition unit 12 which acquires transition information regarding transitions of feature quantity vectors of windows within two or more frames included in each frame set for each of two or more frame sets, and a learning unit 13 that learns the transition information to acquire the learning model; a prediction unit 232 that acquires predicted transition information using the transition information acquired by the information acquisition unit 12; a determination unit 233 that acquires difference information regarding a difference between the transition information and the predicted transition information and acquires a determination result as to whether the difference information satisfies an abnormal condition or a normal condition; and an output unit 24 that outputs the determination result.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an abnormality detection device or the like that detects abnormalities in user operations. [Background technology]

[0002] Conventionally, there has been a technology that captures the moment of a user's screen transition from the characteristics of text and images obtained from screenshots of a personal computer (see Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Y. Ohkawa and T. Nakanishi. “Detection Method of User Behavior Transition on Computer” in: Advanced Data Mining and Applications, 2022, pp 73?85. Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the conventional technology, it is difficult to accurately determine whether or not an abnormal operation has occurred. [Means for solving the problem]

[0005] The learning device of the first invention comprises a teacher data storage unit that stores two or more frame sets, which are a collection of frames in time series of screenshots of a screen output by normal operation; an information acquisition unit that acquires, for each of the two or more frame sets, transition information regarding the transition of feature vectors of windows in each of the two or more frames of the frame set; a learning unit that learns the transition information for each of the two or more frame sets acquired by the information acquisition unit and acquires a learning model; and a storage unit that accumulates the learning model.The information acquisition unit comprises a window detection means that acquires, for each of the two or more frame sets, window images that are images of one or more windows in each of the two or more frames of the frame set; a feature extraction means that acquires, for each of the two or more frame sets, feature vectors of the window images corresponding to each of the two or more frames; and a transition information acquisition means that acquires, for each of the two or more frame sets, transition information regarding the temporal transition of feature vectors corresponding to each of the two or more frames in time series.

[0006] With this configuration, a learning model can be acquired for determining with high accuracy whether or not an abnormal operation has occurred.

[0007] Furthermore, in the learning device of the second invention, compared to the first invention, the information acquisition unit further includes grouping means for acquiring the similarity of temporally consecutive feature vectors among two or more feature vectors in a time series acquired by the feature extraction means, and acquiring a representative vector of the temporally consecutive feature vectors if the similarity satisfies a grouping condition, and the transition information acquisition means acquires transition information regarding the temporal transition of two or more feature vectors including the feature vector that is the representative vector acquired by the grouping means.

[0008] With this configuration, a learning model can be acquired to more accurately determine whether or not an abnormal operation has occurred.

[0009] Furthermore, the anomaly inspection device of the third invention is an anomaly inspection device in which, compared to the first or second invention, the feature extraction means acquires attribute values ​​of two or more window images, and does not acquire feature vectors of window images whose attribute values ​​satisfy the rejection conditions.

[0010] With this configuration, a learning model can be acquired to more accurately determine whether or not an abnormal operation has occurred.

[0011] Furthermore, the learning device of the fourth invention is a learning device according to any one of the first to third inventions, wherein the transition information acquisition means acquires, for each of two or more frame sets, a feature transition matrix which is a matrix representing information on pairs of feature vectors that occur simultaneously in one frame and information on pairs of feature vectors that occur in one frame and feature vectors that occur in the frame following the one frame, in feature vectors corresponding to each of two or more frames in the time series, and acquires transition information related to graph information having a graph structure in which the feature vectors are nodes and related feature vectors in the feature transition matrix are connected by edges from the feature transition matrix.

[0012] With this configuration, a learning model can be acquired to more accurately determine whether or not an abnormal operation has occurred.

[0013] Furthermore, the anomaly inspection device of the fifth invention is an anomaly inspection device according to any one of the first to fourth inventions, comprising: an inspection data receiving unit that receives a frame set to be inspected; an information acquisition unit that acquires transition information related to the transition of feature vectors of windows in each frame of the frame set accepted by the inspection data receiving unit; a prediction unit that performs machine learning prediction processing using a learning model accumulated by the learning device and the transition information acquired by the information acquisition unit to acquire predicted transition information, which is transition information; a judgment unit that acquires difference information related to the difference between the transition information acquired by the information acquisition unit and the predicted transition information acquired by the prediction unit, and acquires a judgment result related to whether the difference information satisfies an abnormal condition or a normal condition; and an output unit that outputs the judgment result, wherein the information acquisition unit comprises window detection means that acquires, for each of two or more frame sets, window images that are images of one or more windows in each of two or more frames included in the frame set; feature extraction means that acquires, for each of the two or more frame sets, feature vectors of the window images corresponding to each of the two or more frames; and transition information acquisition means that acquires, for each of the two or more frame sets, transition information related to the temporal transition of the feature vectors corresponding to each of two or more frames in a time series.

[0014] With this configuration, it is possible to determine with high accuracy whether or not an abnormal operation has occurred.

[0015] Furthermore, the anomaly inspection device of the sixth invention differs from the anomaly inspection device of the fifth invention in that the information acquisition unit acquires a similarity between temporally consecutive feature vectors in two or more time-series feature vectors acquired by the feature extraction means, and further includes grouping means that acquires a representative vector of the temporally consecutive feature vectors if the similarity satisfies a grouping condition, and the transition information acquisition means acquires transition information regarding the temporal transition of two or more feature vectors including the feature vector that is the representative vector acquired by the grouping means.

[0016] With this configuration, it is possible to more accurately determine whether or not an abnormal operation has occurred.

[0017] Furthermore, the anomaly inspection device of the seventh invention is an anomaly inspection device according to the fifth or sixth invention, in which the feature extraction means acquires attribute values ​​of two or more window images, and does not acquire feature vectors of window images whose attribute values ​​satisfy the rejection conditions.

[0018] With this configuration, it is possible to more accurately determine whether or not an abnormal operation has occurred.

[0019] Furthermore, the learning device of the eighth invention is a learning device according to any one of the fifth to seventh inventions, wherein the transition information acquisition means acquires, for each of two or more frame sets, a feature transition matrix which is a matrix representing information on pairs of feature vectors that occur simultaneously in one frame and information on pairs of feature vectors that occur in one frame and feature vectors that occur in the frame following the one frame, in feature vectors corresponding to each of two or more frames in the time series, and acquires transition information related to graph information having a graph structure in which the feature vectors are nodes and related feature vectors in the feature transition matrix are connected by edges from the feature transition matrix.

[0020] With this configuration, it is possible to more accurately determine whether or not an abnormal operation has occurred. [Effects of the Invention]

[0021] The abnormality detection device according to the present invention can accurately determine whether or not an abnormal operation has occurred. [Brief explanation of the drawings]

[0022] [Figure 1] Conceptual diagram of an anomaly detection system A according to the first embodiment [Figure 2] Block diagram of the learning device 1 [Figure 3] Block diagram of the anomaly detection device 2 [Figure 4] A flowchart illustrating an example of the operation of the learning device 1. [Figure 5] Flowchart illustrating an example of the feature vector acquisition process [Figure 6] Flowchart illustrating an example of the transition information acquisition process [Figure 7] Flowchart illustrating an example of a process for obtaining the feature transition matrix [Figure 8] A flowchart illustrating an example of the grouping process [Figure 9] A flowchart illustrating an example of the graph acquisition process [Figure 10] A flowchart illustrating an example of the operation of the abnormality detection device 2 [Figure 11] FIG. 10 is a diagram illustrating the window detection process. [Figure 12] A diagram explaining specific steps in the process leading up to the generation of the graph information. [Figure 13] A diagram explaining the feature transition matrix [Figure 14] A diagram showing the same graph information [Figure 15] Overview of the computer system [Figure 16] Block diagram of the computer system DETAILED DESCRIPTION OF THE INVENTION

[0023] Hereinafter, embodiments of an anomaly detection system and the like will be described with reference to the drawings. Note that components with the same reference numerals in the embodiments perform similar operations, and therefore repeated description may be omitted.

[0024] (Embodiment 1) In this embodiment, a learning device will be described that constructs a learning model that receives transition information of a graph structure and outputs predicted transition information, which will be described later.

[0025] In this embodiment, an anomaly detection device will be described that uses a learning model and a frame set to be inspected to determine whether or not an operation that constitutes the frame set is an abnormal operation.

[0026] In this specification, information X being associated with information Y means that information Y can be obtained from information X, or information X can be obtained from information Y, and the method of association is not important. Information X and information Y may be linked, may exist in the same buffer, information X may be included in information Y, or information Y may be included in information X, etc.

[0027] Furthermore, in this specification, selecting or determining information Z means obtaining information Z, obtaining a pointer to information Z, obtaining the ID of information Z, setting a flag on information Z, etc., and it is sufficient if information Z can be accessed.

[0028] 1 is a conceptual diagram of an anomaly detection system A according to this embodiment. The anomaly detection system A includes a learning device 1, an anomaly detection device 2, and one or more work terminals 3. It is preferable that the learning device 1, the anomaly detection device 2, and each of the one or more work terminals 3 be able to communicate with each other via a network such as the Internet.

[0029] The work terminal 3 is a terminal used by a user. The work terminal 3 is, for example, a terminal used by a user who performs normal operations, and is a terminal that creates training data (described later). The work terminal 3 is, for example, a terminal that is inspected to determine whether an operation is abnormal. The work terminal 3 is, for example, a so-called personal computer, tablet terminal, smartphone, robot, etc., and the type does not matter.

[0030] Fig. 2 is a block diagram of the learning device 1 according to this embodiment. Fig. 3 is a block diagram of the anomaly detection device 2 according to this embodiment.

[0031] The learning device 1 includes a teacher data storage unit 11, an information acquisition unit 12, a learning unit 13, and an accumulation unit 14. The information acquisition unit 12 includes a window detection means 121, a feature extraction means 122, a grouping means 123, and a transition information acquisition means 124. Note that the information acquisition unit 12 does not necessarily have to include the grouping means 123.

[0032] The anomaly detection device 2 includes a learning model storage unit 21, an inspection data receiving unit 22, a processing unit 23, and an output unit 24. The processing unit 23 includes an information acquisition unit 12, a prediction unit 232, and a determination unit 233.

[0033] The training data storage unit 11 constituting the learning device 1 stores two or more training data. Training data is information that serves as the basis for acquiring a learning model. Here, the training data is a frame set. A frame set here is typically a collection of time-series frames of screenshots of a screen output by normal operation. In other words, the training data here is typically a positive example. Furthermore, the screen is typically a computer screen on which an operator performs some operation. A frame is a still image.

[0034] The information acquisition unit 12 acquires transition information for each of two or more frame sets. The transition information is information related to screen transitions. The transition information is information related to the temporal transition of feature vectors corresponding to each of two or more frames in a time series. The transition information is information related to the transition of feature vectors of windows in each of two or more frames included in a frame set. The transition information is, for example, graph information, which will be described later. The transition information is, for example, information indicating the features of the graph information, which will be described later. The transition information is, for example, a feature transition matrix, which will be described later.

[0035] The information acquiring unit 12 acquires transition information, which is information relating to the transition of the feature vector of a window in each frame of a frame set accepted by the test data accepting unit 22, which will be described later.

[0036] The window detection means 121 acquires one or more window images in each of two or more frames contained in the frame set. A window image is an image of a window in a frame. It is preferable that the window detection means 121 acquires one or more window images in each of two or more frames contained in the frame set for each of two or more frame sets. Note that a window image is an image of a window in a frame. In other words, two or more windows may exist in a screenshot. Note that if the screen does not have a window, the window detection means 121 acquires, for example, a screenshot of the screen. In such a case, the screenshot is the window image.

[0037] The window detection means 121 acquires one or more window images from each of two or more frames included in the accepted frame set to be inspected.

[0038] The window detection means 121 can be realized by known techniques, for example, by the FindContour function of OpenCV.

[0039] The feature extraction means 122 acquires feature vectors of window images corresponding to each of two or more frames included in the frame set. For example, the feature extraction means 122 acquires feature vectors of window images corresponding to each of two or more frames for each of two or more frame sets. The processing of the feature extraction means 122 to acquire image feature vectors can be realized by known technology. The processing performed by the window detection means 121 can be realized, for example, by using ResNetRS50, which has been pre-trained on ImageNet.

[0040] It is preferable that the feature extraction means 122 resizes the window image to a fixed size and acquires the feature vector from the resized image. The fixed size is, for example, "512 x 512".

[0041] A feature vector is information having two or more feature amounts of a window image. A feature amount is a feature amount of an image. A feature amount may also be called feature information. The feature amount possessed by a feature vector does not matter. A feature vector is, for example, a 512-dimensional vector, but does not matter.

[0042] The feature extraction means 122 preferably acquires attribute values ​​of two or more window images, and does not acquire feature vectors of window images whose attribute values ​​satisfy the rejection condition. Note that the attribute values ​​here are, for example, information indicating the size of the window (e.g., the size of the area, the number of vertical dots, or the number of horizontal dots), or whether the window is rectangular (whether there is a part hidden by another window). Note that when the feature extraction means 122 does not acquire a feature vector, that means that the transition information acquisition means 124 does not adopt that feature vector when acquiring transition information.

[0043] The rejection conditions are conditions for rejecting a window image. The rejection conditions are, for example, conditions related to the window size or window shape. The rejection conditions are, for example, that the window size is less than or equal to a threshold value. The rejection conditions are, for example, that the window shape is not rectangular.

[0044] The grouping means 123 acquires a similarity between temporally consecutive feature vectors among two or more time-series feature vectors acquired by the feature extraction means 122, and acquires a representative vector of the temporally consecutive feature vectors if the similarity satisfies a grouping condition. The two or more time-series feature vectors are feature vectors corresponding to two or more frames in the time series. The feature vector corresponding to a frame is a feature vector acquired from a window image in the frame.

[0045] The grouping means 123 typically obtains the similarity between two corresponding feature vectors for each of two temporally consecutive frames, and obtains a representative vector of the temporally consecutive feature vectors if the similarity satisfies a grouping condition. When two or more windows exist in each of two temporally consecutive frames, the two feature vectors corresponding to each of the two consecutive frames are the feature vectors obtained from each of the two corresponding windows. The two corresponding windows are windows at the same position on the screen or windows with the same or similar display. Whether two windows are similar is determined by whether the similarity between the feature vectors obtained from each of the two window images is equal to or greater than a threshold.

[0046] The grouping condition is a condition for grouping feature vectors. For example, the grouping condition is that the similarity between two temporally consecutive feature vectors is equal to or greater than a threshold. The representative vector is, for example, an average vector of the two feature vectors. Note that the two temporally consecutive feature vectors are feature vectors acquired from window images of two temporally consecutive frames.

[0047] The transition information acquisition means 124 acquires transition information relating to the temporal transition of feature vectors corresponding to two or more frames in the time series of the frame set. The transition information is, for example, graph information or a feature transition matrix.

[0048] A feature transition matrix is ​​a matrix that represents, for feature vectors corresponding to two or more frames in a time series, information on a set of feature vectors that occur simultaneously in one frame and information on a set of feature vectors that occur in one frame and a feature vector that occurs in the frame following the one frame. Note that if a frame has only one window, there is no set of feature vectors that occur simultaneously in that frame. If a frame does not have a window, the feature vector obtained is the feature vector of the entire frame. Note that the feature transition matrix can be expressed as a vector.

[0049] The information on the set of feature vectors is, for example, a set of identifiers of the feature vectors (for example, (feature vector 1, feature vector 2)).

[0050] The transition information acquisition means 124 acquires, for example, for each of two or more frame sets, transition information relating to the temporal transition of the feature vectors corresponding to each of two or more frames in the time series.

[0051] The transition information acquisition means 124 acquires, for example, transition information relating to the temporal transition of two or more feature amount vectors including the feature amount vector that is the representative vector acquired by the grouping means 123 .

[0052] The transition information acquisition means 124 acquires, for example, for each of two or more frame sets, information on a set of feature vectors that occur simultaneously in one frame, among feature vectors corresponding to two or more frames in a time series. A matrix acquired from such a set is called a simultaneous feature matrix. Furthermore, the transition information acquisition means 124 acquires, for example, information on a set of a feature vector that occurs in one frame and a feature vector that occurs in the frame following the one frame. A matrix acquired from such a set is called a time-step feature matrix. Next, the transition information acquisition means 124 acquires a feature transition matrix using the simultaneous feature matrix and the time-step feature matrix. Note that the feature transition matrix is, for example, the sum of the simultaneous feature matrix and the time-step feature matrix. However, the feature transition matrix may also be a matrix obtained by ORing the simultaneous feature matrix and the time-step feature matrix, or a matrix composed of elements obtained by weighted averaging each element of the simultaneous feature matrix and each element of the time-step feature matrix. The feature transition matrix may be information that expresses information on a set of feature vectors that occur simultaneously in two frames, and information on a set of a feature vector that occurs in one frame and a feature vector that occurs in the frame following the one frame.

[0053] Next, the transition information acquiring means 124 acquires, for example, graph information having a graph structure in which feature vectors are nodes from the feature transition matrix and feature vectors related to each other in the feature transition matrix are connected by edges. Such graph information is an example of transition information. Note that the feature transition matrix may be used as the transition information acquired by the transition information acquiring means 124.

[0054] The transition information acquiring means 124 may acquire transition information that is a vector indicating the characteristics of the graph information. Such a vector is called a transition information feature vector. The transition information feature vector is, for example, a 512-dimensional vector.

[0055] The learning unit 13 learns the transition information for each of the two or more frame sets acquired by the information acquisition unit 12, and acquires a learning model. The learning unit 13 usually performs a machine learning learning process to acquire the learning model. The learning unit 13, for example, learns the features of graph information having a graph structure, and acquires the learning model. The learning model here is, for example, a Graph Autoencoder model. Note that the learning unit 13 and the Graph Autoencoder model are well-known technologies. The learning unit 13 uses, for example, an Encoder with three layers of GIN Conv and a Decoder with a linear layer. Also, the learning model is, for example, a model that receives transition information and outputs a transition information feature vector. Note that the learning model may also be called a learner or the like.

[0056] The storage unit 14 stores the learning model acquired by the learning unit 13. There is no restriction on the area or device in which the storage unit 14 stores the learning model. The storage unit 14 stores the learning model in the learning model storage unit 21 of the anomaly detection device 2, for example.

[0057] A learning model is stored in the learning model storage unit 21 constituting the anomaly detection device 2. This learning model is usually a learning model acquired by the learning device 1. Note that the learning model storage unit 21 may store abnormal conditions or normal conditions.

[0058] The test data receiving unit 22 receives a frame set to be tested. For example, the test data receiving unit 22 receives a frame set acquired at the work terminal 3 during work. In other words, the anomaly detection device 2 may detect anomalies while the user is working. For example, the test data receiving unit 22 receives the frame set from the work terminal 3 after the work at the work terminal 3 is completed. In other words, it is preferable to check whether the user's operation was abnormal when the user finishes their work. For example, the test data receiving unit 22 receives the frame set from the work terminal 3 when the work terminal 3 logs out from a server (not shown) that corresponds to the work. The functions of the anomaly detection device 2 may be provided by the work terminal 3.

[0059] Here, reception is a concept that includes reception of information input from input devices such as a keyboard, mouse, or touch panel, reception of information transmitted via a wired or wireless communication line, and reception of information read from recording media such as an optical disk, magnetic disk, or semiconductor memory.

[0060] The processing unit 23 performs various types of processing. The various types of processing are, for example, processing performed by the information acquisition unit 12, the prediction unit 232, or the determination unit 233.

[0061] The prediction unit 232 performs machine learning prediction processing and acquires predicted transition information using the learning model in the learning model storage unit 21 and the transition information acquired by the information acquisition unit 12. The predicted transition information has the same structure as the transition information, and is, for example, a 512-dimensional vector.

[0062] The judgment unit 233 acquires difference information between the transition information acquired by the information acquisition unit 12 and the predicted transition information acquired by the prediction unit 232, and acquires a judgment result as to whether the difference information satisfies an abnormal condition or a normal condition.

[0063] The difference information is information about the difference between the transition information acquired by the information acquisition unit 12 and the predicted transition information acquired by the prediction unit 232. The difference information is, for example, the mean square error between the transition information that is a vector and the predicted transition information that is also a vector. The difference information is, for example, the distance or similarity between the transition information that is a vector and the predicted transition information that is also a vector.

[0064] The abnormal condition is a condition for detecting whether an operation is abnormal, such as when the difference information is equal to or greater than a threshold value.

[0065] The normal condition is a condition for detecting whether an operation is normal, that is, the difference information is equal to or less than a threshold value.

[0066] The output unit 24 outputs the determination result acquired by the determination unit 233. Here, output is a concept that includes display on a display, projection using a projector, printing on a printer, sound output, transmission to an external device, storage in a recording medium, and delivery of the processing result to another processing device, another program, etc.

[0067] The teacher data storage unit 11 and the learning model storage unit 21 are preferably non-volatile recording media, but can also be realized as volatile recording media.

[0068] There is no restriction on the process by which information is stored in the teacher data storage unit 11 etc. For example, information may be stored in the teacher data storage unit 11 etc. via a recording medium, information transmitted via a communication line etc. may be stored in the teacher data storage unit 11 etc., or information input via an input device may be stored in the teacher data storage unit 11 etc.

[0069] The information acquisition unit 12, learning unit 13, storage unit 14, information acquisition unit 12, window detection means 121, feature extraction means 122, grouping means 123, transition information acquisition means 124, processing unit 23, prediction unit 232, and judgment unit 233 can usually be realized by a processor, memory, etc. The processing procedures of the information acquisition unit 12, etc. are usually realized by software, and the software is recorded on a recording medium such as a ROM. However, they may also be realized by hardware (dedicated circuit). The processor may be a CPU, MPU, GPU, etc., and the type is not important.

[0070] The output unit 24 may or may not include an output device such as a display, a speaker, etc. The output unit 24 may be realized by driver software for an output device, or by a combination of driver software for an output device and the output device, etc.

[0071] Next, an example of the operation of the learning device 1 will be described using the flowchart in Figure 4. It is assumed in Figure 4 that, for example, screenshots of a work terminal 3 used by a user performing normal operations are continuously sent to the learning device 1 from a large number of work terminals 3, and that the learning device 1 accumulates a frame set, which is a collection of frames that are the time-series screenshots. In other words, it is assumed that the learning device 1 stores the teacher data sent from each of two or more work terminals 3 in the teacher data storage unit 11. The trigger for starting the process is, for example, receipt of a start instruction from a user, but this does not matter.

[0072] (Step S401) The information acquiring unit 12 assigns 1 to a counter i.

[0073] (Step S402) The information acquisition unit 12 determines whether or not the i-th frame set exists in the teacher data storage unit 11. If the i-th frame set exists, the process proceeds to step S403; if not, the process proceeds to step S409.

[0074] (Step S403) The information acquiring unit 12 assigns 1 to a counter j.

[0075] (Step S404) The information acquisition unit 12 determines whether the jth frame exists in the ith frame set. If the jth frame exists, the process proceeds to step S405, and if not, the process proceeds to step S407.

[0076] (Step S405) The information acquisition unit 12 acquires one or more feature vectors corresponding to the j-th frame in the i-th frame set. An example of such feature vector acquisition processing will be described with reference to the flowchart in FIG.

[0077] (Step S406) The information acquiring unit 12 increments the counter j by 1. The process returns to step S404.

[0078] (Step S407) The information acquisition unit 12 acquires transition information for the i-th frame set. An example of the transition information acquisition process will be described with reference to the flowchart in FIG.

[0079] (Step S408) The information acquiring unit 12 increments the counter i by 1. The process returns to step S402.

[0080] (Step S409) The learning unit 13 performs a learning process using the two or more pieces of transition information acquired in step S407, and acquires a learning model.

[0081] (Step S410) The storage unit 14 stores the learning model acquired in step S409, and the process ends.

[0082] Next, an example of the feature vector acquisition process in step S405 will be described with reference to the flowchart in FIG.

[0083] (Step S501) Window detection means 121 acquires a frame to be processed. The frame to be processed is the j-th frame of the i-th frame set in step S404 of FIG.

[0084] (Step S502) The window detection means 121 acquires one or more window images from the frame acquired in step S501.

[0085] (Step S503) The feature extraction means 122 determines whether or not one or more window images have been acquired in step S502. If one or more window images have been acquired, the process proceeds to step S504; if not, the process proceeds to step S510.

[0086] (Step S504) The feature extraction means 122 assigns 1 to a counter k.

[0087] (Step S505) The feature extraction means 122 determines whether or not the kth window image exists among the window images acquired in step S502. If the kth window image exists, the process proceeds to step S506, and if not, the process returns to the upper process.

[0088] (Step S506) The feature extraction means 122 acquires one or more attribute values ​​of the k-th window image.

[0089] (Step S507) The feature extraction means 122 determines whether or not one or more attribute values ​​acquired in step S506 satisfy the rejection condition. If the rejection condition is satisfied, the process proceeds to step S509; if the rejection condition is not satisfied, the process proceeds to step S508.

[0090] (Step S508) The feature extraction unit 122 acquires the feature vector of the k-th window image and associates it with the frame to be processed. Associating it with the frame to be processed means, for example, associating it with the identifier of the frame to be processed.

[0091] (Step S509) The feature extraction means 122 increments the counter k by 1. The process returns to step S505.

[0092] (Step S510) The feature extraction means 122 acquires a frame to be processed. Note that the frame here is a window image.

[0093] (Step S511) The feature extraction unit 122 acquires the feature vector of the frame acquired in step S510, associates it with the frame to be processed, and returns to the upper level process.

[0094] Next, an example of the transition information acquisition process in step S407 will be described with reference to the flowchart in FIG.

[0095] (Step S601) The transition information acquisition means 124 performs a feature transition matrix acquisition process. An example of the feature transition matrix acquisition process will be described using the flowchart in Fig. 7. The feature transition matrix acquisition process is a process for acquiring a feature transition matrix that indicates the transition status of frames in a frame set.

[0096] (Step S602) The transition information acquisition means 124 performs graph acquisition processing. An example of the graph acquisition processing will be described using the flowchart in Fig. 9. The graph acquisition processing is processing to acquire graph information that shows the transition status of frames in a frame set in a graph structure.

[0097] It should be noted that step S602 may be omitted in the flowchart of Fig. 6. In other words, the acquired transition information may be a feature transition matrix.

[0098] Next, an example of the feature transition matrix obtaining process in step S601 will be described with reference to the flowchart in FIG.

[0099] (Step S701) The transition information acquiring means 124 assigns 1 to a counter i.

[0100] (Step S702) The transition information acquisition means 124 determines whether the i-th frame exists in the frame set to be processed. If the i-th frame exists, the process proceeds to step S703; if not, the process proceeds to step S705.

[0101] (Step S703) The transition information acquisition means 124 performs grouping processing with the i-th frame as the head. Here, there are cases where grouping is not possible. An example of such grouping processing will be described with reference to the flowchart in FIG. 8.

[0102] (Step S704) The transition information acquisition means 124 advances i to 1 or a value corresponding to the frame next to the frame that has been successfully grouped. The process returns to step S702. In other words, the i-th frame corresponding to the new i is a frame that has not yet been subjected to grouping processing.

[0103] (Step S705) The transition information acquiring means 124 assigns 1 to the counter j.

[0104] (Step S706) The transition information acquiring means 124 determines whether or not the j-th feature vector set exists. If the j-th feature vector set exists, the process proceeds to step S707, and if not, the process proceeds to step S711.

[0105] The jth feature vector set is information corresponding to a frame or a group of frames. A feature vector set consists of one or more feature vectors. If there is only one window or zero windows in a frame, the feature vector set consists of one feature vector. If there are two or more window images used in a frame, the feature vector set consists of two or more feature vectors corresponding to the window images.

[0106] (Step S707) The transition information acquisition means 124 acquires identifiers of one or more feature vectors in the j-th feature vector set (hereinafter referred to as feature vector identifiers). When there are two or more feature vector identifiers among the acquired feature vector identifiers, the transition information acquisition means 124 acquires one or more sets of all combinations of the two or more feature vector identifiers. For example, when the transition information acquisition means 124 acquires feature vector identifiers "1", "2", and "3", it acquires three sets of (1,2), (1,3), and (2,3). Each set is a set of feature vector identifiers corresponding to co-occurring windows in one frame.

[0107] (Step S708) The transition information acquisition means 124 determines whether the (j+1)th feature vector set exists. If the (j+1)th feature vector set exists, the process proceeds to step S709; if not, the process proceeds to step S710.

[0108] (Step S709) The transition information acquiring means 124 acquires one or more pairs of one or more feature vector identifiers of the transition source and one or more feature vector identifiers of the transition destination.

[0109] When one or more feature vector identifiers of the transition source are "1", "2", and "3", and one or more feature vector identifiers of the transition destination are "4", the transition information acquisition means 124 acquires three sets (1,4), (2,4), and (3,4). When one or more feature vector identifiers of the transition source are "1", and one or more feature vector identifiers of the transition destination are "4" and "5", the transition information acquisition means 124 acquires three sets (1,4), and (1,5). When one or more feature vector identifiers of the transition source are "1" and "2", and one or more feature vector identifiers of the transition destination are "4" and "5", the transition information acquisition means 124 acquires four sets (1,4), (2,4), (1,5), and (2,5).

[0110] (Step S710) The transition information acquiring means 124 increments the counter j by 1. The process returns to step S706.

[0111] (Step S711) The transition information obtaining means 124 assigns 1 to the counter i.

[0112] (Step S712) The transition information acquisition means 124 determines whether or not the i-th pair exists among the two or more pairs acquired in step S707 or step S709. If the i-th pair exists, the process proceeds to step S713; if not, the process returns to the upper level process.

[0113] (Step S713) The transition information acquisition means 124 records values ​​in the matrix elements, which are elements corresponding to the two feature vector identifiers of the i-th pair. A feature transition matrix is ​​constructed by recording such values. For example, the pair information indicates (row, column) or (column, row). In other words, the pair (1, 4) indicates that a value is entered in the element in the first row and fourth column or the element in the first column and fourth row.

[0114] Here, even if the same pair appears two or more times, the transition information acquiring means 124 may set the value corresponding to the pair to "1", or may set the value corresponding to the pair to the number of times the same pair appears when the same pair appears two or more times. Note that the initial value of each element of the feature transition matrix is ​​"0".

[0115] (Step S714) The transition information acquiring means 124 increments the counter i by 1. The process returns to step S712.

[0116] Next, an example of the grouping process in step S703 will be described with reference to the flowchart in FIG.

[0117] (Step S801) The grouping means 123 acquires the i-th frame of step S702. It also assigns the value "OFF (for example, "0")" to the variable grouping flag. The grouping flag is a flag that indicates whether or not to group frames. It can be said that grouping of frames is grouping of feature vector sets corresponding to frames. Grouping of feature vector sets corresponding to frames is grouping of feature vector sets corresponding to two or more temporally consecutive frames.

[0118] (Step S802) The grouping means 123 acquires one or more feature vectors corresponding to the frame acquired in step S801 or step S816.

[0119] (Step S803) The grouping means 123 acquires one or more feature vectors corresponding to the frame temporally next to the frame acquired in step S801.

[0120] (Step S804) The grouping means 123 determines whether the number of feature vectors included in the feature vector set acquired in step S802 is the same as the number of feature vectors included in the feature vector set acquired in step S803. If they are the same, the process proceeds to step S805, and if they are not the same, the process proceeds to step S810.

[0121] (Step S805) The grouping means 123 assigns 1 to the counter j.

[0122] (Step S806) The grouping means 123 determines whether or not there are feature vectors for the two corresponding window images in each of the two frames. If there are corresponding feature vectors, the process proceeds to step S807; if there are no corresponding feature vectors, the process proceeds to step S815.

[0123] (Step S807) The grouping means 123 calculates the similarity between the two corresponding feature vectors.

[0124] (Step S808) The grouping means 123 determines whether or not the similarity acquired in step S807 satisfies the grouping condition. If the grouping condition is satisfied, the process proceeds to step S809, and if not, the process proceeds to step S810.

[0125] (Step S809) The grouping means 123 increments the counter j by 1. The process returns to step S806.

[0126] (Step S810) The grouping means 123 determines whether the variable "grouping flag" is ON. If it is ON, the process proceeds to step S811, and if it is OFF, the process returns to the upper level process.

[0127] (Step S811) The grouping means 123 assigns 1 to a counter k.

[0128] (Step S812) The grouping means 123 determines whether or not there exists a set of k-th feature vectors in two or more temporally consecutive frames that should be grouped (satisfying the grouping conditions). If there exists a set of k-th feature vectors, the process proceeds to step S813; if there does not exist a set, the process returns to the upper level process.

[0129] (Step S813) The grouping means 123 acquires a set of k-th feature vectors in each of two or more temporally consecutive frames to be grouped. The grouping means 123 acquires a representative vector of the two or more feature vectors in the set of feature vectors.

[0130] (Step S814) The grouping means 123 increments the counter k by 1. The process returns to step S812.

[0131] (Step S815) The grouping means 123 assigns "ON" to the variable "grouping flag".

[0132] (Step S816) The grouping means 123 obtains the next frame, and the process returns to step S802.

[0133] Next, an example of the graph acquisition process in step S602 will be described with reference to the flowchart in FIG.

[0134] (Step S901) The transition information obtaining means 124 assigns 1 to a counter i.

[0135] (Step S902) The transition information acquisition means 124 determines whether or not the identifier of the i-th feature vector exists in the feature transition matrix. If the identifier exists, the process proceeds to step S903, and if not, the process returns to the upper process.

[0136] (Step S903) The transition information acquisition means 124 determines whether or not a node corresponding to the identifier of the i-th feature vector already exists. If the node exists, the process proceeds to step S905; if not, the process proceeds to step S904. Note that if the node already exists, it means that the node has already been generated by the processing of step S904.

[0137] (Step S904) The transition information acquisition means 124 generates a node corresponding to the identifier of the i-th feature vector. Generating a node means, for example, adding a record of the node to a table that manages nodes. Note that generating a node is one step for constructing graph information, and the processing differs depending on the structure of the graph information. The record of a node may include, for example, an identifier of a transition destination node. The record of a record may further include, for example, an identifier of an edge corresponding to the identifier of the transition destination node.

[0138] (Step S905) The transition information acquiring means 124 acquires the identifiers of one or more other feature vectors associated with the identifier of the i-th feature vector in the feature transition matrix.

[0139] The correspondence between the identifier of a feature vector (identifier 1) and the identifier of one or more other feature vectors (identifier 2) means that, for example, in the feature transition matrix, the value of the element whose row number is "identifier 1" and whose column number is "identifier 2" is a value other than 0 (e.g., "1"), or the value of the element whose row number is "identifier 2" and whose column number is "identifier 1" is a value other than 0 (e.g., "1").

[0140] (Step S906) The transition information acquiring means 124 assigns 1 to the counter j.

[0141] (Step S907) The transition information acquisition means 124 determines whether or not the j-th identifier exists among the identifiers of the one or more other feature vectors acquired in step S905. If the j-th identifier exists, the process proceeds to step S908; if not, the process proceeds to step S914.

[0142] (Step S908) The transition information acquisition means 124 determines whether or not a node corresponding to the identifier of another j-th feature vector exists. If the node exists, the process proceeds to step S910; if not, the process proceeds to step S909.

[0143] (Step S909) The transition information acquiring means 124 generates a node corresponding to the identifier of another j-th feature vector.

[0144] (Step S910) The transition information acquisition means 124 determines whether an edge already exists connecting a node corresponding to the identifier of the i-th feature vector with a node corresponding to the identifier of another j-th feature vector. If the edge exists, the process proceeds to step S912; if not, the process proceeds to step S911. Note that the edge already exists when the edge has already been generated by the processing of step S911.

[0145] (Step S911) The transition information acquiring means 124 generates an edge connecting a node corresponding to the identifier of the i-th feature vector with a node corresponding to the identifier of another j-th feature vector. Note that the weight of the edge here is usually “1.”

[0146] Note that creating an edge is one step in constructing graph information, and the process differs depending on the structure of the graph information. Creating an edge means, for example, adding a record of the edge to a table that manages edges. The record of an edge can be, for example, The edge record includes an identifier of the transition source node and an identifier of the transition destination node. The edge record may further include, for example, a weight.

[0147] (Step S912) The transition information acquisition means 124 increases the weight of the edge connecting the node corresponding to the identifier of the i-th feature vector and the node corresponding to the identifier of another j-th feature vector. Note that the transition information acquisition means 124 normally adds "1" to the weight of the edge. Here, the transition information acquisition means 124 does not need to change the weight of the edge. If the transition information acquisition means 124 does not change the weight of the edge, the weight of the edge in the graph structure is always constant (normally "1").

[0148] (Step S913) The transition information acquiring means 124 increments the counter j by 1. The process returns to step S907.

[0149] (Step S914) The transition information acquiring means 124 increments the counter i by 1. The process returns to step S902.

[0150] Next, an example of the operation of the abnormality detection device 2 will be described with reference to the flowchart of FIG.

[0151] (Step S1001) The test data receiving unit 22 determines whether or not a frame set to be tested has been received. If a frame set has been received, the process proceeds to step S1002, and if not, the process returns to step S1001.

[0152] (Step S1002) The information acquiring unit 12 assigns 1 to a counter j.

[0153] (Step S1003) The information acquisition unit 12 determines whether or not the j-th frame exists in the frame set received in step S1001. If the j-th frame exists, the process proceeds to step S1004; if not, the process proceeds to step S1006.

[0154] (Step S1004) The information acquisition unit 12 acquires a feature vector corresponding to the j-th frame in the frame set. An example of such feature vector acquisition processing has been described with reference to the flowchart in FIG.

[0155] (Step S1005) The information acquiring unit 12 increments the counter i by 1. The process returns to step S1003.

[0156] (Step S1006) The information acquiring unit 12 acquires transition information for the frame set to be inspected. An example of the transition information acquiring process has been described with reference to the flowchart of FIG.

[0157] (Step S1007) The prediction unit 232 acquires the learning model from the learning model storage unit 21.

[0158] (Step S1008) The prediction unit 232 performs machine learning prediction processing using the learning model acquired in step S1007 and the transition information acquired in step S1006, and acquires predicted transition information.

[0159] (Step S1009) The determination unit 233 obtains difference information between the predicted transition information obtained in step S1008 and the transition information obtained in step S1006.

[0160] (Step S1010) The determination unit 233 determines whether or not the difference information satisfies the abnormal condition. If the abnormal condition is satisfied, the process proceeds to step S1011, and if not, the process proceeds to step S1012.

[0161] (Step S1011) The determination unit 233 acquires the determination result "abnormal." The process proceeds to step S1013.

[0162] (Step S1012) The determination unit 233 acquires the determination result "normal."

[0163] (Step S1013) The output unit 24 outputs the determination result. The process returns to step S1001.

[0164] In the flowchart of FIG. 10, the process ends when the power is turned off or an interrupt occurs to end the process.

[0165] Specific examples of the operation of the anomaly detection system A in this embodiment will be described below. Two specific examples will be described below. Specific Example 1 is a processing example of the learning device 1. Specific Example 2 is a processing example of the anomaly detection device 2.

[0166] (Example 1) Now, a frame set, which is a collection of time-series frames including each of the frames (a) and (b) in Figure 11, is stored in the training data storage unit 11 of the learning device 1. This frame set is training data of positive examples acquired at an operating terminal 3 that performed normal operations. It is also assumed that the training data storage unit 11 also stores one or more other frame sets.

[0167] Then, assume that the learning device 1 receives a start instruction from the user. Then, as shown below, the learning device 1 acquires transition information related to the graph information from each of two or more frame sets. The transition information related to the graph information may be graph information itself, or a transition information feature vector related to the features of the graph information.

[0168] That is, the window detection means 121 extracts a window image ((c) in FIG. 11) from each frame (screenshot) of, for example, (a) or (b) in FIG. 11. Here, the window detection means 121 extracts the contour of the frame of (a) in FIG. 11 using, for example, the FindContour function of OpenCV, and obtains the window image of (c) in FIG.

[0169] Next, the feature extraction means 122 acquires a feature vector of the window image acquired by the window detection means 121. The feature extraction means 122 uses, for example, "ResNetRS50," a CNN model that has been pre-trained on "ImageNet," a large-scale image dataset, to resize the window image of FIG. 11(c) and acquire a 512-dimensional feature vector from the resized image. Note that the feature extraction means 122 resizes the window image to, for example, 512 × 512.

[0170] Next, the grouping means 123 groups the frames using one or more feature vectors corresponding to each frame by the grouping process described above in Fig. 8. Note that there may be some frames that cannot be grouped.

[0171] 12, the feature extraction means 122 obtains feature vectors (which may be representative vectors) F1 and F2 corresponding to frame "t=1." The feature extraction means 122 also obtains feature vectors (which may be representative vectors) F3 corresponding to frame "t=2," which is the frame temporally following "t=1." The feature extraction means 122 also obtains feature vectors (which may be representative vectors) F1 and F4 corresponding to frame "t=3," which is the frame temporally following "t=2."

[0172] Next, the transition information acquisition means 124 generates the following graph information based on the transition information of the frame or group (FIG. 12): A group is a set of two or more frames that satisfy the group conditions.

[0173] That is, the transition information acquisition means 124 acquires a set of feature vectors ("F1" and "F2") that occur simultaneously (co-occur) for frame "t=1" and a set of feature vectors ("F1" and "F4") that occur simultaneously (co-occur) for frame "t=3." Then, based on the set of feature vectors ("F1" and "F2"), the transition information acquisition means 124 sets the elements of the "F1" row and "F2" column and the "F2" row and "F1" column of the matrix to "1." Also, based on the set of feature vectors ("F1" and "F4"), the transition information acquisition means 124 sets the elements of the "F1" row and "F4" column and the "F4" row and "F1" column of the matrix to "1." Note that the initial value of each element of this matrix is ​​"0." A matrix based on such co-occurring feature vectors in each frame or group is called a simultaneous feature matrix, and is shown in FIG. 13(a).

[0174] Furthermore, since there is a transition from frame "t=1" to frame "t=2", the transition information acquisition means 124 assumes that there is a transition from feature vector "F1" to feature vector "F3", and also that there is a transition from feature vector "F2" to feature vector "F3", and acquires a pair of information relating to the transition of the feature vectors ("F1", "F3") and ("F2", "F3"). Furthermore, since there is a transition from frame "t=2" to frame "t=3", the transition information acquisition means 124 assumes that there is a transition from feature vector "F3" to feature vector "F1", and also that there is a transition from feature vector "F3" to feature vector "F4", and acquires a pair of information relating to the transition of the feature vectors ("F1", "F3") and ("F3", "F4"). Then, based on the pair of feature vectors ("F1", "F3"), the transition information acquisition means 124 sets the elements of the "F1" row and "F3" column and the "F3" row and "F1" column of the matrix to "1". Also, based on the pair of feature vectors ("F3", "F4"), the transition information acquisition means 124 sets the elements of the "F3" row and "F4" column and the "F4" row and "F3" column of the matrix to "1". Note that the initial value of each element of this matrix is ​​"0". A matrix based on the feature vectors indicating the transition in each frame or group is called a time-step feature matrix, and is shown in FIG. 13(b).

[0175] Next, the transition information acquisition means 124 adds the simultaneous feature matrix and the time-step feature matrix to obtain a feature transition matrix (FIG. 13(c)). Note that the transition information acquisition means 124 only needs to obtain a feature transition matrix that reflects the simultaneous feature matrix and the time-step feature matrix, and the calculation method is not important.

[0176] Next, the transition information acquisition means 124 converts the feature transition matrix into graph information with a graph structure. The graph information obtained by the transition information acquisition means 124 from the feature transition matrix of FIG. 13 is shown in FIG. 14. FIG. 14 graphically illustrates the graph information. The graph information of FIG. 14 can be expressed, for example, as (connection from F1 to F2, connection from F1 to F3, connection from F1 to F4, connection from F2 to F1, connection from F2 to F3, connection from F2 to F4, connection from F3 to F1, connection from F3 to F2, connection from F3 to F4, connection from F4 to F1, connection from F4 to F2, connection from F4 to F3) = (1,1,1,1,1,0,1,0,1,1,0.0). 14 can be expressed as, for example, (first row of feature transition matrix, second row of feature transition matrix, third row of feature transition matrix, fourth row of feature transition matrix) = (0,1,1,1,1,0,1,0,1,0,0,1,1,0,0,0). Through the above processing, the information acquisition unit 12 can acquire transition information related to the graph information.

[0177] The graph information in Fig. 14 is an example of transition information, but the transition information may be the feature transition matrix in Fig. 13(c).

[0178] Then, the information acquisition unit 12 acquires transition information related to the graph information from each of the other one or more frame sets in the teacher data storage unit 11.

[0179] Next, the learning unit 13 performs machine learning learning processing using the two or more pieces of graph information acquired by the information acquisition unit 12, and acquires a learning model.

[0180] Next, the storage unit 14 stores the learning model acquired by the learning unit 13. Through the above processing, the learning model used by the anomaly detection device 2 has been constructed.

[0181] (Example 2) The anomaly detection device 2 uses the learning model constructed in the specific example 1 to perform anomaly detection as follows.

[0182] That is, the inspection data receiving unit 22 of the abnormality detection device 2 receives a frame set, which is a collection of screenshots acquired when a user operates the work terminal 3 (for example, a personal computer).

[0183] Next, the information acquisition unit 12 of the anomaly detection device 2 acquires graph information from the received frame set through the process described in the first specific example.

[0184] Next, the prediction unit 232 performs machine learning prediction processing using the graph information and the learning model to acquire a feature vector of the graph structure. The feature vector of the graph structure is, for example, a 512-dimensional vector.

[0185] Next, the determination unit 233 calculates the mean square error loss between the feature vector of the graph structure acquired by the prediction process and the feature vector of the input data. The mean square error loss is the anomaly score. Next, the determination unit 233 determines whether the anomaly score satisfies the anomaly condition (anomaly score >= threshold A) and acquires the determination result.

[0186] Next, the output unit 24 outputs the determination result.

[0187] As described above, according to this embodiment, it is possible to acquire a learning model for determining with high accuracy whether an operation is abnormal or not.

[0188] Moreover, according to this embodiment, it is possible to determine with high accuracy whether or not an operation is abnormal.

[0189] The processing in this embodiment may be realized by software. This software may be distributed by software download or the like. This software may also be recorded on a recording medium such as a CD-ROM and distributed. This also applies to the other embodiments in this specification. The software that realizes the learning device 1 in this embodiment is the following program. In other words, this program causes a computer that can access a teacher data storage unit in which two or more frame sets, which are a collection of frames in chronological order of screenshots of a screen output by normal operation, are stored to function as an information acquisition unit that acquires, for each of the two or more frame sets, transition information regarding the transition of feature vectors of windows in each of the two or more frames of the frame set; a learning unit that learns the transition information for each of the two or more frame sets acquired by the information acquisition unit and acquires a learning model; and a storage unit that accumulates the learning model, wherein the information acquisition unit is equipped with window detection means that acquires, for each of the two or more frame sets, window images that are images of one or more windows in each of the two or more frames of the frame set; feature extraction means that acquires, for each of the two or more frame sets, feature vectors of the window images corresponding to each of the two or more frames; and transition information acquisition means that acquires, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vectors corresponding to each of the two or more frames in chronological order.

[0190] Furthermore, software that realizes the anomaly detection device 2 in this embodiment is the following program: That is, this program causes a computer to include an inspection data receiving unit that receives a frame set to be inspected, an information acquiring unit that acquires transition information related to transitions of feature vectors of windows in each frame of the frame set accepted by the inspection data receiving unit, a prediction unit that performs machine learning prediction processing using a learning model accumulated by the learning device 1 and the transition information acquired by the information acquiring unit to acquire predicted transition information that is transition information, and a determination unit that acquires difference information related to the difference between the transition information acquired by the information acquiring unit and the predicted transition information acquired by the prediction unit, and acquires a determination result related to whether the difference information satisfies an abnormal condition or a normal condition. and an output unit that outputs the judgment result, and the information acquisition unit comprises: window detection means that acquires, for each of the two or more frame sets, a window image that is an image of one or more windows in each of the two or more frames of the frame set; feature extraction means that acquires, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and transition information acquisition means that acquires, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in a time series.

[0191] FIG. 15 shows the appearance of a computer that executes the programs described herein to realize the learning device 1 or anomaly detection device 2 of the various embodiments described above. The above-described embodiments can be realized by computer hardware and a computer program executed thereon. FIG. 15 is an overview of this computer system 300, and FIG. 16 is a block diagram of the system 300.

[0192] In FIG. 15, a computer system 300 includes a computer 301 including a CD-ROM drive, a keyboard 302, a mouse 303, and a monitor 304.

[0193] 16, computer 301 includes, in addition to CD-ROM drive 3012, MPU 3013, bus 3014 connected to CD-ROM drive 3012 etc., ROM 3015 for storing programs such as a boot-up program, RAM 3016 connected to MPU 3013 for temporarily storing instructions of application programs and providing temporary storage space, and hard disk 3017 for storing application programs, system programs, and data. Although not shown here, computer 301 may further include a network card for providing connection to a LAN.

[0194] A program that causes computer system 300 to execute the functions of learning device 1 and the like of the above-described embodiment may be stored on CD-ROM 3101, inserted into CD-ROM drive 3012, and then transferred to hard disk 3017. Alternatively, the program may be sent to computer 301 via a network (not shown) and stored on hard disk 3017. The program is loaded into RAM 3016 when executed. The program may also be loaded directly from CD-ROM 3101 or the network.

[0195] The program does not necessarily include an operating system (OS) or third-party programs that cause the computer 301 to execute the functions of the learning device 1 of the above-described embodiment. The program need only include instructions that call appropriate functions (modules) in a controlled manner to achieve the desired results. How the computer system 300 operates is well known, and a detailed description will be omitted.

[0196] The computer that executes the program may be a single computer or a plurality of computers, that is, it may perform centralized processing or distributed processing.

[0197] Furthermore, in each of the above embodiments, it goes without saying that two or more communication means present in one device may be physically realized by one medium.

[0198] Furthermore, in each of the above embodiments, each process may be realized by centralized processing in a single device, or may be realized by distributed processing in a plurality of devices.

[0199] The present invention is not limited to the above-described embodiment, and various modifications are possible, and it goes without saying that these modifications are also included within the scope of the present invention. [Industrial Applicability]

[0200] As described above, the anomaly detection device 2 according to the present invention has the effect of being able to determine with high accuracy whether or not an abnormal operation has occurred, and is useful as a server or the like that detects an abnormal operation. [Explanation of symbols]

[0201] A. Anomaly detection system 1 Learning device 2. Anomaly detection device 3 Work terminal 3. Each work terminal 11 Teacher data storage unit 12 Information acquisition department 13 Learning Department 14 Storage unit 21 Learning model storage section 22 Test Data Reception Department 23 Processing section 24 Output section 121 Window detection method 122 Feature Extraction Method 123 Grouping Methods 124 Transition information acquisition means 232 Prediction Department 233 Judgment Department

Claims

1. a training data storage unit that stores two or more frame sets that are a collection of time-series frames of screenshots of a screen output by a normal operation; an information acquisition unit that acquires, for each of the two or more frame sets, transition information regarding transitions of feature vectors of windows in each of the two or more frames included in the frame set; a learning unit that learns the transition information for each of the two or more frame sets acquired by the information acquisition unit and acquires a learning model; a storage unit that stores the learning model, The information acquisition unit a window detection means for acquiring, for each of the two or more frame sets, a window image that is an image of one or more windows in each of the two or more frames included in the frame set; a feature extraction means for acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition means for acquiring, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in a time series.

2. The information acquisition unit further comprising a grouping means for acquiring a similarity between temporally consecutive feature vectors among the two or more feature vectors in the time series acquired by the feature extraction means, and acquiring a representative vector of the temporally consecutive feature vectors when the similarity satisfies a grouping condition; The transition information acquisition means The learning device according to claim 1 , wherein the grouping means acquires the transition information relating to temporal transitions of two or more feature vectors including the feature vector that is the representative vector acquired by the grouping means.

3. The feature extraction means 3. The abnormality inspection device according to claim 1, wherein attribute values ​​of two or more of the window images are acquired, and feature vectors of window images whose attribute values ​​satisfy a rejection condition are not acquired.

4. The transition information acquisition means 4. The learning device according to claim 1, wherein for each of the two or more frame sets, a feature transition matrix is ​​obtained, which is a matrix representing information on a pair of feature vectors that occur simultaneously in one frame and information on a pair of a feature vector that occurs in one frame and a feature vector that occurs in a frame following the one frame, in the feature vectors corresponding to each of the two or more frames in the time series, and from the feature transition matrix, transition information is obtained regarding graph information having a graph structure in which the feature vectors are nodes and the feature vectors that are related in the feature transition matrix are connected by edges.

5. an inspection data receiving unit that receives a frame set to be inspected; an information acquisition unit that acquires transition information regarding transitions of feature vectors of windows in each frame of the frame set accepted by the test data acceptance unit; a prediction unit that performs machine learning prediction processing using a learning model accumulated by the learning device according to any one of claims 1 to 4 and the transition information acquired by the information acquisition unit, and acquires predicted transition information that is transition information; a determination unit that obtains difference information regarding a difference between the transition information obtained by the information obtaining unit and the predicted transition information obtained by the prediction unit, and obtains a determination result regarding whether the difference information satisfies an abnormal condition or a normal condition; an output unit that outputs the determination result, The information acquisition unit a window detection means for acquiring, for each of the two or more frame sets, a window image that is an image of one or more windows in each of the two or more frames included in the frame set; a feature extraction means for acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition means for acquiring, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in the time series.

6. The information acquisition unit further comprising a grouping means for acquiring a similarity between temporally consecutive feature vectors among the two or more feature vectors in the time series acquired by the feature extraction means, and acquiring a representative vector of the temporally consecutive feature vectors when the similarity satisfies a grouping condition; The transition information acquisition means 6. The abnormality inspection device according to claim 5, wherein the grouping means acquires the transition information relating to temporal transitions of two or more feature amount vectors including the feature amount vector that is the representative vector acquired.

7. The feature extraction means 7. The abnormality inspection device according to claim 5, wherein attribute values ​​of two or more of the window images are acquired, and feature vectors of window images whose attribute values ​​satisfy a rejection condition are not acquired.

8. The transition information acquisition means 8. The learning device according to claim 5, wherein for each of the two or more frame sets, a feature transition matrix is ​​obtained, which is a matrix representing information on a pair of feature vectors that occur simultaneously in one frame and information on a pair of a feature vector that occurs in one frame and a feature vector that occurs in a frame following the one frame, in the feature vectors corresponding to each of the two or more frames in the time series, and from the feature transition matrix, transition information is obtained regarding graph information having a graph structure in which the feature vectors are nodes and the feature vectors that are related in the feature transition matrix are connected by edges.

9. A method for producing a learning model, which is realized by a teacher data storage unit that stores two or more frame sets that are a collection of time-series frames of screenshots of a screen output by normal operation, an information acquisition unit, a learning unit, and an accumulation unit, an information acquisition step in which the information acquisition unit acquires, for each of the two or more frame sets, transition information regarding transitions of feature vectors of windows in each of the two or more frames included in the frame set; a learning step in which the learning unit learns the transition information for each of the two or more frame sets acquired by the information acquisition unit and acquires a learning model; a storage step in which the storage unit stores the learning model, The information acquisition step a window detection substep of acquiring, for each of the two or more frame sets, window images that are images of one or more windows in each of the two or more frames included in the frame set; a feature extraction substep of acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition substep of acquiring, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in the time series.

10. An anomaly detection method realized by an inspection data receiving unit, an information acquiring unit, a predicting unit, a determining unit, and an output unit, an examination data receiving step in which the examination data receiving unit receives a frame set to be examined; an information acquisition step in which the information acquisition unit acquires transition information regarding transitions of feature vectors of windows in each frame of the frame set accepted by the test data acceptance unit; a prediction step in which the prediction unit performs a machine learning prediction process using a learning model accumulated by the learning device according to any one of claims 1 to 4 and the transition information acquired by the information acquisition unit, and acquires predicted transition information that is transition information; a determination step in which the determination unit obtains difference information regarding a difference between the transition information obtained by the information obtaining unit and the predicted transition information obtained by the prediction unit, and obtains a determination result regarding whether the difference information satisfies an abnormal condition or a normal condition; an output step in which the output unit outputs the determination result, The information acquisition step includes: a window detection substep of acquiring, for each of the two or more frame sets, window images that are images of one or more windows in each of the two or more frames included in the frame set; a feature extraction substep of acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition substep of acquiring, for each of the two or more frame sets, transition information regarding temporal transition of the feature vector corresponding to each of the two or more frames in a time series.

11. A computer that can access a training data storage unit that stores two or more frame sets that are a collection of time-series frames of screenshots of a screen output by normal operation, an information acquisition unit that acquires, for each of the two or more frame sets, transition information regarding transitions of feature vectors of windows in each of the two or more frames included in the frame set; a learning unit that learns the transition information for each of the two or more frame sets acquired by the information acquisition unit and acquires a learning model; functioning as a storage unit that stores the learning model; The information acquisition unit a window detection means for acquiring, for each of the two or more frame sets, a window image that is an image of one or more windows in each of the two or more frames included in the frame set; a feature extraction means for acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition means for acquiring, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in a time series.

12. Computer, an inspection data receiving unit that receives a frame set to be inspected; an information acquisition unit that acquires transition information regarding transitions of feature vectors of windows in each frame of the frame set accepted by the test data acceptance unit; a prediction unit that performs machine learning prediction processing using a learning model accumulated by the learning device according to any one of claims 1 to 4 and the transition information acquired by the information acquisition unit, and acquires predicted transition information that is transition information; a determination unit that obtains difference information regarding a difference between the transition information obtained by the information obtaining unit and the predicted transition information obtained by the prediction unit, and obtains a determination result regarding whether the difference information satisfies an abnormal condition or a normal condition; functioning as an output unit that outputs the determination result; The information acquisition unit a window detection means for acquiring, for each of the two or more frame sets, a window image that is an image of one or more windows in each of the two or more frames included in the frame set; a feature extraction means for acquiring, for each of the two or more frame sets, a feature vector of the window image corresponding to each of the two or more frames; and a transition information acquisition means for acquiring, for each of the two or more frame sets, transition information regarding the temporal transition of the feature vector corresponding to each of the two or more frames in a time series.

Images