Classification device, classification method, and classification program

Abstract

A classification device (10) according to an embodiment of the present invention collects an operation log of a terminal device (20). The classification device (10) creates, on the basis of the operation log, information indicating a frequency of concurrence between operations on the terminal device (20) and information indicating a similarity degree between captured images of a screen on which operation has been performed. The classification device (10) classifies the operation on the terminal device (20) into a class, using the information indicating the frequency of concurrence and the information indicating the similarity degree.

Description

Technical Field

[0001] The present invention relates to a classification device, a classification method, and a classification program.

Background Art

[0002] In order to effectively improve business in enterprises and the like, it is important to accurately grasp the target business. The person in charge of the business performs a plurality of businesses every day using an information terminal such as a PC or a tablet, and the business performed via the information terminal is composed of a plurality of operations. The operations performed on the PC refer to, for example, a series of information input operations (input to a text box, click of a button, etc.) necessary to perform the business.

[0003] In actual business, the operation procedures vary due to various factors such as the person in charge and the content of the order. The operation procedures are basically defined manually, but there may be a deviation from the manual because the work content has changed since the manual was created or the person in charge has performed the work in an original way.

[0004] As a prerequisite for considering business improvement measures, business analysts need to grasp what kind of work is being performed, how much time is spent, and what procedures (operations) are used. For example, in order to introduce RPA (Robotic Process Automation) and enhance the improvement effect, it is possible to efficiently realize business improvement by grasping the types and amounts of work performed in the business and introducing RPA from the work types with a large amount of work.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Non-Patent Documents

[0006]

Non-Patent Document 1

[0007] However, in the conventional technology, it may be difficult to classify work considering the similarity between operations.

[0008] For example, Non-Patent Document 4 describes a method of classifying operation logs by operation type, then focusing on the co-occurrence of operations to divide the operation logs into segments, and classifying the divided segments into work units using agglomerative clustering.

[0009] Here, consider a system in which the URL and window style change depending on the case. At this time, for example, an operation such as "press the decision button" is preferably classified as the same work even if the URL and window style change.

[0010] On the other hand, in the method of Non-Patent Document 4, since the similarity between operations is not considered, when the URL and window style change, an operation such as "press the decision button" may be classified as a different work.

Means for Solving the Problems

[0011] In order to solve the above-described problems and achieve the object, the classification device of the present invention includes a collection unit that collects an operation log of an information processing device, and based on the operation log, information representing the co-occurrence frequency between operations on the information processing device, and an operation A creation unit that creates information representing the similarity between capture images of the screen on which is performed, and a classification unit that classifies operations on the information processing device into classes using the information representing the co-occurrence frequency and the information representing the similarity. It is characterized by having.

Effects of the Invention

[0012] According to the present invention, it is possible to classify work in consideration of the similarity between operations.

Brief Description of the Drawings

[0013]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Embodiments for Carrying Out the Invention

[0014] Hereinafter, embodiments of the classification device, classification method, and classification program according to the present application will be described in detail with reference to the drawings. Note that the present invention is not limited to the embodiments described below.

[0015] [Configuration of Classification Device] FIG. 1 is a block diagram showing an example of the configuration of a classification device. As shown in FIG. 1, the classification device 10 is connected to the terminal device 20 via a network.

[0016] The terminal device 20 is an information processing device used by a user. The user is, for example, a business operator. The business operator uses various software such as business systems and general-purpose applications on the terminal device 20, for example.

[0017] Note that the terminal device 20 may be any type of information processing device including client devices such as smartphones, desktop PCs, notebook PCs, and tablet PCs.

[0018] Also, in the example of FIG. 1, the classification device 10 and the terminal device 20 are shown as separate devices, but the terminal device 20 may have some or all of the functions of the classification device 10.

[0019] The terminal device 20 acquires the operation log of the user. For example, the terminal device 20 acquires an operation log including the operation date and time, the operation location, the operation position, etc. at the timing when an operation event occurs. The terminal device 20 transmits the acquired operation log to the classification device 10.

[0020] The classification device 10 includes a communication unit 11, a storage unit 12, and a control unit 13.

[0021] The communication unit 11 is realized by a NIC (Network Interface Card) or the like and controls communication with external devices via telecommunication lines such as a LAN (Local Area Network) and the Internet. For example, the communication unit 11 receives an operation log from the terminal device 20.

[0022] The storage unit 12 stores data and programs necessary for various processes by the control unit 13, and has an operation log storage unit 121 and a similarity storage unit 122. For example, the storage unit 12 is a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk.

[0023] The operation log storage unit 121 stores the operation logs acquired from the terminal device 20.

[0024] FIG. 2 is a diagram showing an example of an operation log. As shown in FIG. 2, the operation logs stored in the operation log storage unit 121 include the operation date and time, user information which is information for identifying the user who performed the operation, application information which is information for identifying the application being operated on, window information which is information regarding the window being operated on, the operation location (objects such as buttons, text boxes, cells, etc.), a captured image of the screen when the operation was performed, and the operation position (coordinates within the screen).

[0025] The classification device 10 receives the operation logs from the terminal device 20 and stores the received operation logs in the operation log storage unit 121. Note that the timing at which the classification device 10 receives the operation logs may be any timing. For example, the classification device 10 may receive the operation logs at regular time intervals, or may receive the operation logs each time a new operation log is generated.

[0026] The similarity storage unit 122 stores the similarity between the centroid vectors described later. The centroid vectors and similarity will be described later.

[0027] The control unit 13 has an internal memory for storing programs and required data that define various processing procedures and the like, and executes various processes based on these.

[0028] For example, the control unit 13 includes a collection unit 131, a specification unit 132, a creation unit 133, and a classification unit 134.

[0029] The control unit 13 is an electronic circuit such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit), or an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0030] The collection unit 131 collects the operation logs of the terminal device 20. The terminal device 20 is an example of an information processing device.

[0031] The specifying unit 132 specifies the operation corresponding to the operation log. For example, the specifying unit 132 reads out the operation log table shown in FIG. 2 from the operation log storage unit 121 and adds a column of operation content.

[0032] The specifying unit 132 adds information combining window information and the operation location to the column of operation content and stores it in the operation log storage unit 121.

[0033] For example, the specifying unit 132 adds information such as "Web page 1 + button e" and "Web page 1 + text box b" to the column of operation content.

[0034] Furthermore, for operations with the same operation content, the specifying unit 132 specifies that they are the same operation. The specifying unit 132 can specify the type of operation.

[0035] Taking a specific example to explain, when there are multiple operations with the operation content of "Web page 1 + button a", the specifying unit 12 specifies that these operations are the same operation.

[0036] The creation unit 133 creates information representing the co-occurrence frequency between operations on the terminal device 20 and information representing the similarity between operations based on the operation log.

[0037] For example, the creation unit 133 creates a co-occurrence matrix having the same number of rows and columns as the number of operations on the terminal device 20 and having the co-occurrence frequency between operations as elements, and a similarity matrix having the same number of rows and columns as the number of operations and having the similarity between operations as elements.

[0038] FIG. 5 is a diagram showing an example of a co-occurrence matrix. The creation unit 133 reads operations from the operation log storage unit 121 in chronological order (in the order of earlier operation date and time), counts the n (where n is an integer of 1 or more) operations before and after each operation, and creates a co-occurrence matrix for each operation. Note that each row of the operation log storage unit 121 corresponds to an operation.

[0039] At this time, when the previous and subsequent operations occur within the same window (the web page and file are common), the creation unit 133 may count them with weights. For example, the creation unit 133 counts an operation on a different web page as 1, and counts an operation on the same web page as 0.5.

[0040] FIG. 6 is a diagram showing an example of a similarity matrix. Here, each of a, b, c, d, and e is an operation identified as being the same by the specific unit 132.

[0041] Therefore, for example, there may be a plurality of operation logs corresponding to each operation (for example, operation a) in the operation log storage unit 121.

[0042] As shown in FIG. 6, the similarity matrix is represented as a square matrix in which each operation corresponds to rows and columns.

[0043] Each component of the similarity matrix is the similarity between operations. The similarity is a continuous value from 0 to 1. In the embodiment, by considering such similarity, compared with the case where the relationship between operations is represented by a binary value (1 (identical) or 0 (non-identical)), the operations can be classified more appropriately considering the similarity.

[0044] Note that the greater the similarity, the more similar the operations are. If the similarity between two operations is 1, the two operations are regarded as being the same.

[0045] The similarity of operations may be given in advance by an administrator or the like. Also, the similarity may be calculated by the creation unit 133 based on the similarity of each item in the operation log storage unit 121.

[0046] In the embodiment, the creation unit 133 calculates the similarity between the capture images of the screens where operations are performed as the similarity between operations.

[0047] In this case, the creation unit 133 creates information representing the co-occurrence frequency between operations on the terminal device 20 and information representing the similarity between the capture images of the screens where operations are performed, based on the operation log.

[0048] Using FIG. 3, the method for acquiring the capture image will be described. FIG. 3 is a diagram for explaining the method for acquiring the capture image. It is assumed that the capture image is acquired by the collection unit 131. Note that the capture image may be included in the operation log.

[0049] As shown in FIG. 3, the collection unit 131 acquires a capture image by taking a screenshot of the screen 210 where an operation is performed. At this time, the acquisition range 31 of the screenshot by the collection unit 131 is a certain range centered on the operation location (in the example of FIG. 3, the checkbox labeled "It's red"). The acquisition range 31 of the screenshot by the collection unit 131 may be the entire screen 210.

[0050] Thereby, in the obtained capture image, the ratio of the area where information related to the operation can be obtained can be increased. Note that an area within the acquisition range 31 and outside the range of the screen 210 is not included in the capture image.

[0051] Then, the creation unit 133 creates information representing the similarity between the capture images of a certain range centered on the operation location of the screen where the operation is performed.

[0052] Here, a method for calculating the similarity of the captured images by the creation unit 133 will be described. FIG. 4 is a diagram for explaining a method for calculating the similarity between the captured images.

[0053] In the example of FIG. 4, the creation unit 133 calculates the similarity between the captured image 331i and the captured image 331j to be 0.9.

[0054] For example, the creation unit 133 calculates the similarity between the captured images of the screens where operations are performed by pattern matching (template matching), and creates information representing the calculated similarity.

[0055] For example, the creation unit 133 performs pattern matching using the images of each operation location before and after the operation as patterns. The images serving as patterns are, for example, the image of a checked checkbox, the image of an unchecked checkbox, the image of an empty text box, the image of a text box with a string input, and the like.

[0056] Also, for example, the creation unit 133 calculates the similarity between the captured images of the screens where operations are performed by ImageHush, and creates information representing the calculated similarity. By using ImageHush, the processing load can be reduced.

[0057] Also, for example, the creation unit 133 calculates the similarity between the grayscale captured images of the screens where operations are performed, and creates information representing the calculated similarity. By grayscaling the captured images, for example, the calculation of the similarity can be speeded up.

[0058] Furthermore, the creation unit 133 creates a similarity co-occurrence matrix from the co-occurrence matrix and the similarity matrix.

[0059] Here, the co-occurrence matrix C is represented as in equation (1). For example, c ij is the co-occurrence frequency of operation i and operation j. Note that the creation unit 133 can create the co-occurrence matrix by the method described in Non-Patent Document 4.

[0060] [Mathematics]

[0061] Also, the similarity matrix S is expressed as in equation (2). For example, s ij is the similarity between operation i and operation j.

[0062] [Mathematics]

[0063] In this case, n is the number of operations. And the creation unit 133 calculates the components of the similarity co-occurrence matrix according to equation (3). However, i, j, and k are indices for specifying the components of the matrix.

[0064] [Mathematics]

[0065] Figure 7 is a diagram for explaining the method of creating the similarity co-occurrence matrix. The creation unit 133 applies the method of equation (3) to the co-occurrence matrix in Figure 5 and the similarity matrix in Figure 6, and calculates the components of the row corresponding to operation a as shown in Figure 7.

[0066] In the example of Figure 7, the operation vector of operation a is [0.4, 1.6, 2, 0.7].

[0067] In this way, the creation unit 133 creates the operation vectors of each operation using the similarity co-occurrence matrix.

[0068] Since the length of the operation vector is equal to the number of operation types, the larger the number of operation types, the higher the calculation cost. For this reason, the creation unit 133 may use a dimensionality reduction method such as SVD (Singular Value Decomposition) to perform dimensionality reduction on each operation vector. For example, the creation unit 133 compresses a 1000-dimensional operation vector to 50 dimensions by SVD.

[0069] The classification unit 134 classifies the operations on the terminal device 20 into classes using information representing co-occurrence frequency and information representing similarity. For example, the classification unit 134 classifies the operations on the terminal device 20 into classes using a co-occurrence matrix and a similarity matrix.

[0070] Also, the classification unit 134 classifies the operations on the terminal device 20 into classes using a similar co-occurrence matrix obtained by adding the product of the co-occurrence matrix and the similarity matrix to the co-occurrence matrix.

[0071] Specifically, the classification unit 134 acquires each row of the similar co-occurrence matrix as an operation vector, determines a split point in the sequence of operations based on the operation log, and classifies the set of operations split at the split point into classes based on the change in similarity between the centroids of the operation vectors of a plurality of operations before and after the split point in the sequence.

[0072] First, the classification unit 134 arranges the operations in chronological order. The sequence obtained here is called an operation sequence. Also, each operation in the operation sequence is identified by a number (for example, operation 1, operation i, operation n).

[0073] Then, the classification unit 134 sets the operation i, which is the i-th operation in the operation sequence, as the operation to be judged for splitting (split point), and acquires m operation sequences up to operation i (operation (i - m), operation (i - m + 1), …, operation i) and m operation sequences after operation i (operation (i + 1), operation (i + 2), …, operation (i + m + 1)).

[0074] Let the m operation sequences up to operation i be operation sequence A. Also, let the m operation sequences after operation i be operation sequence B.

[0075] The classification unit 134 acquires the operation vectors of each operation included in operation sequence A and calculates the centroid vector of the acquired operation vectors.

[0076] FIG. 8 is a diagram showing an example of an operation sequence. In the example of FIG. 8, it is assumed that a split point is determined between the operation sequence bdefg (operation sequence A) and the operation sequence opqrs (operation sequence B).

[0077] At this time, the classification unit 134 calculates the centroid of the operation sequence A (centroid vector A) as shown in equation (4).

[0078]

Number

[0079] Also, the classification unit 134 calculates the centroid of the operation sequence B (centroid vector B) as shown in equation (5).

[0080]

Number

[0081] Here, m is the number of operation vectors, and m = 5 in the examples of equations (4) and (5). Note that the classification unit 134 may calculate the sum vector instead of the centroid vector.

[0082] Furthermore, the classification unit 134 calculates the similarity between the calculated centroid vectors as shown in equation (6), and stores the calculated similarity in the similarity storage unit 122. Here, |V| is the dimension number of the centroid vector.

[0083]

Number

[0084] Similarly, the classification unit 134 calculates the similarity between the centroid vectors for each division point of the operation sequence. Note that the similarity may be the cosine similarity shown in equation (6), or the Euclidean distance or the like.

[0085] Figure 9 is a diagram for explaining the change in the similarity between the centroids. The line in Figure 9 represents the change in the similarity between the centroid vectors for each division point.

[0086] The classification unit 134 divides the operation sequence at a division point where the difference in similarity is equal to or greater than the threshold value. The arrows in Fig. 9 represent monotonic decrease and monotonic increase. If (similarity at the start position of monotonic decrease - similarity at the minimum point) + (similarity at the end position of monotonic increase - similarity at the minimum point) is equal to or greater than the threshold value, the classification unit 134 divides the operation sequence at the division point corresponding to the minimum point.

[0087] In the example of Fig. 9, the classification unit 134 obtains the operation sequences abcdefg, opqrstuvwxyz, and hijklmn by division. The operation sequences obtained by such division are called an operation set.

[0088] The classification unit 134 classifies the operation set into classes. First, the classification unit 134 performs classification in descending order of the number of types of operations included.

[0089] Figs. 10 and 11 are diagrams showing an example of the classification result. In the example of Fig. 10, as shown in Fig. 11, the operation set can be divided according to the number of types of operations included.

[0090] First, the classification unit 134 classifies the operation set abcdefgabcdefefg, which has the largest number of types of operations included, into class 1.

[0091] Then, for the operation set opqrssutxwxyz, which has the second largest number of types of operations included, since the number of operations in common with the already classified operation set abcdefgabcdefefg is 0 and is less than or equal to the threshold value (for example, 5), the classification unit 134 classifies the operation set opqrssutxwxyz into class 2, which is a new class.

[0092] Also, for the operation set abcdefg, since the number of operations in common with the already classified operation set abcdefgabcdefefg is 5 and is greater than or equal to the threshold value, the classification unit 134 classifies the operation set abcdefg into class 1.

[0093] [Processing procedure of the classification device] The flow of each process by the classification device 10 will be described using a flowchart.

[0094] Figure 12 is a flowchart showing the flow of the process of collecting operation screen shots. As shown in Figure 12, the classification device 10 acquires operation screen shots from the PC terminal (terminal device 20) until the user of the PC terminal stops the process or shuts down the PC terminal (step S101, No) (step S102).

[0095] Also, when the user of the PC terminal stops the process or shuts down the PC terminal (step S101, Yes), the classification device 10 ends the process of collecting operation screen shots.

[0096] The operation screen shot is a screen shot of the screen where the operation was performed. The acquisition range of the operation screen shot may be a certain range centered on the operation location and does not have to be the entire screen. Also, the classification device 10 acquires an operation log together with the operation screen shot.

[0097] Figure 13 is a flowchart showing the flow of the process of creating a similarity matrix. As shown in Figure 13, the classification device 10 calculates the similarity between operation screen shots until the similarity is substituted into all components of the similarity matrix (step S201, No) (step S202).

[0098] When the calculated similarity does not meet the conditions of the similarity matrix (step S203, No), the classification device 10 converts the similarity into a form that meets the conditions of the similarity matrix (step S204). For example, the classification device 10 normalizes the similarity to a range from 0 to 1.

[0099] When the calculated similarity meets the conditions of the similarity matrix (step S203, Yes), or after converting the similarity, the classification device 10 substitutes the similarity into the target component of the similarity matrix (step S205).

[0100] When the classification device 10 has finished substituting the similarity into all components of the similarity matrix (step S201, Yes), the process of creating the similarity matrix ends.

[0101] FIG. 14 is a flowchart showing the flow of a process for creating a co-occurrence matrix. As shown in FIG. 14, until all operations are targeted (step S301, No), the classification device 10 targets operations in chronological order, counts the n operations before and after, and reflects them in the co-occurrence matrix (step S302).

[0102] When the reflection of the co-occurrence matrix is completed for all operations (step S301, Yes), the classification device 10 ends the process of creating the co-occurrence matrix.

[0103] FIG. 15 is a flowchart showing the flow of a process for creating a similar co-occurrence matrix. As shown in FIG. 15, until all components of the co-occurrence matrix are targeted (step S401, No), the classification device 10 creates a similar co-occurrence matrix from the similarity matrix and the co-occurrence matrix (step S402).

[0104] When the classification device 10 has finished targeting all components of the co-occurrence matrix (step S401, Yes), it generates an operation vector for each operation from the similar co-occurrence matrix (step S403).

[0105] FIG. 16 is a flowchart showing the flow of a process for calculating the similarity of the centroid vectors. As shown in FIG. 16, until all operations are targeted (step S501, No), the classification device 10 targets operations in chronological order and generates centroid vectors for the first m operations including the target operation (step S502).

[0106] Subsequently, the classification device 10 generates centroid vectors for the m operations starting from the operation next to the target operation (step S503). Then, it calculates the similarity of the two centroid vectors and accumulates it in the similarity storage unit 122 (step S504).

[0107] When the classification device 10 has finished targeting all operations (step S501, Yes), it ends the process of calculating the similarity of the centroid vectors.

[0108] FIG. 17 is a flowchart showing the flow of the process of dividing an operation sequence. As shown in FIG. 17, until all operations are targeted (step S601, No), the classification device 10 detects the start of a decrease in the similarity between the centroid vectors arranged in time series (step S602).

[0109] Here, the classification device 10 detects the next start of decrease (= end of increase) and the minimum point therebetween from the start of the decrease in similarity (step S603).

[0110] The classification device 10 calculates the depth (d) = the difference from the start of the decrease in similarity to the minimum point + the difference between the minimum point and the end of the increase (step S604).

[0111] When the calculated depth is equal to or greater than the threshold value (step S605, Yes), the classification device 10 divides between the target operation and the next operation (step S606). When the calculated depth is less than the threshold value (step S605, No), the classification device 10 returns to step S601.

[0112] When the classification device 10 has targeted all operations (step S601, Yes), it ends the process of dividing the operation sequence.

[0113] FIG. 18 is a flowchart showing the flow of the process of classifying operations into classes. As shown in FIG. 18, until all operation sequences are targeted (step S701, No), the classification device 10 determines the target operation sequence (step S702).

[0114] Here, if there is no classified operation sequence (step S703, No), the classification device 10 classifies the target operation sequence into a new class (step S707).

[0115] If there is a classified operation sequence (step S703, Yes), the classification device 10 calculates the number of common operation types between the target operation sequence and the operation sequences within each class (step S704).

[0116] Then, when there is a class in which the number of common operation types is equal to or greater than the threshold value and the number of common operation types is the largest (step S705, Yes), the classification device 10 classifies the target operation sequence into the class that satisfies the conditions (step S706).

[0117] When there is no class in which the number of common operation types is equal to or greater than the threshold value and the number of common operation types is the largest (step S705, No), the classification device 10 classifies the target operation sequence into a new class that satisfies the conditions (step S707).

[0118] When the classification device 10 has finished processing all operation sequences (step S701, Yes), it ends the process of dividing the operation sequences.

[0119] [Effects of the Embodiment] As described above, the classification device 10 of the embodiment includes a collection unit 131, a creation unit 133, and a classification unit 134. The collection unit 131 collects the operation logs of the terminal device 20. The creation unit 133 creates information representing the co-occurrence frequency between operations on the terminal device 20 and information representing the similarity between the captured images of the screens on which the operations are performed, based on the operation logs. The classification unit 134 classifies the operations on the terminal device 20 into classes using the information representing the co-occurrence frequency and the information representing the similarity.

[0120] In this way, the classification device 10 can classify work in consideration of the similarity between operations. As a result, the work can be classified in a form closer to the actual state of the business.

[0121] [Regarding the System Configuration of the Embodiment] Each component of the classification device 10 shown in FIG. 1 is functionally conceptual and does not necessarily have to be physically configured as shown in the figure. That is, the specific form of the distribution and integration of the functions of the classification device 10 is not limited to that shown in the figure, and all or part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage situations, etc.

[0122] In addition, all or any part of each process performed in the classification device 10 may be realized by a program analyzed and executed by the CPU and the CPU. Further, each process performed in the classification device 10 may be realized as hardware by wired logic.

[0123] Also, among the processes described in the embodiments, all or part of the processes described as being automatically performed can be manually performed. Or, all or part of the processes described as being manually performed can be automatically performed by a known method. In addition, the above-described and illustrated process procedures, control procedures, specific names, and information including various data and parameters can be appropriately changed unless otherwise specified.

[0124] [Program] FIG. 19 is a diagram showing an example of a computer that executes a classification program. The computer 1000 has, for example, a memory 1010 and a CPU 1020. The computer 1000 also has a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0125] The memory 1010 includes a ROM 1011 and a RAM 1012. The ROM 1011 stores a boot program such as BIOS (Basic Input Output System), for example. The hard disk drive interface 1030 is connected to the hard disk drive 1090. The disk drive interface 1040 is connected to the disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100, for example. The serial port interface 1050 is connected to, for example, a mouse 1110 and a keyboard 1120. The video adapter 1060 is connected to, for example, a display 1130.

[0126] The hard disk drive 1090 stores, for example, an OS (Operating System) 1091, application programs 1092, program modules 1093, and program data 1094. That is, the programs defining the respective processes of the classification device 10 are implemented as program modules 1093 in which executable code by the computer 1000 is described. The program modules 1093 are stored, for example, in the hard disk drive 1090. For example, program modules 1093 for executing processes similar to the functional configuration in the classification device 10 are stored in the hard disk drive 1090. Note that the hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0127] Also, the setting data used in the processes of the above-described embodiments is stored, for example, in the memory 1010 or the hard disk drive 1090 as program data 1094. Then, the CPU 1020 reads out the program modules 1093 and program data 1094 stored in the memory 1010 or the hard disk drive 1090 to the RAM 1012 and executes them as necessary.

[0128] Note that the program modules 1093 and program data 1094 are not limited to being stored in the hard disk drive 1090, and may be stored, for example, in a removable storage medium and read by the CPU 1020 via a disk drive 1100 or the like. Alternatively, the program modules 1093 and program data 1094 may be stored in another computer connected via a network (LAN (Local Area Network), WAN (Wide Area Network), etc.). Then, the program modules 1093 and program data 1094 may be read by the CPU 1020 from the other computer via the network interface 1070.

[0129] The embodiments to which the invention made by the present inventor has been applied have been described above. However, the present invention is not limited by the description and drawings that form part of the disclosure of the present invention according to this embodiment. That is, all other embodiments, examples, operation techniques, etc. made by those skilled in the art based on this embodiment are included in the scope of the present invention.

Explanation of Signs

[0130] 10 Classification device 11 Communication unit 12 Storage unit 13 Control unit 20 Terminal device 31 Acquisition range 121 Operation log storage unit 122 Similarity storage unit 131 Collection unit 132 Identification unit 133 Creation unit 134 Classification unit 210 Screen 331i, 331j Captured image

Claims

1. A collecting unit that collects operation logs of an information processing device, a creating unit that creates information representing the co-occurrence frequency between operations on the information processing device and information representing the similarity between captured images of the screens on which the operations were performed, based on the operation logs, a classifying unit that classifies operations on the information processing device into classes using the information representing the co-occurrence frequency and the information representing the similarity, and a classification device characterized by comprising the above.

2. The classification device according to claim 1, wherein the creating unit creates information representing the similarity between captured images within a certain range centered on the operation location of the screen on which the operation was performed.

3. The classification device according to claim 1, wherein the creating unit calculates the similarity between captured images of the screens on which the operations were performed by pattern matching and creates information representing the calculated similarity.

4. The classification device according to claim 1, wherein the creating unit calculates the similarity between captured images of the screens on which the operations were performed by ImageHush and creates information representing the calculated similarity.

5. The classification device according to claim 1, wherein the creating unit calculates the similarity between grayscale captured images of the screens on which the operations were performed and creates information representing the calculated similarity.

6. A classification method executed by a classification device, comprising: a collecting step of collecting operation logs of an information processing device; a creating step of creating information representing the co-occurrence frequency between operations on the information processing device and information representing the similarity between captured images of the screens on which the operations were performed, based on the operation logs; a classifying step of classifying operations on the information processing device into classes using the information representing the co-occurrence frequency and the information representing the similarity. and a classification method characterized by including the above.

7. A collecting step of collecting operation logs of an information processing device; a creating step of creating information representing the co-occurrence frequency between operations on the information processing device and information representing the similarity between captured images of the screens on which the operations were performed, based on the operation logs; a classifying step of classifying operations on the information processing device into classes using the information representing the co-occurrence frequency and the information representing the similarity. and a classification program characterized by causing a computer to execute the above.

Images