Skill level evaluation device and program

Abstract

One aspect of the present invention involves: sequentially acquiring operation process data representing each operation history at a time when an evaluation target person repeatedly performs a series of operations in predetermined operation units a plurality of times during an evaluation target period in accordance with a predetermined scenario; calculating a degree of similarity between pieces of adjacent operation process data on a time axis on the basis of the acquired set of operation process data; determining a degree of convergence of the degree of similarity over time; and evaluating a self-proficiency of the evaluation target person for the operation on the basis of the determined degree of convergence.

Description

Proficiency assessment device and program

[0001] One aspect of the present invention relates to a proficiency assessment device for assessing a user's proficiency in a task, for example, and a program used in the device.

[0002] One known method for evaluating the proficiency of operators who perform document creation tasks using personal computers is to refer to the log data of operators (experts) that may serve as best practices (see, for example, Patent Document 1).

[0003] In this method, for example, character strings entered by an expert using, for example, a keyboard and mouse for a given task are captured as an operation log, and a frequency distribution relating to the accuracy and speed of the operations is calculated based on the captured operation log. In this state, an operation log for the same task performed by the operator to be evaluated (evaluated person) is captured, and a frequency distribution is similarly calculated based on this operation log. The calculated frequency distribution of the evaluated person is compared with the frequency distribution of the skilled person. The proficiency of the evaluated person is then evaluated based on the comparison results.

[0004] Japanese Patent No. 7184093

[0005] However, the method described in Patent Document 1 requires that the operation processes of an expert that could become best practices be acquired in advance, and therefore cannot be applied when an expert is not available or when it is difficult to acquire an operation process corresponding to a given task even if an expert is available.

[0006] This invention has been made in light of the above circumstances, and aims to provide a technology that makes it possible to evaluate the self-proficiency of an individual in operations over the evaluation period from the past to the present, even when operation process data that could become best practices cannot be prepared in advance.

[0007] In order to solve the above problems, one aspect of the proficiency assessment device according to the present invention sequentially acquires operation process data representing each operation history when an assessment subject repeatedly performs a series of operations in predetermined operation units according to a predetermined scenario during an assessment period, calculates a similarity between adjacent operation process data on a time axis over the assessment period based on the acquired set of operation process data, calculates a convergence degree of the calculated similarity on the time axis, and evaluates the self-proficiency of the assessment subject based on the calculated convergence degree.

[0008] According to one aspect of the present invention, based on a set of operation process data of the person to be evaluated obtained over a predetermined evaluation period from the past to the present, a similarity between adjacent operation process data on a time axis is calculated, and the person to be evaluated's personal proficiency is evaluated based on the degree of convergence of the calculated similarity in the time series direction, for example, the change in the similarity or the level of the similarity. Therefore, it is possible to objectively evaluate the person to be evaluated's personal proficiency by simply using the person to be evaluated's own operation process data, without having to prepare operation process data corresponding to best practices in advance.

[0009] In other words, according to one aspect of the present invention, it is possible to provide a technology that can evaluate the self-proficiency of an individual in operations over the evaluation period from the past to the present, even if it is not possible to prepare operation process data that could become best practices in advance.

[0010] FIG. 1 is a block diagram showing an example of the hardware configuration of a proficiency assessment device according to an embodiment of the present invention. FIG. 2 is a block diagram showing an example of the software configuration of a proficiency assessment device according to an embodiment of the present invention. FIG. 3 is a flowchart showing an example of the procedure and processing content of proficiency assessment processing executed by a control unit of the proficiency assessment device shown in FIG. 2. FIG. 4 is a flowchart showing an example of the procedure and processing content of similarity calculation processing executed by an operation process comparison processing unit, which is part of the procedure of proficiency assessment processing shown in FIG. 3. FIG. 5 is a flowchart showing an example of the procedure and processing content of self-proficiency estimation processing executed by a self-proficiency estimation processing unit, which is part of the procedure of proficiency assessment processing shown in FIG. 3. FIG. 6 is a diagram showing an example of operation process data. FIG. 7 is a diagram showing an example of a matrix representing co-occurrence relationships between operation process data generated in the operation process comparison processing shown in FIG. 4. FIG. 8 is a diagram showing an example of calculation results of similarities between operation process data in the operation process comparison processing shown in FIG. 4. FIG. 9 is a diagram showing an example of proficiency estimation results relative to similarities obtained by the self-proficiency estimation processing shown in FIG. 5. FIG. 10 is a diagram showing an example of changes in similarity with respect to the number of operations and the level of similarity.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0012] [One Embodiment] (Configuration Example) FIGS. 1 and 2 are block diagrams showing an example of the hardware configuration and software configuration, respectively, of a proficiency assessment device according to one embodiment of the present invention.

[0013] The proficiency assessment device SE according to one embodiment is configured, for example, by a personal computer used by a user. Note that the proficiency assessment device SE may also be configured by a server computer located on the Web or the cloud.

[0014] The proficiency assessment device SE includes a control unit 1 that uses a hardware processor such as a central processing unit (CPU). A storage unit having a program storage unit 2 and a data storage unit 3, and an input / output interface (hereinafter, interface will be abbreviated as I / F) unit 4 are connected to the control unit 1 via a bus 5.

[0015] A user interface (hereinafter referred to as UI) IN and an output device DD are connected to the input / output I / F unit 4. The user interface (UI) IN includes input devices such as a keyboard and mouse, as well as all operation targets such as system screens and graphic UI components. The output device DD is, for example, a display, and is used to display system screens, operation data, etc., and self-proficiency evaluation data.

[0016] The program storage unit 2 is, for example, a combination of a non-volatile memory such as a hard disk drive (HDD) or a solid state drive (SSD) as a storage medium that can be written to and read from at any time, and a non-volatile memory such as a read only memory (ROM), and stores application programs necessary for executing various processes related to one embodiment of the present invention, in addition to middleware such as an operating system (OS).

[0017] The data storage unit 3 is, for example, a combination of a non-volatile memory such as an HDD or SSD as a storage medium that can be written to and read from at any time, and a volatile memory such as a RAM (Random Access Memory), and its storage area is provided with an operation process data storage unit 31, a judgment condition storage unit 32, and an evaluation data storage unit 33.

[0018] The operation process data storage unit 31 stores operation process data representing an operation history of a series of operations performed on a user interface (UI) IN by an evaluation subject. Note that the operation process data may be of any type as long as it represents the operation history of a user performed on the UI.

[0019] The judgment condition storage unit 32 stores judgment conditions used by the control unit 1, which will be described later, when estimating a user's own proficiency in operation based on a set of operation process data of the user to be evaluated.

[0020] The evaluation data storage unit 33 stores the estimation result of the user's own proficiency with respect to the user's operation obtained by the control unit 1 (to be described later) as proficiency evaluation data.

[0021] The control unit 1 includes an operation process data acquisition processing unit 11, an operation process comparison processing unit 12, a self-proficiency estimation processing unit 13, and an evaluation data output processing unit 14 as processing function units necessary for implementing one embodiment of the present invention.

[0022] The processing units 11 to 14 are all realized by causing a hardware processor in the control unit 1 to execute an application program stored in the program storage unit 2. Note that some or all of the processing units 11 to 14 may be realized using hardware such as an LSI (Large Scale Integration) or an ASIC (Application Specific Integrated Circuit).

[0023] The operation process data acquisition processing unit 11 sequentially acquires operation process data representing each operation history when a user as an evaluation target repeatedly performs a series of operations in predetermined operation units according to a predetermined scenario in a user interface (UI) IN during a predetermined evaluation period from the past to the present, and stores a set of the acquired operation process data in the operation process data storage unit 31. Note that the operation process data includes not only operations of input devices such as a mouse and a keyboard, but also operation process data for GUIs on system screens such as an OS, a browser, and various files.

[0024] The operation process comparison processing unit 12 calculates the similarity between adjacent operation process data on the time axis over the evaluation period in the set of operation process data stored in the operation process data storage unit 31, and calculates, for example, the average of the similarity. An example of the similarity calculation process will be described in detail in the operation example.

[0025] The self-proficiency estimation processing unit 13 calculates the degree of change in the similarity and the level of the similarity as information representing the degree of convergence of the similarity, based on the average of the similarities between the operation process data calculated over the evaluation period by the operation process comparison processing unit 12. Then, the self-proficiency estimation processing unit 13 compares the calculated degree of convergence with the judgment conditions stored in the judgment condition storage unit 32, thereby estimating the user's self-proficiency.

[0026] The self-proficiency estimation processing unit 13 stores the estimation result of the self-proficiency as the user's skill evaluation data in the evaluation data storage unit 33. An example of the self-proficiency estimation process will be described in detail in the operation example.

[0027] The evaluation data output processing unit 14 reads out the user's skill evaluation data from the evaluation data storage unit 33, and outputs the read evaluation data from the input / output I / F unit 4 to the output device DD for display.

[0028] (Example of Operation) Next, an example of operation of the skill assessment device SE configured as above will be described.

[0029] FIG. 3 is a flowchart showing an example of the procedure and content of the proficiency evaluation process executed by the control unit 1 of the proficiency evaluation device SE.

[0030] (1) Acquisition of operation process data The user to be evaluated performs a series of operations in a predetermined operation unit on system screens and GUIs such as the OS, browser, and various files, according to a predetermined scenario in the user interface (UI) IN, repeatedly over the evaluation period.

[0031] In response to this, the control unit 1 of the proficiency evaluation device SE sequentially acquires, as operation process data, an operation history of each series of operations each time a series of operations based on the predetermined operation unit is repeatedly performed in the user interface (UI) IN in step S1 under the control of the operation process data acquisition processing unit 11. Then, the acquired operation process data is stored in the operation process data storage unit 31.

[0032] In addition, when a user performs an operation for proficiency assessment on another terminal, the operation process data representing the operation history may be obtained from the other terminal via a network, or may be obtained after having been temporarily stored in a storage medium.

[0033] (2) Comparison of Operation Processes When the process of acquiring the set of operation process data for the evaluation period is completed and the user inputs an evaluation request in this state, the control unit 1 of the proficiency evaluation device SE detects the input of the evaluation request in step S2, and then, under the control of the operation process comparison processing unit 12, executes a process of calculating the similarity between adjacent operation process data on the time axis in step S3 as follows:

[0034] FIG. 4 is a flowchart showing an example of the processing procedure and processing contents of the similarity calculation processing executed by the operation process comparison processing unit 12.

[0035] The operation process comparison processing unit 12 first selects the (i=1)th operation process data in step S31, where m is the number of all operation process data stored in the operation process data storage unit 31 and i is the number of the operation process data. Then, in step S33, the operation process comparison processing unit 12 calculates the similarity between the selected (i=1)th operation process data and the next (i+1)th operation process data. Subsequently, in step S34, the operation process comparison processing unit 12 stores the calculated similarity in a co-occurrence matrix X that represents the co-occurrence relationship between the operation process data.

[0036] When the operation process comparison processing unit 12 completes the calculation of the similarity between the operation process data i and i+1, it increments i (i←i+1) in step S35 and returns to step S32. Then, in step S32, it determines whether the number i of the operation process data is less than the maximum number m of operation process data, and if i<m, it repeatedly executes the similarity calculation processing in steps S33 to S35. Then, when i=m, it completes the calculation of the similarity between all adjacent operation process data.

[0037] The above-mentioned calculation process of the similarity between operation process data will be explained in more detail. Note that this example will be explained on the assumption that the sameness of operations is previously labeled based on the structural characteristics of the operation process data corresponding to the scenario.

[0038] The operation process comparison processing unit 12 first reads scenario (e.g., RPA scenario) data from the data storage unit 3. Fig. 6 shows an example of an operation history corresponding to an RPA scenario.

[0039] The operation process comparison processing unit 12 then creates a co-occurrence matrix using r operation process data before and after the scenario. Figure 7 shows an example of a co-occurrence matrix. r specifies the range of operations before and after that are determined to be co-occurring, and in this example, r is set to 4.

[0040] The operation process comparison processing unit 12 then creates a vector for each operation unit in the operation process data. For example, for each operation unit a, b, c, using the frequency of co-occurrence with all operation units, it creates the vector as follows: Operation unit a = [0, 1, 1, 1, 1, 0, 0, ..., 0, 0] Operation unit b = [1, 0, 1, 1, 1, 0, 0, ..., 0, 0] Operation unit c = [1, 1, 0, 1, 1, 1, ..., 0, 0] ... In this example, all components of the operation vector after operation unit h are 0.

[0041] If there are a large number of types of operations, the dimensions may be compressed using Singular Value Decomposition (SVD). For example, if there are 1,000 types of operations, the dimensions are compressed to a 50-dimensional vector while maintaining the characteristics of the operations.

[0042] The operation process comparison processing unit 12 then reads the operation process data from the operation process data storage unit 31 in chronological order, and acquires operation process data A corresponding to p operation series up to and including the operation unit that is the target of division judgment, and operation process data B corresponding to p operation series after the operation unit that is the target of division judgment.

[0043] For example, when p=3, the following is acquired for division points (1) and (2) shown in FIG. 6: Division point (1): Operation process data A=a, b, c Operation process data B=e, f, g Division point (2): Operation process data A=t, x, w Operation process data B=x, y, z

[0044] Next, the operation process comparison processing unit 12 calculates the center of gravity of the operation vector of each operation included in the operation process data A and B, respectively, as follows: Center of gravity of operation process data A=(operation vectors of a, b, c) / p Center of gravity of operation process data B=(operation vectors of e, f, g) / p

[0045] As a result, in this example, the following calculation results are obtained: Division point (1): Center of gravity of operation process data A = [2, 2, 2, 2, 3, 2, 1, 0, ..., 0, 0] Center of gravity of operation process data B = [1, 2, 3, 3, 2, 2, 1, 0, ..., 0, 0] Division point (2): Center of gravity of operation process data A = [0, 0, 0, 0, 0, 0, 0, ..., 0, 0] Center of gravity of operation process data B = [0, 0, 0, 0, 0, 0, 0, 0, ..., 0, 0]

[0046] The operation process comparison processing unit 12 then calculates the similarity of the centroid vectors between the operation process data A and the operation process data B. For example, when cosine similarity is used as the similarity, if a→ and b→ are the centroid vectors of the operation processes A and B, respectively, the similarity can be calculated by the following formula: where |V| indicates the number of dimensions of the centroid vector.

[0047]

[0048] Fig. 8 shows an example of the result of the similarity calculated for the operation process data of the scenario shown in Fig. 6. In this example, the similarity is shown as two values, "high" and "low".

[0049] (3) Estimation of self-proficiency Next, in step S4, under the control of the self-proficiency estimation processing unit 13, the control unit 1 of the proficiency assessment device SE executes a process to estimate the self-proficiency of the user, who is the subject of assessment, with respect to the operation based on the calculated similarity, as follows.

[0050] FIG. 5 is a flowchart showing an example of the processing procedure and processing content of the skill estimation processing executed by the self skill estimation processing unit 13.

[0051] That is, the self-proficiency estimation processing unit 13 first sets the number of similarities n to the maximum number m-1 of operation processes, sets the similarity number to j, and selects the j=1th similarity and the j+1th similarity from the co-occurrence matrix in step S41. Then, in step S43, the self-proficiency estimation processing unit 13 calculates a differential value indicating the difference between the j+1th similarity X[j+1] and the j=1th similarity X[j] as the degree of change in similarity, as follows: dX[j] = |X[j+1] -X[j] |. The transition of the degree of change in similarity along the time axis follows a growth curve such as a logistic curve or a sigmoid curve.

[0052] Next, in step S44, the self-skill estimation processing unit 13 compares the calculated degree of change in similarity dX[j] with the threshold value α stored in the judgment condition storage unit 32 to determine whether dX[j] > α. If the result of this determination is that the degree of change in similarity dX[j] is greater than the threshold value α, the count value cnt of the counter is counted up in step S47, and if the degree of change dX[j] is equal to or less than the threshold value α, the count value cnt is set to 0 in step S46.

[0053] Next, in step S45, the self-skill estimation processing unit 13 compares the j-th similarity X[j] with the threshold value β stored in the judgment condition storage unit 32 to determine whether X[j] > β, that is, whether the level of similarity is higher than the threshold value β. If the result of this determination is that the similarity X[j] is greater than the threshold value β, the counter value cnt is incremented (cnt←cnt+1) in step S47, whereas if the similarity X[j] is equal to or less than the threshold value β, the counter value cnt is set to 0 in step S46.

[0054] Next, in step S48, the self-proficiency estimation processing unit 13 determines whether the count value cnt is greater than a threshold value γ stored in the determination condition storage unit 32. If the result of this determination is that the count value cnt is equal to or less than the threshold value γ, in step S49, the proficiency label R[j] is set to "0" indicating "unproficient" and stored in the evaluation data storage unit 33. On the other hand, if the count value cnt exceeds the threshold value γ, in step S50, the proficiency label R[j] is set to "1" indicating "proficient" and stored in the evaluation data storage unit 33.

[0055] Each time the self-proficiency estimation processing unit 13 completes the above-described similarity-based proficiency estimation process, it increments j (j←j+1) in step S51 and returns to step S42. Then, in step S42, it determines whether the similarity number j is less than the maximum number n of similarities, and if j<n, it repeatedly executes the proficiency estimation process in steps S43 to S51. Then, when j=n, it terminates all similarity-based proficiency estimation processes.

[0056] (4) Output of proficiency evaluation data In step S5, under the control of the evaluation data output processing unit 14, the control unit 1 of the proficiency evaluation device SE reads out the self-proficiency evaluation data obtained from the set of operation process data by the series of processes described above from the evaluation data storage unit 33, and outputs the read-out evaluation data from the input / output I / F unit 4 to the output device DD for display.

[0057] Figure 9 shows an example of the similarity value calculated based on the operation process data acquired for each operation from the first to the tth time, and the estimated result of self-proficiency, and this evaluation data is displayed on the output device DD.

[0058] In addition, the evaluation data output processing unit 14 may generate a graph showing the progress of self-proficiency, for example, as shown in Figure 10, based on the evaluation results of self-proficiency obtained for each operation from the first to tth times, and display the generated graph on the output device DD.

[0059] (Effects) As described above, in one embodiment, an operation history of a series of operations when an evaluation subject repeatedly performs a series of operations in a predetermined operation unit according to a predetermined scenario multiple times during an evaluation period from the past to the present is sequentially acquired and stored as operation process data, and based on the acquired set of operation process data, a similarity between the operation process data adjacent on a time axis is calculated over the evaluation period. Then, a degree of change in the calculated similarity on the time axis and a level of the similarity are calculated, and the evaluation subject's self-proficiency in the operation is evaluated based on the calculated degree of change in the similarity and the level of the similarity.

[0060] Therefore, it is possible to objectively evaluate the self-proficiency of the person to be evaluated by using the operation process data representing the operation history when the person to be evaluated himself / herself performs operations on the system screen, etc. in the user interface (UI). Therefore, even if it is difficult to obtain the operation process data of an expert corresponding to the best practice, it is possible to evaluate the self-proficiency of the person to be evaluated in operation.

[0061] [Other embodiments] (1) In one embodiment, the self-proficiency level is evaluated in two stages, either “unproficient” or “proficient,” but it may also be evaluated in three or more stages by setting multiple thresholds.

[0062] (2) In one embodiment, the self-proficiency of the person to be evaluated is evaluated. However, if a proficiency evaluation value corresponding to a best practice of a skilled person can be obtained, the self-proficiency evaluation value may be compared with the proficiency evaluation value corresponding to the best practice, and the self-proficiency state of the person to be evaluated may be reevaluated based on the comparison result.

[0063] (3) In addition, the processing functions, processing procedures, processing contents, etc. of the proficiency assessment device can be modified in various ways without departing from the spirit of the present invention.

[0064] Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various improvements and modifications can be made without departing from the scope of the present invention. In other words, when implementing the present invention, specific configurations according to the embodiments may be appropriately adopted.

[0065] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0066] SE...proficiency evaluation device KD...input device DD...output device 1...control unit 2...program storage unit 3...data storage unit 4...input / output I / F unit 5...bus 11...operation process data acquisition processing unit 12...operation process comparison processing unit 13...self-proficiency estimation processing unit 14...evaluation data output processing unit 31...operation process data storage unit 32...determination condition storage unit 33...evaluation data storage unit

Claims

1. A proficiency assessment device comprising: a first processing unit that sequentially acquires operation process data representing each operation history when a person to be assessed repeatedly performs a series of operations in predetermined operation units according to a predetermined scenario multiple times during a period to be assessed; a second processing unit that calculates a similarity between adjacent operation process data on a time axis over the period to be assessed based on the acquired set of operation process data; and a third processing unit that calculates a convergence degree of the calculated similarity on the time axis and evaluates the person to be assessed's self-proficiency in operations based on the calculated convergence degree.

2. The proficiency assessment device of claim 1, wherein the third processing unit calculates the degree of change in the similarity and the magnitude of the similarity as the degree of convergence, and evaluates the self-proficiency of the person being assessed by comparing the calculated degree of change in the similarity and the magnitude of the similarity with predetermined judgment conditions.

3. The proficiency assessment device according to claim 1, wherein the third processing unit generates information representing a change in the self-proficiency over the assessment period.

4. A program that causes a processor provided in the proficiency assessment device to execute at least one of the processes performed by the second processing unit, the second processing unit, and the third processing unit provided in the proficiency assessment device described in any one of claims 1 to 3.

Images