Operation log acquisition device, operation log acquisition method, and program

Abstract

An operation log acquisition device according to an aspect of the present invention is provided with a communication unit, an acquisition unit, an extraction unit, an OCR processing unit, and a storage unit. The communication unit communicates with a terminal device on which a business application is installed. The acquisition unit acquires a captured image of an operation screen of the business application. The extraction unit extracts a character string by performing morphological analysis on document data of a document related to the business application. The OCR processing unit converts the captured image into text data through optical character recognition processing using the extracted character string as a whitelist. The storage unit stores the text data as an operation log of the business application.

Description

Operation log acquisition device, operation log acquisition method, and program

[0001] One aspect of the present invention relates to an operation log acquisition device, an operation log acquisition method, and a program.

[0002] With the advancement of work style reform, business improvement is an important theme in corporate management. To utilize computer systems for business improvement, it is effective to acquire operation logs from the viewpoint of visualizing current business operations. In particular, if operation log data can be acquired in a general-purpose text format, it is suitable for data processing (see, for example, Non-Patent Documents 1 and 2).

[0003] To obtain text data, simple copy / paste is highly dependent on the implementation of various APIs and the target system. Therefore, it has been considered to subject captured images of the operation screen to OCR (Optical Character Recognition) processing (see, for example, Non-Patent Document 3).

[0004] Yokose Fumitaka and five others, "Business Visualization Technology Contributing to the Promotion of DX," February 2020, NTT Technical Journal, pp. 72-75, "Operation Process Classification-Based Business Design Support Technology Contributing to the Promotion of DX," [online], [Retrieved June 20, 2023], Internet <URL: https: / / journal.ntt.co.jp / article / 18032> National Diet Library, Report on the Results of the Implementation of Full-Text Text Conversion of Digital Images Using OCR, FY2010

[0005] The accuracy of OCR depends on the word dictionary. In other words, unless possible values ​​are registered in advance in a whitelist format in a word dictionary, etc., sufficient accuracy may not be achieved with OCR. However, manually creating a word dictionary requires a great deal of effort. In particular, it is expected that a large number of named entities specific to the business domain will appear on the operation screen of a business system, making it increasingly difficult to manually create a word dictionary.

[0006] The present invention has been made in light of the above-mentioned circumstances, and aims to provide a technique that can automatically generate a word dictionary and obtain an operation log with high accuracy from a captured image.

[0007] An operation log acquisition device according to one aspect of the present invention includes a communication unit, an acquisition unit, an extraction unit, an OCR processing unit, and a storage unit. The communication unit communicates with a terminal device on which a business application is installed. The acquisition unit acquires a captured image of an operation screen of the business application. The extraction unit extracts character strings by morphologically analyzing document data of a document related to the business application. The OCR processing unit converts the captured image into text data by optical character recognition processing using the extracted character strings as a whitelist. The storage unit stores the text data as an operation log of the business application.

[0008] According to one aspect of the present invention, a word dictionary can be automatically generated, thereby enabling an operation log to be acquired with high accuracy from a captured image.

[0009] FIG. 1 is a block diagram showing an example of an operation log acquisition device according to an embodiment of the present invention. FIG. 2 is a functional block diagram showing an example of the operation log acquisition device 1 shown in FIG. 1. FIG. 3 is a flowchart showing an example of a processing procedure of the operation log acquisition device 1. FIG. 4 is a diagram showing an example of morphological analysis. FIG. 5 is a diagram showing an example of a list of extracted nouns. FIG. 6 is a diagram for explaining OCR processing of a captured image. FIG. 7 is a diagram for explaining another example of OCR processing. FIG. 8 is a diagram for explaining a modification of the processing procedure of FIG. 3. FIG. 9 is a diagram for explaining an example of a method for creating a common word dictionary. FIG. 10 is a diagram for explaining another example of a method for creating a common word dictionary. FIG. 11 is a diagram for explaining another example of a method for creating a common word dictionary. FIG. 12 is a flowchart showing another example of a processing procedure of the operation log acquisition device 1. FIG. 13 is a diagram for explaining step B-1 (step S22) of FIG. 12. FIG. 14 is a diagram for explaining step B-2 (step S23) of FIG. 12. FIG. 15 is a flowchart showing an example of a processing procedure related to acquisition of screen capture data.

[0010] 1 is a block diagram showing an example of an operation log acquisition device according to an embodiment of the present invention. The operation log acquisition device 1 is a computer including a processor 10 such as a CPU (Central Processing Unit), a storage 20, and a memory 30. The operation log acquisition device 1 further includes a communication unit 40 and an interface unit 50 connected to an input unit 51 and an output unit 52. Of these, the storage 20 stores a program 20a for causing the computer to function as the operation log acquisition device 1. Note that a terminal device 2 may be included in the operation log acquisition device 1.

[0011] The communication unit 40 communicates with a terminal device 2 (computer) to be operated, for example, via a local area network (LAN). A business application is installed on the terminal device 2. The operation log acquisition device 1 reduces the burden on the operator (user) of the terminal device 2 by executing at least a part of the operations of the business application.

[0012] Fig. 2 is a functional block diagram showing an example of the operation log acquisition device 1 shown in Fig. 1. The processor 10 includes an acquisition unit 101, an extraction unit 102, a dictionary generation unit 103, and an OCR processing unit 104 as processing functions according to the embodiment.

[0013] The acquisition unit 101 acquires a captured image of the operation screen of a business application. The acquired image is stored in the memory 30 (captured image data 301). The extraction unit 102 performs morphological analysis on the document data of a document related to the business application to extract character strings. While nouns are typically extracted character strings, any character string related to the business to be subjected to OCR processing, such as verbs, adjectives, adverbs, or other character strings, may be used. The following description will be given taking the extraction of nouns as an example.

[0014] The dictionary generation unit 103 generates a word dictionary by listing the nouns extracted by the morphological analysis. The generated word dictionary is stored in the memory 30 (word dictionary 302). The OCR processing unit 104 converts a captured image acquired from the terminal device 2 into text data by optical character recognition processing using the extracted nouns as a whitelist. In particular, the OCR processing unit 104 converts a captured image acquired from the terminal device 2 into text data by optical character recognition processing using the generated word dictionary 302. The text data obtained in this way is stored in the memory 30 as an operation log of the business application (operation log 303).

[0015] Fig. 3 is a flowchart showing an example of a processing procedure of the operation log acquisition device 1. In Fig. 3, the operation log acquisition device 1 prepares a document related to a task for which a log is to be acquired (preparation 1: step S11).

[0016] That is, in preparation 1, documents to be used for generating a word dictionary are prepared. For example, there is a method of manually specifying documents that are likely to be highly relevant in advance. In addition, there is also a known technique of adding explanatory information to RPA scenarios, and there is a method of automatically specifying documents referenced in an RPA scenario based on the log of running an RPA scenario.

[0017] Next, the operation log acquisition device 1 acquires a capture image triggered by a change in the operation screen of the terminal device 2 (Preparation 2: Step S15). That is, a capture image of the operation screen is acquired in Preparation 2. The operation log acquisition device 1 monitors screen changes and acquires a capture image when a certain amount of change occurs.

[0018] Following preparation 1, the operation log acquisition device 1 performs morphological analysis on the document acquired in preparation 1 (procedure 1: step S12). In procedure 1, the document acquired in preparation 1 is subjected to morphological analysis to extract only nouns. For the morphological analysis, an existing technology such as mecab can be used.

[0019] As shown in Figure 4, when morphological analysis is performed on the statement [Document A] "To learn how to use the product purchasing system, access the product purchasing system and enter your login ID and password," the result obtained is "To learn how to use the product purchasing system, access the product purchasing system and enter your login ID and password."

[0020] Next, the operation log acquisition device 1 extracts noun parts from the results of the morphological analysis and creates a list (Procedure 2: Step S13). As a result, a list like the one shown in Fig. 5 is obtained. For duplicate nouns, one of them is deleted to make one line.

[0021] The operation log acquisition device 1 then performs OCR on the captured image acquired in preparation 2. At this time, the list generated in procedure 2 is used as a word dictionary (procedure 3: step S14).

[0022] As shown in Fig. 6, when performing OCR on the captured image obtained in preparation 2, the list generated in step 2 is used as the word dictionary. As a result, in the example of Fig. 6, the text data obtained as the read character string is [item purchasing system], [login ID], and [password]. If for some reason the word dictionary on the OCR engine side cannot be used, there is a method of using the word dictionary after OCR to perform error correction on a word-by-word basis.

[0023] As shown in FIG. 7, for example, one method is to first perform OCR, and if a character string is found in a word dictionary whose Levenshtein distance to the obtained character string is closer than a threshold, the read character string is corrected to match the word dictionary.

[0024] Fig. 8 is a diagram for explaining a modification of the processing procedure in Fig. 3. For example, a common list of frequently occurring words can be created in advance for each operation log, separate from the document read in procedure 1. The (unique) word dictionary in Fig. 8 indicates the word dictionary used in procedure 3. Further improvement in accuracy can be expected by performing OCR processing using both the (unique) word dictionary and a separately prepared common word dictionary.

[0025] 9 is a diagram illustrating an example of a method for creating a common word dictionary. As shown in FIG. 9, for example, the top N words that are frequently recognized by OCR processing are compiled into a database as a common word dictionary. Words with entry numbers N or higher are then deleted from the common word dictionary as they are considered to be relatively rarely used words. This common word dictionary is then updated every time OCR processing is performed.

[0026] FIG. 10 is a diagram illustrating another example of a method for creating a common word dictionary. As shown in FIG. 10 , for example, if a word misrecognized by OCR is incorporated into the common word dictionary, a function can be considered that allows manual correction. When a correction is made, the word before correction and the corrected vocabulary are registered in a database in a common word dictionary correction table. Even if the frequency of use of the target word decreases and it is deleted from the common word dictionary, if it appears again in the common word dictionary, a similar conversion can be performed based on the common word dictionary correction table.

[0027] For example, for the correct noun "decision" If any incorrect recognition results are obtained, they are registered in the common word dictionary correction table. Then, each time OCR processing is performed, the common word dictionary is updated, increasing the possibility of arriving at a correct answer.

[0028] FIG. 11 is a diagram illustrating another example of a method for creating a common word dictionary. For example, one method is to create a common word dictionary by registering words contained in documents related to the business to be OCRed as a whitelist. Here, the log values ​​entered into the common word dictionary include HTML inner text in the case of a browser, and input values ​​obtained from the value attribute value. As explained in FIG. 9 , words with item numbers N or higher (relatively infrequently used) are deleted from the common word dictionary, and the common word dictionary is updated each time logs are acquired. Character strings appearing in documents related to the business are naturally likely to be terms frequently used in the business domain. Using these character strings as a whitelist can further dramatically improve OCR accuracy.

[0029] 12 is a flowchart showing another example of the processing procedure of the operation log acquisition device 1. As the number of words registered in the word dictionary increases, the possibility of misrecognition between registered words increases, so it is desirable to keep the number of words registered in the dictionary to a minimum. Therefore, when using nouns from a manual for a business domain as a dictionary, only pages close to pages that describe the OCR target are used as the dictionary.

[0030] 12, the operation log acquisition device 1 prepares a document related to the business for which a log is to be acquired (Preparation 1: Step S21). Meanwhile, the operation log acquisition device 1 acquires a captured image triggered by a screen change (Preparation 2: Step S24). Next, the operation log acquisition device 1 matches the captured image of Preparation 2 with a captured image in the manual (Procedure B-1: Step S22). Then, the operation log acquisition device 1 extracts a character string located near the matching page in the manual (Procedure B-2: Step S23). After that, the operation log acquisition device 1 jumps to Step S12 of the flowchart in FIG. 3 (Step S25).

[0031] 13 is a diagram for explaining step B-1 (step S22) in FIG. 12. In step B-1, the operation log acquisition device 1 matches the captured image in preparation 2 with the captured image in the manual. Template matching such as AKAZE (Accelerated-KAZE) can be used for matching, and matching is performed in descending order of the degree of similarity between the images.

[0032] 14 is a diagram for explaining step B-2 (step S23) in FIG. 12. In step B-2, the operation log acquisition device 1 extracts a character string within M characters near the page in the manual matched in step B-1. Then, the extracted character string is used as an input, and the process proceeds to step 1.

[0033] 15 is a flowchart showing an example of a processing procedure for acquiring screen capture data. For example, when a process using the terminal device 2 is continuously executed without the user stopping the process (No in step S31) and there is a change in the operation screen that is equal to or greater than a predetermined threshold (Yes in step S32), the operation log acquisition device 1 acquires screen capture data (step S33).

[0034] As described above, in the embodiment, a document related to the business for which a log is to be acquired is input, and the input document is subjected to morphological analysis. The resulting nouns are compiled into a list, enabling automatic generation of a word dictionary specialized for the business domain. Furthermore, a captured image is acquired using a screen change on the terminal device 2 on which a business application is installed as a trigger. The generated word dictionary is then used to subject the captured image to OCR processing, thereby obtaining text data for the operation log. This makes it possible to acquire a business operation log from a captured image of the operation screen, without the need to manually create a word dictionary.

[0035] In particular, in the embodiment, character strings extracted from documents related to the business to be OCR processed are used as the whitelist, which further improves the accuracy of OCR.

[0036] From these facts, according to the embodiment, it is possible to automatically generate a word dictionary and to acquire an operation log from a captured image with high accuracy.

[0037] It should be noted that the present invention is not limited to the above-described embodiments, and that the components can be modified and embodied in practice without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0038] 1...Operation log acquisition device 2...Terminal device 10...Processor 20...Storage 20a...Program 30...Memory 40...Communication unit 50...Interface unit 51...Input unit 52...Output unit 101...Acquisition unit 102...Extraction unit 103...Dictionary generation unit 104...OCR processing unit 301...Captured image data 302...Word dictionary 303...Operation log.

Claims

1. An operation log acquisition device comprising: a communication unit that communicates with a terminal device on which a business application is installed; an acquisition unit that acquires a captured image of the operation screen of the business application; an extraction unit that performs morphological analysis on document data of a document related to the business application to extract character strings; an OCR processing unit that converts the captured image into text data by optical character recognition processing using the extracted character strings as a whitelist; and a memory unit that stores the text data as an operation log of the business application.

2. The operation log acquisition device according to claim 1, further comprising a dictionary generation unit that lists the extracted character strings and generates a word dictionary, and the OCR processing unit converts the captured image into text data using the word dictionary.

3. The operation log acquisition device according to claim 1, wherein the acquisition unit acquires the capture image when a change in the operation screen is triggered.

4. The operation log acquisition device of claim 2, wherein when the word dictionary contains a character string whose Levenshtein distance to a character string obtained from the captured image by optical character recognition processing without using the word dictionary is closer than a threshold, the OCR processing unit corrects the character string to match the word dictionary to obtain the text data.

5. The operation log acquisition device of claim 2, wherein the OCR processing unit converts the captured image into text data by optical character recognition processing using the word dictionary and a common word dictionary prepared separately from the word dictionary.

6. A method for acquiring an operation log, comprising the steps of: a computer communicating with a terminal device on which a business application is installed, and acquiring a captured image of an operation screen of the business application; the computer performing morphological analysis of document data of a document related to the business application to extract character strings; the computer converting the captured image into text data by optical character recognition processing using the extracted character strings as a whitelist; and the computer storing the text data in a memory unit as an operation log of the business application.

7. A program including instructions for causing a computer to function as an operation log acquisition device according to any one of claims 1 to 5.

Images