Information processing device, information processing method, and information processing program

Abstract

An information processing device according to one embodiment of the present invention comprises: a scenario information acquisition unit that acquires first scenario information including a first automatic operation scenario; an operation information extraction unit that extracts, from the first automatic operation scenario, first operation information to be executed according to the first automatic operation scenario; a storage unit that stores a plurality of labels and a machine learning model; an explanatory information imparting unit that imparts any of the plurality of labels as explanatory information by using the machine learning model with the first operation information as input; and a transmission control unit that transmits the explanatory information.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] In order to achieve organizational improvements in business operations, companies and other business entities must integrate bottom-up and top-down approaches. To achieve this integration, it is particularly important to understand the business operations of the bottom-up side.

[0003] Robotic Process Automation (RPA) and other automation technologies have been introduced as a method for streamlining work on information processing devices such as computers, and are used to perform operations on information processing devices instead of users. The automation scenarios created using automation technology contain a lot of operational information that indicates the business content, making them useful for understanding the business from the bottom up. Therefore, if we can understand the business using automation scenarios, we can expect to be able to horizontally deploy the scenarios or ascertain needs, thereby improving business efficiency.

[0004] For example, Non-Patent Document 1 discloses a business visualization technology that supports objective and quantitative analysis of business operations. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Yokose, Urabe, Yagi, Tsuchikawa, Masuda, and Oishi, "Business Visualization Techniques Contributing to Digital Transformation Promotion," NTT Technical Journal, 2020, Vol. 32, No. 2, pp. 72-75. Summary of the Invention [Problem to be solved by the invention]

[0006] However, the operational information that can be confirmed using the automated scenario created on-site (bottom side) often does not include explanatory information, and there is a problem in that it takes a long time to decipher the scenario.

[0007] This invention has been made in light of the above circumstances, and its purpose is to provide a technology that can automatically generate user-understandable explanatory information for operations executed by an automatic operation scenario, in order to promote business understanding using the automatic operation scenario. [Means for solving the problem]

[0008] In order to solve the above problem, one aspect of the present invention is an information processing device comprising: a scenario information acquisition unit that acquires first scenario information including a first automatic operation scenario; an operation information extraction unit that extracts first operation information executed by the first automatic operation scenario from the first automatic operation scenario; a memory unit that stores a plurality of labels and a machine learning model; an explanatory information assignment unit that uses the machine learning model to input the first operation information and assigns a label as explanatory information from one of the plurality of labels; and a transmission control unit that transmits the explanatory information. [Effects of the Invention]

[0009] According to one aspect of the present invention, it is possible to automatically generate user-understandable explanation information for operation information that can be confirmed from an automatic operation scenario, thereby enabling users to promote business understanding using the automatic scenario. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device and a mobile device 2 according to an embodiment. [Figure 2] FIG. 2 is a block diagram showing the software configuration of the information processing device according to the embodiment in relation to the hardware configuration shown in FIG. [Figure 3] FIG. 3 is a flowchart showing an example of an operation performed by an information processing device to construct an appropriate machine learning model. [Figure 4] FIG. 4 is a diagram showing an example of text information extracted from operation information. [Figure 5] FIG. 5 is a diagram showing an example of image information extracted from operation information. [Figure 6] FIG. 6 is a flowchart showing an example of an operation for outputting a label (explanatory information) from an unknown automatic operation scenario. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments of the present invention will be described with reference to the drawings. Hereinafter, elements that are identical or similar to elements already described will be designated by the same or similar reference numerals, and duplicate descriptions will generally be omitted. For example, when there are multiple identical or similar elements, a common reference numeral may be used to describe each element without distinguishing between them, or a subnumber may be used in addition to the common reference numeral to describe each element with distinction between them.

[0012] [Embodiment] (composition) FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing device 1 and a mobile device 2 according to the embodiment. The information processing device 1 is a computer that performs learning based on input data and generates and outputs predetermined output data from the input data. The information processing device 1 can create explanatory information that can be understood by the administrator (user) regarding operation information that can be confirmed from the automatic scenario, based on various information input by the administrator who manages the information processing device 1. Here, the information processing device 1 may be included in the mobile device 2. In other words, the mobile device 2 may be a terminal that can execute the processing executed by the information processing device 1, which will be described later.

[0013] The portable device 2 is a device on which a user performs a task using an automatic operation technique. The portable device 2 may be a portable terminal that the user can carry, including a smartphone, a tablet, a wearable device, etc. However, the portable device 2 is not limited to a portable terminal, and may be a stationary personal computer on which the user can perform a task. Therefore, the portable device 2 may be a device equipped with general hardware such as a CPU and memory.

[0014] 1, the information processing device 1 includes a control unit 10, a program storage unit 20, a data storage unit 30, a communication interface 40, an input / output interface 50, an input device 51, and an output device 52. The control unit 10, the program storage unit 20, the data storage unit 30, the communication interface 40, and the input / output interface 50 are communicatively connected to one another via a bus. Furthermore, the communication interface 40 is communicatively connected to a mobile device 2 via a network. Furthermore, the input / output interface 50 is communicatively connected to the input device 51 and the output device 52.

[0015] The control unit 10 controls the information processing device 1. The control unit 10 includes a hardware processor such as a central processing unit (CPU). For example, the control unit 10 may be an integrated circuit capable of executing various programs.

[0016] The program storage unit 20 may use, as a storage medium, a combination of nonvolatile memory that can be written to and read from at any time, such as an EPROM (Erasable Programmable Read Only Memory), an HDD (Hard Disk Drive), or an SSD (Solid State Drive), and a nonvolatile memory such as a ROM (Read Only Memory). The program storage unit 20 stores programs necessary for executing various processes. That is, the control unit 10 can realize various controls and operations by reading and executing the programs stored in the program storage unit 20.

[0017] The data storage unit 30 is a storage that uses a combination of nonvolatile memory such as a HDD or memory card, which can be written to and read from at any time, and volatile memory such as RAM (Random Access Memory), as a storage medium. The data storage unit 30 is used to store data acquired and generated in the process of the control unit 10 executing programs and performing various processes.

[0018] The communication interface 40 includes one or more wired or wireless communication modules. For example, the communication interface 40 includes a communication module that wirelessly connects to the mobile device 2 via a network. The communication interface 40 may also include a wired communication module that enables direct connection to the mobile device 2 without going through a network. The communication interface 40 may also include a wireless communication module that uses short-range wireless technology to wirelessly connect to the mobile device 2. In other words, the communication interface 40 may be any general communication interface as long as it can communicate with the mobile device 2 under the control of the control unit 10 and send and receive various information.

[0019] The input / output interface 50 is connected to the input device 51, the output device 52, etc. The input / output interface 50 is an interface that enables transmission and reception of information between the input device 51, the output device 52, etc. The input / output interface 50 may be integrated with the communication interface 40. For example, the information processing device 1 and at least one of the input device 51 and the output device 52, etc. may be wirelessly connected using short-range wireless technology, etc., and information may be transmitted and received using the short-range wireless technology.

[0020] The input device 51 includes, for example, a keyboard, a pointing device, etc., which are used by the administrator of the information processing device 1 to input various information, including past performance data, to the information processing device 1, which information is acquired by the portable device 2. The input device 51 may also include a reader for reading data to be stored in the program storage unit 20 or the data storage unit 30 from a memory medium such as a USB memory, or a disk device for reading such data from a disk medium.

[0021] The output device 52 includes a display that displays output data to be presented to the user from the information processing device 1, a printer that prints the output data, and the like.

[0022] FIG. 2 is a block diagram showing the software configuration of the information processing device 1 according to the embodiment in relation to the hardware configuration shown in FIG. The control unit 10 includes a scenario information acquisition unit 101, an operation information extraction unit 102, a label assignment unit 103, a learning unit 104, an explanation information assignment unit 105, and a transmission control unit 106. The data storage unit 30 includes an acquired information storage unit 301, a label storage unit 302, and a learning result storage unit 303.

[0023] The scenario information acquisition unit 101 acquires scenario information from the portable device 2 via the communication interface 40. The scenario information includes a first automatic operation scenario or a second automatic operation scenario. For example, the first automatic operation scenario may be an automatic operation scenario newly created by the user. For example, the second automatic operation scenario may be a scenario to be used as training data. Therefore, the scenario information acquired by the scenario information acquisition unit 101 may include multiple second automatic operation scenarios.

[0024] The operation information extraction unit 102 extracts operation information, which is information on the operation executed by the first automatic operation scenario or the second automatic operation scenario. The method for extracting operation information will be described in detail later.

[0025] The labeling unit 103 assigns a label to the operation information. The labels include labels that indicate the operation content itself on the portable device 2, such as "turn off the terminal" or "launch the browser," as well as labels that indicate actions in the work performed by the user of the portable device 2, such as "input book information." The labeling unit 103 assigns a label to the operation information extracted from the second operation scenario using labels stored in the label storage unit 302, which will be described later. The detailed method of assigning labels will be described later.

[0026] The learning unit 104 learns the relationship between the operation information extracted from the second automatic operation scenario and the labels assigned by the label assignment unit 103. For example, the learning unit 104 constructs a machine learning model using supervised learning in which the operation information is input and the labels are output. Details of the method for constructing the machine learning model will be described later.

[0027] The explanatory information assigning unit 105 assigns one of the labels stored in the label storage unit 302 to the operation information extracted from the first automatic operation scenario, using a machine learning model stored in the learning result storage unit 303 (described later). The assigned label becomes explanatory information for the operation information of the first automatic operation scenario.

[0028] The transmission control unit 106 transmits the explanatory information through the communication interface 40. When the information processing device 1 is incorporated in the mobile device 2, the transmission control unit 106 operating as an output control unit may display the explanatory information on a display of the output device 52 or the like through the input / output interface 50.

[0029] The acquired information storage unit 301 is used to store the scenario information acquired by the scenario information acquisition unit 101 .

[0030] The label storage unit 302 is used to store labels to be assigned to operation information by the label assignment unit 103 or the explanation information assignment unit 105. Note that the labels stored in the label storage unit 302 may be labels input in advance by an administrator who manages the information processing device 1.

[0031] The learning result storage unit 303 is used to store the machine learning model that has been learned by the learning unit 104.

[0032] (operation) FIG. 3 is a flowchart showing an example of the operation of the information processing device 1 for constructing an appropriate machine learning model. The control unit 10 of the information processing device 1 reads out and executes the program stored in the program storage unit 20, thereby realizing the operation of this flowchart.

[0033] The operation is started by an instruction from an administrator of the information processing device 1. Alternatively, it may be started by receiving a predetermined instruction from the mobile device 2.

[0034] It is assumed that an RPA scenario (automatic operation scenario) is created using RPA, which is an automatic operation technology, in the mobile device 2. The RPA scenario is stored in the mobile device 2. A general method may be used to create the RPA scenario, which is an automatic operation scenario, and therefore detailed description thereof will be omitted here.

[0035] It is also assumed that a predetermined number of labels are stored in advance by an administrator in the label storage unit 302. As described above, the labels include labels that represent the operation content itself on the mobile device 2, such as "turn off the terminal" or "launch the browser," as well as labels that represent the user's work actions, such as "enter book information."

[0036] In step ST101, the scenario information acquisition unit 101 acquires scenario information. For example, the scenario information acquisition unit 101 acquires scenario information from the portable device 2 through the communication interface 40. The scenario information includes, for example, an RPA scenario (second automatic operation scenario) used by the portable device 2. The RPA scenario may be a scenario to be used as training data. Therefore, the scenario information acquired by the scenario information acquisition unit 101 may include multiple RPA scenarios. The scenario information acquisition unit 101 stores the acquired scenario information in the acquired information storage unit 301. Note that when the portable device 2 performs the processing of the information processing device 1, the scenario information is already stored in the data storage unit 30, and therefore step ST101 can be omitted.

[0037] In step ST102, the operation information extraction unit 102 extracts operation information from the RPA scenario. The operation information extraction unit 102 acquires the RPA scenario included in the scenario information stored in the acquired information storage unit 301, and extracts operation information, which is information about the operation executed by the RPA scenario. For example, the operation information extraction unit 102 extracts the operation information as text information or image information. When extracting operation information as text information, the operation information extraction unit 102 extracts at least one of attributes for identifying detailed operation content or processing associated with each node of the RPA scenario. On the other hand, when extracting operation information as image information, the operation information extraction unit 102 extracts at least one of image information indicating each node of the RPA tool or the processing content of each node. The operation information extraction unit 102 outputs the extracted operation information and RPA scenario to the label assignment unit 103.

[0038] The operation information extraction unit 102 may extract both text information and image information as operation information. Furthermore, if there are multiple RPA scenarios, the operation information extraction unit 102 extracts operation information for each of the multiple RPA scenarios.

[0039] Fig. 4 is a diagram showing an example of text information extracted from operation information, and Fig. 5 is a diagram showing an example of image information extracted from operation information. 4 shows a specific example of text information extracted as operation information by the operation information extraction unit 102. For example, the text information is associated with each node. The example in FIG. 4 shows that the operation information extraction unit 102 extracts both detailed operation content and attributes associated with each node.

[0040] 5 shows a specific example of image information extracted as operation information by the operation information extraction unit 102. For example, the image information indicates that both image information of each node in the RPA tool and image information indicating the processing content of each node are extracted.

[0041] In step ST103, the labeling unit 103 assigns a label to the operation information. The labeling unit 103 assigns a label stored in the label storage unit 302 to the operation information extracted from the scenario information (RPA scenario) used as training data. For example, if the operation information is text information, the labeling unit 103 assigns an appropriate label representing the processing executed by the automatic operation scenario (RPA scenario) based on at least one of detailed operation content or attributes, such as the type of operation event (click, key input) or identification information of the window where the operation was performed. Alternatively, if the operation information is image information, the labeling unit 103 assigns an appropriate label representing the processing executed by the automatic operation scenario (RPA scenario) based on at least one of image information representing each node of the RPA tool or the processing content of each node. Note that the assignment of labels may be performed by an administrator (user) of the information processing device 1. For example, the control unit 10 may display the operation information and the labels stored in the label storage unit 302 on the display of the output device 52, and ask the administrator to assign an appropriate label to the operation information based on the child information. Then, the information processing device 1 may receive the information (input information). The label assignment unit 103 outputs the operation information with the assigned label to the learning unit 104.

[0042] In step ST104, the learning unit 104 learns the relationship between the operation information and the label. For example, the learning unit 104 receives at least one of text information and image information (i.e., only text information, only image information, or both text information and image information) as input, and learns the relationship between the operation information and the label using the labels assigned by the label assignment unit 103 as output. That is, the learning unit 104 constructs a machine learning model through supervised learning. For supervised learning, a machine learning algorithm such as BERT (Bidirectional Encoder Representation from Transformers) or CNN (Convolutional Neural Network) is used. The learning unit 104 stores the machine learning model after learning in the learning result storage unit 303. Note that when there are multiple pieces of training data, i.e., multiple RPA scenarios, the learning unit 104 may train the machine learning model using all of the training data.

[0043] Through the above-described operations, the information processing device 1 can construct a machine learning model that has learned how to assign labels (explanatory information) to RPA scenarios (automatic operation scenarios) used as training data.

[0044] Next, we will explain how to use the trained machine learning model to input operation information extracted from an unknown RPA scenario (automated operation scenario) with no explanatory information and assign one of the prepared labels as explanatory information.

[0045] FIG. 6 is a flowchart showing an example of an operation for outputting a label (explanatory information) from an unknown automatic operation scenario. The control unit 10 of the information processing device 1 reads out and executes the program stored in the program storage unit 20, thereby realizing the operation of this flowchart.

[0046] The operation is started by an instruction from an administrator of the information processing device 1. Alternatively, it may be started by receiving a predetermined instruction from the mobile device 2.

[0047] In step ST201, the scenario information acquisition unit 101 acquires scenario information. The scenario information acquisition unit 101 acquires scenario information (second scenario information) through the communication interface 40. The acquisition method may be the same as that of step ST101. The RPA scenario included in the scenario information may be an RPA scenario without explanatory information, i.e., an unknown RPA scenario. For example, the RPA scenario may be an RPA scenario newly created by a user of the mobile device 2, etc. Then, the scenario information acquisition unit 101 stores the scenario information in the acquired information storage unit 301.

[0048] In step ST202, the operation information extraction unit 102 extracts operation information. The operation information extraction unit 102 extracts operation information in the same manner as in step ST102. Then, the operation information extraction unit 102 outputs the extracted operation information to the explanation information addition unit 105.

[0049] In step ST203, the explanatory information assigning unit 105 assigns a label. Upon receiving the operation information, the explanatory information assigning unit 105 acquires an appropriate machine learning model stored in the learning result storage unit 303. Then, the explanatory information assigning unit 105 assigns the label stored in the label storage unit 302 to the operation information using the machine learning model. Then, the explanatory information assigning unit 105 outputs the assigned label to the transmission control unit 106 as explanatory information. Note that the explanatory information assigning unit 105 may also output the operation information to the transmission control unit 106.

[0050] In step ST204, the transmission control unit 106 transmits the explanatory information through the communication interface 40. The mobile device 2 can display the received explanatory information on a display or the like. If the information processing device 1 is incorporated in the mobile device 2, the transmission control unit 106 operating as an output control unit may display the explanatory information on a display or the like of the output device 52 through the input / output interface 50. The transmission control unit 106 may transmit (output) operation information in addition to the explanatory information. By having the mobile device 2 (or the output device 52) display the operation information in addition to the explanatory information, the user (administrator) can understand whether the explanatory information corresponds to the operation information extracted by the operation information extraction unit 102.

[0051] (Effects of the embodiment) According to this embodiment, the information processing device 1 performs machine learning using operation information that can be confirmed from an automatic operation scenario and preset labels, and constructs a trained machine learning model.The information processing device 1 can then generate explanatory information for an unknown automatic operation scenario using the machine learning model.This makes it possible to add explanatory information to an automatic operation scenario without the need for human (user) operation, allowing the user to promote business understanding in a short period of time.

[0052] [Other embodiments] In the above embodiment, the learning unit 104 performs learning using all input RPA scenarios (automatic operation scenarios). However, learning may be performed for each individual automatic operation scenario. For example, a machine learning model may be constructed for each automatic operation scenario used in a specific usage scenario. This allows the information processing device 1 to construct a more accurate machine learning model.

[0053] The techniques described in the above embodiments can be stored as a program (software means) that can be executed by a computer on a storage medium such as a magnetic disk (e.g., a floppy disk, a hard disk, etc.), an optical disk (e.g., a CD-ROM, a DVD, an MO, etc.), or a semiconductor memory (e.g., a ROM, a RAM, a flash memory, etc.), and can also be distributed by transmitting the program via a communication medium. The program stored on the medium also includes a configuration program that configures the software means (including not only executable programs but also tables and data structures) that the computer executes. The computer that implements this device loads the program stored on the storage medium and, in some cases, configures the software means using the configuration program, and executes the above-described processing by controlling the operation of the software means. The term "storage medium" as used herein is not limited to storage media for distribution, but also includes storage media such as magnetic disks and semiconductor memories installed inside the computer or in devices connected via a network.

[0054] In short, this invention is not limited to the above-described embodiments, and various modifications can be made in the implementation stage without departing from the spirit of the invention. Furthermore, the embodiments may be implemented in combination as appropriate as possible, and in such cases, the combined effects can be obtained. Furthermore, the above-described embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining the disclosed multiple constituent elements. [Explanation of symbols]

[0055] 1...Information processing device 10...Control unit 101...Scenario information acquisition unit 102...Operation information extraction section 103...Label assignment unit 104…Study Department 105...Explanatory information assignment section 106...Transmission control unit 20...Program memory section 30...Data storage unit 301...Acquired information storage unit 302...Label storage unit 303...Learning result memory unit 40...Communication interface 50...Input / output interface 51...Input device 52...Output device 2. Portable device

Claims

1. a scenario information acquisition unit that acquires first scenario information including a first automatic operation scenario; an operation information extraction unit that extracts, from the first automatic operation scenario, first operation information to be executed by the first automatic operation scenario; a storage unit that stores a plurality of labels and a machine learning model; an explanatory information assigning unit that uses the machine learning model to input the first operation information and assigns a label as explanatory information from one of the plurality of labels; a transmission control unit that transmits the explanation information; An information processing device comprising:

2. The information processing device according to claim 1 , wherein the first operation information is text information including at least one attribute for identifying an operation content or a process associated with each node of the first automatic operation scenario.

3. The information processing apparatus according to claim 1 , wherein the first operation information is image information indicating at least one of each node of the first automatic operation scenario or processing content of each node.

4. the scenario information acquisition unit acquires second scenario information including a second automatic operation scenario to be used as training data; The information processing apparatus according to claim 1 , wherein the operation information extraction unit extracts second operation information executed according to the second automatic operation scenario.

5. The information processing apparatus according to claim 4 , further comprising a label assignment unit that assigns one of the plurality of labels to the second operation information.

6. 6. The information processing apparatus according to claim 5, further comprising a learning unit that uses the second operation information as an input, the labels assigned by the label assignment unit as an output, learns a relationship between the operation information and the labels as supervised data, and constructs the machine learning model.

7. An information processing method executed by a processor of an information processing device, Obtaining first scenario information including a first automatic operation scenario; extracting, from the first automatic operation scenario, first operation information to be executed by the first automatic operation scenario; storing a plurality of labels and a machine learning model; using the machine learning model, the first operation information is input, and a label is assigned as explanatory information from one of the plurality of labels; transmitting the description information; An information processing method comprising:

8. An information processing program comprising instructions to be executed by a processor of an information processing device, the instructions comprising: Obtaining first scenario information including a first automatic operation scenario; extracting, from the first automatic operation scenario, first operation information to be executed by the first automatic operation scenario; storing a plurality of labels and a machine learning model; using the machine learning model, the first operation information is input, and a label is assigned as explanatory information from one of the plurality of labels; transmitting the description information; An information processing program comprising:

Images