Support device, support method, and support program

Abstract

An assistance device (10) according to an embodiment of the present invention acquires environment information pertaining to an environment in work and action information pertaining to actions in the work. The assistance device (10) associates the environment information and the action information and performs extraction thereof. An action sequence treated so as to be more effective for learning actions until processing pertaining to the work is executed is generated by the assistance device (10) from the actions included in the action information on the basis of the environment information and the action information. The assistance device (10) uses information relating to the treatment and the combination of the associated environment information and action information as a basis to design an environment and a reward in reinforcement learning of a model for executing the work. In addition, after performing preliminary learning for determining what kinds of actions are appropriate using the action sequence generated in the generation step, the assistance device (10) performs reinforcement learning of the model on the basis of the environment and the reward designed in the design step.

Description

[Technical field]

[0001] The present invention relates to an assistance device, an assistance method, and an assistance program. [Background technology]

[0002] Reinforcement learning is a type of machine learning in which an agent (the entity that takes action) takes various actions against the environment (the object that is being influenced) and learns the optimal action through trial and error.

[0003] A technology has been proposed that applies reinforcement learning to work to support the automation of work, etc. This technology acquires information on the environment during the worker's work and the actions taken in that environment from cameras, microphones, sensors, or terminals operated by the worker, and by associating these, it designs the rewards necessary for reinforcement learning and creates an environment for learning. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Misa Fukai, Masashi Tadokoro, Haruo Oishi, “Study on Automating Decision-Making by Learning Optimal Processes from PC Work”, IS1-5, APNOMS 2021 (2021). Summary of the Invention [Problem to be solved by the invention]

[0005] However, the work environment and the actions taken by workers are extremely complex, and there are cases where the agent cannot arrive at the correct action by trial and error alone, and learning does not progress.

[0006] In addition, there is a method in which the agent learns in advance about the actions taken by the worker before searching, but the actions of the worker are not necessarily optimal, and some of them contain meaningless actions or noise such as machine operation errors. For this reason, there was an issue that efficient learning could not be performed when applying reinforcement learning to business operations.

[0007] The present invention has been made in consideration of the above, and has an object to provide an assistance device, an assistance method, and an assistance program that realize efficient learning in reinforcement learning targeted at a business. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems and achieve the objectives, the assistance device is characterized by having an acquisition unit that acquires environmental information, which is information about the environment in a business, and behavioral information, which is information about behavior in the business; an extraction unit that associates and extracts the environmental information and the behavioral information; a generation unit that generates a sequence of actions that have been processed based on the environmental information and the behavioral information to be more effective for learning actions from the actions included in the behavioral information to performing processing related to the business; a design unit that designs an environment and reward in reinforcement learning of a model that performs the business based on the combination of the environmental information and behavioral information associated by the extraction unit and information about the processing performed by the generation unit; and a learning unit that uses the behavior sequence generated by the generation unit to perform pre-learning on what actions are appropriate, and then performs reinforcement learning of the model based on the environment and reward designed by the design unit. Effect of the Invention

[0009] According to the present invention, efficient learning in reinforcement learning targeted at business is realized. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a support device according to the first embodiment. [Diagram 2]FIG. 2 is a diagram showing an example of the environmental information and the behavioral information. [Diagram 3] FIG. 3 is a diagram showing an example of the environmental information and the behavioral information. [Figure 4] FIG. 4 is a diagram showing an example of the environment information and the behavior information. [Diagram 5] FIG. 5 is a diagram showing an example of the environment information and the behavior information. [Figure 6] FIG. 6 is a diagram illustrating the process of the generation unit. [Figure 7] FIG. 7 is a diagram showing an example of an environment design and an action design. [Figure 8] FIG. 8 is a flowchart showing the flow of the acquisition process. [Figure 9] FIG. 9 is a flowchart showing the flow of the extraction process. [Figure 10] FIG. 10 is a flowchart showing an example of the flow of the behavior deletion process. [Figure 11] FIG. 11 is a flowchart showing an example of the flow of the behavior deletion process. [Figure 12] FIG. 12 is a flowchart showing the flow of the process of minimizing the number of steps of a behavior. [Figure 13] FIG. 13 is a flowchart showing the flow of the learning process. [Figure 14] FIG. 14 is a flowchart showing the flow of the execution process. [Figure 15] FIG. 15 is a diagram illustrating an example of a computer that executes the assistance program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of an assisting device, an assisting method, and an assisting program according to the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiment described below.

[0012] Conventionally, reinforcement learning of a model to support a business is performed using an environment in which the business is actually performed. In such a case, it is considered that the business may be adversely affected. On the other hand, according to the embodiment, it is possible to perform reinforcement learning of a model to support a business while suppressing adverse effects on the business.

[0013] In the embodiment, the tasks include all tasks performed by humans, such as inputting data into a terminal device such as a personal computer (PC), responding to customer inquiries by voice or text, and inspecting equipment.

[0014] For example, a model for automating input operations into a terminal device automatically performs operations on the terminal device that mimic human input operations based on a captured image of a screen displayed by the terminal device. The model may also use, for example, a neural network.

[0015] [Configuration of the first embodiment] First, the configuration of the support device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the support device according to the first embodiment.

[0016] As shown in Fig. 1, the support system 1 includes a support device 10 and a terminal device 20. The support device 10 is connected to the terminal device 20. The terminal device 20 is a device such as a PC that allows a worker to perform work related to the worker's business. The support device 10 may be connected to a camera, a microphone, a wearable device, or the like held by the worker.

[0017] Here, a description will be given of each unit of the support device 10. As shown in FIG.

[0018] The input / output unit 11 is an interface for inputting and outputting data. For example, the input / output unit 11 is a network interface card (NIC). The input / output unit 11 can transmit and receive data to and from other devices.

[0019] The input / output unit 11 may also be connected to input devices such as a mouse, a keyboard, etc. The input / output unit 11 may also be connected to output devices such as a display and a speaker.

[0020] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0021] The storage unit 12 stores an OS (Operating System) and various programs executed by the assistance device 10. The storage unit 12 stores learning information 121 and model information 122.

[0022] The learning information 121 is information for performing reinforcement learning. The learning information 121 includes a reward and an environment in reinforcement learning.

[0023] The model information 122 is information such as parameters for constructing a model that supports a business operation. When the model is a neural network, the model information 122 includes weights and biases for each node.

[0024] The model receives input of information indicating the work environment and outputs information related to the action. The assistance device 10 assists the work based on the output information.

[0025] The control unit 13 controls the entire support device 10. The control unit 13 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0026] The control unit 13 also has an internal memory for storing programs that define various processing procedures and control data, and executes each process using the internal memory. The control unit 13 also functions as various processing units by the operation of various programs. For example, the control unit 13 has an acquisition unit 131, an extraction unit 132, a generation unit 133, a design unit 134, a learning unit 135, and an execution unit 136.

[0027] The acquisition unit 131 acquires environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in a business.

[0028] The acquisition unit 131 can acquire information on the behavior of a worker performing a task as behavior information, and acquire information on the environment related to the worker as environmental information. The acquisition unit 131 can also acquire the content of an operation performed by the worker on the terminal device 20 as behavior information, and acquire the state of the terminal device 20 that changes in response to the operation performed by the worker as environmental information.

[0029] For example, the acquisition unit 131 acquires, as environmental information, an image captured by a camera in the viewpoint direction of the worker and sounds around the worker collected by a microphone.

[0030] Furthermore, for example, the acquisition unit 131 acquires, as environmental information, information such as images and sounds output from the terminal device 20 operated by the worker.

[0031] The image acquired by the acquisition unit 131 may be a still image such as a captured image, or may be a moving image. The acquisition unit 131 may acquire the collected voice as a voice file, or may acquire text obtained by converting the collected voice.

[0032] For example, the acquisition unit 131 acquires, as behavior information, information on the physical movement of the worker sensed by a sensor attached to the worker and the contents of the worker's speech collected by a microphone.

[0033] Also, for example, the acquiring unit 131 acquires, as the behavioral information, the contents of the operation performed by the worker on the terminal device 20. The contents of the operation include the time when the keyboard was pressed, the type of the pressed key, the trajectory of the mouse movement, the position and time when the mouse was clicked, and the like.

[0034] At this time, the acquisition unit 131 can acquire the operation content of the worker on the terminal device 20 as behavior information, and acquire a capture image of the screen of the terminal device 20 that changes in response to the operation as environmental information.

[0035] Furthermore, the acquisition unit 131 may acquire information for identifying an application or a window operated by the operator on the terminal device 20 together with the operation content as behavior information.

[0036] The extraction unit 132 extracts the environmental information and the behavioral information in association with each other. In other words, the extraction unit 132 extracts a combination of the environmental information and the behavioral information acquired by the acquisition unit 131.

[0037] The extraction unit 132 can extract behavior information related to behavior in a business operation in association with environment information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior.

[0038] The association between the environmental information and the behavioral information will be described with reference to Fig. 2 to Fig. 5. Fig. 2, Fig. 3, Fig. 4, and Fig. 5 are diagrams showing examples of the environmental information and the behavioral information.

[0039] 2 and 3 show examples of environmental information and behavior information related to a task in which the terminal device 20 is used.

[0040] 2, the extraction unit 132 extracts a captured image 51a and an operation content 52a in association with each other. The captured image 51a corresponds to environmental information. The operation content 52a corresponds to behavior information.

[0041] It is assumed that the worker inputs information into various items and clicks (presses) button 511a with the mouse. In this case, operation content 52a includes information that the type of the operation event is a click, and the coordinates of the cursor when the click is performed.

[0042] In this case, for example, the environment before the action is taken is a captured image of the screen before the button 511a is pressed, while the environment influenced by the action is a captured image of the screen to which the action transitions after the button 511a is pressed.

[0043] The extraction unit 132 may extract the entire captured image 51a as the environmental information, or may extract only the operated button 511a, which is a part of the captured image 51a, as the environmental information.

[0044] 2 is an image obtained by cutting out a part of the screen displayed on the display, whereas the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specific application.

[0045] 3, the extraction unit 132 extracts a captured image 51b and an operation content 52b in association with each other. The captured image 51b corresponds to environmental information. The operation content 52b corresponds to behavior information.

[0046] It is assumed that, in order to input "yokosuka" in the roman alphabet, the worker first types (presses) the "y" key on the keyboard in text box 511b, and then types the "o" key on the keyboard.

[0047] In this case, the operation content 52b includes the type of the operation event being pressing of the "o" key.

[0048] In this case, for example, the environment before the action is taken is a captured image of the screen before the "o" key is pressed. On the other hand, the environment influenced by the action is a captured image of the screen after the "o" key is pressed and "yo" is entered in text box 511b. In romaji input, the hiragana "yo" is displayed by entering "o" after "y".

[0049] The extraction unit 132 may extract the entire captured image 51b as the environmental information, or may extract a part of the captured image 51b, that is, a text box 511b that is the target of an operation, as the environmental information.

[0050] 3 is an image obtained by cutting out a part of the screen displayed on the display, whereas the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specific application.

[0051] The extraction unit 132 may extract a series of multiple pieces of environmental information in association with one piece of behavioral information. For example, the extraction unit 132 can extract, as environmental information, capture images of multiple frames of screens in a time series until a predetermined operation content occurs on the terminal device 20.

[0052] The extraction unit 132 may further extract environmental information relating to an environment whose similarity to the environment corresponding to the extracted environmental information is equal to or greater than a threshold value, in association with the behavioral information.

[0053] For example, in addition to the captured image 51b, a previously captured image similar to the captured image 51b may be extracted in association with the operation content 52b.

[0054] For example, the number of words commonly shown in two captured images is taken as the degree of similarity, and the extraction unit 132 considers captured images having a degree of similarity equal to or greater than a threshold value to be similar to each other.

[0055] That is, the extraction unit 132 extracts not only the environmental information at the moment when the behavior indicated by the behavior information was performed, but also similar environmental information from the past.

[0056] FIG. 4 shows an example of environmental information and behavioral information related to telephone response work.

[0057] 4, the extraction unit 132 extracts a voice 51c of a telephone inquiry from a customer and a voice 52c of an operator's reply in association with each other. The voice 51c corresponds to environmental information. The voice 52c corresponds to behavioral information.

[0058] At this time, the extraction unit 132 may extract text obtained by transcribing the speech instead of the speech.

[0059] In this case, for example, the environment before the action is taken is the voice 51c of the telephone inquiry from the customer, while the environment influenced by the action is the voice uttered by the customer in response to the voice 52c of the operator's response.

[0060] FIG. 5 shows an example of environmental information and behavioral information related to equipment inspection work.

[0061] 5, the extraction unit 132 extracts an image 51d from the viewpoint of a moving worker and a position 52d of a destination to which the worker has moved in association with each other. The image 51d corresponds to environmental information. The position 52d corresponds to behavior information.

[0062] In this case, for example, the environment before the action is taken is an image 51d from the viewpoint of the worker while he is moving, whereas the environment influenced by the action is an image from the viewpoint of the worker after he has moved.

[0063] The generation unit 133 generates an action sequence that is processed to be more effective for learning an action from an action indicated by the action information to performing a process related to a business, based on the environment information and the action information associated by the extraction unit 132. The generation unit 133 performs deletion and / or addition processing on the information associated by the extraction unit 132, thereby eliminating unnecessary information and generating an action sequence that is more suitable for learning.

[0064] The generation unit 133 determines which behavior corresponds to a mistake among the behaviors indicated by the behavior information based on a predetermined judgment rule, and deletes information about the behavior corresponding to the mistake. For the behavior corresponding to the mistake, the generation unit 133 deletes not only the behavior information indicating this behavior, but also the environmental information corresponding to this behavior information. The generation unit 133 makes the judgment based on, for example, the following judgment rule.

[0065] The generation unit 133 extracts a specific action from the action sequence of the worker, and determines that the corresponding action or actions before and after it are errors.

[0066] For example, when a worker is inputting text at a terminal and takes the action of pressing the "backspace key" or "delete key," it can be determined that the key input made before that was an operational error.

[0067] For this reason, the judgment rule is set so that if the behavior indicated by the behavior information is a text input operation in terminal operation, such as pressing the "backspace key" or pressing the "delete key", then the generation unit 133 judges that the key input made before this behavior is an operation error if the behavior indicated by the behavior information is a text input operation in terminal operation, such as pressing the "backspace key" or pressing the "delete key", and deletes information about this behavior.

[0068] For example, if a worker writes words on paper with a pen, and then erases or blacks out the words he or she wrote, the previous writing can be determined to be a mistake.

[0069] For this reason, the judgment rule is set such that if the behavior indicated by the behavior information is the behavior of writing words on paper with a pen and erasing the written word or blacking out the written word, the writing made before this behavior is judged to be a mistake. Then, if the behavior indicated by the behavior information is the behavior of writing words on paper with a pen and erasing the written word or blacking out the written word, the generation unit 133 judges that the writing made before this behavior is a mistake and deletes information about this behavior.

[0070] In addition, the generation unit 133 compares the environmental information before and after the first environmental information, and if an identical environment exists, determines that the behavior corresponding to the first environmental information or the behavior corresponding to the environment before or after the first environmental information that was identical to the first environmental information is a mistake.

[0071] For example, in the case of a page transition action in terminal operation, if the operator immediately clicks the "back" button, this page transition operation can be determined to be an operational error.

[0072] For this reason, the judgment rule is set such that if the behavior indicated by the behavior information is a page transition behavior in a terminal operation, and if the "Back" button is clicked within a predetermined time, the judgment is that the page transition behavior is an operation error. Then, if the behavior indicated by the behavior information is a page transition behavior in a terminal operation, and if the "Back" button is clicked within a predetermined time, the generation unit 133 judges that the page transition behavior is an operation error, and deletes information related to this behavior.

[0073] For example, when an operator repeatedly gives the same explanation over the phone, it can be determined that the explanations except for the last one were insufficient in content or otherwise incomprehensible to the other party, i.e., were mistakes.

[0074] For this reason, the judgment rule is set such that, if the behavior indicated by the behavior information is a behavior in which the operator repeatedly gives the same explanation during telephone conversation, the behavior other than the final explanation is judged to be a mistake.Then, if the behavior indicated by the behavior information is a behavior in which the operator repeatedly gives the same explanation during telephone conversation, the generation unit 133 judges that the behavior other than the final explanation is a mistake, and deletes information related to this behavior.

[0075] Furthermore, when specific information is included in the environmental information, the generation unit 133 determines that the behavior indicated by the behavior information associated with this environmental information, or the behaviors before and after this behavior, are errors.

[0076] For example, when operating a terminal, if the environmental information contains a message such as "Please try again," it can be inferred that one of the operations performed previously contained an error.

[0077] For this reason, the judgment rule is set such that, if the environmental information contains a phrase such as "please redo ~", it is judged that any of the operations performed before that contains an error. In this case, the generation unit 133 judges which of the operations performed before that contained an error. Specifically, the generation unit 133 extracts an action similar to the next action (operation) performed by the worker from among the operations performed before, identifies the error from the difference between the next action performed by the worker and the extracted action, and deletes information about the action identified as the error. For example, if there is an error in inputting an incorrect value into some items in a system input operation, when redoing the system input, it is expected that the same value will be input in the redo operation for items in which a correct value was input in the previous operation, and a correct value different from the previous operation will be input in items in which an incorrect value was input in the previous operation. Therefore, if an environment similar to an action that may contain an error appears during the redo operation, if the same action is taken, it can be judged that the previous action is correct, and if a different action is taken, it can be judged that the previous action was incorrect. In this way, the generation unit 133 identifies which operation was an error based on the difference between the information associated with the action that may include an error and the information associated with the action taken during the redo work.

[0078] Furthermore, for example, if the environmental information includes a voice saying "Please try again," it can be inferred that one of the actions taken prior to that included a mistake.

[0079] For this reason, the determination rule is set such that, when the environmental information contains a voice with the content of "Please redo...", it is determined that a mistake was made in any of the actions taken before that. When the environmental information contains a voice with the content of "Please redo...", the generation unit 133 extracts information similar to information on the next action taken by the worker from among any of the actions taken before that. Then, the generation unit 133 identifies a mistake from the difference between the next action taken by the worker and the extracted action, and deletes information on the action identified as a mistake.

[0080] The generation unit 133 determines which of the behaviors indicated by the behavior information corresponds to a meaningless and wasteful behavior based on a predetermined judgment rule, and deletes information about the behavior if it is a meaningless behavior. The generation unit 133 deletes the behavior information indicating the meaningless behavior and the environmental information associated with the behavior information. The generation unit 133 makes the judgment based on, for example, the following judgment rule.

[0081] The generation unit 133, for example, sets environmental information before and after the behavior based on the collected behavioral information and environmental information, compares them, and if there is no change in the environment, determines that the behavior is a meaningless behavior.

[0082] For example, when a worker performs some operation on a terminal, if there is no difference between the screen before and after the operation, it can be assumed that the operation is meaningless.

[0083] For this reason, the judgment rule is set such that when the worker performs some operation on the terminal, if no difference occurs on the screen before and after the operation, the operation is judged to be a meaningless operation. If no difference occurs on the screen before and after the operation, the generation unit 133 judges the operation to be a meaningless operation and deletes information related to this behavior.

[0084] For example, if a worker is inspecting a machine at an on-site job and moves to the front of the machine to visually inspect it, but then does not take any action such as reporting it verbally or filling out an inspection form, it can be inferred that the actions of moving to the machine and visually inspecting it are meaningless.

[0085] For this reason, the judgment rule is set such that, in a case where a worker, in an inspection task of a machine at a site, moves to the front of the machine and visually inspects the machine, and then does not take an action such as reporting by voice or filling in an inspection sheet, the action of the movement and visual inspection is judged to be a meaningless action. In a case where a worker, in an inspection task of a machine at a site, moves to the front of the machine and visually inspects the machine, and then does not take an action such as reporting by voice or filling in an inspection sheet, the generation unit 133 judges that the action of the movement and visual inspection is a meaningless operation, and deletes information regarding this action.

[0086] The generation unit 133 then acquires actions from the operation log in chronological order and generates an action sequence that minimizes the number of steps of the acquired actions. The generation unit 133 arranges the actions taken by the worker in chronological order based on the operation log, and sets an action sequence that minimizes the number of steps of those actions.

[0087] 6 is a diagram for explaining the process of the generation unit 133. For example, the case of operation content 62a in which a text box is clicked in a terminal operation will be explained as an example.

[0088] The generation unit 133 extracts a capture image 61a of the terminal screen associated with the operation content 62a ((1) in FIG. 6). Then, the generation unit 133 detects coordinates 611c that are closest to the cursor coordinates on the clicked text box 611b from cursor coordinates 611a of the capture image 61a. Then, the generation unit 133 generates a cursor movement action that approaches the coordinates 611c.

[0089] The generation unit 133 generates a behavior sequence that minimizes the number of movement steps from the cursor coordinate 611a to the coordinate 611c. Specifically, the generation unit 133 generates a behavior sequence in which the cursor moves two steps to the left from the cursor coordinate 611a and one step down to reach the coordinate 611c. Note that one step can move a predetermined unit distance on the terminal screen.

[0090] The design unit 134 designs an environment and reward in reinforcement learning of a model that executes a task, based on both the information associated by the extraction unit 132 and information related to processing performed by the generation unit 133. The design unit 134 designs an environment and reward for the behavior information and environmental information associated by the extraction unit 132, other than the information deleted by the generation unit 133. Therefore, the design unit 134 designs an environment and reward using behaviors other than mistakes and behaviors other than meaningless behaviors. Note that behaviors such as cursor movement set in the generation unit 133 to minimize the number of steps of a behavior are not used in the design of reinforcement learning.

[0091] The design unit 134 assumes that the behavior indicated by the behavior information is the "correct behavior" and performs a design so that a reward is given if the agent takes a similar behavior in the environment in which the behavior was taken.

[0092] 7 is a diagram showing an example of an environment design and a behavior design. As shown in FIG. 7, the design unit 134 performs an environment design and a reward design.

[0093] For example, if clicking a specific button is the "right action," the design unit 134 performs a design so that a positive reward is given to clicking on that button and moving the cursor onto that button. On the other hand, the design unit 134 performs a design so that a negative reward (penalty) is given to clicking anywhere other than on the button.

[0094] Furthermore, the design unit 134 designs the agent so that when the same operation as that performed by the worker, i.e., clicking a button, is performed, the environment is transitioned to a captured image after the operation, and when an operation other than clicking a button is performed, the environment is not transitioned and the agent is caused to perform the operation again on the same screen.

[0095] Here, the extraction unit 132 extracts both environmental information related to the environment before the action is taken and environmental information related to the environment influenced by the action. At this time, the design unit 134 performs a design such that the environment before the action is taken is presented to the agent, and the agent transitions to the environment influenced by the action when the agent takes the "correct action."

[0096] The design unit 134 stores the design content in the storage unit 12 as learning information 121.

[0097] The learning unit 135 constructs a learning environment for performing reinforcement learning of the model according to the learning information 121. Before the agent performs trial and error in the environment prepared by reinforcement learning, the learning unit 135 performs pre-learning to determine what behavior is appropriate, using the behavior sequence generated by the generation unit 133. Thereafter, the learning unit 135 performs reinforcement learning of the model based on the environment and rewards designed by the design unit 134. The learning unit 135 updates the model information 122 based on the results of having the agent perform actions in the learning environment.

[0098] If the task involves the use of the terminal device 20, the learning unit 135 presents the agent with a captured image of the screen of the terminal device 20 as the environment, and has the agent select an action to be taken on the captured image (such as clicking or moving the cursor).

[0099] If the task involves movement by walking, the learning unit 135 presents the agent with a moving image from the viewpoint of the worker while walking or a still image cut out from the moving image as the environment, and allows the agent to select the direction in which to move.

[0100] In this way, by using environmental information about the worker's work environment as a substitute for a learning environment and having the agent act in that environment, it is possible to carry out learning without affecting actual work.

[0101] An agent is a simulated subject that selects an action according to the output of a model constructed from model information 122.

[0102] The learning unit 135 may also perform learning based on the environment information and behavior information of one worker. In this case, it is expected that the learning of behavior that reflects the characteristics of each worker can be achieved.

[0103] On the other hand, the learning unit 135 may perform learning by combining environmental information and behavioral information related to a plurality of workers. In this case, it is expected that a more efficient learning of work procedures can be achieved.

[0104] The execution unit 136 generates a sequence of actions related to the task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 134.

[0105] For example, the execution unit 136 inputs environmental information in an actual business operation into a trained model constructed from the model information 122 and identifies an action based on the output obtained.

[0106] Specifically, the execution unit 136 generates a sequence of actions from environmental information during the worker's work using a trained model, and supports the work based on the generated sequence of actions.

[0107] The business support may be a direct execution of a task, or a provision of actions to be taken in the task to the worker.

[0108] For example, the execution unit 136 automatically inputs information into the items based on a captured image of the screen of the terminal device. Also, for example, the execution unit 136 may infer the next task to be performed from the viewpoint video of the worker and provide information about the inferred task by voice.

[0109] [Processing of the first embodiment] Fig. 8 is a flowchart showing the flow of the acquisition process. As shown in Fig. 7, when the worker has not finished the work (step S101, No), the acquisition unit 131 acquires environmental information during the worker's work (step S102).

[0110] Then, when the worker has taken an action (Yes at step S103), the acquisition unit 131 acquires the action information of the worker (step S104).

[0111] If the worker has not taken any action (step S103, No), the acquisition unit 131 returns to step S101.

[0112] Here, if the worker has finished the work (Yes at step S101), the acquisition unit 131 ends the process.

[0113] 9 is a flowchart showing the flow of the extraction process. When the corresponding environment is not extracted for all the behavioral information acquired by the acquisition unit 131 (step S201, No), the extraction unit 132 determines the behavioral information to be targeted (step S202).

[0114] Then, the extraction unit 132 extracts environmental information corresponding to the targeted behavioral information (step S203).

[0115] If the extraction unit 132 extracts environments corresponding to all pieces of behavior information acquired by the acquisition unit 131 (Yes at step S201), the extraction unit 132 ends the process.

[0116] Fig. 10 is a flowchart showing an example of the flow of the behavior deletion process. As shown in Fig. 10, the generation unit 133 sets a determination rule to be used when determining that an behavior includes a mistake (step S301).

[0117] When it has not been determined whether or not all actions include a mistake (step S302: No), the generation unit 133 determines action information to be a target (step S303).

[0118] The generation unit 133 determines whether the targeted behavior includes a mistake based on the set judgment rule (step S304). When the generation unit 133 determines that the targeted behavior includes a mistake (step S304: Yes), the generation unit 133 deletes the targeted behavior as a behavior including a mistake (step S305).

[0119] If the generation unit 133 determines that the targeted action does not include a mistake (step S304: No) or after the process of step S305, the generation unit 133 returns to step S302.

[0120] When it is determined whether or not all actions include a mistake (step S302: Yes), the generation unit 133 ends the action deleting process including a mistake.

[0121] Fig. 11 is a flowchart showing an example of the flow of the behavior deletion process. As shown in Fig. 11, when it is not determined whether or not each behavior is a meaningful operation for all behaviors (step S401: No), the generation unit 133 determines behavior information to be a target (step S402).

[0122] The generation unit 133 associates the target behavior with environmental information before and after the behavior (step S403). The generation unit 133 determines whether the target behavior is a meaningless behavior (step S404). If there is no change in the environment of the environmental information before and after the behavior, the generation unit 133 determines that the behavior is a meaningless behavior.

[0123] If the target behavior is a meaningless behavior (step S404: Yes), the generation unit 133 deletes this behavior as a meaningless behavior (step S405).

[0124] If the targeted behavior is not a meaningless behavior (step S404: No) or after the process of step S405, the generation unit 133 returns to step S401.

[0125] When it has been determined for all the actions whether or not the actions are meaningful operations (step S401: Yes), the generation unit 133 ends the process of deleting meaningless actions.

[0126] Fig. 12 is a flowchart showing the flow of the process of minimizing the number of steps of an action. As shown in Fig. 12, when the generation unit 133 does not set all actions to have the minimum number of steps (step S501: No), the generation unit 133 determines the target action information (step S502).

[0127] The generation unit 133 detects the coordinates to be moved to in order to execute the targeted action and the current cursor coordinates based on the operation log (step S503). The generation unit 133 generates a cursor movement operation based on the detected coordinates (step S504).

[0128] When the actions that have the minimum number of steps are set for all actions (step S501: Yes), the generation unit 133 ends the process of minimizing the number of steps of the actions.

[0129] 13 is a flowchart showing the flow of the learning process. Here, it is assumed that the reward and the environment for reinforcement learning have already been designed by the design unit 134.

[0130] 13, the learning unit 135 performs pre-learning to determine what behavior is appropriate using the behavior sequence effective for learning generated by the generation unit 133 (step S601). After that, if it is not possible to generate a behavior similar to that of the worker for the acquired environmental information (step S602, No), the learning unit 135 determines the environmental information to be targeted (step S603).

[0131] The learning unit 135 uses the environmental information as a target and an environment for reinforcement learning to learn actions to be taken by trial and error (step S604). The learning unit 135 updates the model information 122 based on the results of the learning.

[0132] When it becomes possible to generate an action similar to that of the worker for the acquired environmental information (Yes in step S602), the learning unit 135 ends the process.

[0133] 14 is a flowchart showing the flow of the execution process. Here, it is assumed that the execution unit 136 constructs a trained model from the model information 122.

[0134] As shown in FIG. 14, if the worker has not finished the task (step S701, No), the execution unit 136 acquires the worker's environment information (step S702).

[0135] Then, if the execution unit 136 can generate an appropriate behavior sequence for the environment (Yes at step S703), it executes the generated behavior sequence using the model (step S704).

[0136] If the execution unit 136 cannot generate an appropriate behavior sequence for the environment (step S703, No), the processing returns to step S701.

[0137] If the worker has finished the task (Yes at step S701), the execution unit 136 ends the process.

[0138] [Advantages of the First Embodiment] As described above, the acquisition unit 131 acquires environmental information, which is information about the environment in a task, and behavioral information, which is information about behavior in a task. The extraction unit 132 extracts the environmental information and behavioral information in association with each other. The generation unit 133 generates a behavior sequence that is processed based on the environmental information and behavioral information associated by the extraction unit 132 so as to be more effective in learning behavior from the behavior indicated by the behavioral information to performing processing related to the task. The design unit 134 appropriately performs design for reinforcement learning by designing an environment and a reward in reinforcement learning of a model that performs a task, based on both the information associated by the extraction unit 132 and the information processed by the generation unit 133.

[0139] Then, the learning unit 135 uses the behavioral sequence generated by the generation unit 133 to perform pre-learning to determine what behavior is appropriate, and then performs reinforcement learning of the model based on the environment and rewards designed by the design unit 134.

[0140] In this way, the support device 10 performs pre-learning using a behavior sequence that has been processed to be more effective for learning behaviors up to performing processing related to a task. Therefore, by applying reinforcement learning to a task, the support device 10 performs pre-learning by processing the worker's behavior into a behavior sequence that is more effective for learning, and can learn appropriate behaviors for complex environments and behaviors. As a result, the support device 10 can provide support such as automation for complex tasks performed by humans.

[0141] The acquisition unit 131 acquires information about the behavior of a worker performing a task as behavior information, and acquires information about the environment related to the worker as environment information. In this way, by focusing on the behavior and environment of the worker, it is possible to easily acquire behavior information and environment information.

[0142] The acquisition unit 131 also acquires the contents of the operation of the terminal device 20 by the worker as behavioral information, and acquires the state of the terminal device 20 that changes in response to the operation by the worker as environmental information. This makes it possible to easily acquire environmental information related to the work using the terminal device.

[0143] In addition, the extraction unit 132 extracts the first behavioral information related to the behavior in the business and the first environmental information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior, in association with each other. This makes it easy to design reinforcement learning from the related behavioral information and environmental information.

[0144] In addition, the extraction unit 132 further extracts second environmental information related to an environment whose similarity to the environment is equal to or greater than a threshold value, in association with the first behavioral information. In this manner, by associating a plurality of pieces of similar environmental information with the behavioral information, the accuracy of reinforcement learning can be improved.

[0145] In addition, the generating unit 133 determines the behavior corresponding to the mistake, and deletes the behavior information related to the behavior corresponding to the mistake and the environmental information associated with the behavior information. As a result, the support device 10 can more efficiently learn the behavior leading up to the execution of the task-related process.

[0146] In addition, the generating unit 133 determines meaningless behaviors that have no change in the environment indicated by the previous and next environmental information, and deletes the behavior information related to the meaningless behaviors and the environmental information associated with the behavior information. As a result, the support device 10 can learn the behaviors up to the execution of the task-related processing more efficiently.

[0147] Moreover, the generation unit 133 generates a behavior sequence in which the number of steps of the behavior is minimized. As a result, the support device 10 performs pre-learning based on the behavior in which the number of steps is minimized, and therefore, it becomes possible to perform pre-learning efficiently.

[0148] Furthermore, the execution unit 136 generates a sequence of actions related to the task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 134. This makes it possible to reduce human work and judgment related to the task.

[0149] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0150] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0151] [program] As an embodiment, the support device 10 can be implemented by installing a support program that executes the above support processing as package software or online software on a desired computer. For example, the above support program can be executed by an information processing device, causing the information processing device to function as the support device 10. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDA (Personal Digital Assistant), etc.

[0152] The support device 10 may also be implemented as a support server device that provides a service related to the support process to a client, the client being a terminal device used by a user. For example, the support server device is implemented as a server device that provides a support service that receives action information and environmental information in a business and outputs a trained model for supporting the business. In this case, the support server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the support process by outsourcing.

[0153] 15 is a diagram showing an example of a computer that executes an assistance program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0154] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0155] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the support device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the support device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0156] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0157] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0158] 1. Support System 10 Support equipment 11 Input / output section 12 Storage section 13 Control section 20 Terminal Equipment 51a, 51b, 61a Captured images 51c, 52c Audio 51d Video 52a, 52b, 62a Operation details 52d position 121 Learning Information 122 Model Information 131 Acquisition Department 132 Extraction part 133 Generation part 134 Design Department 135 Learning Department 136 Executive Department 511a Button 511b Text Box

Claims

1. an acquisition unit that acquires environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business; an extraction unit that extracts the environmental information and the behavioral information in association with each other; a generation unit that generates an action sequence that is processed based on the environmental information and the action information so as to be more effective for learning an action from an action included in the action information to an action performed in relation to the business; A design unit that designs an environment and a reward in reinforcement learning of a model that performs the task based on a combination of the environment information and the behavior information associated by the extraction unit and information about the processing performed by the generation unit; a learning unit that performs pre-learning of appropriate behaviors using the behavior sequence generated by the generation unit, and then performs reinforcement learning of the model based on the environment and rewards designed by the design unit; having The support device is characterized in that the generation unit determines, based on a predetermined judgment rule, an action that corresponds to a mistake among the actions indicated by the action information, and deletes the action information related to the action that corresponds to the mistake and the environmental information associated with the action information.

2. The support device according to claim 1, characterized in that the acquisition unit acquires information regarding the behavior of a worker performing the task as the behavioral information, acquires information regarding the environment related to the worker as the environmental information, acquires content of operations performed by the worker on a terminal device as the behavioral information, and acquires a state of the terminal device that changes in response to operations performed by the worker as the environmental information.

3. The support device according to claim 1 or 2, characterized in that the extraction unit extracts first behavioral information related to the behavior in the work in association with first environmental information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior, and further extracts second environmental information related to an environment whose similarity to the environment is equal to or greater than a threshold value in association with the first behavioral information.

4. The support device according to any one of claims 1 to 3, characterized in that the generation unit deletes behavior information relating to an action in which there is no change in the environment indicated by the environmental information before and after the action, and environmental information associated with the behavior information.

5. The support device according to any one of claims 1 to 4, characterized in that the generation unit acquires the actions in chronological order and generates the action sequence that minimizes the number of steps of the acquired actions.

6. The support device according to any one of claims 1 to 5, further comprising an execution unit that generates a series of actions related to a task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit.

7. An assistance method executed by an assistance device, comprising: An acquisition step of acquiring environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business; an extraction step of extracting the environmental information and the behavioral information in association with each other; a generation step of generating an action sequence, which is processed based on the environmental information and the action information, so as to be more effective for learning an action from the action included in the action information to the execution of a process related to the business; a design process of designing an environment and a reward in reinforcement learning of a model that executes the task, based on a combination of the environment information and the behavior information associated in the extraction process and information about the processing performed in the generation process; a learning step in which, after pre-learning of what actions are appropriate using the action sequence generated in the generation step, reinforcement learning of the model is performed based on the environment and rewards designed in the design step; Including, The support method is characterized in that the generation process determines, based on a predetermined judgment rule, an action that corresponds to a mistake among the actions indicated by the behavior information, and deletes the behavior information related to the action that corresponds to the mistake and the environmental information associated with the behavior information.

8. A support program for causing a computer to function as the support device according to any one of claims 1 to 6.

Images