Support device, support method, and support program

Abstract

An assistance device (10) of one embodiment acquires: environment information which is information pertaining to an environment with respect to an operation; and action information which is information pertaining to an action in the operation. The assistance device (10) associates the environment information and the action information and extracts the same. On the basis of a combination of the environment information and the action information which have been associated, the assistance device (10) designs an environment and a reward with respect to reinforced learning of a model in which the operation is executed. Further, the assistance device (10) uses the designed environment and the reward to assist the operation by using the model with which reinforced learning was carried out.

Description

[Technical field]

[0001] The present invention relates to an assistance device, an assistance method, and an assistance program. [Background technology]

[0002] When using real-world environments for reinforcement learning of models, care must be taken to avoid irreparably damaging the environment.

[0003] For example, reinforcement learning is used to control autonomous cars and robots. However, during the reinforcement learning process using real-world environments, it is possible that the hardware of the robot may be damaged or that the car may have a collision.

[0004] In response to this, Non-Patent Document 1 discloses a reinforcement learning technique based on the Lyapunov method, which rewards safe actions.

[0005] In addition, Non-Patent Document 2 discloses a technique called CARL (Cautious Adaptation For Reinforcement Learning), in which uncertainties regarding transition dynamics and catastrophic states are trained in a sandbox, and in a real environment, the agent learns to avoid catastrophic states. [Prior art documents] [Non-patent literature]

[0006] [Non-Patent Document 1] Yinlam Chow, Ofir Nachum, Edgar Duenez-Guzman, Mohammad Ghavamzadeh, "A Lyapunov-based Approach to Safe Reinforcement Learning", arXiv:1805.07708v1 [cs.LG] 20 May 2018 [Non-Patent Document 2] Jesse Zhang, Brian Cheung, Chelsea Finn, Sergey Levine, Dinesh Jayaraman, "Cautious Adaptation For Reinforcement Learning in Safety-Critical Settings", arXiv:2008.06622v1 [cs.LG] 15 Aug 2020 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the conventional techniques have a problem in that it may be difficult to perform reinforcement learning to support business operations easily and at low cost.

[0008] In reinforcement learning to support business operations, an agent is made to take various actions in the business environment through trial and error, and the optimal action is learned. In this case, it is necessary to prepare the environment for the agent to take the action.

[0009] For example, in business, it is often unclear what should be set as a safe behavior. For this reason, it is difficult to apply the technology of Non-Patent Document 1 to business and design appropriate rewards.

[0010] In addition, preparing a simulation environment such as the sandbox described in Non-Patent Document 2 requires a great deal of cost, and it is difficult to prepare an optimal simulation environment for learning about a business. [Means for solving the problem]

[0011] In order to solve the above-mentioned problems and achieve the objectives, the assistance device is characterized by having an acquisition unit that acquires environmental information, which is information regarding the environment in a business, and behavioral information, which is information regarding behavior in the business, an extraction unit that associates and extracts the environmental information and the behavioral information, and a design unit that designs an environment and reward in reinforcement learning of a model that performs the business based on the combination of the environmental information and the behavioral information associated by the extraction unit. Effect of the Invention

[0012] According to the present invention, reinforcement learning for supporting business can be performed easily and at low cost. [Brief description of the drawings]

[0013] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of a support device according to the first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the environmental information and the behavioral information. [Diagram 3] FIG. 3 is a diagram showing an example of the environmental information and the behavioral information. [Figure 4] FIG. 4 is a diagram showing an example of the environment information and the behavior information. [Diagram 5] FIG. 5 is a diagram showing an example of the environment information and the behavior information. [Figure 6] FIG. 6 is a diagram showing an example of an environment design and an action design. [Figure 7] FIG. 7 is a flowchart showing the flow of the acquisition process. [Figure 8] FIG. 8 is a flowchart showing the flow of the extraction process. [Figure 9] FIG. 9 is a flowchart showing the flow of the learning process. [Figure 10] FIG. 10 is a flowchart showing the flow of the execution process. [Figure 11] FIG. 11 is a diagram illustrating an example of a computer that executes the assistance program. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, an embodiment of an assisting device, an assisting method, and an assisting program according to the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiment described below.

[0015] Conventionally, reinforcement learning of a model to support a business is performed using an environment in which the business is actually performed. In such a case, it is considered that the business may be adversely affected. On the other hand, according to the embodiment, it is possible to perform reinforcement learning of a model to support a business while suppressing adverse effects on the business.

[0016] In the embodiment, the tasks include all tasks performed by humans, such as inputting data into a terminal device such as a personal computer (PC), responding to customer inquiries by voice or text, and inspecting equipment.

[0017] For example, a model for automating input operations into a terminal device automatically performs operations on the terminal device that mimic human input operations based on a captured image of a screen displayed by the terminal device. The model may also use, for example, a neural network.

[0018] [Configuration of the first embodiment] First, the configuration of the support device according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing an example of the configuration of the support device according to the first embodiment.

[0019] As shown in Fig. 1, the support system 1 includes a support device 10 and a terminal device 20. The support device 10 is connected to the terminal device 20. The terminal device 20 is a device such as a PC that allows a worker to perform work related to the worker's business. The support device 10 may be connected to a camera, a microphone, a wearable device, or the like held by the worker.

[0020] Here, a description will be given of each unit of the support device 10. As shown in FIG.

[0021] The input / output unit 11 is an interface for inputting and outputting data. For example, the input / output unit 11 is a network interface card (NIC). The input / output unit 11 can transmit and receive data to and from other devices.

[0022] The input / output unit 11 may also be connected to input devices such as a mouse, a keyboard, etc. The input / output unit 11 may also be connected to output devices such as a display and a speaker.

[0023] The storage unit 12 is a storage device such as a hard disk drive (HDD), a solid state drive (SSD), an optical disk, etc. The storage unit 12 may be a semiconductor memory in which data can be rewritten, such as a random access memory (RAM), a flash memory, or a non-volatile static random access memory (NVSRAM).

[0024] The storage unit 12 stores an OS (Operating System) and various programs executed by the assistance device 10. The storage unit 12 stores learning information 121 and model information 122.

[0025] The learning information 121 is information for performing reinforcement learning. The learning information 121 includes a reward and an environment in reinforcement learning.

[0026] The model information 122 is information such as parameters for constructing a model that supports a business operation. When the model is a neural network, the model information 122 includes weights and biases for each node.

[0027] The model receives input of information indicating the work environment and outputs information related to the action. The assistance device 10 assists the work based on the output information.

[0028] The control unit 13 controls the entire support device 10. The control unit 13 is, for example, an electronic circuit such as a central processing unit (CPU), a micro processing unit (MPU), or a graphics processing unit (GPU), or an integrated circuit such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

[0029] The control unit 13 also has an internal memory for storing programs and control data that define various processing procedures, and executes each process using the internal memory. The control unit 13 also functions as various processing units by the operation of various programs. For example, the control unit 13 has an acquisition unit 131, an extraction unit 132, a design unit 133, a learning unit 134, and an execution unit 135.

[0030] The acquisition unit 131 acquires environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in a business.

[0031] The acquisition unit 131 can acquire information on the behavior of a worker performing a task as behavior information, and acquire information on the environment related to the worker as environmental information. The acquisition unit 131 can also acquire the content of an operation performed by the worker on the terminal device 20 as behavior information, and acquire the state of the terminal device 20 that changes in response to the operation performed by the worker as environmental information.

[0032] For example, the acquisition unit 131 acquires, as environmental information, an image captured by a camera in the viewpoint direction of the worker and sounds around the worker collected by a microphone.

[0033] Furthermore, for example, the acquisition unit 131 acquires, as environmental information, information such as images and sounds output from the terminal device 20 operated by the worker.

[0034] The image acquired by the acquisition unit 131 may be a still image such as a captured image, or may be a moving image. The acquisition unit 131 may acquire the collected voice as a voice file, or may acquire text obtained by converting the collected voice.

[0035] For example, the acquisition unit 131 acquires, as behavior information, information on the physical movement of the worker sensed by a sensor attached to the worker and the contents of the worker's speech collected by a microphone.

[0036] Also, for example, the acquiring unit 131 acquires, as the behavioral information, the contents of the operation performed by the worker on the terminal device 20. The contents of the operation include the time when the keyboard was pressed, the type of the pressed key, the trajectory of the mouse movement, the position and time when the mouse was clicked, and the like.

[0037] At this time, the acquisition unit 131 can acquire the operation content of the worker on the terminal device 20 as behavior information, and acquire a capture image of the screen of the terminal device 20 that changes in response to the operation as environmental information.

[0038] Furthermore, the acquisition unit 131 may acquire information for identifying an application or a window operated by the operator on the terminal device 20 together with the operation content as behavior information.

[0039] The extraction unit 132 extracts the environmental information and the behavioral information in association with each other. In other words, the extraction unit 132 extracts a combination of the environmental information and the behavioral information acquired by the acquisition unit 131.

[0040] The extraction unit 132 can extract behavior information related to behavior in a business operation in association with environment information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior.

[0041] The association between the environmental information and the behavioral information will be described with reference to Fig. 2 to Fig. 5. Fig. 2, Fig. 3, Fig. 4, and Fig. 5 are diagrams showing examples of the environmental information and the behavioral information.

[0042] 2 and 3 show examples of environmental information and behavior information related to a task in which the terminal device 20 is used.

[0043] 2, the extraction unit 132 extracts a captured image 51a and an operation content 52a in association with each other. The captured image 51a corresponds to environmental information. The operation content 52a corresponds to behavior information.

[0044] It is assumed that the worker inputs information into various items and clicks (presses) button 511a with the mouse. In this case, operation content 52a includes information that the type of the operation event is a click, and the coordinates of the cursor when the click is performed.

[0045] In this case, for example, the environment before the action is taken is a captured image of the screen before the button 511a is pressed, while the environment influenced by the action is a captured image of the screen to which the action transitions after the button 511a is pressed.

[0046] The extraction unit 132 may extract the entire captured image 51a as the environmental information, or may extract only the operated button 511a, which is a part of the captured image 51a, as the environmental information.

[0047] 2 is an image obtained by cutting out a part of the screen displayed on the display, whereas the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specific application.

[0048] 3, the extraction unit 132 extracts a captured image 51b and an operation content 52b in association with each other. The captured image 51b corresponds to environmental information. The operation content 52b corresponds to behavior information.

[0049] It is assumed that, in order to input "yokosuka" in the roman alphabet, the worker first types (presses) the "y" key on the keyboard in text box 511b, and then types the "o" key on the keyboard.

[0050] In this case, the operation content 52b includes the type of the operation event being pressing of the "o" key.

[0051] In this case, for example, the environment before the action is taken is a captured image of the screen before the "o" key is pressed. On the other hand, the environment influenced by the action is a captured image of the screen after the "o" key is pressed and "yo" is entered in text box 511b. In romaji input, the hiragana "yo" is displayed by entering "o" after "y".

[0052] The extraction unit 132 may extract the entire captured image 51b as the environmental information, or may extract a part of the captured image 51b, that is, a text box 511b that is the target of an operation, as the environmental information.

[0053] 3 is an image obtained by cutting out a part of the screen displayed on the display, whereas the environmental information may be an image obtained by capturing the entire screen displayed on the display, including the task bar of the OS and the toolbar of the browser or a specific application.

[0054] The extraction unit 132 may extract a series of multiple pieces of environmental information in association with one piece of behavioral information. For example, the extraction unit 132 can extract, as environmental information, capture images of multiple frames of screens in a time series until a predetermined operation content occurs on the terminal device 20.

[0055] The extraction unit 132 may further extract environmental information relating to an environment whose similarity to the environment corresponding to the extracted environmental information is equal to or greater than a threshold value, in association with the behavioral information.

[0056] For example, in addition to the captured image 51b, a previously captured image similar to the captured image 51b may be extracted in association with the operation content 52b.

[0057] For example, the number of words commonly shown in two captured images is taken as the degree of similarity, and the extraction unit 132 considers captured images having a degree of similarity equal to or greater than a threshold value to be similar to each other.

[0058] That is, the extraction unit 132 extracts not only the environmental information at the moment when the behavior indicated by the behavior information was performed, but also similar environmental information from the past.

[0059] FIG. 4 shows an example of environmental information and behavioral information related to telephone response work.

[0060] 4, the extraction unit 132 extracts a voice 51c of a telephone inquiry from a customer and a voice 52c of an operator's reply in association with each other. The voice 51c corresponds to environmental information. The voice 52c corresponds to behavioral information.

[0061] At this time, the extraction unit 132 may extract text obtained by transcribing the speech instead of the speech.

[0062] In this case, for example, the environment before the action is taken is the voice 51c of the telephone inquiry from the customer, while the environment influenced by the action is the voice uttered by the customer in response to the voice 52c of the operator's response.

[0063] FIG. 5 shows an example of environmental information and behavioral information related to equipment inspection work.

[0064] 5, the extraction unit 132 extracts an image 51d from the viewpoint of a moving worker and a position 52d of a destination to which the worker has moved in association with each other. The image 51d corresponds to environmental information. The position 52d corresponds to behavior information.

[0065] In this case, for example, the environment before the action is taken is an image 51d from the viewpoint of the worker while he is moving, whereas the environment influenced by the action is an image from the viewpoint of the worker after he has moved.

[0066] The design unit 133 designs an environment and a reward in reinforcement learning of a model that executes a task, based on a combination of the environment information and the behavioral information associated by the extraction unit 132.

[0067] The design unit 133 assumes that the behavior indicated by the behavior information is the "correct behavior" and performs a design so that a reward is given if the agent takes a similar behavior in the environment in which the behavior was taken.

[0068] 6 is a diagram showing an example of an environment design and a behavior design. As shown in FIG. 6, the design unit 133 performs an environment design and a reward design.

[0069] For example, if clicking a specific button is the "right action," the design unit 133 performs a design so that a positive reward is given to the action of clicking on that button and moving the cursor onto that button. On the other hand, the design unit 133 performs a design so that a negative reward (penalty) is given to the action of clicking anywhere other than on that button.

[0070] Furthermore, the design unit 133 designs the system so that when the same operation as that performed by the worker, i.e., clicking a button, is performed, the environment is transitioned to a captured image after the operation, and when an operation other than clicking a button is performed, the environment is not transitioned and the agent is caused to perform the operation again on the same screen.

[0071] Here, the extraction unit 132 extracts both environmental information related to the environment before the action is taken and environmental information related to the environment influenced by the action. At this time, the design unit 133 performs a design such that the environment before the action is taken is presented to the agent, and the agent transitions to the environment influenced by the action when the agent takes the "correct action."

[0072] The design unit 133 stores the design content in the storage unit 12 as learning information 121.

[0073] The learning unit 134 constructs a learning environment for carrying out reinforcement learning of the model in accordance with the learning information 121. Furthermore, the learning unit 134 updates the model information 122 based on the results of having the agent take an action in the learning environment.

[0074] If the task involves the use of the terminal device 20, the learning unit 134 presents the agent with a captured image of the screen of the terminal device 20 as the environment, and has the agent select an action to be taken on the captured image (such as clicking or moving the cursor).

[0075] If the task involves movement by walking, the learning unit 134 presents the agent with a moving image from the viewpoint of the worker while walking or a still image extracted from the moving image as the environment, and allows the agent to select the direction in which to proceed.

[0076] In this way, by using environmental information about the worker's work environment as a substitute for a learning environment and having the agent act in that environment, it is possible to carry out learning without affecting actual work.

[0077] An agent is a simulated subject that selects an action according to the output of a model constructed from model information 122.

[0078] The learning unit 134 may also perform learning based on the environment information and behavior information of one worker. In this case, it is expected that the learning of behavior that reflects the characteristics of each worker can be achieved.

[0079] On the other hand, the learning unit 134 may perform learning by combining environmental information and behavioral information related to a plurality of workers. In this case, it is expected that a more efficient learning of work procedures can be achieved.

[0080] The execution unit 135 generates a sequence of actions related to a task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 133.

[0081] For example, the execution unit 135 inputs environmental information in an actual business operation into a trained model constructed from the model information 122 and identifies an action based on the output obtained.

[0082] Specifically, the execution unit 135 generates a sequence of actions from environmental information during the worker's work using a trained model, and supports the work based on the generated sequence of actions.

[0083] The business support may be a direct execution of a task, or a provision of actions to be taken in the task to the worker.

[0084] For example, the execution unit 135 automatically inputs information into the items based on a captured image of the screen of the terminal device. Also, for example, the execution unit 135 may infer the next task to be performed from the viewpoint video of the worker and provide information about the inferred task by voice.

[0085] [Processing of the first embodiment] Fig. 7 is a flowchart showing the flow of the acquisition process. As shown in Fig. 7, if the worker has not finished the work (step S101, No), the acquisition unit 131 acquires environmental information during the worker's work (step S102).

[0086] Then, when the worker has taken an action (Yes at step S103), the acquisition unit 131 acquires the action information of the worker (step S104).

[0087] If the worker has not taken any action (step S103, No), the acquisition unit 131 returns to step S101.

[0088] Here, if the worker has finished the work (Yes at step S101), the acquisition unit 131 ends the process.

[0089] 8 is a flowchart showing the flow of the extraction process. When the corresponding environment is not extracted for all the behavioral information acquired by the acquisition unit 131 (step S201, No), the extraction unit 132 determines the behavioral information to be targeted (step S202).

[0090] Then, the extraction unit 132 extracts environmental information corresponding to the targeted behavioral information (step S203).

[0091] If the extraction unit 132 extracts environments corresponding to all pieces of behavior information acquired by the acquisition unit 131 (Yes at step S201), the extraction unit 132 ends the process.

[0092] 9 is a flowchart showing the flow of the learning process. It is assumed here that the reward and environment for reinforcement learning have already been designed by the design unit 133.

[0093] As shown in FIG. 9, when it is not possible to generate a behavior similar to that of the worker for the acquired environmental information (step S301, No), the learning unit 134 determines environmental information to be a target (step S302).

[0094] The learning unit 134 uses the environmental information as a target and an environment for reinforcement learning to learn actions to be taken by trial and error (step S303). The learning unit 134 updates the model information 122 based on the results of the learning.

[0095] When it becomes possible to generate an action similar to that of the worker for the acquired environmental information (Yes in step S301), the learning unit 134 ends the process.

[0096] 10 is a flowchart showing the flow of the execution process. Here, it is assumed that the execution unit 135 constructs a trained model from the model information 122.

[0097] As shown in FIG. 10, if the worker has not finished the task (step S401, No), the execution unit 135 acquires the worker's environment information (step S402).

[0098] Then, if the execution unit 135 can generate an appropriate behavior sequence for the environment (Yes at step S403), it executes the generated behavior sequence using the model (step S404).

[0099] If the execution unit 135 cannot generate an appropriate behavior sequence for the environment (step S403, No), the processing returns to step S401.

[0100] If the worker has finished the task (Yes at step S401), the execution unit 135 ends the process.

[0101] [Advantages of the first embodiment] As described above, the acquisition unit 131 acquires environmental information, which is information about the environment in a task, and behavioral information, which is information about behavior in the task. The extraction unit 132 extracts the environmental information and the behavioral information in association with each other. The design unit 133 designs an environment and a reward in reinforcement learning of a model that executes a task, based on a combination of the environmental information and the behavioral information associated by the extraction unit 132.

[0102] In this way, the assistance device 10 can perform design for reinforcement learning based on the actions and environment related to the work. As a result, according to the embodiment, reinforcement learning for assisting the work can be performed easily and at low cost.

[0103] The acquisition unit 131 also acquires information on the behavior of the worker performing the task as behavior information, and acquires information on the environment related to the worker as the environment information. In this way, the behavior information and the environment information can be easily acquired by focusing on the behavior and environment of the worker.

[0104] The acquisition unit 131 also acquires the contents of the operation of the terminal device 20 by the worker as behavioral information, and acquires the state of the terminal device 20 that changes in response to the operation by the worker as environmental information. This makes it possible to easily acquire environmental information related to the work using the terminal device.

[0105] In addition, the extraction unit 132 extracts the first behavioral information related to the behavior in the business and the first environmental information related to at least one of the environment before the behavior was taken and the environment influenced by the behavior, in association with each other. This makes it easy to design reinforcement learning from the related behavioral information and environmental information.

[0106] In addition, the extraction unit 132 further extracts second environmental information related to an environment whose similarity to the environment is equal to or greater than a threshold value, in association with the first behavioral information. In this manner, by associating a plurality of pieces of similar environmental information with the behavioral information, the accuracy of reinforcement learning can be improved.

[0107] Furthermore, the execution unit 135 generates a sequence of actions related to the task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit 133. This makes it possible to reduce human work and judgment related to the task.

[0108] [System configuration, etc.] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or a part of it can be functionally or physically distributed or integrated in any unit according to various loads, usage conditions, etc. Furthermore, each processing function performed by each device can be realized in whole or in any part by a CPU (Central Processing Unit) and a program analyzed and executed by the CPU, or can be realized as hardware by wired logic. Note that the program may be executed not only by the CPU but also by other processors such as a GPU.

[0109] Furthermore, among the processes described in this embodiment, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified.

[0110] [program] As an embodiment, the support device 10 can be implemented by installing a support program that executes the above support processing as package software or online software on a desired computer. For example, the above support program can be executed by an information processing device, causing the information processing device to function as the support device 10. The information processing device here includes desktop or notebook personal computers. In addition, the information processing device also includes mobile communication terminals such as smartphones, mobile phones, and PHS (Personal Handyphone System), and even slate terminals such as PDA (Personal Digital Assistant), etc.

[0111] The support device 10 may also be implemented as a support server device that provides a service related to the support process to a client, the client being a terminal device used by a user. For example, the support server device is implemented as a server device that provides a support service that receives action information and environmental information in a business and outputs a trained model for supporting the business. In this case, the support server device may be implemented as a Web server, or may be implemented as a cloud that provides a service related to the support process by outsourcing.

[0112] 11 is a diagram showing an example of a computer that executes an assistance program. The computer 1000 includes, for example, a memory 1010 and a CPU 1020. The computer 1000 also includes a hard disk drive interface 1030, a disk drive interface 1040, a serial port interface 1050, a video adapter 1060, and a network interface 1070. These components are connected by a bus 1080.

[0113] The memory 1010 includes a read only memory (ROM) 1011 and a random access memory (RAM) 1012. The ROM 1011 stores a boot program such as a basic input output system (BIOS). The hard disk drive interface 1030 is connected to a hard disk drive 1090. The disk drive interface 1040 is connected to a disk drive 1100. A removable storage medium such as a magnetic disk or an optical disk is inserted into the disk drive 1100. The serial port interface 1050 is connected to a mouse 1110 and a keyboard 1120, for example. The video adapter 1060 is connected to a display 1130, for example.

[0114] The hard disk drive 1090 stores, for example, an OS 1091, an application program 1092, a program module 1093, and program data 1094. That is, a program that defines each process of the support device 10 is implemented as a program module 1093 in which computer-executable code is written. The program module 1093 is stored, for example, in the hard disk drive 1090. For example, a program module 1093 for executing the same process as the functional configuration of the support device 10 is stored in the hard disk drive 1090. The hard disk drive 1090 may be replaced by an SSD (Solid State Drive).

[0115] Furthermore, setting data used in the processing of the above-described embodiment is stored as program data 1094, for example, in the memory 1010 or the hard disk drive 1090. Then, the CPU 1020 reads out the program module 1093 or the program data 1094 stored in the memory 1010 or the hard disk drive 1090 into the RAM 1012 as necessary, and executes the processing of the above-described embodiment.

[0116] Note that the program module 1093 and the program data 1094 are not limited to being stored in the hard disk drive 1090, but may be stored in, for example, a removable storage medium and read by the CPU 1020 via the disk drive 1100 or the like. Alternatively, the program module 1093 and the program data 1094 may be stored in another computer connected via a network (such as a local area network (LAN) or wide area network (WAN)). The program module 1093 and the program data 1094 may then be read by the CPU 1020 from the other computer via the network interface 1070. [Explanation of symbols]

[0117] 1. Support System 10 Support equipment 11 Input / output section 12 Storage section 13 Control section 20 Terminal Equipment 51a, 51b Captured images 51c, 52c Audio 51d Video 52a, 52b Operation details 52d position 121 Learning Information 122 Model Information 131 Acquisition Department 132 Extraction part 133 Design Department 134 Learning Department 135 Executive Department 511a Button 511b Text Box

Claims

1. an acquisition unit that acquires environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business; an extracting unit that extracts, in association with each other, first behavioral information related to the behavior in the business and first environmental information related to at least one of an environment before the behavior was taken and an environment influenced by the behavior, and further extracts, in association with the first behavioral information, second environmental information related to an environment having a similarity to the environment equal to or greater than a threshold; a design unit that designs an environment and a reward in reinforcement learning of a model that performs the task based on a combination of the first environmental information, the second environmental information, and the first behavior information associated by the extraction unit; An assistance device comprising:

2. The support device according to claim 1 , wherein the acquisition unit acquires information on an action of a worker performing the task as the action information, and acquires information on an environment related to the worker as the environment information.

3. The support device according to claim 2, characterized in that the acquisition unit acquires the content of operations performed by the worker on the terminal device as the behavioral information, and acquires a state of the terminal device that changes in response to the operation performed by the worker as the environmental information.

4. 4. The support device according to claim 1, further comprising an execution unit that generates a series of actions related to a task using a model that has undergone reinforcement learning based on the environment and rewards designed by the design unit.

5. An assistance method executed by an assistance device, comprising: An acquisition step of acquiring environmental information, which is information about an environment in a business, and behavioral information, which is information about behavior in the business; an extraction step of extracting, in association with each other, first behavioral information related to the behavior in the business and first environmental information related to at least one of an environment before the behavior was taken and an environment influenced by the behavior, and further extracting, in association with the first behavioral information, second environmental information related to an environment having a similarity to the environment equal to or greater than a threshold; a design process of designing an environment and a reward in reinforcement learning of a model that executes the task based on a combination of the first environmental information, the second environmental information, and the first behavioral information that are associated by the extraction process; Assistance method comprising:

6. A support program for causing a computer to function as the support device according to any one of claims 1 to 4.

Images