Learning device, utilization device, learning method, control method, learning program, and control program

Abstract

A learning device (100) comprises: an acquisition unit (120) that acquires an image and an image generation model; an image generation unit (130) that uses the image and the image generation model to generate an image in which an attention mask has been applied to the image; a word vector generation unit (140) that generates a word vector of an attention region, which is a region of the attention mask; and a learning generation unit (150) that generates an operation output model on the basis of the word vector and information indicating an operation of a device corresponding to the word vector.

Description

Technical Field

[0001] The present disclosure relates to a learning device, an application device, a learning method, a control method, a learning program, and a control program.

Background Art

[0002] It is known that AI (Artificial Intelligence) is used for robot control. For example, in Non-Patent Document 1, it is proposed that by using the machine learning framework "CLIPort" that understands the abstract concept of an object in natural language and combining it with appropriate image abstraction and prior information on spatial meaning, a robot can quickly learn new skills.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, there is a method of performing learning based on information included in the entire image. In this method, in the image, information in an image area other than the learning target area may be learned. When the information in the image area is learned, the judgment accuracy decreases.

[0005] An object of the present disclosure is to improve learning efficiency and judgment accuracy.

Means for Solving the Problems

[0006] A learning device according to one aspect of the present disclosure is provided. The learning device includes an acquisition unit that acquires an image and an image generation model; an image generation unit that uses the image and the image generation model to generate an image by applying an attention mask to the image; a word vector generation unit that generates word vectors of an attention region which is the region of the attention mask; and a learning generation unit that generates an operation output model based on the word vectors and information indicating the operation of the device corresponding to the word vectors. [Effects of the Invention]

[0007] According to this disclosure, it is possible to improve learning efficiency and judgment accuracy. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows the learning device and robot according to Embodiment 1. [Figure 2] This figure shows the hardware of the learning device according to Embodiment 1. [Figure 3] This is a block diagram showing the functions of the learning device according to Embodiment 1. [Figure 4] Figures (A) and (B) show examples of the processing of the image generation unit in Embodiment 1. [Figure 5] This flowchart shows an example of the processing performed by the learning device of Embodiment 1. [Figure 6] This diagram shows the hardware of the robot according to Embodiment 1. [Figure 7] This is a block diagram showing the functions of the robot according to Embodiment 1. [Figure 8] This flowchart shows an example of the process performed by the robot of Embodiment 1. [Figure 9] This flowchart shows an example of the processing performed by the learning device of Embodiment 2. [Figure 10] This flowchart shows an example of the process performed by the robot in the modified example 1 of Embodiment 2. [Figure 11]This flowchart shows an example of the process performed by the robot in the modified example 2 of Embodiment 2. [Figure 12] This is a block diagram showing the functions of the robot according to Embodiment 3. [Modes for carrying out the invention]

[0009] The embodiments will be described below with reference to the drawings.

[0010] Embodiment 1. Figure 1 shows the learning device and robot according to Embodiment 1. Figure 1 shows the learning device 100 and the robot 200. The learning device 100 is a device that executes a learning method. The learning device 100 generates a trained model. The trained model is stored in the robot 200. For example, the trained model is stored in the robot 200 via a network. Robot 200 executes the control method. For example, robot 200 is a harvesting robot. Also, robot 200 is an example of an application device. Furthermore, robot 200 may simply be called a device.

[0011] The learning phase and the application phase will be explained below. In the learning phase, the learning device 100 will be described. In the application phase, the robot 200 will be described. The robot 200 may also have the functions of the learning device 100. In other words, the learning phase may be performed by the robot 200.

[0012] <Learning Phase> The hardware of the learning device 100 will be described. Figure 2 shows the hardware of the learning device according to Embodiment 1. The learning device 100 is also called a computer. The learning device 100 has a processor 101, a volatile memory device 102, and a non-volatile memory device 103.

[0013] Processor 101 controls the entire learning device 100. For example, processor 101 can be a CPU (Central Processing Unit), an FPGA (Field Programmable Gate Array), etc. Processor 101 can also be a multi-processor. Additionally, learning device 100 may have a processing circuit.

[0014] Volatile memory device 102 is the main memory device of learning device 100. For example, volatile memory device 102 is a RAM (Random Access Memory). Non-volatile memory device 103 is the auxiliary storage device of learning device 100. For example, non-volatile memory device 103 is an HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0015] Next, the functions of learning device 100 will be described. FIG. 3 is a block diagram showing the functions of the learning device according to Embodiment 1. Learning device 100 includes a storage unit 110, an acquisition unit 120, an image generation unit 130, a word vector generation unit 140, a learning generation unit 150, an adjustment unit 160, and an output unit 170.

[0016] Storage unit 110 may be realized as a storage area secured in volatile memory device 102 or non-volatile memory device 103. Storage unit 110 may also be referred to as a memory. Part or all of acquisition unit 120, image generation unit 130, word vector generation unit 140, learning generation unit 150, adjustment unit 160, and output unit 170 may be realized by a processing circuit. Also, part or all of acquisition unit 120, image generation unit 130, word vector generation unit 140, learning generation unit 150, adjustment unit 160, and output unit 170 may be realized as modules of a program executed by processor 101. For example, the program executed by processor 101 is also called a learning program or a learning program product. For example, the learning program is recorded on a recording medium.

[0017] The memory unit 110 stores various types of information. For example, the memory unit 110 stores images, which are training data.

[0018] The acquisition unit 120 acquires images that are training data. For example, the acquisition unit 120 acquires the image from the storage unit 110. Alternatively, the acquisition unit 120 may acquire the image from an external device. The external device is a device located outside the learning device 100. For example, the external device could be a cloud server or external memory. The diagram of the external device is omitted.

[0019] The acquisition unit 120 acquires the image generation model from the storage unit 110 or an external device. The structure of the image generation model is similar to that of a CNN (Convolutional Neural Network). The image generation model also has a spatial attention function.

[0020] The image generation unit 130 uses the image and the image generation model to generate an image with an attention mask applied to the acquired image. Specifically, when the image generation unit 130 inputs an image to the image generation model, the image generation model outputs an image with an attention mask applied to the acquired image. An example of an image with an attention mask applied to the acquired image is shown here.

[0021] Figures 4(A) and 4(B) show an example of the processing of the image generation unit in Embodiment 1. Figure 4(A) is an image acquired by the acquisition unit 120. Figure 4(B) is an image with an attention mask applied to the acquired image. As shown in Figure 4(B), the image generation unit 130 generates an image in which the area where the apple is located is the attention region. The areas other than where the apple is located are then masked.

[0022] The word vector generation unit 140 generates word vectors for the attention region, which is the area of ​​the attention mask. These word vectors may also be called language vectors or vector representations. For example, the word vector generation unit 140 receives an image obtained by applying an attention mask to the acquired image shown in Figure 4(B).

[0023] The word vector generation process is described below. For example, the word vector generation unit 140 generates word vectors using a VLM (Vision and Language Model). For example, if the memory unit 110 stores a VLM, the word vector generation unit 140 generates word vectors using that VLM. Alternatively, if the VLM is stored in an external device, the acquisition unit 120 may acquire the VLM from the external device. The word vector generation unit 140 then generates word vectors using that VLM. Furthermore, if the VLM is stored in an external device, the word vector generation unit 140 may transmit the generated image to the external device. The word vector generation unit 140 then transmits a word vector generation instruction to the external device. The word vector generation unit 140 receives the word vectors from the external device.

[0024] For example, if the attention area indicates an apple, the word vector generation unit 140 generates a word vector that indicates an apple.

[0025] The learning generation unit 150 generates an action output model based on word vectors and information indicating the actions of the robot 200 corresponding to those word vectors.

[0026] For example, the learning generation unit 150 generates an action output model based on a word vector representing "apple" and information representing the action of "grabbing an apple." Alternatively, for example, the learning generation unit 150 generates an action output model based on a word vector representing "small apple" and information representing the action of "moving forward."

[0027] Information indicating the operation of the robot 200 corresponding to the word vector may be stored in advance in the memory unit 110. Furthermore, the learning generation unit 150 may generate an operation output model based on the word vector and the information indicating the operation of the device corresponding to the word vector.

[0028] The adjustment unit 160 performs an evaluation of the action. For example, the evaluation of the action is a positive or negative reward value. If, as a result of the action evaluation, the reward value for the entire episode is higher than the baseline value and stable, the evaluation ends. If the evaluation is lower than the baseline value for the entire episode or unstable, the adjustment unit 160 adjusts the image generation model. For example, the adjustment unit 160 adjusts the weights of the neural network that constitutes the image generation model.

[0029] The output unit 170 outputs an image generation model and a motion output model. For example, the output unit 170 outputs the image generation model and the motion output model to the storage unit 110. Alternatively, for example, the output unit 170 outputs the image generation model and the motion output model to the robot 200 via the network.

[0030] Next, the processes performed by the learning device 100 will be explained using a flowchart. Figure 5 is a flowchart showing an example of the processing performed by the learning device of Embodiment 1. (Step S11) The acquisition unit 120 acquires an image. (Step S12) The acquisition unit 120 acquires the image generation model. (Step S13) The image generation unit 130 uses the image and the image generation model to generate an image in which an attention mask has been applied to the image. (Step S14) The word vector generation unit 140 generates word vectors for the attention region, which is the region of the attention mask.

[0031] (Step S15) The learning generation unit 150 generates an action output model based on word vectors and information indicating the actions of the robot 200 corresponding to those word vectors. (Step S16) The adjustment unit 160 performs an evaluation of the operation. If the evaluation of the operation shows that the reward value for the entire episode is higher than the baseline value and stable, the evaluation is terminated. If the evaluation is lower than the baseline value for the entire episode or unstable, the adjustment unit 160 adjusts the image generation model. (Step S17) The adjustment unit 160 determines whether or not to terminate the process. For example, the adjustment unit 160 terminates the process if it receives a termination instruction from the user. Also, for example, the adjustment unit 160 terminates the process if a state in which a high reward value is obtained in the evaluation of the operation continues (i.e., it is in a convergence state). If the process is not terminated, the process proceeds to step S13.

[0032] After the processing shown in Figure 5 is completed, the output unit 170 outputs an image generation model and an action output model. The output image generation model and action output model are also called trained models.

[0033] One method involves learning based on the information contained in the entire image. However, this method may involve learning information from areas of the image other than the target area for learning. For example, it may involve learning information from areas of the image other than the area where the apple in Figure 4 is located. If information from such areas is learned, the accuracy of determining which area of ​​the image is appropriate to use will decrease. Furthermore, if information from areas other than the target area is learned, the accuracy of the learning process will decrease. In addition, if information from areas other than the target area is learned, it will take time for the accuracy of the learning model to reach a certain level. In other words, if information from areas other than the target area is learned, the learning process will not progress.

[0034] According to Embodiment 1, the learning device 100 learns information about the attention region. That is, the learning device 100 learns information about the region to be learned. Therefore, the learning device 100 can improve the accuracy of its learning. Furthermore, because the accuracy of learning is improved, the learning device 100 can reduce the learning time. That is, the learning efficiency is improved. As will be described later, since the region to be learned is learned, the robot 200 can detect the appropriate region in the image. Therefore, the judgment accuracy of the robot 200 is improved.

[0035] Next, we will explain the utilization phase. For the purposes of this explanation, we will assume that Robot 200 is located in an apple orchard. <Utilization Phase> This section describes the hardware of robot 200. Figure 6 shows the hardware of the robot according to Embodiment 1. The robot 200 includes a processor 201, a volatile memory device 202, a non-volatile memory device 203, and a camera 204. The robot 200 also includes a drive device and the like.

[0036] The processor 201 controls the entire robot 200. For example, the processor 201 could be a CPU, FPGA, etc. The processor 201 may also be a multiprocessor. The robot 200 may also have processing circuits.

[0037] The volatile memory device 202 is the main memory of the robot 200. For example, the volatile memory device 202 is RAM. The non-volatile memory device 203 is the auxiliary memory of the robot 200. For example, the non-volatile memory device 203 is an HDD or SSD. Camera 204 takes pictures of objects.

[0038] Next, we will explain the functions of robot 200. Figure 7 is a block diagram showing the functions of the robot according to Embodiment 1. The robot 200 includes a storage unit 210, an acquisition unit 220, an image generation unit 230, a word vector generation unit 240, and a control unit 250.

[0039] The storage unit 210 may be implemented as a storage area reserved in a volatile storage device 202 or a non-volatile storage device 203. The storage unit 210 may also be called a memory. Some or all of the acquisition unit 220, image generation unit 230, word vector generation unit 240, and control unit 250 may be implemented by processing circuits. Alternatively, some or all of the acquisition unit 220, image generation unit 230, word vector generation unit 240, and control unit 250 may be implemented as modules of a program executed by the processor 201. For example, the program executed by the processor 201 is also called a control program or control program product. For example, the control program is recorded on a recording medium.

[0040] The memory unit 210 stores various types of information. For example, the memory unit 210 stores the image generation model and the motion output model output by the learning device 100.

[0041] The acquisition unit 220 acquires images from the camera 204.

[0042] The acquisition unit 220 acquires the image generation model output by the learning device 100 from the storage unit 210 or an external device. The external device is a device located outside the robot 200. For example, the external device is the learning device 100 or a cloud server.

[0043] The image generation unit 230 uses the image and the image generation model to generate an image by applying an attention mask to the acquired image.

[0044] The word vector generation unit 240 generates word vectors in the attention region. The function of the word vector generation unit 240 is the same as that of the word vector generation unit 140. Therefore, a description of the function of the word vector generation unit 240 will be omitted.

[0045] The acquisition unit 220 acquires the operation output model output by the learning device 100 from the storage unit 210 or an external device.

[0046] The control unit 250 uses a word vector and an action output model to identify the action of the robot 200 corresponding to the word vector. More specifically, when the control unit 250 inputs a word vector to the action output model, the action output model outputs the action of the robot 200 corresponding to the word vector. For example, the control unit 250 uses a word vector indicating "apple" and the action output model to identify the action of "grab an apple." Alternatively, for example, the control unit 250 uses a word vector indicating "small apple" and the action output model to identify the action of "move forward."

[0047] The control unit 250 controls the robot 200 so that it performs a specified action. For example, the control unit 250 controls the robot 200 so that it grasps an apple. Alternatively, for example, the control unit 250 controls the robot 200 so that it moves forward (i.e., so that the robot 200 approaches the apple).

[0048] Next, we will explain the processes that robot 200 will perform using a flowchart. Figure 8 is a flowchart showing an example of the process performed by the robot of Embodiment 1. (Step S21) The acquisition unit 220 acquires an image. (Step S22) The acquisition unit 220 acquires the image generation model from the storage unit 210. (Step S23) The image generation unit 230 uses the image and the image generation model to generate an image in which an attention mask is applied to the acquired image. (Step S24) The word vector generation unit 240 generates word vectors in the attention area.

[0049] (Step S25) The acquisition unit 220 acquires the operation output model from the storage unit 210. (Step S26) The control unit 250 uses the word vector and the motion output model to determine the operation of the robot 200 corresponding to the word vector. (Step S27) The control unit 250 controls the robot 200 so that it performs the specified operation.

[0050] This allows robot 200 to perform actions according to word vectors.

[0051] A modified example of Embodiment 1. <Learning Phase> Embodiment 1 describes a case in which the learning generation unit 150 generates an action output model based on word vectors and information indicating the operation of the robot 200 corresponding to those word vectors. When generating the action output model, the learning generation unit 150 may further use feature quantities based on images acquired by the acquisition unit 120 to generate the action output model. Detailed modifications are described below.

[0052] The image generation unit 130 acquires feature quantities, including positional information of objects contained in the image, based on the image. For example, the image generation unit 130 acquires information (i.e., feature quantities) before the information is input to the pooling layer of the image generation model. In other words, the image generation unit 130 acquires information (i.e., feature quantities) output after the convolutional layer. Note that the objects contained in the image are those located in the attention region.

[0053] The learning generation unit 150 generates an action output model based on feature quantities, word vectors, and information indicating the actions of the robot 200 corresponding to the feature quantities and word vectors. As a result, the action output model can output actions based on the word indicated by the word vector and the positional information indicated by that word. For example, the action output model outputs the action of "grab the apple with the right hand" based on the word vector indicating "apple" and the feature quantity indicating that "apple" is on the right side of the image.

[0054] According to a modified version of Embodiment 1, the learning device 100 can improve the accuracy of its output by further using feature quantities.

[0055] <Utilization Phase> Next, we will explain when features are used in the utilization phase. The image generation unit 230 acquires feature quantities, including the positional information of objects contained in the image, based on the image acquired by the acquisition unit 220. The objects contained in the image are those located in the attention region. That is, the image generation unit 230 acquires feature quantities, including the positional information of objects located in the attention region.

[0056] The control unit 250 uses the feature vectors, word vectors, and motion output model to identify the robot 200's actions corresponding to the feature vectors and word vectors. The control unit 250 then controls the robot 200 so that it performs the identified actions.

[0057] Embodiment 2. Next, Embodiment 2 will be described. Embodiment 2 will mainly describe the differences from Embodiment 1. In Embodiment 2, the explanation of matters common to Embodiment 1 will be omitted.

[0058] <Learning Phase> The acquisition unit 120 acquires additional information, which is information other than word vectors. For example, the acquisition unit 120 acquires additional information from the storage unit 110 or an external device. Alternatively, for example, the acquisition unit 120 acquires additional information through user input.

[0059] Additional information includes user instructions and information obtained from the manual. For example, user instructions and information obtained from the manual include information indicating prohibited actions and restricted actions. In other words, additional information is information that cannot be obtained from the image acquired by the acquisition unit 120.

[0060] The learning generation unit 150 generates an action output model based on the additional information, word vectors, and information indicating the robot 200's movements. As a result, the learning generation unit 150 generates an action output model in which movements that take the additional information into account have been learned.

[0061] Next, the processes performed by the learning device 100 will be explained using a flowchart. Figure 9 is a flowchart showing an example of the processing performed by the learning device of Embodiment 2. The processing in Figure 9 differs from the processing in Figure 5 in that steps S14a and S15a are performed. Therefore, steps S14a and S15a will be explained in Figure 9. The explanation of the processing other than steps S14a and S15a will be omitted.

[0062] (Step S14a) The acquisition unit 120 acquires additional information. (Step S15a) The learning generation unit 150 generates an action output model based on the additional information, word vectors, and information indicating the actions of the robot 200. The generated operational output model will be used in the utilization phase.

[0063] According to Embodiment 2, the learning device 100 can generate an action output model in which actions that take additional information into consideration have been learned.

[0064] Modification 1 of Embodiment 2. <Utilization Phase> Embodiment 2 describes a case in which, during the learning phase, an action output model is generated in which actions that take additional information into consideration are learned. Then, the case in which the generated action output model is used in the utilization phase is described.

[0065] In the first modification of Embodiment 2, we describe a case in which, during the utilization phase, the operation of the robot 200 is controlled using the operation output model generated in Embodiment 1 (i.e., the operation output model without learned additional information) and the additional information.

[0066] Figure 10 is a flowchart showing an example of the process performed by the robot in Modification 1 of Embodiment 2. The process in Figure 10 differs from the process in Figure 8 in that steps S26a, 26b, and 27a are performed. Therefore, steps S26a, 26b, and 27a will be explained in Figure 10. The explanation of processes other than steps S26a, 26b, and 27a will be omitted.

[0067] (Step S26a) The acquisition unit 220 acquires additional information. For example, the acquisition unit 220 acquires additional information from the storage unit 210 or an external device. (Step S26b) The control unit 250 determines an operation that takes the additional information into consideration, based on the identified operation and the additional information. (Step S27a) The control unit 250 controls the robot 200 so that it performs the determined operation.

[0068] Let's explain with a specific example. For instance, suppose the specified action is "grab the apple." The additional information indicates the constraints on grasping the apple. The constraint on grasping the apple is "do not grip the apple too tightly." The control unit 250 controls the robot 200 so that it does not grip the apple too tightly when it grasps it.

[0069] According to Modification 1 of Embodiment 2, the robot 200 can perform actions that take additional information into consideration.

[0070] Modification 2 of Embodiment 2. <Utilization Phase> Modification 1 of Embodiment 2 describes a case in which the operation of the robot 200 is controlled using additional information after the operation has been identified. Modification 2 of Embodiment 2 describes a case in which the operation of the robot 200 is controlled by changing the word vector using additional information before the operation has been identified.

[0071] Figure 11 is a flowchart showing an example of a process performed by a robot in a modified example 2 of Embodiment 2. The process in Figure 11 differs from the process in Figure 8 in that steps S25a, 25b, and 26c are performed. Therefore, steps S25a, 25b, and 26c will be explained in Figure 11. The explanation of processes other than steps S25a, 25b, and 26c will be omitted.

[0072] (Step S25a) The acquisition unit 220 acquires additional information. For example, the acquisition unit 220 acquires additional information from the storage unit 210 or an external device. (Step S25b) The word vector generation unit 240 modifies the word vector generated in step S24 into a word vector that takes the additional information into account, based on the additional information. (Step S26c) The control unit 250 uses the modified word vector and the motion output model to determine the operation of the robot 200 corresponding to the modified word vector.

[0073] According to Modification 2 of Embodiment 2, the robot 200 can perform actions that take additional information into consideration.

[0074] Embodiment 3. Next, Embodiment 3 will be described. Embodiment 3 will mainly describe the differences from Embodiments 1 and 2. In Embodiment 3, the explanation of matters common to Embodiments 1 and 2 will be omitted.

[0075] <Utilization Phase> Figure 12 is a block diagram showing the functions of the robot according to Embodiment 3. The robot 200 further has an output unit 260. Part or all of the output unit 260 may be implemented by a processing circuit. Alternatively, part or all of the output unit 260 may be implemented as a module of a program executed by a processor 201.

[0076] The output unit 260 outputs information indicating the attention area and information indicating the action performed by the robot 200. For example, the output unit 260 outputs the information indicating the attention area and the information indicating the action to the storage unit 210. Alternatively, for example, the output unit 260 outputs the information indicating the attention area and the information indicating the action to an external device. Furthermore, if the robot 200 has a display, the output unit 260 outputs the information indicating the attention area and the information indicating the action to the display.

[0077] For example, the user can recognize that the robot 200 focused on the attention area and performed an action based on that area, as information indicating the attention area and information indicating the action are output to the display. In other words, the user can recognize the reason for controlling the robot 200.

[0078] Furthermore, the output unit 260 may generate and output an explanatory text based on information indicating the attention area and information indicating the operation. For example, the explanatory text may be, "Robot 200 detected an apple and grasped the apple."

[0079] Furthermore, the output unit 260 may output the operation history of the robot 200 to the storage unit 210, an external device, or a display. For example, the user can recognize a series of actions of the robot 200 by having the operation history of the robot 200 output to the display.

[0080] Furthermore, the output unit 260 may generate a summary of the robot 200's movements based on the robot 200's movement history. For example, the output unit 260 generates a summary of the robot 200's movements for a day using the robot 200's movement history for a day and LLM (Large Language Models). The output unit 260 outputs the summary of the robot 200's movements to the storage unit 210, an external device, or a display. For example, a user can easily understand the robot 200's movements for a day by seeing the summary of the robot 200's movements output to the display.

[0081] The utilization phase described cases where a robot performs the processing. However, in the utilization phase, devices other than robots may also perform the processing. For example, such devices could be devices that perform autonomous driving (i.e., cars) or devices that identify image scenes. Such devices are also called utilization devices.

[0082] Each embodiment is merely an example, and various modifications are possible within the scope of this disclosure. Furthermore, each embodiment and each variation thereof can be combined with each other as appropriate. For example, a variation of Embodiment 1 can be combined with Embodiment 2. [Explanation of symbols]

[0083] 100 Learning device, 101 Processor, 102 Volatile memory device, 103 Non-volatile memory device, 110 Storage unit, 120 Acquisition unit, 130 Image generation unit, 140 Word vector generation unit, 150 Learning generation unit, 160 Adjustment unit, 170 Output unit, 200 Robot, 201 Processor, 202 Volatile memory device, 203 Non-volatile memory device, 204 Camera, 210 Storage unit, 220 Acquisition unit, 230 Image generation unit, 240 Word vector generation unit, 250 Control unit, 260 Output unit.

Claims

1. An acquisition unit that acquires images and image generation models, An image generation unit that generates an image by applying an attention mask to the image using the aforementioned image and the image generation model, A word vector generation unit generates word vectors for the attention region, which is the region of the attention mask, A learning generation unit that generates an operation output model based on the word vector and information indicating the operation of the device corresponding to the word vector, A learning device having the following features.

2. The learning generation unit generates the operation output model based on feature quantities including positional information of objects in the attention region, word vectors, and information indicating the operation of the device in accordance with the feature quantities and word vectors. The learning device according to claim 1.

3. The acquisition unit acquires additional information, which is information other than the word vector, The learning generation unit generates the operation output model based on the additional information, the word vectors, and information indicating the operation of the device corresponding to the word vectors. The learning device according to claim 1 or 2.

4. It is a device for utilization, An acquisition unit that acquires an image, an image generation model, and an operation output model, An image generation unit that generates an image by applying an attention mask to the image using the aforementioned image and the image generation model, A word vector generation unit generates word vectors for the attention region, which is the region of the attention mask, A control unit that uses the word vector and the operation output model to identify the operation of the utilization device corresponding to the word vector, and controls the utilization device so that it performs the identified operation, A device that utilizes the features of a device.

5. The control unit uses feature quantities including positional information of objects in the attention area, word vectors, and the operation output model to determine the operation of the utilization device in accordance with the feature quantities and word vectors. The utilization device according to claim 4.

6. The acquisition unit acquires additional information, which is information other than the word vector, The control unit determines an operation that takes the additional information into consideration, based on the identified operation and the additional information, and controls the utilization device so that the utilization device performs the determined operation. The utilization device according to claim 4 or 5.

7. The acquisition unit acquires additional information, which is information other than the word vector, The word vector generation unit modifies the word vector into a word vector that takes the additional information into consideration. The control unit uses the modified word vector and the operation output model to identify the operation of the utilization device corresponding to the modified word vector. The utilization device according to claim 4 or 5.

8. The system further includes an output unit that outputs information indicating the attention region and information indicating the operation. The utilization device according to claim 4 or 5.

9. The system further includes an output unit that generates an explanatory text and outputs the explanatory text based on the information indicating the attention area and the information indicating the operation. The utilization device according to claim 4 or 5.

10. The device further includes an output unit that outputs the operating history of the aforementioned utilization device. The utilization device according to claim 4 or 5.

11. The system further includes an output unit that outputs a summary of the operation of the utilization device based on the operation history of the utilization device. The utilization device according to claim 4 or 5.

12. The learning device, Obtain the image and the image generation model, Using the aforementioned image and the image generation model, an image is generated by applying an attention mask to the aforementioned image. A word vector is generated for the attention region, which is the region of the attention mask. Based on the word vector and information indicating the operation of the device corresponding to the word vector, an operation output model is generated. Learning methods.

13. The device being used is, The system acquires an image, an image generation model, and an action output model; uses the image and the image generation model to generate an image with an attention mask applied to it; generates word vectors for the attention region, which is the region of the attention mask; and uses the word vectors and the action output model to identify the operation of the utilization device corresponding to the word vectors. Controlling the utilization device so that it performs a specified operation, Control method.

14. In the learning device, Obtain the image and the image generation model, Using the aforementioned image and the aforementioned image generation model, an image is generated by applying an attention mask to the aforementioned image. A word vector is generated for the attention region, which is the region of the attention mask. Based on the word vector and information indicating the operation of the device corresponding to the word vector, an operation output model is generated. A learning program that executes a process.

15. In the device being used, The system acquires an image, an image generation model, and an action output model; uses the image and the image generation model to generate an image with an attention mask applied to it; generates word vectors for the attention region, which is the region of the attention mask; and uses the word vectors and the action output model to identify the operation of the utilization device corresponding to the word vectors. Controlling the utilization device so that it performs a specified operation, A control program that executes a process.

Images