Learning device, utilization device, learning method, control method, learning program, and control program

Abstract

A learning device (100) comprises: an acquisition unit (120) that acquires an image and an image generation model; an image generation unit (130) that uses the image and the image generation model to generate an image in which an attention mask has been applied to the image; a word vector generation unit (140) that generates a word vector of an attention region, which is a region of the attention mask; and a learning generation unit (150) that generates an operation output model on the basis of the word vector and information indicating an operation of a device corresponding to the word vector.

Description

Learning device, utilization device, learning method, control method, learning program, and control program

[0001] The present disclosure relates to a learning device, a utilization device, a learning method, a control method, a learning program, and a control program.

[0002] It is known that AI (Artificial Intelligence) is used to control robots. For example, Non-Patent Document 1 proposes that a robot can quickly learn new skills by using CLIPort, a machine learning framework that understands the abstract concepts of objects in natural language, in combination with appropriate image abstraction and prior information on spatial meaning.

[0003] Mohit Shridhar et al. "CLIPORT: What and Where Pathways for Robotic Manipulation"

[0004] There is a method for learning based on information contained in the entire image. In this method, information from an image region other than the learning target region may be learned. If information from the image region is learned, the accuracy of the judgment will be low.

[0005] An object of the present disclosure is to improve learning efficiency and judgment accuracy.

[0006] According to one aspect of the present disclosure, there is provided a learning device including: an acquisition unit that acquires an image and an image generation model; an image generation unit that uses the image and the image generation model to generate an image by applying an attention mask to the image; a word vector generation unit that generates word vectors for an attention region that is a region of the attention mask; and a learning generation unit that generates an action output model based on the word vectors and information indicating an action of the device corresponding to the word vectors.

[0007] According to the present disclosure, it is possible to improve learning efficiency and judgment accuracy.

[0008] FIG. 1 is a diagram illustrating a learning device and a robot according to a first embodiment. FIG. 2 is a diagram illustrating hardware possessed by the learning device according to the first embodiment. FIG. 3 is a block diagram illustrating functions of the learning device according to the first embodiment. (A) and (B) are diagrams illustrating an example of processing by an image generation unit according to the first embodiment. FIG. 4 is a flowchart illustrating an example of processing performed by the learning device according to the first embodiment. FIG. 5 is a diagram illustrating hardware possessed by the robot according to the first embodiment. FIG. 6 is a block diagram illustrating functions of the robot according to the first embodiment. FIG. 7 is a flowchart illustrating an example of processing performed by the robot according to the first embodiment. FIG. 8 is a flowchart illustrating an example of processing performed by the learning device according to the second embodiment. FIG. 9 is a flowchart illustrating an example of processing performed by the robot according to the first modification of the second embodiment. FIG. 10 is a block diagram illustrating functions of the robot according to the third embodiment.

[0009] Hereinafter, an embodiment will be described with reference to the drawings.

[0010] Embodiment 1. FIG. 1 is a diagram showing a learning device and a robot according to embodiment 1. FIG. 1 shows a learning device 100 and a robot 200. The learning device 100 is a device that executes a learning method. The learning device 100 generates a trained model. The trained model is stored in the robot 200. For example, the trained model is stored in the robot 200 via a network. The robot 200 executes a control method. For example, the robot 200 is a harvesting robot. The robot 200 is also an example of a utilization device. Furthermore, the robot 200 may simply be called a device.

[0011] The learning phase and the utilization phase will be described below. In the learning phase, the learning device 100 will be described. In the utilization phase, the robot 200 will be described. The robot 200 may also have the functions of the learning device 100. That is, the learning phase may be performed by the robot 200.

[0012] <Learning Phase> The hardware of the learning device 100 will be described. Fig. 2 is a diagram showing the hardware of the learning device of embodiment 1. The learning device 100 is also called a computer. The learning device 100 has a processor 101, a volatile storage device 102, and a non-volatile storage device 103.

[0013] The processor 101 controls the entire learning device 100. For example, the processor 101 is a central processing unit (CPU) or a field programmable gate array (FPGA). The processor 101 may be a multiprocessor. The learning device 100 may also include a processing circuit.

[0014] The volatile memory device 102 is a main memory device of the learning device 100. For example, the volatile memory device 102 is a RAM (Random Access Memory). The non-volatile memory device 103 is an auxiliary memory device of the learning device 100. For example, the non-volatile memory device 103 is a HDD (Hard Disk Drive) or an SSD (Solid State Drive).

[0015] Next, we will explain the functions of the learning device 100. Figure 3 is a block diagram showing the functions of the learning device of embodiment 1. The learning device 100 has a storage unit 110, an acquisition unit 120, an image generation unit 130, a word vector generation unit 140, a learning generation unit 150, an adjustment unit 160, and an output unit 170.

[0016] The storage unit 110 may be realized as a storage area secured in the volatile storage device 102 or the non-volatile storage device 103. The storage unit 110 may also be called a memory. Some or all of the acquisition unit 120, image generation unit 130, word vector generation unit 140, learning generation unit 150, adjustment unit 160, and output unit 170 may be realized by a processing circuit. Furthermore, some or all of the acquisition unit 120, image generation unit 130, word vector generation unit 140, learning generation unit 150, adjustment unit 160, and output unit 170 may be realized as program modules executed by the processor 101. For example, the program executed by the processor 101 is also called a learning program or a learning program product. For example, the learning program is recorded on a recording medium.

[0017] The storage unit 110 stores various information, for example, images that are learning data.

[0018] The acquisition unit 120 acquires images that are learning data. For example, the acquisition unit 120 acquires the images from the storage unit 110. Alternatively, for example, the acquisition unit 120 acquires the images from an external device. Note that the external device is a device that exists outside the learning device 100. For example, the external device is a cloud server, an external memory, or the like. The external device is not shown in the drawing.

[0019] The acquisition unit 120 acquires the image generation model from the storage unit 110 or an external device. The structure of the image generation model is similar to that of a convolutional neural network (CNN). The image generation model also has a spatial attention function.

[0020] The image generation unit 130 uses the image and the image generation model to generate an image by applying an attention mask to the acquired image. Specifically, when the image generation unit 130 inputs an image to the image generation model, the image generation model outputs an image by applying the attention mask to the acquired image. Here, an example of an image by applying the attention mask to the acquired image is shown.

[0021] 4A and 4B are diagrams illustrating an example of processing by the image generation unit in the first embodiment. Fig. 4A shows an image acquired by the acquisition unit 120. Fig. 4B shows an image in which an attention mask is applied to the acquired image. As shown in Fig. 4B, the image generation unit 130 generates an image in which the area where the apple is present is the attention area. Areas other than the area where the apple is present are masked.

[0022] The word vector generation unit 140 generates word vectors for the attention region, which is the region of the attention mask. Note that the word vectors may also be called language vectors or vector representations. For example, the word vector generation unit 140 receives an image obtained by applying the attention mask to the acquired image shown in FIG. 4(B).

[0023] The process of generating word vectors will now be described. For example, the word vector generation unit 140 generates word vectors using a VLM (Vision and Language Model). For example, if the storage unit 110 stores a VLM, the word vector generation unit 140 generates word vectors using the VLM. Alternatively, if the VLM is stored in an external device, the acquisition unit 120 may acquire the VLM from the external device. Then, the word vector generation unit 140 generates word vectors using the VLM. Furthermore, if the VLM is stored in an external device, the word vector generation unit 140 may transmit the generated image to the external device. Then, the word vector generation unit 140 transmits an instruction to generate word vectors to the external device. The word vector generation unit 140 receives word vectors from the external device.

[0024] For example, if the attention region indicates an apple, the word vector generation unit 140 generates a word vector indicating an apple.

[0025] The learning generation unit 150 generates an action output model based on the word vector and information indicating the action of the robot 200 according to the word vector.

[0026] For example, the learning generation unit 150 generates a movement output model based on a word vector indicating "apple" and information indicating the movement of "grabbing an apple." Also, for example, the learning generation unit 150 generates a movement output model based on a word vector indicating "small apple" and information indicating the movement of "moving forward."

[0027] Note that information indicating the behavior of the robot 200 corresponding to the word vector may be stored in advance in the storage unit 110. Furthermore, the learning generation unit 150 may generate a behavior output model based on the word vector and information indicating the behavior of the device corresponding to the word vector.

[0028] The adjustment unit 160 performs an evaluation of the action. For example, the evaluation of the action is a positive or negative reward value. If the result of the evaluation of the action is that the reward value for the entire episode is higher than the reference value and stable, the evaluation ends. If the evaluation is lower than the reference value for the entire episode or is unstable, the adjustment unit 160 adjusts the image generation model. For example, the adjustment unit 160 adjusts the weights of the neural network that constitutes the image generation model.

[0029] The output unit 170 outputs the image generation model and the action output model. For example, the output unit 170 outputs the image generation model and the action output model to the storage unit 110. Furthermore, for example, the output unit 170 outputs the image generation model and the action output model to the robot 200 via a network.

[0030] Next, the processing executed by the learning device 100 will be described using a flowchart. FIG. 5 is a flowchart showing an example of processing executed by the learning device of embodiment 1. (Step S11) The acquisition unit 120 acquires an image. (Step S12) The acquisition unit 120 acquires an image generation model. (Step S13) The image generation unit 130 uses the image and the image generation model to generate an image by applying an attention mask to the image. (Step S14) The word vector generation unit 140 generates word vectors for the attention region, which is the region of the attention mask.

[0031] (Step S15) The learning generation unit 150 generates a movement output model based on the word vector and information indicating the movement of the robot 200 corresponding to the word vector. (Step S16) The adjustment unit 160 performs movement evaluation. If the result of the movement evaluation shows that the reward value for the entire episode is higher than the reference value and stable, the evaluation ends. If the evaluation is lower than the reference value for the entire episode or is unstable, the adjustment unit 160 adjusts the image generation model. (Step S17) The adjustment unit 160 determines whether to end the process. For example, the adjustment unit 160 ends the process when it receives an end instruction through a user operation. Furthermore, for example, the adjustment unit 160 ends the process when a state in which a high reward value is obtained in the movement evaluation continues (i.e., when the state is converged). If the process does not end, the process proceeds to step S13.

[0032] 5 is completed, the output unit 170 outputs the image generation model and the action output model. The output image generation model and the action output model are also called trained models.

[0033] Here, there is a method of performing learning based on information contained in the entire image. In this method, information on image regions other than the learning target region may be learned. For example, information on image regions other than the image region in Figure 4 where the apple is present may be learned. When information on such image regions is learned, the accuracy of determining which region in the image is appropriate to use decreases. Furthermore, when information on image regions other than the learning target region is learned, the accuracy of learning decreases. Furthermore, when information on image regions other than the learning target region is learned, it takes time for the accuracy of the learning model to reach a certain level. In other words, when information on image regions other than the learning target region is learned, learning does not progress.

[0034] According to the first embodiment, the learning device 100 learns information about the attention region. That is, the learning device 100 learns information about the learning target region. Therefore, the learning device 100 can improve the accuracy of learning. Furthermore, because the learning accuracy is improved, the learning device 100 can reduce the learning time. That is, the learning efficiency is improved. As the learning target region is learned, as will be described later, the robot 200 detects appropriate regions in the image. Therefore, the judgment accuracy of the robot 200 is improved.

[0035] Next, the utilization phase will be described. In the following description, it is assumed that the robot 200 is in an apple orchard. <Utilization Phase> The hardware of the robot 200 will be described. FIG. 6 is a diagram showing the hardware of the robot of embodiment 1. The robot 200 has a processor 201, a volatile storage device 202, a non-volatile storage device 203, and a camera 204. The robot 200 also has a drive device and the like.

[0036] The processor 201 controls the entire robot 200. For example, the processor 201 is a CPU, an FPGA, or the like. The processor 201 may be a multiprocessor. The robot 200 may also have a processing circuit.

[0037] The volatile storage device 202 is a main storage device of the robot 200. For example, the volatile storage device 202 is a RAM. The nonvolatile storage device 203 is an auxiliary storage device of the robot 200. For example, the nonvolatile storage device 203 is an HDD or SSD. The camera 204 captures images of objects.

[0038] Next, a description will be given of the functions of the robot 200. Fig. 7 is a block diagram showing the functions of the robot according to embodiment 1. The robot 200 has a storage unit 210, an acquisition unit 220, an image generation unit 230, a word vector generation unit 240, and a control unit 250.

[0039] The storage unit 210 may be realized as a storage area secured in the volatile storage device 202 or the non-volatile storage device 203. The storage unit 210 may also be called a memory. Some or all of the acquisition unit 220, image generation unit 230, word vector generation unit 240, and control unit 250 may be realized by processing circuits. Furthermore, some or all of the acquisition unit 220, image generation unit 230, word vector generation unit 240, and control unit 250 may be realized as program modules executed by the processor 201. For example, the program executed by the processor 201 is also called a control program or a control program product. For example, the control program is recorded on a recording medium.

[0040] The storage unit 210 stores various information, such as the image generation model and the action output model output by the learning device 100.

[0041] The acquisition unit 220 acquires an image from the camera 204 .

[0042] The acquisition unit 220 acquires the image generation model output by the learning device 100 from the storage unit 210 or an external device. The external device is a device that exists outside the robot 200. For example, the external device is the learning device 100 or a cloud server.

[0043] The image generation unit 230 uses the image and the image generation model to generate an image by applying an attention mask to the acquired image.

[0044] The word vector generation unit 240 generates word vectors for the attention region. The function of the word vector generation unit 240 is the same as that of the word vector generation unit 140. Therefore, a description of the function of the word vector generation unit 240 will be omitted.

[0045] The acquisition unit 220 acquires the action output model output by the learning device 100 from the storage unit 210 or an external device.

[0046] The control unit 250 uses the word vector and the action output model to identify an action of the robot 200 corresponding to the word vector. In particular, when the control unit 250 inputs a word vector into the action output model, the action output model outputs an action of the robot 200 corresponding to the word vector. For example, the control unit 250 identifies the action of "grabbing an apple" using the word vector indicating "apple" and the action output model. Also, for example, the control unit 250 identifies the action of "moving forward" using the word vector indicating "small apple" and the action output model.

[0047] The control unit 250 controls the robot 200 so that the robot 200 performs a specified action. For example, the control unit 250 controls the robot 200 so that the robot 200 grabs an apple. Furthermore, for example, the control unit 250 controls the robot 200 so that the robot 200 moves forward (i.e., so that the robot 200 approaches the apple).

[0048] Next, the processing executed by the robot 200 will be explained using a flowchart. FIG. 8 is a flowchart showing an example of processing executed by the robot of embodiment 1. (Step S21) The acquisition unit 220 acquires an image. (Step S22) The acquisition unit 220 acquires an image generation model from the storage unit 210. (Step S23) The image generation unit 230 uses the image and the image generation model to generate an image by applying an attention mask to the acquired image. (Step S24) The word vector generation unit 240 generates a word vector for the attention region.

[0049] (Step S25) The acquisition unit 220 acquires the action output model from the storage unit 210. (Step S26) The control unit 250 uses the word vector and the action output model to identify an action of the robot 200 corresponding to the word vector. (Step S27) The control unit 250 controls the robot 200 so that the robot 200 performs the identified action.

[0050] This allows the robot 200 to perform an action according to the word vector.

[0051] Modifications of the First Embodiment <Learning Phase> In the first embodiment, the learning generation unit 150 generates an action output model based on a word vector and information indicating the action of the robot 200 corresponding to the word vector. When generating the action output model, the learning generation unit 150 may further use feature amounts based on the image acquired by the acquisition unit 120 to generate the action output model. Modifications will be described in detail below.

[0052] Based on the image, the image generation unit 130 acquires features including position information of objects included in the image. For example, the image generation unit 130 acquires information (i.e., features) before the information is input to the pooling layer of the image generation model. In other words, the image generation unit 130 acquires information (i.e., features) output after the convolution layer. Note that the objects included in the image are objects present in the attention area.

[0053] The learning generation unit 150 generates an action output model based on the feature amounts, the word vectors, and information indicating the action of the robot 200 according to the feature amounts and the word vectors. As a result, for example, the action output model can output an action based on the word indicated by the word vector and the position information indicated by the word. For example, the action output model outputs the action of "grabbing an apple with the right hand" based on the word vector indicating "apple" and the feature amounts indicating that the "apple" is located on the right side of the image.

[0054] According to the modification of the first embodiment, the learning device 100 can improve the accuracy of the output by further using features.

[0055] <Utilization Phase> Next, a case where feature amounts are used in the utilization phase will be described. Based on an image acquired by the acquisition unit 220, the image generation unit 230 acquires feature amounts including position information of objects included in the image. Note that the objects included in the image are objects present in the attention area. In other words, the image generation unit 230 acquires feature amounts including position information of objects present in the attention area.

[0056] The control unit 250 uses the feature amounts, the word vectors, and the action output model to identify an action of the robot 200 according to the feature amounts and the word vectors. The control unit 250 controls the robot 200 so that the robot 200 performs the identified action.

[0057] Second Embodiment Next, a second embodiment will be described. In the second embodiment, differences from the first embodiment will be mainly described. Furthermore, in the second embodiment, descriptions of the commonalities between the first embodiment and the second embodiment will be omitted.

[0058] <Learning Phase> The acquisition unit 120 acquires additional information other than the word vectors. For example, the acquisition unit 120 acquires the additional information from the storage unit 110 or an external device. Also, for example, the acquisition unit 120 acquires the additional information through an input operation by the user.

[0059] The additional information is user instruction information, information obtained from a manual, etc. For example, the user instruction information and information obtained from a manual are information indicating prohibited actions, restricted actions, etc. In other words, the additional information is information that cannot be obtained from the image acquired by the acquisition unit 120.

[0060] The learning generation unit 150 generates an action output model based on the additional information, the word vector, and information indicating the action of the robot 200. In this way, the learning generation unit 150 generates an action output model in which an action that takes the additional information into consideration is learned.

[0061] Next, the processing executed by the learning device 100 will be described using a flowchart. Fig. 9 is a flowchart showing an example of processing executed by the learning device of embodiment 2. The processing in Fig. 9 differs from the processing in Fig. 5 in that steps S14a and S15a are executed. Therefore, steps S14a and S15a will be described in Fig. 9. Further, a description of the processing other than steps S14a and S15a will be omitted.

[0062] (Step S14a) The acquisition unit 120 acquires additional information. (Step S15a) The learning generation unit 150 generates a behavior output model based on the additional information, the word vector, and information indicating the behavior of the robot 200. The generated behavior output model is used in the utilization phase.

[0063] According to the second embodiment, the learning device 100 can generate an action output model in which actions that take additional information into consideration are learned.

[0064] Variation 1 of Embodiment 2 <Utilization Phase> In Embodiment 2, a case has been described in which a motion output model is generated in which motions taking additional information into consideration are learned in the learning phase. Then, a case has been described in which the generated motion output model is used in the utilization phase.

[0065] In variant example 1 of embodiment 2, a case is described in which the movement of robot 200 is controlled in the utilization phase using the movement output model generated in embodiment 1 (i.e., the movement output model in which no additional information has been learned) and additional information.

[0066] Fig. 10 is a flowchart showing an example of processing executed by a robot according to Modification 1 of Embodiment 2. The processing in Fig. 10 differs from the processing in Fig. 8 in that steps S26a, 26b, and 27a are executed. Therefore, only steps S26a, 26b, and 27a will be described in Fig. 10. Description of processing other than steps S26a, 26b, and 27a will be omitted.

[0067] (Step S26a) The acquisition unit 220 acquires additional information. For example, the acquisition unit 220 acquires the additional information from the storage unit 210 or an external device. (Step S26b) The control unit 250 determines an action that takes the additional information into consideration, based on the identified action and the additional information. (Step S27a) The control unit 250 controls the robot 200 so that the robot 200 performs the determined action.

[0068] A specific example will be described. For example, it is assumed that the identified action is "grabbing an apple." It is assumed that the additional information indicates a constraint when grabbing an apple. It is assumed that the constraint when grabbing an apple is "do not grab the apple too tightly." The control unit 250 controls the robot 200 so that when the robot 200 grabs an apple, it does not grab the apple too tightly.

[0069] According to the first modification of the second embodiment, the robot 200 can perform an action that takes into account the additional information.

[0070] Variation 2 of Embodiment 2 <Utilization Phase> Variation 1 of Embodiment 2 describes a case where the movement of the robot 200 is controlled using additional information after the movement is identified. Variation 2 of Embodiment 2 describes a case where the movement of the robot 200 is controlled by changing the word vector using additional information before the movement is identified.

[0071] Fig. 11 is a flowchart showing an example of processing executed by a robot according to Modification 2 of Embodiment 2. The processing in Fig. 11 differs from the processing in Fig. 8 in that steps S25a, 25b, and 26c are executed. Therefore, steps S25a, 25b, and 26c will be described in Fig. 11. Descriptions of processing other than steps S25a, 25b, and 26c will be omitted.

[0072] (Step S25a) The acquisition unit 220 acquires additional information. For example, the acquisition unit 220 acquires the additional information from the storage unit 210 or an external device. (Step S25b) The word vector generation unit 240 changes the word vector generated in step S24 based on the additional information to a word vector that takes the additional information into consideration. (Step S26c) The control unit 250 uses the changed word vector and the action output model to identify an action of the robot 200 that corresponds to the changed word vector.

[0073] According to the second modification of the second embodiment, the robot 200 can perform an action that takes into account the additional information.

[0074] Embodiment 3 Next, embodiment 3 will be described. In embodiment 3, differences from embodiments 1 and 2 will be mainly described. Furthermore, in embodiment 3, descriptions of matters common to embodiments 1 and 2 will be omitted.

[0075] 12 is a block diagram showing the functions of the robot according to the third embodiment. The robot 200 further includes an output unit 260. A part or all of the output unit 260 may be realized by a processing circuit. Alternatively, a part or all of the output unit 260 may be realized as a program module executed by the processor 201.

[0076] The output unit 260 outputs information indicating the attention area and information indicating the action performed by the robot 200. For example, the output unit 260 outputs the information indicating the attention area and the information indicating the action to the storage unit 210. Also, for example, the output unit 260 outputs the information indicating the attention area and the information indicating the action to an external device. Also, if the robot 200 has a display, the output unit 260 outputs the information indicating the attention area and the information indicating the action to the display.

[0077] For example, when information indicating the attention area and information indicating the action are output to the display, the user can recognize that the robot 200 has focused on the attention area and performed an action based on the attention area. In other words, the user can recognize the reason for controlling the robot 200.

[0078] Furthermore, the output unit 260 may generate an explanatory sentence based on the information indicating the attention area and the information indicating the action, and output the explanatory sentence. For example, the explanatory sentence may be, "The robot 200 detected the apple and picked up the apple."

[0079] Furthermore, the output unit 260 may output the operation history of the robot 200 to the storage unit 210, an external device, a display, or the like. For example, the operation history of the robot 200 is output to a display, allowing the user to recognize a series of operations of the robot 200.

[0080] Furthermore, the output unit 260 may generate a summary of the operations of the robot 200 based on the operation history of the robot 200. For example, the output unit 260 generates a summary of the operations of the robot 200 for one day by using the operation history of the robot 200 for one day and LLM (Large Language Models). The output unit 260 outputs the summary of the operations of the robot 200 to the storage unit 210, an external device, a display, or the like. For example, the user can easily understand the operations of the robot 200 for one day by outputting the summary of the operations of the robot 200 for one day on a display.

[0081] In the utilization phase, a case where a robot performs the processing has been described. In the utilization phase, a device other than a robot may perform the processing. For example, the device in question may be a device that performs autonomous driving (i.e., a car) or a device that identifies an image scene. The device in question is also called a utilization device.

[0082] The embodiments are merely examples, and various modifications are possible within the scope of the present disclosure. Furthermore, the embodiments and their variations can be combined with each other as appropriate. For example, a variation of the first embodiment can be combined with the second embodiment.

[0083] 100 Learning device, 101 Processor, 102 Volatile storage device, 103 Non-volatile storage device, 110 Storage unit, 120 Acquisition unit, 130 Image generation unit, 140 Word vector generation unit, 150 Learning generation unit, 160 Adjustment unit, 170 Output unit, 200 Robot, 201 Processor, 202 Volatile storage device, 203 Non-volatile storage device, 204 Camera, 210 Storage unit, 220 Acquisition unit, 230 Image generation unit, 240 Word vector generation unit, 250 Control unit, 260 Output unit.

Claims

1. A learning device having an acquisition unit that acquires an image and an image generation model; an image generation unit that uses the image and the image generation model to generate an image by applying an attention mask to the image; a word vector generation unit that generates word vectors for an attention region that is an area of ​​the attention mask; and a learning generation unit that generates an action output model based on the word vector and information indicating the action of the device according to the word vector.

2. The learning device described in claim 1, wherein the learning generation unit generates the action output model based on features including positional information of objects present in the attention area, the word vector, and information indicating the action of the device according to the features and the word vector.

3. The learning device described in claim 1 or 2, wherein the acquisition unit acquires additional information other than the word vector, and the learning generation unit generates the behavior output model based on the additional information, the word vector, and information indicating the behavior of the device according to the word vector.

4. A utilization device comprising: an acquisition unit that acquires an image, an image generation model, and an action output model; an image generation unit that uses the image and the image generation model to generate an image by applying an attention mask to the image; a word vector generation unit that generates word vectors for an attention region that is a region of the attention mask; and a control unit that uses the word vector and the action output model to identify an action of the utilization device according to the word vector, and controls the utilization device so that the utilization device performs the identified action.

5. The utilization device described in claim 4, wherein the control unit uses features including positional information of objects present in the attention area, the word vector, and the action output model to identify the action of the utilization device according to the features and the word vector.

6. The utilization device described in claim 4 or 5, wherein the acquisition unit acquires additional information other than the word vector, and the control unit determines an action that takes into account the additional information based on the identified action and the additional information, and controls the utilization device so that the utilization device performs the determined action.

7. The utilization device described in claim 4 or 5, wherein the acquisition unit acquires additional information other than the word vector, the word vector generation unit changes the word vector to a word vector that takes into account the additional information, and the control unit uses the changed word vector and the action output model to identify an action of the utilization device corresponding to the changed word vector.

8. The utilization device according to any one of claims 4 to 7, further comprising an output unit that outputs information indicating the attention area and information indicating the action.

9. The utilization device according to any one of claims 4 to 7, further comprising an output unit that generates an explanatory sentence based on the information indicating the attention area and the information indicating the action, and outputs the explanatory sentence.

10. The utilization device according to any one of claims 4 to 7, further comprising an output unit that outputs the operation history of the utilization device.

11. The utilization device according to any one of claims 4 to 7, further comprising an output unit that outputs a summary of the operation of the utilization device based on the operation history of the utilization device.

12. A learning method in which a learning device acquires an image and an image generation model, uses the image and the image generation model to generate an image by applying an attention mask to the image, generates word vectors for an attention region that is the region of the attention mask, and generates an action output model based on the word vector and information indicating the action of the device according to the word vector.

13. A control method in which a utilization device acquires an image, an image generation model, and an action output model, uses the image and the image generation model to generate an image by applying an attention mask to the image, generates word vectors for an attention region that is a region of the attention mask, uses the word vector and the action output model to identify an action of the utilization device according to the word vector, and controls the utilization device so that the utilization device performs the identified action.

14. A learning program that causes a learning device to perform the following processes: acquire an image and an image generation model; use the image and the image generation model to generate an image by applying an attention mask to the image; generate word vectors for an attention region that is the region of the attention mask; and generate an action output model based on the word vector and information indicating the action of the device according to the word vector.

15. A control program that causes a utilization device to execute the following process: acquire an image, an image generation model, and an action output model; use the image and the image generation model to generate an image by applying an attention mask to the image; generate word vectors for an attention region that is the region of the attention mask; use the word vector and the action output model to identify an action of the utilization device according to the word vector; and control the utilization device so that the utilization device performs the identified action.

Images