Motion command generating device and motion command generating method

Abstract

To enable an object to be selected in autonomous learning type end-to-end operation command generation.SOLUTION: An operation command generation device 100 comprises: an acquisition unit 111 which acquires an image including a candidate of an object of the work by a robot and sensor information; and a command generation unit 113 which outputs an operation command of the robot with the image and the sensor information as an input by using a machine learning model (command generation model 140). The machine learning model comprises: a position extraction block which includes a parameter to be referred to when calculating a position or region of a candidate of the object; and an operation command generation block which includes a parameter to be referred to when calculating an operation command of the work of a robot on the object. An explanatory variable includes the image and sensor information and a target variable is generated by using learning data 130 including the operation command.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a motion command generating device and a motion command generating method for generating a motion command for a robot. [Background technology]

[0002] In order to improve production efficiency and reduce labor costs, there are increasing efforts to use robots to perform tasks previously performed by humans, such as assembling industrial products, welding, and transporting products. However, previous robot systems required extensive programming and a high level of specialized knowledge, which was an obstacle to the introduction of robots.

[0003] To cope with such situations, an autonomous learning robot control system has been proposed, in which the robot itself determines its own behavior based on information from various sensors attached to the robot. Compared to previous robot systems, this system does not require extensive programming or advanced specialized knowledge, and is expected to enable robots to be easily introduced. Furthermore, this autonomous learning robot control system is expected to enable the robot to memorize and learn from its own behavioral experience, enabling it to flexibly generate behavior (generate behavior commands) in response to diverse environmental changes.

[0004] Examples of how a robot can gain experience with its movements include a robot operator / manager directly teaching the robot movements and having it memorize them, or a robot watching and imitating the movements of the operator / manager or other robots. Furthermore, autonomous learning robot control systems are generally equipped with a learning device called a learner, which stores sensor information from movement experience and adjusts parameters to generate movements. These stored movements are called learning data, and parameter adjustment is called learning, and the learner learns using the learning data. The learner defines the input-output relationship in advance and repeatedly learns (adjusts parameters) so that the expected output value is output for the input value to the learner.

[0005] An example of learning data is time-series data of a robot's joint angle information and captured images of the task when it experiences a certain action. If the task involves grasping an object, the captured images will show the object being grasped, the robot arm, and the robot hand. Using this learning data, joint angle information and images at time (t) are input into a learning device, and time-series learning is performed to predict joint angle information and images at time (t+1). Then, by sequentially inputting robot joint angle information and images into the learning device once learning has completed, the autonomous learning robot control system will be able to automatically generate actions according to the task status.

[0006] These methods generate robot behavior directly from sensor information at a given time without explicitly going through object recognition, and are therefore called autonomous learning-based end-to-end behavior generation methods. With these methods, if an object that was the target of work during learning or a similar object is detected in the captured image, the robot will autonomously begin working on that object. However, these methods do not take into account the situation where multiple target objects exist at the same time, which could result in the robot being unable to perform the work or not knowing which object to work on.

[0007] On the other hand, there are cases where robots that require autonomy have to work on multiple objects simultaneously. Examples of such tasks include removing rubble during decommissioning work, opening and closing valves in a plant, and transporting cardboard boxes. In rubble removal work, rubble with similar shapes and textures is expected to be scattered, and there are cases where it is sufficient to remove the rubble in any order, but there are also cases where the rubble overlaps and must be removed in the correct order. In addition, in valve opening and closing tasks, the valves must be opened and closed in the correct order. For this reason, an autonomous learning-based end-to-end behavior generation method is required that allows the robot to specify an object and perform the task on that object. Patent Document 1 describes a robot device as a technique for specifying an object to be worked on, in which an operator specifies the position and posture (direction) of an object in an image captured by the robot device. [Prior art documents] [Patent documents]

[0008] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-173209 Summary of the Invention [Problem to be solved by the invention]

[0009] The robot device described in Patent Document 1 plans the activation of an arm to grasp an object whose position and posture are specified, and controls its movement according to this trajectory. In autonomous learning-based end-to-end movement generation, the robot's movement (movement command) is generated directly without explicit object recognition (recognition of the object's position and posture). For this reason, it is not possible to specify the position and posture of an object, as with the robot device described in Patent Document 1. The present invention has been made in consideration of the above background, and aims to provide an action command generation device and an action command generation method that enable selection of an object in autonomous learning type end-to-end action command generation. [Means for solving the problem]

[0010] In order to solve the above-mentioned problems, the motion command generation device according to the present invention includes an acquisition unit that acquires images and sensor information including candidate objects for a robot to work on, and a command generation unit that uses a machine learning model to input the images and the sensor information and outputs motion commands for the robot, the machine learning model including a position extraction block including parameters that are referenced when calculating the position or area of ​​the candidate object, and a motion command generation block including parameters that are referenced when calculating a motion command for the robot to work on the object, and explanatory variables include the images and the sensor information, and a target variable is generated using learning data including the motion commands. The command generation unit acquires work object designation information that designates an object to be worked on by the robot from among the object candidates, replaces the information on the position or area of ​​the object candidate calculated using the position extraction block with the information on the position or area of ​​the object indicated by the work object designation information, and outputs the operation command. . [Effects of the Invention]

[0011] According to the present invention, it is possible to provide a motion command generation device and a motion command generation method that enable selection of an object in autonomous learning type end-to-end motion command generation. Problems, configurations, and effects other than those described above will become clear from the description of the following embodiments. [Brief explanation of the drawings]

[0012] [Figure 1] 1 is an overall configuration diagram of a robot control system according to a first embodiment. [Figure 2] 1 is a functional block diagram of a motion command generating device according to a first embodiment. [Figure 3] FIG. 2 is a configuration diagram of a command generation model according to the first embodiment. [Figure 4] 4 is a flowchart of an operation command generation process according to the first embodiment. [Figure 5] 10 is a photographed image according to a modified example of the first embodiment. [Figure 6] 10 is an image for a mask according to a modified example of the first embodiment. [Figure 7] The captured image according to the modified example of the first embodiment is a captured image masked with an image. [Figure 8] 10 is a flowchart of an operation command generation process according to the second embodiment. [Figure 9] 11 is a flowchart of an operation command generation process according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0013] <<Outline of the motion command generation device>> The following describes a motion command generation device in a mode (embodiment) for carrying out the present invention. The motion command generation device receives a captured image including a work object as input and generates motion commands for a robot using a command generation model, which is a trained machine learning model.

[0014] The command generation model includes a position extraction block and an action command generation block. The position extraction block includes parameters used in the process of recognizing one or more work targets (objects) in a captured image. In other words, the position extraction block includes parameters referenced when calculating the position or area of ​​a work target (a candidate work target). The action command generation block includes parameters used in the process of generating an action command that instructs the robot to operate to grasp one work target. In other words, the action command generation block includes parameters referenced when calculating an action command for the robot to work on an object.

[0015] Note that the command generation model, position extraction block, and action command generation block are parameters used in processes such as recognition and inference using machine learning technology, but in the following they may be described as if the parameters themselves are the entities that use the parameters for processing. For example, they may be described as "the command generation model calculates the action command" or "the captured image that is the input to the position extraction block."

[0016] The motion command generation device trains a machine learning model including a position extraction block and a motion command generation block using learning data to generate a command generation model, which is a trained machine learning model. The explanatory variables (input) of the learning data include captured images, and the objective variables (output, correct answer) include a motion command for a robot to grasp one object (work target, target object) in the captured images.

[0017] The learning data may be teaching-type learning data including captured images of a person operating a robot to grasp an object and motion commands indicating how the robot should be operated to grasp the object. Alternatively, the learning data may be obtained by calculating an optimal gripping motion for an object without human operation, for example, using machine learning technology, and recording the calculated motion as a motion command. Note that the number of work objects captured in the captured images of the teaching-type learning data may be one.

[0018] The motion command generation device acquires one or more work objects (position information) from a captured image using (the position extraction block of) the command generation model. Next, the motion command generation device queries the operator to select one work object from the one or more work objects. Subsequently, the motion command generation device generates a motion command for a robot to grasp the selected work object using (the motion command generation block of) the command generation model. In other words, the motion command generation device replaces the one or more work objects acquired using (the position extraction block of) the command generation model with one work object and generates a motion command to grasp the one work object.

[0019] With this type of motion command generation device, a motion command is generated to grasp a work object designated by a worker from among multiple work objects, using a command generation model generated using easily obtainable teaching / autonomous learning learning data. When using a machine learning model generated using conventional teaching-type learning data, it was not possible to guarantee the operation when there were multiple work objects (it was unclear how the work would be performed). With the motion command generation device, the order in which the work objects are worked on can be guaranteed. For example, it becomes possible to use a robot for work with a fixed work order.

[0020] <Overall configuration of the robot control system> 1 is a diagram showing the overall configuration of a robot control system 10 according to the first embodiment. The robot control system 10 includes a robot 300, a control device 310, cameras 371 and 372, and an operation command generation device 100. The robot 300 is capable of handling a work target 380, which is an object, and performs predetermined tasks such as assembling and transporting parts. The configuration of the robot 300 is not important, and it may be a robot arm alone, or may be equipped with a moving device such as crawlers or wheels.

[0021] The control device 310 is a device that outputs control commands to the robot 300 based on motion commands such as the joint angles of the robot 300 and the posture (position) and force (torque) of the end effector 301 (robot hand) input from the motion command generating device 100, to control the motion of the robot 300. The control commands are signals that indicate, for example, current values ​​and voltage values ​​for actuators (motors, etc.) provided in the joints of the robot arm and the end effector 301 of the robot 300. When the robot 300 receives a control command from the control device 310, a built-in drive circuit supplies a drive signal to the corresponding actuator.

[0022] Cameras 371 and 372 are imaging devices for capturing images of the working environment and surrounding environment of robot 300. In FIG. 1, two cameras 371 and 372 are installed, but the number may be one or three or more. Camera 371 is attached to robot arm 302, and camera 372 is installed around robot 300 (for example, on a workroom or a wall of a building). Here, the working environment corresponds to the movable region (work area) of robot 300, and the surrounding environment corresponds to the surrounding region outside the movable region of the robot.

[0023] The motion command generation device 100 is a device that plans the motion of the robot 300 based on images (photographed images) taken by the cameras 371 and 372 and information obtained from sensors arranged in the robot 300, the work environment, and the surrounding environment (hereinafter referred to as sensor information), and transmits motion commands such as joint angles and forces of the robot 300 to the control device 310. Here, the type of robot 300 and the type of sensor information are not important. For example, the sensor information may be the current value of an actuator provided in the joints of the robot 300, or the output signal of a tactile sensor or inertial sensor externally attached to the robot 300. Furthermore, the sensor information may be a temperature sensor that measures the work environment or the surrounding environment. In this way, each sensor detects the state of the robot 300 and the state of the environment, and outputs a detection signal according to the detected content.

[0024] <Configuration of the motion command generating device> 2 is a functional block diagram of the motion command generation device 100 according to the first embodiment. The motion command generation device 100 is a computer, and includes a control unit 110, a storage unit 120, and an input / output unit 180. User interface devices such as a display, keyboard, and mouse are connected to the input / output unit 180. For example, a touch panel display is connected to the input / output unit 180, and images captured by cameras 371 and 372 are displayed on the touch panel display. The operator of the robot designates one of the work objects 380 to be grasped by the robot 300 on the touch panel display. The input / output unit 180 also includes a communication device, and is capable of sending and receiving data (signals) to and from devices such as the control device 310 and the cameras 371 and 372.

[0025] The storage unit 120 includes storage devices such as a read-only memory (ROM), a random access memory (RAM), and a solid-state drive (SSD). The storage unit 120 stores learning data 130, a command generation model 140, and a program 128. The program 128 includes a description of an operation command generation process (see FIG. 4, which will be described later).

[0026] <Motion command generator: learning data> The training data 130 is training data used for training the command generation model 140, which is a machine learning model. The explanatory variables (input) of the training data 130 are photographed images of the work environment and surrounding environment of the robot 300, acquired in time series while the robot 300 is working to grasp the work object 380, and sensor information of the robot 300, the work environment, and the surrounding environment. The objective variables (output) of the training data 130 are operation commands to the robot 300, acquired in time series while the robot 300 is working to grasp the work object 380.

[0027] In this task, the operator of the robot 300 connects an operation unit such as a joystick to the input / output unit 180 to operate (control) the robot 300. The operation unit is a device that receives input from the operator and transmits operation commands corresponding to the input to the control device 310. Instead of the operator operating the robot 300, the robot 300 may reproduce a pre-planned operation of the robot 300. Note that even if it is expected that there will be multiple task targets 380 when the robot 300 is in operation, there may only be one task target 380 when the learning data 130 is acquired.

[0028] <Motion command generation device: command generation model> 3 is a configuration diagram of a command generation model 140 according to the first embodiment. The command generation model 140 is a machine learning model, and includes a position extraction block 141 and an action command generation block 142. The position extraction block 141 includes parameters used in the process of recognizing one or more work objects 380 and end effectors 301 in the captured image. The position extraction block 141 includes, for example, parameters related to a convolutional neural network (CNN) that extracts image features and spatial softmax that extracts coordinate information of the position with the strongest intensity from the obtained feature map, and is referenced when processing to extract the position information of the work object 380 from the captured image. Spatial softmax is a type of function used in machine learning, such as the softmax function, tanh function, or sigmoid function, and is also called soft argmax. The number of position coordinates extracted by spatial softmax is determined based on the number of channels of the previous CNN.

[0029] The motion command generation block 142 includes parameters used in the process of generating motion commands that instruct the robot 300 to grip a work object. The motion command generation block 142 generates motion commands from the position information and sensor information of the work object obtained from the position extraction block, using, for example, a fully connected layer or a recurrent neural network (RNN).

[0030] <<Motion command generating device: control unit>> Returning to Fig. 2, the explanation of the control unit 110 will continue. The control unit 110 is configured to include a CPU (Central Processing Unit) and is provided with an acquisition unit 111, a learning unit 112, and a command generation unit 113. The acquisition unit 111 acquires images captured by cameras 371 and 372, and sensor information from sensors provided in the robot 300, the work environment, and the surrounding environment.

[0031] The learning unit 112 trains the command generation model 140 using the learning data 130 (makes the command generation model 140 learn the learning data 130). As a result of the training / learning, the parameters included in the command generation model 140 are adjusted, and an action command for grasping the work object 380 that minimizes the error based on the captured image and sensor information is output.

[0032] Since minimizing the error in the motion commands is equivalent to accomplishing the task shown in the training data 130, it is expected that important position information for accomplishing the task will be obtained as the output of the position extraction block 141. For example, it is expected that position information of the work object 380 and the end effector 301 will be obtained. Note that the training data 130 includes captured images, sensor information, and motion commands, but does not include position information, and is not explicitly trained / learned.

[0033] Furthermore, each channel of the CNN obtained as a result of learning responds to a specific shape. For example, the first channel responds to the work object 380, and the second channel responds to the end effector 301. The action command generation block 142 predicts action commands based on the coordinates obtained from each channel, so a change in the coordinates of the first channel is equivalent to a change in the position of the work object. Note that which channel responds to the work object can be determined by inputting learning data after training / learning and examining the channel that outputs the position coordinates of the work object. In this way, the position information output by the position extraction block 141 includes the position coordinates of each work object 380 and end effector 301 included in the captured image.

[0034] The command generation unit 113 uses the command generation model 140 to make inferences (predictions) based on the captured image, sensor information, and position information of the work target object designated by the operator, and outputs an operation command for the robot 300. This operation command is sent to the control device 310, and the robot 300 operates.

[0035] The position information of the work object specified by the operator is not an input (explanatory variable) of the machine learning model, command generation model 140. The command generation unit 113 replaces the position information of the work object calculated using the position extraction block 141 with the position information of the work object specified by the operator, and calculates an action command using the action command generation block 142.

[0036] <<Motion command generation process>> 4 is a flowchart of the operation command generation process according to the first embodiment. In the following description of the operation command generation process, the main task is the task of the robot 300 grasping one or more work objects 380 (e.g., rubble) one by one with the end effector 301 and transferring (removing) them to a predetermined container (not shown). Gripping one work object 380 and transferring it to the container is one sub-task. In the first embodiment, the operator of the robot 300 indicates the work object 380 to be removed, and the robot 300 grasps the instructed work object 380 and transfers it to the container in accordance with the generated operation command.

[0037] In step S11, the command generation unit 113 starts the process of repeating steps S12 to S17 for each sub-task. More specifically, the command generation unit 113 calculates position information from the captured image using the position extraction block 141, and if the position information does not include position information for the work object 380, ends the action command generation process, and if the position information does include the work object 380, proceeds to step S12.

[0038] In step S12, the command generation unit 113 displays images captured by cameras 371 and 372 showing work objects 380 on a touch panel display and asks the operator which work object 380 to grasp. When the operator touches and indicates one of the work objects 380 to be grasped, the command generation unit 113 acquires its position. In other words, the command generation unit 113 acquires work object designation information that specifies an object to be worked on by the robot 300 from among candidate objects to be worked on. Hereinafter, the one designated work object 380 will be referred to as the work object.

[0039] In step S13, the command generating unit 113 repeats the processes of steps S14 to S17 until the sub-task is completed (the robot grasps the work target object instructed in step S12 and transfers it to a container). In step S14, the command generation unit 113 acquires position information by inputting the captured image to the position extraction block 141. Specifically, the command generation unit 113 calculates position information from the captured image using the position extraction block 141. The position information includes the position coordinates of each work object 380 and end effector 301.

[0040] In step S15, the command generation unit 113 replaces the position coordinates of the work target object included in the position information with the position information acquired in step S12. Note that the position of the work target object changes as the command generation unit 113 repeats steps S14 to S17 and the robot 300 grasps and moves the work target object. The command generation unit 113 tracks the position of the work target object instructed in step S12 and replaces the position coordinates with the position of the work target object that is the tracking result. The position of the work target object after grasping can be obtained from the position of the end effector 301 calculated from the operation command acquired in step S16, which will be described later.

[0041] In step S16, the command generation unit 113 inputs the replaced position information and sensor information to the action command generation block 142 to obtain an action command. Specifically, the command generation unit 113 calculates an action command from the replaced position information and sensor information using the action command generation block 142. In step S17, the command generator 113 transmits the operation command calculated in step S16 to the control device 310.

[0042] <Features of the motion command generation device> The motion command generation device 100 generates motion commands for the robot 300 from the captured image and sensor information using the command generation model 140. The command generation model 140 is a machine learning model, and its learning data is data that can be acquired from the work performed by the operator operating the robot 300, and can be created at low cost.

[0043] The command generation model 140 includes a position extraction block 141 and a motion command generation block 142, and is configured to calculate the position of the work target when generating a motion command. When generating a motion command, the motion command generation device 100 replaces the position of the work target with the position of the work target specified by the operator. In this way, even if there are multiple objects to be worked on, the robot 300 will perform the work on the object specified by the operator. When creating learning data (when the operator operates the robot 300), the number of work objects may be one. Compared to preparing learning data for multiple work objects and then training / learning, learning data can be created at low cost and in a short time.

[0044] <<Variation: Location Information>> In the first embodiment described above, position information (position coordinates) of the work object is calculated (see FIG. 3) when calculating an operation command using the command generation model 140. By obtaining a heat map using a sigmoid function instead of the Spatial Softmax function in the position extraction block 141, area information of the work object and the end effector 301 may be calculated instead of the position information.

[0045] <<Variation: Replacing location information>> In the first embodiment described above, the command generation unit 113 replaces the position information of the work object output from the position extraction block 141 with the position information of the work object specified by the operator (see step S15 in FIG. 4). The captured image to be input may include only the work object.

[0046] 5 shows a captured image 510 according to a modified example of the first embodiment. Suppose the operator specifies the right work object 511 to be grasped among three work objects at the bottom right of the captured image 510. The command generation unit 113 then generates a masking image 520 (see FIG. 6, described later) that is an image that masks the area including the work object 511 and the rest, and that masks work objects other than the work object 511 that the operator did not specify.

[0047] 6 shows a masking image 520 according to a modification of the first embodiment. Area 521 is an area that includes work object 511 and does not include work objects other than work object 511 that have not been specified by the operator. In image 520, the area other than area 521 is masked.

[0048] 7 shows a captured image 530 in which a captured image 510 according to a modified example of the first embodiment is masked with an image 520. The command generation unit 113 does not replace the position information, but instead inputs the captured image 530, in which the specified work object 511 is left and the rest is masked, into the command generation model 140 (position extraction block 141) to calculate an action command. By inputting an image in which only the work object 511 is captured, the position information output by the position extraction block 141 includes only the position information of the work object 511, and an action command to grasp the work object 511 is generated.

[0049] <<Variation: Selection of work type>> In the first embodiment described above, the operator selects the work target (see step S12 in FIG. 4), but the operator may also select one of a plurality of work types. The work type in rubble removal work refers to, for example, the container to which the rubble of the work target that has been selected and grasped by the robot 300 is to be transferred. Similarly, the work type in valve opening and closing work refers to, for example, whether the selected valve is to be opened or closed.

[0050] There are multiple command generation models 140 corresponding to the task type, and each is trained / learned to generate operation commands corresponding to the task type. In step S12, the command generation unit 113 acquires the task type along with the position information of the task object. In steps S14 to S17, the command generation unit 113 calculates operation commands using the command generation model 140 corresponding to the task type. In this way, the robot 300 can perform the work of the work type instructed by the operator on the work object.

[0051] Second Embodiment In the first embodiment described above, the operator is queried about the work object to be grasped (see step S12 in FIG. 4). If it is possible to identify a single work object to be grasped, the work may be performed without querying the operator. For example, if the work object can be regarded as being in one location in the captured image, the work may be performed without querying the operator. This is the case when there is one or more pieces of position information (position coordinates) calculated by the position extraction block 141, but the dispersion (variance) of the positions is small and they can be regarded as being in one location.

[0052] The functional configuration of the motion command generation device 100 according to the second embodiment is the same as that of the first embodiment, except for the command generation unit 113 (motion command generation processing). The command generation unit according to the second embodiment is referred to as a command generation unit 113A. Fig. 8 is a flowchart of the motion command generation processing according to the second embodiment. In step S31, the command generating unit 113A starts the process of repeating steps S32 to S40 for each sub-task. In step S32, the command generation unit 113A inputs the captured image to the position extraction block 141 to acquire position information and calculates the variance of the position information. The variance is, for example, the sum of the variances of the X coordinate and Y coordinate that indicate the position of the work object.

[0053] In step S33, if the variance calculated in step S32 is greater than a predetermined value (step S33→YES), the command generating unit 113A proceeds to step S34, and if it is equal to or less than the predetermined value (step S33→NO), the command generating unit 113A proceeds to step S35. Step S34 is the same process as step S12 in FIG. In step S35, the command generation unit 113A repeats the processes of steps S36 to S40 until the robot grasps the work target object instructed in step S34 and transfers it to the container. If step S34 is skipped (step S33→NO), the work target object is an object to be worked on that is located at a position whose position information has small variance and can be regarded as a single location.

[0054] Step S36 is the same process as step S14. In step S37, if the command generation unit 113A acquires the position of the object to be worked on, which is the operator's instruction, in step S34 (see step S12) (step S37 → YES), it proceeds to step S38, and if it has not acquired the position (step S37 → NO), it proceeds to step S39. Steps S38 to S40 are the same processes as steps S15 to S17, respectively.

[0055] <Features of the second embodiment> If the work object can be regarded as being in one place in the captured image, the robot 300 performs the work without asking the operator, thereby improving work efficiency. Note that step S38 may be executed following step S36 without branching to NO in step S37 and skipping step S38. In this case, in step S38, the command generator 113A may replace the position coordinates of the work target object with position information according to the movement of the gripped work target object (see step S15).

[0056] Third Embodiment In the first embodiment, the order of work objects is fixed. If the order is not fixed, the work may be performed one after the other without asking the operator. For example, in a rubble removal work, if rubble of similar shape and appearance is scattered and can be removed in any order, the rubble may be picked up one after the other and placed in a container.

[0057] The functional configuration of the motion command generation device 100 according to the third embodiment is the same as that of the first embodiment, except for the command generation unit (motion command generation process). The command generation unit according to the third embodiment is referred to as a command generation unit 113B. Fig. 9 is a flowchart of the motion command generation process according to the third embodiment. In step S51, the command generating unit 113B starts the process of repeating steps S52 to S58 for each sub-task. In step S52, the command generation unit 113B inputs the captured image to the position extraction block 141 to obtain position information of the work object.

[0058] In step S53, command generation unit 113B randomly selects one of the work targets in the position information acquired in step S52. Hereinafter, this selected work target will be referred to as the work target object. Steps S54 to S58 are respectively similar to steps S13 to S 17. However, in step S56, the position coordinates of the work target object included in the position information are replaced with the position information of the work target object selected in step S53.

[0059] <Features of the third embodiment> The robot 300 performs the work without asking the operator, thereby improving work efficiency.

[0060] <<Variations>> Although several embodiments of the present invention have been described above, these embodiments are merely illustrative and do not limit the technical scope of the present invention. For example, the removal of rubble (grasping the rubble and moving it to a container) has been taken as an example of the task of the robot 300, but the task is not limited to this and may be other tasks. For example, the task may be the opening and closing of multiple valves in a plant in a set sequence, or the transport of an object (box). In the case of the task of opening and closing a valve, a sub-task is the opening and closing of one valve.

[0061] In the first embodiment described above, the operator of the robot 300 selects the object to be worked on, but this is not limiting. There is a system that determines the order of the objects to be worked on, and this system may select the objects to be worked on. In the first embodiment described above, the command generator 113 determines whether or not the work object 380 (position information) is present based on the position information (see step S11 in FIG. 4), but the operator may also determine this based on the captured image. Also, the operator may also determine the end of the sub-task (see step S13).

[0062] The motion command generation device 100 in the above-described embodiment includes a learning unit 112 and command generation units 113, 113A, and 113B, generates a command generation model 140, and outputs motion commands using the command generation model 140. The device that generates the command generation model 140 and the device that outputs motion commands may be separated. For example, a model generation device including the learning unit 112 may generate the command generation model 140 and transmit it to multiple motion command devices, and each motion command device may output a motion command to a control device of each robot. Alternatively, the motion command generation device 100 and the control device 310 may be integrated.

[0063] The present invention can take on various other embodiments, and various modifications such as omissions and substitutions can be made without departing from the spirit of the present invention. These embodiments and modifications are included in the scope and spirit of the invention described in this specification, etc., and are also included in the invention described in the claims and their equivalents. [Explanation of symbols]

[0064] 100 Operation command generation device 111 Acquisition Department 112 Learning Department 113,113A,113B Command generation section 128 programs 130 training data 140 Command Generation Model (Machine Learning Model) 141 Position Extraction Block 142 Operation command generation block 300 robots 380 Work Object (Object)

Claims

1. an acquisition unit that acquires images and sensor information including candidates for objects on which the robot will work; a command generation unit that uses a machine learning model to input the image and the sensor information and outputs an operation command for the robot; The machine learning model is a position extraction block including parameters to be referenced when calculating a position or area of ​​the candidate object; and an operation command generation block including parameters to be referenced when calculating an operation command for the robot to perform a task on the object, an explanatory variable including the image and the sensor information, and a target variable including the operation command, The command generation unit acquiring work object designation information that designates an object to be worked on by the robot from among the candidate objects; The information on the position or area of ​​the candidate object calculated using the position extraction block is replaced with the information on the position or area of ​​the object indicated by the work object designation information, and the operation command is output.

1. A motion command generating device comprising:

2. The machine learning model is Multiple machine learning models for each type of work, The command generation unit acquiring the type of work to be performed by the robot on the object corresponding to the work object designation information; Using a machine learning model corresponding to the task type from among the plurality of machine learning models, the image and the sensor information are input and an operation command for the robot is output.

2. The motion command generating device according to claim 1.

3. The command generation unit Using the machine learning model, an image in which a region different from the region of the object indicated by the work object designation information is masked is input, and the operation command is output.

2. The motion command generating device according to claim 1.

4. The command generation unit When the variance calculated from the information on the position or area of ​​the candidate object calculated using the position extraction block is greater than a predetermined value, The information on the position or area of ​​the candidate object calculated using the position extraction block is replaced with the information on the position or area of ​​the object indicated by the work object designation information, and the operation command is output.

2. The motion command generating device according to claim 1.

5. The command generation unit The image is displayed on a display device, and information on a position or area in the image of a candidate object designated from among the displayed candidate objects is used as the work object designation information.

2. The motion command generating device according to claim 1.

6. The command generation unit selecting one of the position or area information of the candidate object calculated using the position extraction block as the work object designation information; The information on the position or area of ​​the candidate object calculated using the position extraction block is replaced with the information on the position or area of ​​the object indicated by the work object designation information, and the operation command is output.

2. The motion command generating device according to claim 1.

7. The motion command generating device acquiring image and sensor information including candidate objects for the robot to work on; and using a machine learning model, outputting an operation command for the robot using the image and the sensor information as input. The machine learning model is a position extraction block including parameters to be referenced when calculating a position or area of ​​the candidate object; and an operation command generation block including parameters to be referenced when calculating an operation command for the robot to perform a task on the object, an explanatory variable including the image and the sensor information, and a target variable including the operation command, In the step of outputting the operation command, acquiring work object designation information that designates an object to be worked on by the robot from among the candidate objects; The information on the position or area of ​​the candidate object calculated using the position extraction block is replaced with the information on the position or area of ​​the object indicated by the work object designation information, and the operation command is output. A motion command generating method comprising:

Images