Robot holding mode determination device, holding mode determination method, and robot control system

Abstract

A holding mode determination device for a robot comprises a control unit and an interface. The control unit outputs, to a display device, at least one holding mode for a target object of the robot, together with an image of the target object, the holding mode being estimated by classifying the target object into at least one of a plurality of holding categories that can be estimated. The interface obtains a user's selection with respect to the at least one holding mode, via the display device.

Description

Cross-reference to related applications

[0001] This application claims the priority of Japanese Patent Application No. 2021-134391 (filed on August 19, 2021), and the entire disclosure of the application is incorporated herein by reference for that purpose.

Technical Field

[0002] This disclosure relates to an object holding mode determination device for a robot, an object holding mode determination method, and a robot control system.

Background Art

[0003] Conventionally, a method of appropriately holding an object with an arbitrary shape by a robot hand is known (see, for example, Patent Document 1).

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

[0005] The object holding mode determination device for a robot according to an embodiment of the present disclosure includes a control unit and an interface. The control unit outputs at least one holding mode of the object of the robot estimated by classifying the object into at least one of a plurality of estimated holding sections, together with an image of the object, to a display device. The interface acquires a user's selection for the at least one holding mode via the display device.

[0006] A method for determining a robotic holding mode according to one embodiment of the present disclosure includes outputting to a display device, along with an image of the object, at least one holding mode of the object estimated by classifying the object into at least one of a plurality of presumable holding categories. The method for determining a holding mode includes obtaining a user's selection for the at least one holding mode via the display device.

[0007] A robot control system according to one embodiment of the present disclosure comprises a robot, a robot control device for controlling the robot, a holding mode determination device for determining how the robot holds an object, and a display device. The holding mode determination device outputs to the display device, along with an image of the object, at least one holding mode of the object estimated by the robot, which is determined by classifying the object into at least one of a plurality of presumable holding categories. The display device displays the image of the object and the at least one holding mode of the object. The display device receives input of a user's selection for the at least one holding mode and outputs it to the holding mode determination device. The holding mode determination device obtains the user's selection for the at least one holding mode from the display device. [Brief explanation of the drawing]

[0008] [Figure 1] This block diagram shows an example configuration of a robot control system according to one embodiment. [Figure 2] This is a schematic diagram showing an example configuration of a robot control system according to one embodiment. [Figure 3] This figure shows an example of a model under training. [Figure 4] This figure shows an example of a retention mode inference model. [Figure 5] This figure shows an example configuration in which the retention method is determined by a trained model. [Figure 6] This figure shows an example configuration in which the retention mode is determined by a retention mode inference model. [Figure 7A]This figure shows examples of gripping positions when the object is classified as Class 1. [Figure 7B] This figure shows examples of gripping positions when the object is classified as Class 2. [Figure 7C] This figure shows examples of gripping positions when the object is classified as Class 3. [Figure 8A] This figure shows an example of a gripping position determined when an object is classified into Class 1. [Figure 8B] This figure shows an example of the gripping position determined when an object is classified into the second class. [Figure 8C] This figure shows an example of the gripping position determined when an object is classified into the third class. [Figure 9] This flowchart shows an example of the procedure for determining the holding mode according to one embodiment. [Modes for carrying out the invention]

[0009] In a trained model that takes an image of an object as input and outputs the object's holding method, the user cannot control the output of the trained model and therefore cannot control the object's holding method. There is a need to improve the usability of the trained model that outputs the object's holding method. According to a robot holding method determination device, holding method determination method, and robot control system according to one embodiment of this disclosure, the usability of the trained model that outputs the object's holding method can be improved.

[0010] (Example configuration of robot control system 100) As shown in Figure 1, a robot control system 100 according to one embodiment of the present disclosure comprises a robot 2, a robot control device 110, a holding mode determination device 10, a display device 20, and a camera 4. The robot 2 performs work by holding the object to be held 80 with an end effector 2B. The object to be held 80 is also simply referred to as the object. The robot control device 110 controls the robot 2. The holding mode determination device 10 determines the mode in which the robot 2 holds the object to be held 80 and outputs it to the robot control device 110. The mode in which the robot 2 holds the object to be held 80 is also simply referred to as the holding mode. The holding mode includes the position in which the robot 2 contacts the object to be held 80 or the force that the robot 2 applies to the object to be held 80.

[0011] As shown in Figure 2, in this embodiment, the robot 2 holds the object to be held 80 at the work start point 6. That is, the robot control device 110 controls the robot 2 to hold the object to be held 80 at the work start point 6. The robot 2 may move the object to be held 80 from the work start point 6 to the work target point 7. The object to be held 80 is also called the work target. The robot 2 operates within the operating range 5.

[0012] <Robot 2> Robot 2 comprises an arm 2A and an end effector 2B. Arm 2A may be configured as, for example, a 6-axis or 7-axis vertical articulated robot. Arm 2A may be configured as a 3-axis or 4-axis horizontal articulated robot or a SCARA robot. Arm 2A may be configured as a 2-axis or 3-axis Cartesian robot. Arm 2A may be configured as a parallel link robot, etc. The number of axes constituting Arm 2A is not limited to those exemplified. In other words, Robot 2 has an arm 2A connected by multiple joints and operates by driving the joints.

[0013] The end effector 2B may include, for example, a gripping hand configured to grip the object to be held 80. The gripping hand may have a plurality of fingers. The number of fingers of the gripping hand may be two or more. The fingers of the gripping hand may have one or more joints. The end effector 2B may include a suction hand configured to suck and hold the object to be held 80. The end effector 2B may include a scooping hand configured to scoop and hold the object to be held 80. The end effector 2B is also referred to as a holding portion that holds the object to be held 80. The end effector 2B is not limited to these examples and may be configured to perform various other operations. In the configuration illustrated in FIG. 1, the end effector 2B is assumed to include a gripping hand.

[0014] The robot 2 can control the position of the end effector 2B by operating the arm 2A. The end effector 2B may have an axis that serves as a reference for the direction in which it acts on the object to be held 80. When the end effector 2B has an axis, the robot 2 can control the direction of the axis of the end effector 2B by operating the arm 2A. The robot 2 controls the start and end of the operation in which the end effector 2B acts on the object to be held 80. The robot 2 can move or process the object to be held 80 by controlling the operation of the end effector 2B while controlling the position of the end effector 2B or the direction of the axis of the end effector 2B. In the configuration illustrated in FIG. 1, the robot 2 causes the end effector 2B to hold the object to be held 80 at the work start point 6 and moves the end effector 2B to the work target point 7. The robot 2 releases the object to be held 80 from the end effector 2B at the work target point 7. By doing so, the robot 2 can move the object to be held 80 from the work start point 6 to the work target point 7.

[0015] <Sensor> The robot control system 100 further includes sensors. The sensors detect the physical information of the robot 2. The physical information of the robot 2 may include information regarding the actual position or posture of each component of the robot 2, or the speed or acceleration of each component of the robot 2. The physical information of the robot 2 may include information regarding the force acting on each component of the robot 2. The physical information of the robot 2 may include information regarding the current flowing through the motor that drives each component of the robot 2 or the torque of the motor. The physical information of the robot 2 represents the result of the actual operation of the robot 2. That is, the robot control system 100 can grasp the result of the actual operation of the robot 2 by acquiring the physical information of the robot 2.

[0016] The sensors may include a force sensor or a tactile sensor that detects, as the physical information of the robot 2, a force, a distributed pressure, a slip, etc. acting on the robot 2. The sensors may include a motion sensor that detects the position or posture, or the speed or acceleration of the robot 2 as the physical information of the robot 2. The sensors may include a current sensor that detects the current flowing through the motor that drives the robot 2 as the physical information of the robot 2. The sensors may include a torque sensor that detects the torque of the motor that drives the robot 2 as the physical information of the robot 2.

[0017] The sensors may be installed at the joints of the robot 2 or at the joint drive units that drive the joints. The sensors may also be installed on the arm 2A or the end effector 2B of the robot 2.

[0018] The sensors output the detected physical information of the robot 2 to the robot control device 110. The sensors detect and output the physical information of the robot 2 at a predetermined timing. The sensors output the physical information of the robot 2 as time-series data.

[0019] <Camera 4> In the configuration example shown in Figure 1, the robot control system 100 is assumed to include a camera 4 attached to the end effector 2B of the robot 2. The camera 4, for example, photographs the object 80 to be held from the end effector 2B toward the object 80. In other words, the camera 4 photographs the object 80 from the direction in which the end effector 2B holds the object 80. The image of the object 80 photographed from the direction in which the end effector 2B holds the object 80 is also called the held object image. The camera 4 may also be equipped with a depth sensor and configured to acquire depth data of the object 80 to be held. Depth data refers to distance data for each direction within the field of view of the depth sensor. More specifically, depth data can also be said to be information about the distance from the camera 4 to the measurement point. The image captured by the camera 4 may include monochrome luminance information, or it may include luminance information for each color represented by RGB (Red, Green, and Blue), etc. The number of cameras 4 is not limited to one, but may be two or more.

[0020] Furthermore, the camera 4 is not limited to being mounted on the end effector 2B, but may be installed at any position capable of capturing images of the object to be held 80. In configurations where the camera 4 is mounted on a structure other than the end effector 2B, the image of the object to be held may be synthesized based on the image captured by the camera 4 mounted on the structure. The image of the object to be held may be synthesized by image transformation based on the relative position and orientation of the end effector 2B with respect to the mounting position and orientation of the camera 4. Alternatively, the image of the object to be held may be generated from CAD and drawing data.

[0021] <Holding Mode Determination Device 10> As shown in Figure 1, the holding mode determination device 10 comprises a control unit 12 and an interface (I / F) 14. The interface 14 acquires information or data related to the object to be held 80 from an external device and outputs information or data to an external device. The interface 14 also acquires images of the object to be held 80 taken from the camera 4. The interface 14 outputs information or data to the display device 20, causing the control unit of the display device 20, which will be described later, to display the information or data. Based on the information or data acquired by the interface 14, the control unit 12 determines the holding mode of the object to be held 8 by the robot 2 and outputs it to the robot control device 110 via the interface 14.

[0022] The control unit 12 may include at least one processor to provide control and processing capabilities for performing various functions. The processor may execute programs that realize the various functions of the control unit 12. The processor may be implemented as a single integrated circuit. An integrated circuit is also called an IC (Integrated Circuit). The processor may be implemented as a plurality of communicably connected integrated circuits and discrete circuits. The processor may be implemented based on various other known technologies.

[0023] The control unit 12 may include a storage unit. The storage unit may include an electromagnetic storage medium such as a magnetic disk, or a memory such as a semiconductor memory or magnetic memory. The storage unit stores various types of information. The storage unit stores programs executed by the control unit 12, etc. The storage unit may be configured as a non-temporary readable medium. The storage unit may function as the work memory of the control unit 12. At least a part of the storage unit may be configured separately from the control unit 12.

[0024] Interface 14 may include a communication device configured to communicate via wired or wireless connection. The communication device may be configured to communicate using communication methods based on various communication standards. The communication device can be configured using known communication technologies.

[0025] Interface 14 may be configured to include an input device that accepts information or data from a user. The input device may include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may also include physical keys. The input device may also include an audio input device such as a microphone. Interface 14 may be configured to be connectable to an external input device. Interface 14 may be configured to acquire information input to an external input device from the external input device.

[0026] Interface 14 may be configured to include an output device that outputs information or data to the user. The output device may include, for example, an audio output device such as a speaker that outputs auditory information such as sound. The output device may also include a vibration device that vibrates to provide tactile information to the user. The output device is not limited to these examples and may include various other devices. Interface 14 may be configured to be connectable to an external output device. Interface 14 may output information to an external output device so that the external output device outputs information to the user. Interface 14 may be configured to be connectable to a display device 20, which will be described later, as an external output device.

[0027] <Display device 20> The display device 20 displays information or data to the user. The display device 20 may include a control unit for controlling the display. The display device 20 may be configured to include, for example, an LCD (Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a PDP (Plasma Display Panel). The display device 20 is not limited to these displays and may be configured to include various other types of displays. The display device 20 may be configured to include light-emitting devices such as LEDs (Light Emission Diodes) or LDs (Laser Diodes). The display device 20 may be configured to include various other devices.

[0028] Furthermore, the display device 20 may be configured to include an input device that receives information or data from the user. The input device may include, for example, a touch panel or touch sensor, or a pointing device such as a mouse. The input device may also include physical keys. The input device may also include an audio input device such as a microphone. The input device may be connected to, for example, interface 14.

[0029] <Robot control device 110> The robot control device 110 acquires information that identifies the holding mode from the holding mode determination device 10, and controls the robot 2 to hold the object to be held 80 in the holding mode determined by the holding mode determination device 10.

[0030] The robot control device 110 may be configured to include at least one processor in order to provide control and processing capabilities for performing various functions. Each component of the robot control device 110 may be configured to include at least one processor. Multiple components of the robot control device 110 may be implemented by one processor. The entire robot control device 110 may be implemented by one processor. The processor can execute programs that implement the various functions of the robot control device 110. The processor may be configured to be the same as or similar to the processor used in the holding mode determination device 10.

[0031] The robot control device 110 may include a memory unit. The memory unit may be configured to be the same as or similar to the memory unit used in the storage mode determination device 10.

[0032] The robot control device 110 may include a holding mode determination device 10. The robot control device 110 and the holding mode determination device 10 may be configured as separate units.

[0033] (Example of operation of the holding mode determination device 10) The control unit 12 of the holding mode determination device 10 determines the holding mode of the object to be held 80 based on a trained model. The trained model is configured to accept an image of the object to be held 80 as input and output an estimated result of the holding mode of the object to be held 80. The control unit 12 generates the trained model. The control unit 12 acquires an image of the object to be held 80 from the camera 4, inputs the image into the trained model, and obtains an estimated result of the holding mode of the object to be held 80 from the trained model. The control unit 12 determines the holding mode based on the estimated result of the holding mode of the object to be held 80 and outputs it to the robot control device 110. The robot control device 110 controls the robot 2 to hold the object to be held 80 in the holding mode acquired from the holding mode determination device 10.

[0034] <Generating a pre-trained model> The control unit 12 learns from images of the object to be held 80, or from images generated from CAD data of the object to be held 80, as training data, and generates a trained model that estimates the holding method of the object to be held 80. The training data may include training data used in so-called supervised learning. The training data may also include data generated by the device performing the learning, which is used in so-called unsupervised learning. Images of the object to be held 80, or images generated as images of the object to be held 80, are collectively referred to as object images. As shown in Figure 3, the inference model includes a class inference model 40 and a holding method inference model 50. The control unit 12 updates the model being learned by learning using object images as training data, and generates a trained model.

[0035] The class inference model 40 accepts an image of the object as input. Based on the input image of the object, the class inference model 40 estimates the holding category into which the holding object 80 should be classified. In other words, the class inference model 40 classifies the object into one of several possible holding categories. The multiple holding categories that can be estimated by the class inference model 40 are the categories that the class inference model 40 can output. A holding category is a category that represents a difference in the shape of the holding object 80, and is also called a class.

[0036] The class inference model 40 classifies the input target image into a predetermined class. Based on the target image, the class inference model 40 outputs a classification result that classifies the retained object 80 into a predetermined class. In other words, the class inference model 40 estimates the class to which the retained object 80 belongs when it is classified into a class.

[0037] The class inference model 40 outputs the class estimation result as class information. The classes that can be estimated by the class inference model 40 may be determined based on the shape of the object to be held 80. The classes that can be estimated by the class inference model 40 may be determined based on various characteristics of the object to be held 80, such as surface condition, material, or hardness. In this embodiment, the number of classes that can be estimated by the class inference model 40 is assumed to be four. The four classes are referred to as the first class, the second class, the third class, and the fourth class, respectively. The number of classes may be three or less, or five or more.

[0038] The retention mode inference model 50 accepts an object image and class information output from the class inference model 40 as input. Based on the input object image and class information, the retention mode inference model 50 estimates the retention mode of the retained object 80 and outputs the estimated retention mode result.

[0039] The class inference model 40 and the retention mode inference model 50 are each configured, for example, as a Convolutional Neural Network (CNN) with multiple layers. The layers of the class inference model 40 are represented as processing layers 42. The layers of the retention mode inference model 50 are represented as processing layers 52. For the information input to the class inference model 40 and the retention mode inference model 50, a convolution operation based on predetermined weighting coefficients is performed in each layer of the CNN. The weighting coefficients are updated during the training of the class inference model 40 and the retention mode inference model 50. The class inference model 40 and the retention mode inference model 50 may be configured as VGG16 or ResNet50. The class inference model 40 and the retention mode inference model 50 are not limited to these examples and may be configured as various other models.

[0040] The retention mode inference model 50 includes a multiplier 54 that accepts class information as input. If there are four classes, the retention mode inference model 50 branches the retained objects 80 included in the input object image to processing layers 52 corresponding to each of the four classes, as shown in Figure 4. The multiplier 54 includes a first multiplier 541, a second multiplier 542, a third multiplier 543, and a fourth multiplier 544, each corresponding to one of the four classes. The first multiplier 541, the second multiplier 542, the third multiplier 543, and the fourth multiplier 544 multiply the output of the processing layer 52 corresponding to each of the four classes by a weighting coefficient. The retention mode inference model 50 includes an adder 56 that adds the outputs of the first multiplier 541, the second multiplier 542, the third multiplier 543, and the fourth multiplier 544, respectively. The output of the adder 56 is input to the processing layer 52. The retention mode inference model 50 outputs the output of the processing layer 52 located after the adder 56 as the estimated retention mode.

[0041] The weighting coefficients that the first multiplier 541, second multiplier 542, third multiplier 543, and fourth multiplier 544 each multiply by the output of the processing layer 52 are determined based on the class information input to the multiplier 54. The class information indicates which of the four classes the object to be held 80 is classified into. If the object to be held 80 is classified into the first class, the weighting coefficient of the first multiplier 541 is 1, and the weighting coefficients of the second multiplier 542, third multiplier 543, and fourth multiplier 544 are 0. In this case, the output of the processing layer 52 corresponding to the first class is output from the adder 56.

[0042] The class information may be expressed as the probability that the object to be held 80 is classified into each of the four classes. For example, the probabilities that the object to be held 80 is classified into the first class, second class, third class, and fourth class, respectively, may be expressed as X1, X2, X3, and X4. In this case, the adder 56 adds the output obtained by multiplying the output of the processing layer 52 corresponding to the first class by X1, the output obtained by multiplying the output of the processing layer 52 corresponding to the second class by X2, the output obtained by multiplying the output of the processing layer 52 corresponding to the third class by X3, and the output obtained by multiplying the output of the processing layer 52 corresponding to the fourth class by X4, and outputs the result.

[0043] The control unit 12 generates a class inference model 40 and a holding mode inference model 50 as trained models by training on object images as training data. The control unit 12 requires ground truth data to generate the trained models. For example, in order to generate the class inference model 40, the control unit 12 requires training data associated with information indicating which class the image used as training data should be classified into. In addition, in order to generate the holding mode inference model 50, the control unit 12 requires training data associated with information indicating the correct gripping position of the object depicted in the image used as training data.

[0044] The trained models used in the robot control system 1 according to this embodiment may include models that have learned the class or gripping position of an object in an object image during training, or they may include models that have not learned the class or gripping position. Even if the trained model has not learned the class or gripping position, the control unit 12 may estimate the class and gripping position of the object by referring to the class and gripping position of a similar trained object. Therefore, in the robot control system 1 according to this embodiment, the control unit 12 may or may not learn the object itself in order to generate a trained model.

[0045] <Estimation of retention patterns using a pre-trained model> As shown in Figure 5, the control unit 12 configures a trained model such that class information from the class inference model 40 is input to the retention mode inference model 50 via the conversion unit 60. In other words, the trained model includes a conversion unit 60 located between the class inference model 40 and the retention mode inference model 50. The control unit 12 inputs an object image to the class inference model 40 and causes the class inference model 40 to output class information. The control unit 12 inputs an object image to the retention mode inference model 50 and inputs class information from the class inference model 40 to the retention mode inference model 50 via the conversion unit 60, causing the retention mode inference model 50 to output the estimated retention mode of the retained object 80.

[0046] The class information output from the class inference model 40 indicates which of the first to fourth classes the retained object 80 included in the input object image should be classified into. Specifically, the class inference model 40 is configured to output "1000" as class information when it estimates that the retained object 80 should be classified into the first class. The class inference model 40 is configured to output "0100" as class information when it estimates that the retained object 80 should be classified into the second class. The class inference model 40 is configured to output "0010" as class information when it estimates that the retained object 80 should be classified into the third class. The class inference model 40 is configured to output "0001" as class information when it estimates that the retained object 80 should be classified into the fourth class.

[0047] The conversion unit 60 converts the class information from the class inference model 40 and inputs it to the multiplier 54 of the retention mode inference model 50. For example, if the class information input from the class inference model 40 is "1000", the conversion unit 60 may be configured to convert it to another string such as "0100" and output the converted class information to the retention mode inference model 50. The conversion unit 60 may also be configured to output the class information "1000" input from the class inference model 40 to the retention mode inference model 50 as is. The conversion unit 60 may have a table that identifies rules for converting class information and may be configured to convert class information based on the table. The conversion unit 60 may be configured to convert class information using matrices or mathematical formulas.

[0048] As shown in Figure 6, the control unit 12 may apply only the retained mode inference model 50 as the trained model and directly input the class information to the retained mode inference model 50.

[0049] In this embodiment, a shape image 81 representing the shape of the object to be held 80, as exemplified in Figures 7A, 7B, and 7C, is input to the trained model as the object image. In this case, the object to be held 80 included in the shape image 81 is classified into each class based on the shape of the object to be held 80. If the shape of the object to be held 80 is O-shaped, the object to be held 80 included in the shape image 81 is classified into the first class. If the shape of the object to be held 80 is I-shaped, the object to be held 80 included in the shape image 81 is classified into the second class. If the shape of the object to be held 80 is J-shaped, the object to be held 80 included in the shape image 81 is classified into the third class. The J-shape can also be said to be a shape that combines the I-shape and the O-shape. If the shape of the object to be held 80 is any other shape, the object to be held 80 included in the shape image 81 is classified into the fourth class.

[0050] If the class inference model 40 estimates that the object to be held 80 included in the shape image 81 should be classified into the first class, it outputs "1000" as class information. If the holding manner inference model 50 obtains "1000" as class information, it estimates the holding position 82 to be inside the O-shaped object to be held 80 in the shape image 81. If the class inference model 40 estimates that the object to be held 80 included in the shape image 81 should be classified into the second class, it outputs "0100" as class information. If the holding manner inference model 50 obtains "0100" as class information, it estimates the holding positions 82 to be on both sides near the center of the I-shape in the shape image 81. If the class inference model 40 estimates that the object to be held 80 included in the shape image 81 should be classified into the third class, it outputs "0010" as class information. Depending on the class to which the object to be held 80 included in the shape image 81 is classified, the holding position 82 of the object to be held 80 included in the shape image 81 is estimated to be inside the O-shaped object to be held 80. For example, if the retention mode inference model 50 obtains "0010" as class information, it estimates the positions on both sides near the ends of the J-shape in the shape image 81 as retention positions 82. In other words, within the combined shape of the I-shape and O-shape, the retention mode inference model 50 estimates the positions on both sides near the end of the I-shape that is farther from the O-shape as retention positions 82.

[0051] <Setting the conversion rules for the conversion unit 60> The control unit 12 can set rules for converting class information in the conversion unit 60. The control unit 12 receives user input through the interface 14 and sets rules based on the user input.

[0052] The control unit 12 inputs an object image to the trained model and outputs the estimated holding patterns when the holding object 80 is classified into multiple classes from the trained model. Specifically, as shown in Figures 8A, 8B, and 8C, the control unit 12 inputs an image of a screw as the holding object 80 as the object image to the trained model. The screw has a screw head 83 and a screw shaft 84. If the trained model classifies the screw as the holding object 80 included in the screw shape image 81 into the first class, it cannot identify the corresponding position inside the screw and therefore cannot estimate the holding position 82. Consequently, the holding position 82 is not displayed in the shape image 81 containing the screw classified into the first class. If the trained model classifies the screw as the holding object 80 included in the screw shape image 81 into the second class, it estimates both sides near the center of the screw shaft 84 as the holding position 82. If the trained model assumes that the screw, as the object to be held 80 included in the screw shape image 81, is classified as a third class, it estimates the positions on both sides near the end of the screw shaft 84 that is furthest from the screw head 83 as the holding positions 82.

[0053] The control unit 12 presents the user with the results of estimating the holding position 82 in the shape image 81 when the trained model classifies the object to be held 80 contained in the shape image 81 into each class. Specifically, the control unit 12 displays images on the display device 20, as illustrated in Figures 8A, 8B, and 8C, in which the holding position 82 is superimposed on the shape image 81, as candidates for the holding position 82 when holding the object to be held 80, allowing the user to see the image. The user can check the holding position 82 (candidates for holding position 82) when the object to be held 80 contained in the shape image 81 is classified into each class, and determine which holding position 82 is suitable for holding the screw as the object to be held 80. The control unit 12 prompts the user to input information through the I / F 14 that identifies which of the candidate holding position 82 the user has determined to be suitable. Based on the user's input to the I / F 14, the control unit 12 obtains the class corresponding to the holding position 82 that the user has determined to be suitable.

[0054] When the control unit 12 has the robot 2 hold a screw as the object to be held 80, it causes the holding mode inference model 50 to estimate a holding position 82 of a class that the user has determined to be suitable. Specifically, the control unit 12 can control the holding position 82 that the holding mode inference model 50 estimates by controlling the class information input to the holding mode inference model 50.

[0055] For example, if the object to be held 80 included in the object image is classified into a class corresponding to a screw (e.g., the second class), the control unit 12 may cause the holding manner inference model 50 to estimate the holding position 82 as the estimated result, assuming that the object to be held 80 is classified into the third class. In this case, the control unit 12 inputs "0010" to the holding manner inference model 50 as the class information to which the object to be held 80 belongs, indicating the third class. Even if the class information output by the class inference model 40 is originally "0100" indicating the second class, the control unit 12 inputs "0010" as the class information to the holding manner inference model 50. In other words, normally, the holding manner of an object to be held 80 classified into the second class would be estimated to be the holding manner corresponding to the second class, but the control unit 12 causes the holding manner inference model 50 to estimate the holding manner as if the object to be held 80 were classified into the third class. The control unit 12 sets the conversion rule in the conversion unit 60 so that when the class inference model 40 outputs "0100" as the class information of the object to be held 80, it inputs "0010" as the class information to the holding mode inference model 50.

[0056] The control unit 12 can cause the holding mode inference model 50 to estimate that the holding mode inference model 50 is a class that has been converted to another class and output as an estimation result. For example, even if the holding mode inference model 50 is a class 2, the control unit 12 can cause the holding mode inference model 50 to estimate that the holding position 82 is a class 3, and output as an estimation result. Even if the holding mode inference model 50 is a class 3, the control unit 12 can cause the holding mode inference model 50 to estimate that the holding position 82 is a class 2, and output as an estimation result. In this case, the control unit 12 sets the conversion rules in the conversion unit 60 so that when the class information from the class inference model 40 is "0100", it is converted to "0010" and input to the holding mode inference model 50, and when the class information from the class inference model 40 is "0010", it is converted to "0100" and input to the holding mode inference model 50.

[0057] As described above, by configuring the control unit 12 to be able to set conversion rules, it is possible to reduce the need for the user to select a holding mode each time, for example, when processing multiple objects in succession. In this case, for example, once the holding mode has been determined by the user, the conversion rules may be set so that the holding modes of multiple objects become the determined holding mode.

[0058] Furthermore, the class inference model 40 may have a class set as the retention category that does not have a retention mode. In this case, the retention mode inference model 50 may estimate that a certain class does not have a retention mode when it is input. In this case, for example, when processing multiple objects in succession, the control unit 12 may determine that an object classified into a class other than the class expected by the user is a foreign object. The control unit 12 can then control the process so as not to retain foreign objects by converting the class of an object determined to be a foreign object to a class that does not have a retention mode according to the set conversion rules.

[0059] <Example of a procedure for determining the retention method> The control unit 12 of the storage mode determination device 10 may execute a storage mode determination method that includes the steps of the flowchart illustrated in Figure 9. The storage mode determination method may be implemented as a storage mode determination program to be executed by the processor constituting the control unit 12. The storage mode determination program may be stored on a non-temporary computer-readable medium.

[0060] The control unit 12 inputs the object image into the trained model (step S1).

[0061] The control unit 12 classifies the retained objects 80 included in the object image into classes using the class inference model 40 (step S2). Specifically, the class inference model 40 estimates the class to which the retained objects 80 belong and outputs the estimation result as class information.

[0062] The control unit 12 estimates the holding mode for the object to be held 80, assuming it is one of the classes estimated by the class inference model 40 (e.g., the third class) or another class (e.g., the first, second, or fourth class). The control unit 12 generates the estimation result as a candidate for the holding mode and displays it on the display device 20 (step S3). Specifically, the control unit 12 displays an image in which the holding position 82 is superimposed on the object image. In this case, the display may be made in a manner that allows identification of which class the class inference model 40 estimated.

[0063] The control unit 12 prompts the user to select a holding mode from the candidates (step S4). Specifically, the control unit 12 prompts the user to input which holding mode to select from the candidates via the I / F 14, and obtains information to identify the holding mode selected by the user. In other words, if the user believes that the holding mode for the object to be held 80 in the class estimated by the class inference model 40 is not appropriate, the control unit 12 prompts the user to select another holding mode.

[0064] The control unit 12 causes the trained model to estimate the holding mode so that the object to be held 80 is held in the holding mode selected by the user, and outputs the estimated holding mode to the robot control device 110 (step S5). After executing the procedure in step S5, the control unit 12 finishes executing the procedure in the flowchart in Figure 9.

[0065] <Summary> As described above, the holding mode determination device 10 according to this embodiment allows the user to select a holding mode estimated by the trained model and controls the robot 2 with the holding mode selected by the user. In this way, the user can control the holding mode performed by the robot 2 even if they cannot control the results of the training model's estimation of the holding mode. As a result, the convenience of the training model that outputs the holding mode of the object to be held 80 is improved.

[0066] (Other embodiments) Other embodiments are described below.

[0067] In the embodiments described above, a configuration has been described in which the holding mode determination device 10 determines the holding position 82 as the holding mode. The holding mode determination device 10 can determine other modes as the holding mode, not just the holding position 82.

[0068] The holding mode determination device 10 may determine, for example, the force that the robot 2 applies to hold the object 80 as the holding mode. In this case, the holding mode inference model 50 estimates and outputs the force that will be applied to hold the object 80.

[0069] Furthermore, the holding mode determination device 10 may determine, for example, the type of hand used by the robot 2 to hold the object 80 as the holding mode. In this case, the holding mode inference model 50 estimates and outputs the type of hand used to hold the object 80.

[0070] The holding mode determination device 10 may display the holding mode estimated by the holding mode inference model 50 superimposed on the object image on the display device 20. Furthermore, if the robot 2 has at least two fingers that grip the object 80, the holding mode determination device 10 may display the position where the fingers grip the object 80 as the holding position 82. The holding mode determination device 10 may also display on the display device 20 a position within a predetermined range from the gripping position of the object 80.

[0071] While embodiments of the holding mode determination device 10 and the robot control system 100 have been described above, embodiments of the present disclosure may also include not only methods or programs for implementing the system or device, but also a storage medium on which a program is recorded (for example, an optical disc, magneto-optical disc, CD-ROM, CD-R, CD-RW, magnetic tape, hard disk, or memory card).

[0072] Furthermore, the implementation form of the program is not limited to application programs such as object code compiled by a compiler or program code executed by an interpreter, but may also be in the form of a program module embedded in an operating system. In addition, the program may or may not be configured so that all processing is performed only on the CPU on the control board. The program may also be configured so that some or all of its processing is performed by another processing unit implemented on an expansion board or expansion unit attached to the board, as needed.

[0073] While embodiments relating to this disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can make various modifications or alterations based on this disclosure. Therefore, it should be noted that these modifications or alterations are within the scope of this disclosure. For example, the functions and other elements included in each component can be rearranged in a logically consistent manner, and multiple components can be combined into one or separated.

[0074] All of the constituent elements described in this disclosure, and / or all of the disclosed methods or steps of processing, can be combined in any combination except for any combination in which these features are mutually exclusive. Furthermore, each of the features described in this disclosure can be replaced by an alternative feature that works for the same, equivalent, or similar purposes, unless expressly disregarded. Thus, unless expressly disregarded, each of the disclosed features is merely an example of a comprehensive set of identical or equivalent features.

[0075] Furthermore, the embodiments relating to this disclosure are not limited to any specific configuration of the embodiments described above. The embodiments relating to this disclosure can be extended to all novel features or combinations thereof described herein, or all novel methods or processing steps or combinations thereof described herein.

[0076] In this disclosure, the designations "First," "Second," etc., are identifiers used to distinguish the configurations. Configurations distinguished by the designations "First," "Second," etc., in this disclosure may have their numbers swapped. For example, Class 1 may swap the identifiers "First" and "Second" with Class 2. The swapping of identifiers occurs simultaneously. The configurations remain distinguishable even after the swapping of identifiers. Identifiers may be deleted. Configurations from which identifiers have been deleted are distinguished by codes. The designations "First," "Second," etc., in this disclosure should not be used alone to interpret the order of the configurations or to justify the existence of smaller numbered identifiers. [Explanation of Symbols]

[0077] 10. Holding mode determination device (12: control unit, 14: interface) 20 Display device 40-class inference model (42: processing layer) 50. Retention Mode Inference Model (52: Processing Layer, 54: Multiplier (541-544: 1st-4th Multipliers), 56: Adder) 60 Conversion section 80. Object to be held (83: screw head, 84: screw shaft) 81 Shape Images 82 Holding position 100 Robot control system (2: robot, 2A: arm, 2B: end effector, 3: sensor, 4: camera, 5: robot's influence range, 6: work start point, 7: work target point, 110: robot control device)

Claims

1. A control unit outputs to a display device, along with an image of the object, at least one holding mode of the object by the robot, which is estimated by classifying the object into at least one of a plurality of presumable holding categories. An interface for obtaining the user's selection for the at least one holding mode via the display device. Equipped with, The multiple presumably estimated retention categories are retention categories representing the general shape of the object, which can be output by a class inference model that outputs a classification result that classifies the object into a predetermined class based on an image of the object. A robot holding mode determination device, wherein the control unit outputs the holding mode for each of the classification results to the display device when the classification result is input to a holding mode inference model that outputs the holding mode of the object based on the input holding category.

2. The robot holding mode determination device according to claim 1, wherein the interface outputs the holding mode selected by the user to a robot control device that controls the operation of the robot so that the robot holds the object in the holding mode selected by the user.

3. The robot holding mode determination device according to claim 1, wherein the holding mode inference model determines the position in which the robot holds the object as a mode in which the robot holds the object.

4. The robot holding mode determination device according to claim 1, wherein the holding mode inference model determines the force applied by the robot to hold the object, as a mode in which the robot holds the object.

5. The robot holding mode determination device according to claim 1, wherein the holding mode inference model determines the type of hand the robot uses to hold the object, as the mode in which the robot holds the object.

6. The robot has a holding unit that holds the object based on an image of the object acquired from a camera and the output of a trained model including the class inference model and the holding mode inference model. The robot holding mode determination device according to claim 1, wherein the control unit outputs the user's selection to the robot so that the robot holds the object in the holding mode output by the holding mode inference model based on the class corresponding to the user's selection.

7. The holding portion has at least two fingers for gripping the object, The robot holding mode determination device according to claim 6, wherein the control unit outputs to the display device the position in which the fingers grip the object as the holding mode, or a position within a predetermined range from the position in which the object is gripped.

8. The robot holding mode determination device according to any one of claims 1 to 7, wherein the control unit outputs to the display device an image in which a mode of holding the object is superimposed on an image of the object.

9. The robot holding mode determination device according to any one of claims 1 to 7, wherein the control unit sets a conversion rule to convert the classification result of a holding category output from a class inference model that classifies an object into at least one of a plurality of presumable holding categories into other holding categories.

10. The robot holding mode determination device according to any one of claims 1 to 7, wherein the control unit determines a foreign object based on the result of classifying the object into at least one of a plurality of presumable holding categories.

11. The robot outputs to a display device, along with an image of the object, at least one of the estimated holding modes of the object, which is determined by classifying the object into at least one of a plurality of presumable holding categories. The user's selection for the at least one holding mode is obtained via the display device. Includes, The multiple presumably estimated retention categories are retention categories representing the general shape of the object, which can be output by a class inference model that outputs a classification result that classifies the object into a predetermined class based on an image of the object. A method for determining the holding mode of a robot, further comprising outputting the holding mode for each of the classification results to the display device when the classification result is input to a holding mode inference model that outputs the holding mode of the object based on the input holding category.

12. The system comprises a robot, a robot control device for controlling the robot, a holding mode determination device for determining the manner in which the robot holds an object, and a display device. The holding mode determination device outputs to the display device, along with an image of the object, at least one holding mode of the robot for the object, which is estimated by classifying the object into at least one of a plurality of presumable holding categories. The display device displays an image of the object and at least one holding mode of the object, receives input from the user regarding the at least one holding mode, and outputs it to the holding mode determination device. The holding mode determination device obtains the user's selection for the at least one holding mode from the display device, The multiple presumably estimated retention categories are retention categories representing the general shape of the object, which can be output by a class inference model that outputs a classification result that classifies the object into a predetermined class based on an image of the object. The holding mode determination device outputs the holding mode for each of the classification results to the display device when the classification result is input to a holding mode inference model that outputs the holding mode of the object based on the input holding category. Robot control system.

Images