Robot control device and robot control method

Abstract

A robot control device includes a control unit that controls a robot having a holding device. The control unit acquires a first image in which at least one holding object is captured. The control unit selects, as a selected object, at least one holding object held by the holding device from among the at least one holding object recognized in the first image. The control unit determines a holding position of the selected object on the basis of a second image obtained by capturing the selected object. The control unit causes the holding device to hold the selected object at the determined holding position.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit of priority of Japanese Patent Application No. 2022-35713 (filed on March 8, 2022), the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] The present invention relates to a robot control device and a robot control method. Background Art

[0004] Conventionally, there is known a method for appropriately holding an object of an arbitrary shape using a robot arm (see, for example, Patent Document 1).

[0005] Prior art literature

[0006] Patent Literature

[0007] Patent Document 1: Japanese Patent Application Publication No. 2005-169564 Summary of the invention

[0008] A robot control device according to one embodiment of the present invention includes a control unit for controlling a robot having a holding device. The control unit obtains a first image obtained by photographing at least one holding object. The control unit selects at least one holding object to be held by the holding device as a selected object from among the at least one holding object identified in the first image. The control unit determines a holding position of the selected object based on a second image obtained by photographing the selected object. The control unit causes the holding device to hold the selected object at the determined holding position.

[0009] A robot control method according to an embodiment of the present invention includes a process of controlling a robot having a holding device. The robot control method includes a process of obtaining a first image obtained by photographing at least one held object. The robot control method includes a process of selecting at least one held object to be held by the holding device as a selected object from the at least one held object identified in the first image. The robot control method includes a process of determining a holding position of the selected object based on a second image obtained by photographing the selected object. The robot control method includes a process of causing the holding device to hold the selected object at the determined holding position. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram showing a configuration example of a robot control system according to one embodiment.

[0011] Figure 2This is a block diagram showing a configuration example of a robot control system according to an embodiment.

[0012] Figure 3 It is a diagram showing an example of recognition of a tray in a bird's-eye view image.

[0013] Figure 4A This is an example of a first image obtained by photographing a tray.

[0014] Figure 4B It is shown Figure 4A FIG. 1 is a diagram showing an example of recognition of a held object in a first image.

[0015] Figure 5 It is a top view and a side view which show an example of a holding target object.

[0016] Figure 6 FIG. 1 is a diagram showing an example of a selection object selected in the first image.

[0017] Figure 7 It is a diagram showing the normal direction of the plane included in the holding object.

[0018] Figure 8 This is an example of a second image obtained by photographing the held object from the normal direction.

[0019] Fig. 9 1 is a schematic diagram showing an example in which the area where the robot places and holds an object is changed.

[0020] Fig.10 The present invention is a flowchart of a robot control method including an example of a procedure for determining a holding position.

[0021] Fig.11 The present invention is a flowchart of a robot control method including a procedure example of holding a holding object and arranging it at a determined holding position. DETAILED DESCRIPTION

[0022] (Configuration Example of Robot Control System 100)

[0023] like Figure 1 as well as Figure 2As shown, a robot control system 100 of one embodiment includes a robot 1, a robot control device 2, and a system control device 10. The various structural parts of the robot control system 100 can be connected to each other so as to be communicable with the help of a network, or can be connected to each other so as to be communicable without the help of a network. At least one structural part of the robot control system 100 can also be connected to each other so as to be communicable in a wired or wireless manner. At least one structural part of the robot control system 100 can also be connected to each other so as to be communicable with the help of a dedicated line. At least one structural part of the robot control system 100 is not limited to these examples, and can also be connected to each other so as to be communicable in various other ways. The following describes in detail the various structures of the robot control system 100.

[0024] <System Control Device 10>

[0025] The system control device 10 includes a control unit 12 and an interface 14 .

[0026] The control unit 12 may be configured to include at least one processor in order to implement various functions of the system control device 10. The processor can execute programs that implement various functions of the system control device 10. The processor may be implemented as a single integrated circuit. An integrated circuit is also referred to as an IC (Integrated Circuit). The processor may be implemented as a plurality of integrated circuits and discrete circuits that can be connected in communication. The processor may be configured to include a CPU (Central Processing Unit). The processor may also be configured to include a DSP (Digital Signal Processor) or a GPU (Graphics Processing Unit). The processor may be implemented based on various other known technologies.

[0027] The system control device 10 may further include a storage unit. The storage unit may be configured to include an electromagnetic storage medium such as a magnetic disk, or may be configured to include a memory such as a semiconductor memory or a magnetic memory. The storage unit may also be configured as an HDD (Hard Disk Drive) or an SSD (Solid State Drive). The storage unit stores various information and programs executed by the control unit 12. The storage unit may function as a working memory of the control unit 12. At least a portion of the storage unit may also be included in the control unit 12. At least a portion of the storage unit may also be configured as a storage device separate from the system control device 10.

[0028] The interface 14 may be configured to include a communication device configured to communicate in a wired or wireless manner. The communication device may be configured to communicate in a communication manner based on various communication standards.

[0029] The system control device 10 may be configured to include one or more servers. The system control device 10 may be configured to enable multiple servers to perform parallel processing. The system control device 10 does not need to be configured to include a physical shell, and may also be configured based on virtualization technologies such as virtual machines or container orchestration systems. The system control device 10 may also be configured using cloud services. In the case where the system control device 10 is configured using cloud services, it can be configured by combining management services. That is, the functions of the system control device 10 can be implemented as cloud services.

[0030] The system control device 10 may include at least one server group and at least one database group. The server group functions as a control unit 12. The database group functions as a storage unit. The number of server groups may be one or more than two. When the number of server groups is one, the functions implemented by one server group include the functions implemented by each server group. The server groups are connected to each other in a wired or wireless manner so as to be able to communicate. The number of database groups may be one or more than two. The number of database groups may be appropriately increased or decreased based on the capacity of data managed by the system control device 10 and the availability requirements required for the system control device 10. The database group is connected to each server group in a wired or wireless manner so as to be able to communicate. The system control device 10 may also be connected to an external database. An information processing system including the system control device 10 and an external database may also be constructed.

[0031] The system control device 10 Figure 1 as well as Figure 2 Although it is recorded as one structure, multiple structures can be operated as one system as needed. That is, the system control device 10 is configured as a platform with variable capacity. By using multiple structures as the system control device 10, even if one structure becomes inoperable due to unpredictable events such as natural disasters, other structures can be used to continue the operation of the system. In this case, the multiple structures are connected by lines regardless of whether they are wired or wireless, and are configured to communicate with each other. The multiple structures can also be constructed across cloud services and local deployment environments.

[0032] The system control device 10 is connected to at least one component of the robot control system 100 via a wired or wireless line so as to be communicable. The system control device 10 and at least one component of the robot control system 100 have interfaces using a standard protocol to enable bidirectional communication.

[0033] The robot control system 100 may also include an external device 6. When the robot control system 100 includes an external device 6, the external device 6 may be controlled by, for example, the system control device 10. In this case, the system control device 10 may include, for example, a PLC (Programmable Logic Controller) or the like, and communicate with the robot control device 2 described later, and control the external device 6 in accordance with the action of the robot 41.

[0034] <Robot control device 2>

[0035] The robot control device 2 includes a robot control unit 22 (also referred to as a control unit). The robot control unit 22 may be configured to include at least one processor in order to realize various functions or controls of the robot control device 2. The robot control unit 22 is configured to be able to control at least one robot 40. The robot control unit 22 may be configured the same as or similar to the control unit 12 of the system control device 10. The robot control device 2 may also be configured to further include a storage unit. The storage unit of the robot control device 2 may be configured the same as or similar to the storage unit of the system control device 10.

[0036] The robot control device 2 obtains information that specifies the action to be performed by the robot 1 from the system control device 10. The information that specifies the action to be performed by the robot 1 is also called action information. The robot control device 2 causes the robot 1 to act based on the action information. Figure 1 As shown in the example, the robot 1 can perform the operation of holding the object 80 in the tray 7 and moving it to the external device 6. The robot control device 2 is not limited to this, and can make the robot 1 perform various operations. The robot control device 2 may be connected to the cloud computing environment or not. When the robot control device 2 is not connected to the cloud computing environment, the operation of the robot control device 2 is completed in the local deployment environment.

[0037] The robot control device 2 may include a communication device for acquiring motion information from the system control device 10. The communication device of the robot control device 2 may be configured identically or similarly to the communication device of the system control device 10. The robot control unit 22 of the robot control device 2 can generate information for controlling the motion of the robot 1 by executing a control program based on the motion information.

[0038] The robot control device 2 may include a robot interface 24. The robot interface 24 may include a communication device that can communicate in a wired or wireless manner. The communication device may be configured to communicate in a communication method based on various communication standards.

[0039] The robot interface 24 may also have an input device and an output device. The input device may be configured to include, for example, a touch panel or a contact sensor, or a pointing device such as a mouse. The input device may also be configured to include a physical key. The input device may also be configured to include a sound input device such as a microphone. The input device is not limited to these examples, and may also be configured to include various other devices.

[0040] The output device may be configured to include a display device. For example, the display device may be configured to include a liquid crystal display (LCD: Liquid Crystal Display), an organic EL (Electro-Luminescence) display or an inorganic EL display, or a plasma display (PDP: Plasma Display Panel). The display device is not limited to these displays, and may be configured to include displays of various other types. The display device may be configured to include a light-emitting device such as an LED (Light Emitting Diode). The display device may be configured to include various other devices. The output device may also be configured to include a sound output device such as a speaker that outputs auditory information such as sound. The output device is not limited to these examples, and may be configured to include various other devices.

[0041] exist Figure 1 In the illustrated structure, one robot control device 2 is connected to one robot 1. One robot control device 2 may be connected to two or more robots 1. One robot control device 2 may control only one robot 1 or may control two or more robots 1. The number of robot control devices 2 and robots 1 is not limited to two, and may be one or three or more.

[0042] <Robot 1>

[0043] The robot 1 may include an arm including joints and links. The arm may be, for example, a 6-axis or 7-axis vertical multi-joint robot. The arm may also be a 3-axis or 4-axis horizontal multi-joint robot or a SCARA robot. The arm may also be a 2-axis or 3-axis orthogonal robot. The arm may also be a parallel linkage robot, etc. The number of axes constituting the arm is not limited to the number illustrated.

[0044] The robot 1 has a holding device 20 installed at the front end of the arm or a predetermined position. The holding device 20 may also include a suction hand configured to suck and hold the object 80. The suction hand may have one or more suction parts. The holding device 20 may include a gripping hand configured to hold the object 80. The gripping hand may have a plurality of fingers. The number of fingers of the gripping hand may also be two or more. The fingers of the gripping hand may have one or more joints. The holding device 20 may also include a scooping hand configured to scoop and hold the object 80.

[0045] The robot 1 can control the position and posture of the holding device 20 by moving the arm. The posture of the holding device 20 can be represented by an angle that determines the direction in which the holding device 20 acts relative to the held object 80. The posture of the holding device 20 is not limited to an angle, and can also be represented by various other methods such as a space vector. The direction in which the holding device 20 acts relative to the held object 80 can be, for example, the direction in which the holding device 20 approaches the held object 80 when the holding device 20 adsorbs and holds the held object 80, or it can be the direction in which the adsorption is performed. The direction in which the holding device 20 acts relative to the held object 80 can also be the direction in which the holding device 20 approaches the held object 80 when the held object 80 is held by multiple fingers, or it can be the direction in which the holding is performed. The direction in which the holding device 20 acts relative to the held object 80 is not limited to these examples, and can be various directions.

[0046] The robot 1 may further include a sensor for detecting the state of an arm including a joint or a link, or the state of the holding device 20. The sensor may detect the actual position or posture of the arm or the holding device 20, or information related to the speed or acceleration of the arm or the holding device 20, as the state of the arm or the holding device 20. The sensor may also detect the force acting on the arm or the holding device 20. The sensor may also detect the current flowing in the motor driving the joint or the torque of the motor. The sensor can detect information obtained as a result of the actual action of the robot 1. The robot control device 2 can grasp the result of the actual action of the robot 1 by obtaining the detection result of the sensor.

[0047] <Handheld Camera 30>

[0048] The robot control system 100 also includes a hand camera 30 installed at the front end of the arm of the robot 1 or at a predetermined position. The hand camera 30 can be configured to capture RGB images or to obtain depth data. The robot 1 can control the position and posture of the hand camera 30 by moving its arm. The posture of the hand camera 30 can be represented by an angle that determines the direction in which the hand camera 30 takes a picture. The posture of the hand camera 30 is not limited to an angle, and can also be represented by various other methods such as a space vector. The hand camera 30 outputs the image captured at the position and posture determined by the movement of the robot 1 to the robot control device 2.

[0049] <Overhead Camera 3>

[0050] The robot control system 100 may further include a bird's-eye view camera 3. The bird's-eye view camera 3 may be configured to capture RGB images, or may be configured to obtain depth data. The bird's-eye view camera 3 may be configured to capture the working range 4 of the robot 1. The bird's-eye view camera 3 may also be configured to capture a range including at least a portion of the working range 4. The bird's-eye view camera 3 may be configured to capture a range including a range where the pallet 7 can be placed. The bird's-eye view camera 3 outputs the captured image to the robot control device 2 as a bird's-eye view image. The bird's-eye view camera 3 is also referred to as a first imaging device. On the other hand, the hand camera 30 is also referred to as a second imaging device.

[0051] <External device 6>

[0052] The robot control system 100 may further include an external device 6. The external device 6 may be controlled by the system control device 10. In this case, the system control device 10 communicates with the robot control device 2, so that the external device 6 and each structural part under the control of the robot control device 2 can be controlled to be able to communicate. The external device 6 has a conveying device. The conveying device is Figure 1 The middle structure is a belt conveyor.

[0053] The robot control system 100 may also include a flipping device as an external device 6. The flipping device can flip the component when the robot 1 picks up the component with the back side facing. The device for flipping the component may also be included in the conveying device. The flipping device may be configured to put the flipped component into the conveying device, or may be configured to allow the flipped component to be picked up by the robot 1.

[0054] (Operation Example of Robot Control System 100)

[0055] In the robot control system 100 of the present embodiment, the picking and placing operation of bulk components is performed. That is, the robot control device 2 causes the robot 1 to pick up the bulk components and arrange the components at a predetermined position. The robot control device 2 can detect the posture of the component when the robot 1 is caused to pick up. The robot control device 2 causes the robot 1 to perform the picking and placing operation on the component based on the detection result of the posture of the component.

[0056] The robot control device 2 may also determine the area and / or placement method of the component in the pick and place operation based on the posture of the component. For example, the robot control device 2 may also control the robot 1 in such a way that the posture (front and back or rotation direction) of the component when the component is arranged at a specified position conforms to a specified posture in order to perform machine tending (inputting a workpiece into another device and taking a workpiece out of another device).

[0057] The robot control device 2 can, for example, arrange the component in different postures when the robot 1 picks up the component with the surface facing and when the robot 1 picks up the component with the back facing. Specifically, the component can be arranged in the first mode when the robot 1 picks up the component with the surface facing. The component can also be arranged in the second mode when the robot 1 picks up the component with the back facing. In addition, the second mode can be set to match the orientation or posture of the component arranged in the first mode when the component arranged in the second mode is reversed by the external device 6.

[0058] In addition, the robot control device 2 can, for example, arrange the components at different positions when the robot 1 picks up the components with the surface facing or the back facing, in order to distinguish the components according to the surface facing or the back facing. When the robot control system 100 has a flipping device, the robot control device 2 can make the robot 1 put the components picked up with the back facing into the flipping device, and pick up the components that are reversed to the surface facing by the flipping device again. The robot control device 2 can make the robot 1 arrange the components picked up with the surface facing in a predetermined position while maintaining the state.

[0059] The robot control device 2 can detect the position and posture of the tray 7 storing bulk components and control the robot 1 based on the position and posture of the tray 7. That is, in the robot control system 100, it is not necessary to position the tray 7 storing bulk components.

[0060] In the robot control system 100 of the present embodiment, the robot control device 2 can estimate the posture of the component without performing three-dimensional matching based on the image obtained by photographing the component in a bulk state. The estimation accuracy of the posture of the component in the present embodiment can be higher than the estimation accuracy of the case where a technique for estimating the posture of the component using deep learning such as PoseCNN (Convolution Neural Network) is assumed.

[0061] The robot control system 100 of the present embodiment can be applied, for example, to the picking and placing of parts with a relatively stable posture. Parts with a relatively stable posture include, for example, parts with a relatively large plane that can clearly distinguish whether the part is facing the front or the back, that is, flat parts. When the robot control device 2 causes the robot 1 to perform the picking and placing of parts with a relatively stable posture, it can improve the accuracy of the operation by decoupling the tasks. In addition, when the robot control device 2 causes the robot 1 to perform the picking and placing of flat parts, it can determine the posture of the part by rotating the flat part around the normal line of the flat part as the axis.

[0062] The system control device 10 outputs operation information to the robot control device 2 so that the robot 1 executes the operation of holding the holding object 80 placed in the tray 7 by the holding device 20 and moving it to the external device 6. A specific operation example of the robot control device 2 will be described below.

[0063] <Obtaining Bird's-Eye Image>

[0064] In the robot control system 100, the bird's-eye view camera 3 can capture a range including the working range 4 of the robot 1 and output it as a bird's-eye view image. The robot control unit 22 of the robot control device 2 can obtain the bird's-eye view image and detect the pallet 7 shown in the bird's-eye view image. The robot control unit 22 can detect the position and posture of the pallet 7 by detecting the feature points of the pallet 7. Figure 3 . The pallet 7 is shown in the bird's-eye view image. The robot control unit 22 can detect the pallet 7 in a manner represented by an identification frame 76. The identification frame 76 is generated in a manner surrounding the detected pallet 7. The shape of the identification frame 76 is not limited to a rectangle, and may be other polygons, or may be any figure including a curve.

[0065] The bird's-eye view image shows the objects 80 stored in bulk on the tray 7. The bird's-eye view image shows the robot 1 including the holding device 20 and the hand camera 30. The external device 6 to which the robot 1 moves the objects 80 is shown in the bird's-eye view image.

[0066] The robot control unit 22 may display the bird's-eye view image using the robot interface 24. The robot control unit 22 may display an image in which the recognition frame 76 is superimposed on the bird's-eye view image. This allows the operator to confirm the recognition result of the tray 7. As a result, the convenience of the robot 1 can be improved.

[0067] <Photographing of Tray 7 by Hand Camera 30>

[0068] The robot control unit 22 controls the hand camera 30 of the robot 1 to photograph the tray 7. The robot control unit 22 controls the position and posture of the hand camera 30 based on the position and posture of the tray 7 so that the size of the tray 7 in the image obtained by the hand camera 30 photographing the tray 7 becomes a prescribed size, or the orientation of the tray 7 becomes a prescribed orientation. The position and posture of the hand camera 30 when photographing the tray 7 in a prescribed size or in a prescribed orientation are also referred to as a first position and a first posture, respectively. Specifically, the hand camera 30 photographs the tray 7 at a prescribed distance from a prescribed direction, so that the tray 7 is photographed in a prescribed size and orientation. The robot control unit 22 can set the prescribed direction to the direction directly above the tray 7. The robot control unit 22 can control the position and posture of the hand camera 30 in such a way that the hand camera 30 photographs the tray 7 at a prescribed distance from a prescribed direction.

[0069] The robot control unit 22 can detect the position and posture of the tray 7 based on the overhead image when acquiring the overhead image. In this case, the place and posture where the operator arranges the tray 7 can also be uncertain. That is, the operator can freely arrange the tray 7 within the range shown in the overhead image. The robot control unit 22 can calculate the first position and the first posture of the hand camera 30 based on the overhead image, and control the robot 1 in such a manner that the hand camera 30 photographs the tray 7 at the first position and the first posture. The robot control unit 22 can acquire the position and posture of the tray 7 that is predetermined as the arrangement rule of the tray 7 without acquiring the overhead image. In this case, it is determined in such a manner that the operator arranges the tray 7 at a predetermined position with a predetermined posture.

[0070] <Selection of Object 80 and Detection of Normal Line>

[0071] The robot control unit 22 obtains an image obtained by the hand camera 30 photographing the tray 7 at the first position and the first posture, that is, an image obtained by photographing the tray 7 at a predetermined angle of view, as a first image. Figure 4A An example of the first image is shown in FIG. The first image shows the tray 7 and the holding objects 80 contained in the tray 7 in a bulk state. The robot control unit 22 recognizes the holding objects 80 shown in the first image. Figure 4BAs illustrated, the robot control unit 22 can detect the holding object 80 shown in the first image in a manner represented by the identification frame 86. That is, the robot control unit 22 can generate the identification frame 86 as the recognition result of the holding object 80. The identification frame 86 can also be displayed in a manner surrounding the holding object 80 shown in the first image. In addition, the identification frame 86 can also be generated for each holding object 80. The shape of the identification frame 86 is not limited to a rectangle, and can also be other polygons, or can be any figure including a curve.

[0072] It should be noted that the robot control unit 22 may detect the holding object 80 shown in the first image in a manner of being represented by a mask. That is, the robot control unit 22 may generate a mask as a recognition result of the holding object 80. The mask is displayed in a manner of covering the holding object 80 shown in the first image. It should be noted that the mask may also be displayed along the outline of the holding object 80 shown in the first image. In addition, the mask may also be generated for each holding object 80.

[0073] The outer shape of the holding object 80 in this embodiment is as follows: Figure 5 The top view and the side view are shown. The object 80 to be held is divided into a first part 81 and a second part 82 with the bent portion as a boundary. The first part 81 has a surface 81A and a back surface 81B. The second part 82 has a surface 82A and a back surface 82B. The direction in which the second part 82 is bent is set to the front side of the object 80 to be held.

[0074] The robot control unit 22 Figure 4B The holding object 80 that is to be held next by the holding device 20 is selected from among the holding objects 80 identified as shown in the example. The selected holding object 80 is also referred to as a selected object 84. Figure 6 As illustrated, one holding object 80 is selected from the holding objects 80 as the selected object 84 .

[0075] The robot control unit 22 may perform plane estimation, for example, in order to select the selected object 84 from the held objects 80 stored in bulk in the tray 7. The robot control unit 22 may detect the plane with the most point group records from the planes of the object shown in the first image, and select the held object 80 having the detected plane as the selected object 84. It should be noted that the robot control unit 22 may also perform plane estimation of the held object 80 for each mask. In this case, information that becomes noise during plane estimation can be reduced, and the accuracy of plane estimation can be improved.

[0076] The robot control unit 22 is as follows Figure 7As illustrated, normal line 83A or normal line 83B of front surface 81A or back surface 81B of first portion 81 of selected object 84 is detected. Robot controller 22 detects the normal line of the plane containing the most point groups of selected object 84 as normal line 83A or 83B of selected object 84.

[0077] The robot control unit 22 may not determine whether the selected object 84 faces the front side or the back side in the first image. The robot control unit 22 may detect the normal line 83A or 83B of the first part 81 of the selected object 84 regardless of whether the selected object 84 faces the front side or the back side. When the selected object 84 faces the front side, the robot control unit 22 detects the normal line 83A of the surface 81A. When the selected object 84 faces the back side, the robot control unit 22 detects the normal line 83B of the back side 81B. The robot control unit 22 can estimate the normal line 83A or 83B by image analysis of the first image.

[0078] <Photographing, Recognition, and State Detection of Holding Object 80 from the Normal Direction>

[0079] The robot control unit 22 controls the robot 1 and controls the position and posture of the hand camera 30 so that the hand camera 30 captures the selected object 84 from the direction of the detected normal line 83A or 83B. The robot control unit 22 obtains the image captured by the hand camera 30 from the direction of the detected normal line 83A or 83B as the second image. Figure 8 An example of the second image is shown in FIG. Figure 8 The surface 81A of the first portion 81 and the surface 82A of the second portion 82 of the selected object 84 located on the surface side are reflected.

[0080] The robot control unit 22 can be as follows in the second image Figure 8The robot control unit 22 may recognize the selected object 84 again as indicated as a recognition frame 86 in the image. The robot control unit 22 may detect whether the selected object 84 faces the front side or the back side based on the second image. The robot control unit 22 may also detect the direction of the normal line 83A or 83B of the first part 81 of the selected object 84 again based on the second image. The robot control unit 22 may detect the direction of the normal line 83A of the surface 81A when the selected object 84 faces the front side. The robot control unit 22 may detect the direction of the normal line 83B of the back side 81B when the selected object 84 faces the back side. The robot control unit 22 may detect the rotation angle around the normal line 83A or 83B of the selected object 84 based on the second image. The robot control unit 22 may detect information including the direction of the normal line 83A or 83B of the selected object 84 and the rotation angle around the normal line 83A or 83B as information on the posture of the selected object 84. That is, the posture of the selected object 84 can be determined by the direction of the normal line 83A or 83B and the rotation angle around the normal line 83A or 83B. The posture of the selected object 84 can be determined by the direction of the normal line 83A and the rotation angle around the normal line 83A based on the direction of the normal line 83A. When the selected object 84 faces the back side, the robot control unit 22 can detect the direction of the normal line 83B and regard the opposite direction of the normal line 83B as the direction of the normal line 83A.

[0081] <Determination of holding position and holding action>

[0082] The robot control unit 22 determines the surface of the selected object 84 to be held by the holding device 20 based on the detection result of the state of the selected object 84. The surface on which the holding device 20 holds is also called a holding surface. When the selected object 84 faces the front side, the robot control unit 22 determines the surface 81A of the first part 81 as the holding surface. When the selected object 84 faces the back side, the robot control unit 22 determines the back side 81B of the first part 81 as the holding surface. The robot control unit 22 determines the position on the holding surface at which the holding device 20 is held. The position at which the holding device 20 is held is also called a holding position. The robot control unit 22 controls the robot 1 in such a manner that the holding device 20 holds the held object 80 at the determined holding position.

[0083] The robot control unit 22 can correct the rotation angle of the selected object 84 around the normal line 83A or 83B when the holding device 20 holds the selected object 84 and arranges it at the prescribed position. In other words, the robot control unit 22 can control the robot 1 in such a manner that the direction of the selected object 84 is consistent and arranged at the prescribed position. The robot control unit 22 can determine the correction amount of the rotation angle around the normal line 83A or 83B when the holding device 20 arranges the selected object 84 at the prescribed position after holding the selected object 84 based on the posture of the selected object 84 projected in the second image. The robot control unit 22 can determine the posture of the holding device 20 when holding the selected object 84, taking into account the posture of the holding device 20 when arranging the selected object 84 at the prescribed position after holding the selected object 84. The robot control unit 22 can control the robot 1 in such a manner that the holding device 20 holds the selected object 84 in the determined posture.

[0084] <Arrangement of Holding Object 80>

[0085] The robot control unit 22 controls the robot 1 so as to arrange the held object 80 at different positions when the held object 80 selected as the selected object 84 faces the front side and faces the back side. Fig. 9 As shown, the robot control unit 22 can control the robot 1 so as to place the held object 80 in the first area 6A when the held object 80 faces the front side. The robot control unit 22 can control the robot 1 so as to place the held object 80 in the second area 6B when the held object 80 faces the back side.

[0086] exist Fig. 9 In the example of , the first area 6A and the second area 6B are set as areas located on the belt conveyor constituting the external device 6. When the external device 6 is configured as a conveying device, the holding object 80 arranged in the first area 6A or the second area 6B is conveyed in the conveying direction. The robot control unit 22 can arrange the next holding object 80 in the first area 6A or the second area 6B after the holding object 80 is conveyed in the conveying direction and the first area 6A and the second area 6B are vacant.

[0087] The first area 6A or the second area 6B is not limited to being set on the belt conveyor as described above, and can be set in other places. For example, the second area 6B can be set in a flipping device included in the external device 6. Thus, the object 80 can be flipped by the flipping device when it is facing the back side. As a result, the object 80 can be arranged on the external device 6 in a state facing the front side. The first area 6A or the second area 6B can also be set in a place where a transport vehicle or a drone is on standby, or can be set in a place where a transport vehicle picks up the object 80.

[0088] (Procedure example of robot control method)

[0089] The robot control unit 22 of the robot control device 2 may also execute the following operations: Fig.10 11. The robot control method may be implemented as a robot control program executed by a processor constituting the robot control unit 22 of the robot control device 2. The robot control program may be stored in a non-transitory computer-readable medium.

[0090] The robot control unit 22 obtains a bird's-eye view image from the bird's-eye view camera 3 ( Fig.10 The robot control unit 22 recognizes the pallet 7 from the bird's-eye view image (step S2). Based on the recognition result of the pallet 7, the robot control unit 22 operates the robot 1 so that the hand camera 30 images the pallet 7, and obtains the first image captured by the hand camera 30 (step S3).

[0091] The robot control unit 22 recognizes the objects that are stored in bulk in the tray 7 and are shown in the first image (step S4). The robot control unit 22 selects the holding object 80 to be held by the holding device 20 from the recognized objects (step S5). The robot control unit 22 detects the normal direction of the holding object 80 based on the image of the holding object 80 included in the first image (step S6). The robot control unit 22 operates the robot 1 so that the hand camera 30 captures the holding object 80 from the detected normal direction of the holding object 80, and obtains the second image captured by the hand camera 30 (step S7).

[0092] The robot control unit 22 recognizes the holding object 80 based on the second image and detects the posture of the holding object 80 (step S8). The robot control unit 22 determines the position of the holding device 20 holding the holding object 80 based on the recognition result of the holding object 80 and the detection result of the posture of the holding object 80 (step S9). As described above, the robot control unit 22 performs Fig.10 In the example of the sequence, it is possible to select the holding object 80 and determine its holding position.

[0093] The robot control unit 22 moves the holding device 20 to the position where the holding object 80 is held ( Fig.11 The robot control unit 22 causes the holding device 20 to hold the holding object 80 (step S11). The robot control unit 22 determines whether the holding surface of the holding object 80 is the front side of the holding object 80 (step S12).

[0094] When the holding surface of the object 80 is the front side of the object 80 (step S12: Yes), the robot control unit 22 moves the holding device 20 so as to place the object 80 in the first area 6A (step S13). When the holding surface of the object 80 is not the front side of the object 80 (step S12: No), that is, when the holding surface of the object 80 is the back side of the object 80, the robot control unit 22 moves the holding device 20 so as to place the object 80 in the second area 6B (step S14).

[0095] After the robot control unit 22 moves the holding device 20 in the order of step S13 or S14, the holding device 20 places the holding object 80 in the first area 6A or the second area 6B (step S15). After the robot control unit 22 executes the order of step S15, the process ends. Fig.10 as well as Fig.11 Execution in the order of the flowchart.

[0096] (summary)

[0097] As described above, in the robot control system 100 of the present embodiment, the robot control device 2 includes a robot control unit 22 that controls the robot 1 having the holding device 20. The robot control unit 22 obtains a first image obtained by photographing the holding object 80 by the hand camera 30, and recognizes the holding object 80 in the first image. The robot control unit 22 selects the holding object 80 to be held by the holding device 20 from among the recognized holding objects 80 as the selected object 84. The robot control unit 22 obtains a second image obtained by photographing the selected object 84 by the hand camera 30. The robot control unit 22 recognizes the selected object 84 in the second image and determines the holding position of the selected object 84. The robot control unit 22 causes the holding device 20 to hold the selected object 84 at the determined holding position. As a result, the holding of the holding object 80 by the holding device 20 can be stably performed. As a result, the convenience of the robot 1 can be improved.

[0098] (Other embodiments)

[0099] Hereinafter, a robot control system 100 according to another embodiment will be described.

[0100] <Filter>

[0101] The robot control unit 22 may perform selection candidate narrowing to select the selection object 84 from the held objects 80 stored in bulk on the tray 7. The robot control unit 22 may select the selection object 84 from the held objects 80 remaining as selection candidates through narrowing.

[0102] The robot control unit 22 may generate a mask for holding the object 80 in the first image. The robot control unit 22 may select the held object 80 based on the score. When the score of the generated mask is greater than a threshold value, the held object 80 may be left as a selection candidate. When the area of ​​the mask is greater than a threshold value, the robot control unit 22 may leave the held object 80 as a selection candidate. The robot control unit 22 may calculate the ratio of non-missing data for the depth data in the mask, and leave the held object 80 as a selection candidate when the ratio is greater than a threshold value.

[0103] When the center coordinates of the recognition frame 86 generated when recognizing the held object 80 are located inside the tray 7 , the robot control unit 22 may keep the held object 80 as a selection candidate.

[0104] <Selection based on rating>

[0105] The robot control unit 22 may also calculate the score of the holding object 80 shown in the first image, and evaluate the ease of acquisition based on the score. Furthermore, the robot control unit 22 may also select the selection object 84 based on the evaluation result. In this case, for example, the selection objects 84 may be selected in descending order of the scores. The robot control unit 22 may calculate the score of the holding object 80 based on the depth data. The robot control unit 22 may also calculate the score of the holding object 80 as a higher value when the depth data of the holding object 80 includes a point indicating that it is located above the tray 7. The robot control unit 22 may also calculate the score of the holding object 80 that is retained after screening.

[0106] The robot control unit 22 may also calculate the score based on the mask. For example, the robot control unit 22 may also calculate the score of the held object 80 based on the area of ​​the mask. The robot control unit 22 may calculate the score as a higher value as the area of ​​the mask is larger. The robot control unit 22 may also calculate the score based on whether the mask can accurately reflect the shape of the held object 80. For example, in a case where the held object 80 has a hole, the score of the held object 80 may be calculated based on whether the hole of the mask appears correctly. In a case where the held object 80 is tilted greatly, it is difficult to reflect the hole in the first image. In a case where a hole is omitted, it is difficult to overcome gravity because the adsorption force or the holding force acts obliquely when the held object 80 is held by the holding device 20. The robot control unit 22 may calculate the score as a higher value when the hole of the mask appears correctly.

[0107] The robot control unit 22 may calculate a higher score for a held object 80 that does not overlap another held object 80. The robot control unit 22 may calculate the accuracy of the depth data and calculate a higher score for a held object 80 with higher accuracy.

[0108] The robot control unit 22 may calculate an evaluation score by combining the scores reflecting the above-mentioned various factors for each held object 80, and select a held object 80 having a larger evaluation score as the selected object 84. The robot control unit 22 may calculate the evaluation score by weighting the scores reflecting the various factors.

[0109] The above describes an implementation mode of the robot control system 100, but as an implementation mode of the present invention, in addition to the method or program for implementing the device, an implementation mode can also be adopted as a storage medium with a program recorded thereon (for example, a CD, a magneto-optical disk, CD-ROM, CD-R, CD-RW, a magnetic tape, a hard disk, or a memory card, etc.).

[0110] In addition, the installation method of the program is not limited to application programs such as target codes compiled by a compiler and program codes executed by an interpreter, but may also be a program module compiled into an operating system. Furthermore, the program may be configured so that all processing is performed only in the CPU on the control substrate. The program may also be configured so that part or all of it is performed by an expansion board attached to the substrate or other processing units installed in the expansion unit as needed.

[0111] The embodiments of the present invention are described based on the drawings and embodiments. However, it should be noted that a person skilled in the art can make various modifications or changes based on the present invention. Therefore, please note that these modifications or changes are included in the scope of the present invention. For example, the functions included in each structural part can be reconfigured in a logically non-contradictory manner, and multiple structural parts can be combined into one or divided.

[0112] For example, in the above description, an example is described in which the robot control system 100 includes one robot control device 2. The robot control device 2 may be divided. That is, the robot control system 100 may actually include two robot control devices 2. In this case, the two robot control devices 2 may each be responsible for different controls of the robot 1.

[0113] In addition, in the above description, after the robot control unit 22 selects the selection object 84 based on the first image, it determines the holding position of the selection object 84 based on the second image, but the robot control unit 22 may also determine the holding position of the selection object 84 based on the first image. In this case, for example, the robot control unit 22 may determine whether to obtain the second image after selecting the selection object 84 based on the first image. More specifically, the robot control unit 22 may, for example, calculate the score of the holding object 80 shown in the first image, select the selection object 84 based on the score, and determine whether to obtain the second image. For example, the robot control unit 22 may also evaluate the ease of obtaining the holding object 80, and when the ease of obtaining the holding object 80 determined by the first image is high, the robot control unit 22 does not obtain the second image, but holds the holding object 80 based on the first image. In addition, the robot control unit 22 may evaluate the proximity between the direction of the normal line 83A or 83B obtained by image analysis of the first image and the shooting direction from the camera to the holding object 80 when the first image is obtained, and when it is estimated that the shooting direction and the direction of the normal line 83A or 83B are sufficiently close based on a predetermined threshold value, the holding object 80 is held based on the first image. It should be noted that the holding of the holding object 80 based on the first image may be performed in the same manner as the holding of the holding object 80 based on the second image. In addition, when it is determined that the second image needs to be obtained, the robot control unit 22 may hold the holding object 80 based on the second image as described above.

[0114] The robot control unit 22 may select at least one holding object 80 as the selected object 84 based on the first image, and determine whether to capture the second image. In addition, when it is determined that it is not necessary to capture the second image, the robot control unit 22 may determine the holding position of the selected object 84 based on the first image, and cause the holding device 20 to hold the selected object 84 at the determined holding position.

[0115] All of the constituent elements recorded in the present invention and / or all of the disclosed methods or all of the steps of the processing can be combined in any combination except for the mutually exclusive combination of these features. In addition, the features recorded in the present invention can be replaced with alternative features that play a role for the same purpose, equivalent purpose or similar purpose as long as they are not explicitly denied. Therefore, as long as they are not explicitly denied, the disclosed features are only an example of a series of generalized identical or equal features.

[0116] Furthermore, the embodiments of the present invention are not limited to any specific structure in the above-mentioned embodiments. The embodiments of the present invention can be extended to all new features described in the present invention, or their combinations, or all new methods described, or processing steps, or their combinations.

[0117] In the present invention, records such as "first" and "second" are identifiers for distinguishing the structure. The structures distinguished by records such as "first" and "second" in the present invention can exchange the numbers in the structures. For example, the first image can exchange "first" and "second" as identifiers with the second image. The exchange of identifiers is performed simultaneously. The structure is also distinguished after the exchange of identifiers. The identifier can be deleted. The structure with the identifier deleted is distinguished by the figure mark. The records based only on the identifiers such as "first" and "second" in the present invention cannot be used to interpret the order of the structure or the basis for the existence of identifiers with small numbers.

[0118] Description of Reference Numerals

[0119] 100 Robot control system

[0120] 1 robot (4: working range)

[0121] 2 Robot control device (22: robot control unit, 24: robot interface)

[0122] 3. Overlooking the camera

[0123] 6 External device (6A: first area, 6B: second area)

[0124] 7 Tray (76: Identification frame)

[0125] 10 Robot control device (12: control unit, 14: interface)

[0126] 20 Holding device

[0127] 30 Hand Camera

[0128] 80 Holding object

[0129] 81 Part 1 (81A: Surface, 81B: Back)

[0130] 82 Part 2 (82A: surface, 82B: back)

[0131] 83A Surface normal

[0132] 83B Backface normal

[0133] 84 Select object

[0134] 86 Identification box.

Claims

1. A robot control device, wherein: The robot control device includes a control unit for controlling the robot having the holding device. The control unit obtains a first image obtained by photographing at least one held object, the control unit selects, as a selected object, the at least one object to be held by the holding device from among the at least one object to be held recognized in the first image, The control unit determines a holding position of the selected object based on a second image obtained by photographing the selected object. The control unit causes the holding device to hold the selected object at the determined holding position.

2. The robot control device according to claim 1, wherein: The control unit detects a normal direction of a holding surface of the selected object, and determines the holding position based on a second image captured from the detected normal direction.

3. The robot control device according to claim 1 or 2, wherein: The control unit calculates the posture of the selection object shown in the second image, and determines a region in which the selection object is placed based on the posture.

4. The robot control device according to any one of claims 1 to 3, wherein: The control unit calculates a posture of the selection object shown in the second image, and determines a placement method of the selection object based on the posture.

5. The robot control device according to claim 4, wherein: The control unit arranges the selection object in a first mode when the selection object faces the front side, The control unit arranges the selection object in a second mode when the selection object faces the rear side.

6. The robot control device according to claim 5, wherein: The second pattern is set so that, when the selection object arranged in the second pattern is reversed, the orientation of the selection object matches the orientation of the selection object arranged in the first pattern.

7. The robot control device according to claim 3, wherein: The control unit controls the robot so that the holding device places the selected object in the first area when the selected object faces the front side. The control unit controls the robot so that the holding device places the selected object in the second area when the selected object faces the rear side.

8. The robot control device according to any one of claims 1 to 7, wherein: The control unit determines a correction amount of a rotation angle when the holding device places the selected object after holding the selected object, based on the posture of the selected object shown in the second image.

9. The robot control device according to any one of claims 1 to 8, wherein: The control unit acquires, as a bird's-eye view image, an image captured by a first imaging device arranged to capture at least a portion of a working range of the robot. The control unit calculates a first position and a first posture at which the first image was captured based on the bird's-eye view image. The control unit controls the robot so that a second imaging device installed on the robot captures images at the first position and the first posture. The control unit acquires, as the first image, an image captured by the second imaging device at the first position and the first posture.

10. The robot control device according to any one of claims 1 to 9, wherein: The control unit generates an identification frame surrounding the held object as a result of identification of the held object.

11. The robot control device according to any one of claims 1 to 10, wherein: The control unit selects the selection object based on the score indicating the recognition result of the held object.

12. The robot control device according to claim 11, wherein: The control unit evaluates the easiness of acquisition based on the score, and selects the selection object based on a result of the evaluation.

13. The robot control device according to claim 11 or 12, wherein: The control unit generates a mask for covering each of the held objects shown in the first image as a recognition result of the held objects, The control unit calculates the score based on the mask.

14. The robot control device according to claim 1, wherein: The control unit selects the at least one held object as a selected object based on the first image, and determines whether to capture the second image.

15. The robot control device according to claim 14, wherein: The control unit determines the holding position of the selected object based on the first image when it is determined that it is not necessary to capture the second image. The control unit causes the holding device to hold the selected object at the determined holding position.

16. A robot control method, which controls a robot having a holding device, wherein: The robot control method comprises the following processing: Acquire a first image obtained by photographing at least one held object; selecting, as a selected object, the at least one object to be held by the holding device from among the at least one object to be held recognized in the first image; determining a holding position of the selected object based on a second image obtained by photographing the selected object; and The holding device is caused to hold the selected object at the determined holding position.

Images