ROBOT CONTROL METHOD, ROBOT CONTROL SYSTEM, CONTROL DEVICE, AND CONTROL PROGRAM

Abstract

To provide a robot control method, a robot control system, a control device and a control program which enable even a mobile-type robot to efficiently perform operation.SOLUTION: A robot control method is executed in a system which is provided with a first photographing part arranged above a work-piece, a mobile-type robot which performs predetermined operation to the work-piece using a movable arm in which a second photographing part is arranged and a control device that controls the robot on the basis of images photographed by the first photographing part and the second photographing part respectively. The control device executes: a first step of storing correspondence information concerning a predetermined correspondence relation; a second step of storing identification information for identifying a position of the work-piece and a direction of the work-piece, on the basis of an image of the work-piece photographed by the first photographing part; and a third step of storing posture information concerning a posture of the arm at the time when the arm of the robot performs the predetermined operation to the work-piece, when the robot is at a reference position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a robot control method, a robot control system, a control device, and a control program. [Background technology]

[0002] Conventionally, there have been robots that grip a workpiece. The technology described in Patent Document 1 includes a wide-area sensor and a narrow-area sensor. The wide-area sensor captures a wide range including the entire storage box in which the workpieces are stored, and can detect the rough position of each workpiece included in the collection of workpieces. The narrow-area sensor captures a narrow range including a specific workpiece with high resolution and precisely measures the three-dimensional position and orientation of the workpiece. The technology described in Patent Document 1 selects a specific workpiece to be gripped by the robot based on the rough position of the workpiece detected by the wide-area sensor, precisely measures the selected specific workpiece using the narrow-area sensor, and controls the robot based on the measurement result to grip and transport the workpiece. The technology described in Patent Document 1 repeats the above process until there are no more works stored in the storage box. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2008-087074 A Summary of the Invention [Problem to be solved by the invention]

[0004] Generally, most robots installed in factories are stationary. In other words, even if a robot installed in a factory has a movable member such as an arm, it is the arm that moves, and the robot itself is often stationary and cannot move. When considering the manufacturing cost and manufacturing process of the product, it is sometimes desirable to make the robot itself mobile. However, when the robot itself is mobile, for example, in the case of a robot that grasps and transports a workpiece as described in the above-mentioned Patent Document 1, the positional relationship between the workpiece and the robot and the positional relationship between the sensor and the robot change every time the robot is moved, and it takes time and effort to calibrate the above-mentioned positional relationships before starting the robot's operation.

[0005] An object of the present invention is to provide a robot control method, a robot control system, a control device, and a control program that enable even a mobile robot to perform work efficiently. [Means for solving the problem]

[0006] A robot control method according to one aspect is a method executed in a robot control system including a movable robot having a first imaging unit disposed above a workpiece and a movable arm on which a second imaging unit is disposed, the robot performing a predetermined operation on the workpiece using the arm, and a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, the method including a first step in which the control device having a memory unit stores correspondence information relating to a predetermined correspondence relationship in the memory unit, and a second step in which the control device stores correspondence information relating to a predetermined correspondence relationship in the memory unit, and The method executes a second step of storing in a memory unit identification information that identifies the orientation of the workpiece, and a third step of storing in the memory unit posture information relating to the posture of the arm of the robot when the arm of the robot performs a predetermined operation on the workpiece when the robot is in a reference position. In the first step, information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to the reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot is stored in the memory unit as correspondence information.

[0007] A robot control system according to one embodiment is a system including a movable robot having a first imaging unit arranged above a workpiece and a movable arm on which a second imaging unit is arranged, the robot performing a predetermined operation on the workpiece using the arm, and a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, the control device storing correspondence information relating to the predetermined correspondence relationship, identification information for identifying the position and orientation of the workpiece, and posture information relating to the posture of the arm when the robot is in a reference position and the robot is in a reference position, and a control device that controls the robot to move to a position where the robot will perform the predetermined operation on the workpiece when the robot actually performs the predetermined operation on the workpiece. the workpiece imaged by the first imaging unit and the correspondence information, identification information, and posture information stored in the memory unit; and a third control unit that controls the robot's arm to perform a predetermined operation on the workpiece based on the image of the workpiece imaged by the second imaging unit. The memory unit stores, as the correspondence information, information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot.

[0008] A control device according to one embodiment is a control device for a robot in a system including a first imaging unit arranged above a workpiece, a movable robot equipped with a movable arm on which a second imaging unit is arranged, and performing a predetermined operation on the workpiece using the arm, and includes a memory unit that stores correspondence information relating to a predetermined correspondence relationship, identification information for identifying the position of the workpiece and the orientation of the workpiece, and posture information relating to the posture of the arm when the robot is in a reference position and performing a predetermined operation on the workpiece using the robot's arm, a first control unit that controls the robot to move to a position where the predetermined operation on the workpiece is performed when the robot actually performs the predetermined operation on the workpiece, and a work image captured by the first imaging unit. and a second control unit that controls the arm of the robot to assume a posture for capturing an image of the workpiece captured by the first imaging unit with the second imaging unit based on an image of the workpiece and the correspondence information, identification information and posture information stored in the memory unit, and a third control unit that controls the arm of the robot to perform a predetermined operation on the workpiece based on the image of the workpiece captured by the second imaging unit, wherein the memory unit stores, as the correspondence information, information including a first correspondence relationship that corresponds the position of the first imaging unit with the position of the workpiece captured by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit with a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit with the position of the robot.

[0009] A control program of one embodiment is a control program for a robot of a system including a movable robot having a first imaging unit arranged above a workpiece and a movable arm on which a second imaging unit is arranged, the robot performing a predetermined operation on the workpiece using the arm, and a control device for controlling the robot, the control device for controlling the robot having a memory function for storing correspondence information regarding the predetermined correspondence relationship, identification information for identifying the position and orientation of the workpiece, and posture information regarding the posture of the arm when the robot is in a reference position and performing the predetermined operation on the workpiece using the robot's arm, and a first control function for controlling the robot to move to a position where the robot will perform the predetermined operation on the workpiece when the robot actually performs the predetermined operation on the workpiece, The system realizes a second control function that controls the arm of the robot to take a posture for imaging the workpiece imaged by the first imaging unit by the second imaging unit based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information and posture information stored in the memory function, and a third control function that controls the arm of the robot to perform a predetermined operation on the workpiece based on the image of the workpiece imaged by the second imaging unit, and the memory function stores information as correspondence information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot. Effect of the Invention

[0010] One embodiment of the robot control method is a method executed in a robot control system including a mobile robot having a first imaging unit arranged above a workpiece, a movable arm on which a second imaging unit is arranged, and which performs a predetermined operation on the workpiece using the arm, and a control device that controls the robot based on images captured by the first imaging unit and images captured by the second imaging unit, in which correspondence information regarding the predetermined correspondence relationship, identification information that identifies the position and orientation of the workpiece based on the image of the workpiece captured by the first imaging unit, and posture information regarding the posture of the arm when the robot is in a reference position and the robot's arm performs a predetermined operation on the workpiece are stored in a memory unit, so that even a mobile robot can perform work efficiently based on the above-mentioned multiple pieces of information. The robot control system, the control device, and the control program according to one aspect can achieve the same effects as the robot control method according to the aspect described above. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram for explaining a robot control system according to an embodiment. [Diagram 2] FIG. 1 is a block diagram for explaining a robot control system according to an embodiment. [Diagram 3] FIG. 1 is a diagram for explaining an example of a workpiece. [Figure 4] FIG. 13 is a diagram for explaining an example of a gripping position of a workpiece. [Diagram 5] 1 is a first flowchart illustrating a robot control method according to an embodiment. [Figure 6] 11 is a second flowchart illustrating a robot control method according to an embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, one embodiment of the present invention will be described.

[0013] [Outline of the robot control system 1] FIG. 1 is a diagram illustrating a robot control system 1 according to an embodiment.

[0014] The robot control system 1 is used to control the robot 200 in various facilities where the robot 200 is used, such as factories, warehouses and stores for storing goods, and logistics warehouses used for transporting goods, etc. In the example shown in Fig. 1, the robot control system 1 controls the robot 200 that moves a workpiece 400 such as an item (or a part that constitutes an item) in a manufacturing facility for goods, such as a factory.

[0015] The robot control system 1 includes a first imaging unit 100, a robot 200, and a control device 300.

[0016] The first imaging section 100 is, for example, a camera device that captures an image of the workpiece 400 and generates image information (first image information).

[0017] The robot 200 is movable, for example, in a planar direction, and can move relatively closer to or relatively farther away from the workpiece 400. The robot 200 includes, for example, an arm 220, and a gripping unit 230 at the tip of the arm 220 for gripping the workpiece 400. The robot 200 also includes a second imaging unit 210 at the tip of the arm 220 (at a position relatively close to the tip (grip 230)). The second imaging unit 210 is, for example, a camera device that captures an image of the workpiece 400 and generates image information (second image information).

[0018] The control device 300 controls the robot 200 based on, for example, image information (first image information) generated by the first imaging section 100 and image information (second image information) generated by the second imaging section 210. The control device 300 controls, for example, the robot 200 located relatively close to the workpiece 400 so as to grip the workpiece 400 based on the first image information and the second image information. Note that the control device 300 may be mounted on the robot 200, or may be disposed independently of the robot 200 without being mounted on the robot 200.

[0019] [Detailed configuration of robot control system 1] Next, the robot control system 1 will be described in detail. FIG. 2 is a block diagram for explaining a robot control system 1 according to an embodiment.

[0020] [First imaging unit 100] FIG. 3 is a diagram for explaining an example of the workpiece 400. As shown in FIG.

[0021] The first imaging unit 100 is fixedly disposed relative to the location where the workpiece 400 is disposed. As an example, the first imaging unit 100 is disposed above the workpiece 400. For example, when one or more workpieces 400 are contained in a container 401 such as a pallet (see FIG. 3), the first imaging unit 100 may be disposed above the location where the container 401 is disposed (for example, a table, etc.). The term "above" may include concepts such as "approximately above" and "diagonally above". That is, the first imaging unit 100 is disposed at a position where the workpiece 400 can be imaged, as a specific example, at a position where the workpiece 400 contained in the container 401 can be imaged.

[0022] The first imaging section 100 may have an angle of view capable of capturing an image of the entire container 401 in the planar direction, for example. The first imaging unit 100 captures still images or video of the workpiece 400. When capturing still images, the first imaging unit 100 may capture the still images continuously in time (for example, at a predetermined timing). For example, the first imaging unit 100 generates first image information when performing an imaging operation, and transmits the first image information to the control device 300.

[0023] [Robot 200] The robot 200 may be movable by being provided with a running member such as the wheels 201. The robot 200 may be self-propelled by being provided with a driving unit (not shown) that rotates the wheels 201. The robot 200 may be autonomous (automatic driving). The robot 200 may be automatically driven by using various known methods.

[0024] The robot 200 includes a movable arm 220 on which the second imaging unit 210 is disposed. As a specific example, the robot 200 may include a multi-joint arm 220. The robot 200 includes an operation unit at the tip of the arm 220 for performing a predetermined operation on the workpiece 400. The predetermined operation may be various operations including, for example, gripping the workpiece 400. That is, the robot 200 may include, for example, a gripping unit 230 (operation unit) at the tip of the arm 220. As an example, the gripping unit 230 may be a type that includes two fingers and grips the workpiece 400 with those fingers.

[0025] At the tip (near the tip) of the arm 220, i.e., near the gripping unit 230, for example, the second imaging unit 210 is disposed. The second imaging unit 210 can image the workpiece 400 that is the target of a predetermined operation performed by the operating unit, i.e., the workpiece 400 that is the target of gripping by the gripping unit 230. The second imaging unit 210 images the workpiece 400 as a still image or a video. When capturing a still image, the second imaging unit 210 may capture the still images continuously in time (for example, at a predetermined timing). For example, the second imaging unit 210 may generate second image information when performing an imaging operation, and transmit the second image information to the control device 300.

[0026] [Control device 300] The control device 300 controls the robot 200 based on an image (first image information) captured by the first imaging unit 100 and an image (second image information) captured by the second imaging unit 210. As a specific example, the control device 300 may determine a workpiece to be subjected to a predetermined operation by the robot 200, and instruct the robot 200 on the position and orientation of the arm 220 (second imaging unit 210) so that the second imaging unit 210 can capture an image of the workpiece when the robot 200 moves to a position to perform the predetermined operation on the workpiece. The control device 300 includes a communication unit 320, a storage unit 330, a first control unit 311, a second control unit 312, and a third control unit 313 in order to control the robot 200. The first control unit 311, the second control unit 312, and the third control unit 313 may be realized as one function of the control unit 310 (for example, an arithmetic processing device, etc.) of the control device 300.

[0027] The communication unit 320 communicates with each of the first imaging unit 100, the second imaging unit 210, and the robot 200. That is, the communication unit 320 may receive, for example, first image information transmitted from the first imaging unit 100 and second image information transmitted from the second imaging unit 210. The communication unit 320 may transmit, for example, control information for controlling the robot 200 to the robot 200 based on the control of the control unit 310.

[0028] The storage unit 330 stores, for example, various information and programs. The storage unit 330 stores, for example, correspondence information, identification information, and attitude information.

[0029] The correspondence information is information about a predetermined correspondence relationship, that is, the correspondence information includes, for example, information about a first correspondence relationship, a second correspondence relationship, and a third correspondence relationship.

[0030] The first correspondence relationship is a correspondence relationship that associates the position of the first imaging unit 100 with the position of the workpiece 400 imaged by the first imaging unit 100. The first correspondence relationship may be, for example, a positional relationship in a planar direction between the first imaging unit 100 and the position of the workpiece 400, that is, a relationship between the position of the first imaging unit 100 and the position of the workpiece 400 when viewed in a planar view.

[0031] The position of the workpiece 400 is not limited to a specific point, and may be an area having a spread in a planar direction. The position of the workpiece 400 may be a predetermined position where the workpiece 400 is placed. That is, the position of the workpiece 400 may be a position where a container 401 for containing the workpiece 400 is placed (for example, a table, etc.). In this case, the first correspondence relationship may be generated based on a predetermined positional relationship between the first imaging unit 100 and the workpiece 400, and then may be stored in the storage unit 330. That is, when the control unit 310 or the like generates the first correspondence relationship, the control unit 310 or the like may include the first correspondence relationship in the correspondence information, or may store the information of the first correspondence relationship as the correspondence information in the storage unit 330.

[0032] Alternatively, the first correspondence relationship may be set each time the workpiece 400 is placed in the imaging range of the first imaging unit 100 (for example, below the first imaging unit 100). That is, the first imaging unit 100 may image the workpiece 400 when the workpiece 400 is placed, for example. In this case, the control unit 310 or the like may generate the first correspondence relationship that associates the position of the first imaging unit 100 with the position of the workpiece 400 based on the image (image information) of the workpiece 400 imaged by the first imaging unit 100. As an example, since the first imaging unit 100 is fixed, the control unit 310 can obtain the positional relationship (for example, the positional relationship in the planar direction) between the first imaging unit 100 and the workpiece 400 and generate the first correspondence relationship even when the workpiece 400 is imaged by the first imaging unit 100 when the workpiece 400 is placed. by setting in advance the positional relationship with the first imaging unit 100 in the image imaged by the first imaging unit 100. When the control unit 310 generates the first correspondence relationship, the control unit 310 can store the first correspondence relationship in the correspondence information or store the information on the first correspondence relationship as the correspondence information in the storage unit 330. The first correspondence relationship is not limited to being set as in the above example, and may be set by various methods.

[0033] The second correspondence is a correspondence (positional relationship between the first imaging unit 100 and the robot 200) that associates the position of the first imaging unit 100 with the reference position of the robot 200. The reference position of the robot 200 may be, for example, a position where the mobile robot 200 can move relatively close to the workpiece 400 (first imaging unit 100) and perform a predetermined operation (for example, gripping, etc.) on the workpiece 400. That is, the reference position of the robot 200 may be, for example, a position that is a destination (arrival point) of the robot 200 performing a predetermined operation, or may be a position that is set in advance. The reference position is not limited to, for example, a specific point, and may be an area that spreads in a planar direction. In this case, the second correspondence may be, for example, a positional relationship in a planar direction between the position of the first imaging unit 100 and the reference position, that is, a positional relationship between the first imaging unit 100 and the reference position when viewed in a planar view, and may be generated based on the positional relationship being set in advance, and then stored in the storage unit 330. That is, for example, when the control unit 310 etc. generates the second correspondence relationship, the control unit 310 etc. may store the second correspondence relationship in the correspondence information or store the information on the second correspondence relationship as the correspondence information in the storage unit 330. The second correspondence relationship is not limited to being set as in the above example, and may be set by various methods.

[0034] The third correspondence relationship is a correspondence relationship in which the position of the second imaging unit 210 and the position of the robot 200 are associated with each other. That is, the third correspondence relationship may be, for example, a positional relationship in a planar direction between the position of the second imaging unit 210 and the position of the robot 200, that is, a positional relationship between the position of the second imaging unit 210 and the position of the robot 200 when viewed in a planar direction. The second imaging unit 210 is installed on the arm 220 of the robot 200. Therefore, even if the arm 220 operates in various patterns, the relationship between the position of the second imaging unit 210 and the position of the robot 200 can be acquired in advance. The third correspondence relationship may be, for example, a relationship in which the position of the second imaging unit 210 and the position of the robot 200 are associated in advance according to various operation patterns of the arm 220, and may be generated based on the relationship being set in advance, and then stored in the storage unit 330. That is, for example, when the control unit 310 etc. generates the third correspondence relationship, the control unit 310 etc. may store the third correspondence relationship in the correspondence information or store the information on the third correspondence relationship as the correspondence information in the storage unit 330. The third correspondence relationship is not limited to being set as in the above example, and may be set by various methods.

[0035] The identification information is information for identifying the position of the workpiece 400 and the orientation of the workpiece 400. The workpiece 400 has various shapes. As a specific example, when there is a ring-shaped workpiece 400, the workpiece 400 may appear in various shapes depending on the orientation. In addition, when multiple workpieces 400 overlap, the workpieces 400 may appear in various shapes depending on how the workpieces 400 overlap. The shapes (visual shapes) of these workpieces 400 can be specified in advance. That is, the control unit 310, etc. may generate the identification information and store it in the storage unit 330, for example. The identification information may be set by various methods other than the above example.

[0036] The identification information may be information for specifying the orientation of the workpiece 400 according to the shape of the workpiece 400 recorded in the first image information obtained by imaging the workpiece 400 when the first imaging unit 100 images the workpiece 400. In addition, the identification information may be information for specifying the positional relationship of the workpieces 400 (for example, the workpiece 400 located at the top when there are multiple workpieces 400, and the workpiece 400 located below, etc.) according to the shapes of the multiple workpieces 400 recorded in the first image information obtained by imaging the multiple workpieces 400 when the first imaging unit 100 images the multiple workpieces 400.

[0037] The posture information is information on the posture of the arm 220 of the robot 200 when the arm 220 of the robot 200 performs a predetermined operation on the workpiece 400 when the robot 200 is in the reference position. The posture information is, for example, information on the posture of the arm 220 when the arm 220 operates in various patterns. As an example, the posture information is information on various postures of the arm 220, including the posture of the arm 220 when the multi-joint arm 220 is extended and the posture of the arm 220 when the multi-joint arm 220 is bent. That is, the posture information may be information on various postures of the arm 220 when performing a predetermined operation on the workpiece 400 when the robot 200 is in the reference position, such as the posture of the arm 220 when gripping the workpiece 400, the posture when the second imaging unit 210 can image the target workpiece 400, and the posture of the arm 220 when transporting the workpiece 400. These postures of the arm 220 can be specified in advance. For this reason, the posture information may be generated based on various postures of the arm 220 being set in advance, and then may be stored in the storage unit 330. That is, the control unit 310, etc. may, for example, store the posture information in the storage unit 330 after generating it. The posture information is not limited to being set as in the above example, and may be set by various methods.

[0038] The above-mentioned correspondence information, identification information, and attitude information are not limited to the example in which they are generated by the control unit 310 or the like as described above, and may be generated outside (by an external device) of the control device 300. In this case, the control device 300 may acquire the correspondence information, identification information, and attitude information via the communication unit 320, for example, and then store them in the storage unit 330.

[0039] When the control unit 310 stores the correspondence information, the identification information, and the posture information in the memory unit 330 as described above, the control unit 310 controls the robot 200 as follows so that the robot 200 performs a predetermined operation on the workpiece 400. In this case, the control unit 310 selects one of the multiple overlapping workpieces 400 as follows: The workpiece 400 on which a predetermined operation is easy to perform may be identified from the image captured by the first imaging section 100, and identified as the workpiece 400 on which the predetermined operation is to be performed.

[0040] When the robot 200 actually performs a predetermined operation on the workpiece 400, the first control unit 311 controls the robot 200 to move to a position where the robot 200 performs the predetermined operation on the workpiece 400. That is, the first control unit 311 controls the robot 200 to move to a position relatively close to the workpiece 400 in order to make the robot 200 perform a predetermined operation on the workpiece 400. The position where the predetermined operation is performed (the position relatively close to the workpiece 400) may be, for example, the reference position described above. The first control unit 311 moves the robot 200 by rotating a running member such as the wheel 201, for example, based on controlling the drive unit of the robot 200. In this case, the robot 200 may move to the reference position autonomously (automatically) when it receives an instruction to move to the reference position based on a control signal transmitted from the first control unit 311, for example.

[0041] The control signal may be, for example, a signal transmitted to the robot 200 via the communication unit 320 in order for the control unit 310 to control the robot 200, and may include various control contents for controlling the robot 200.

[0042] The second control unit 312 controls the arm 220 of the robot 200 to assume a posture for imaging the workpiece 400 imaged by the first imaging unit 100 using the second imaging unit 210, based on the image of the workpiece 400 imaged by the first imaging unit 100 and the correspondence information, identification information and posture information stored in the memory unit 330. For example, when the second control unit 312 acquires an image (first image information) captured by the first imaging unit 100 at a predetermined timing, the second control unit 312 performs image recognition based on the first image information and recognizes the workpiece 400 recorded in the first image information. The second control unit 312 can use various known methods for image recognition, for example. The second control unit 312 transmits a control signal to the robot 200 so as to assume a posture of the arm 220 for performing a predetermined operation on the workpiece 400, for example, based on the first image information (result of image recognition), the correspondence information, the identification information, and the posture information. That is, the second control unit 312 can specify the position of the workpiece 400, for example, based on the first image information and the correspondence information, and transmits a control signal to the robot 200 so as to assume a posture of the arm 220 for performing a predetermined operation on the workpiece 400, based on the specified position, the identification information, and the posture information of the workpiece 400. In this case, the second control unit 312 may control the arm 220 to assume the posture described above, so as to assume a posture of the arm 220 such that the second imaging unit 210 can capture an image of the workpiece 400.

[0043] Further, the second control unit 312 may transmit to the robot 200 a control signal including a content specifying an operation position when performing a predetermined operation on the workpiece 400, for example, when the predetermined operation is gripping the workpiece 400, a location where the workpiece 400 is gripped. In addition, when the relationship between the gripping direction by the gripping unit 230 (the direction of the dashed line illustrated in FIG. 4) and a position in an image captured by the second imaging unit 210 (the posture of the arm 220 and the orientation in which the workpiece 400 is imaged by the second imaging unit 210, etc.) is recorded in advance in posture information, etc., the second control unit 312 may transmit to the robot 200 a control signal including a content specifying a location where the workpiece 400 is gripped, based on the posture information, etc. In other words, the posture information may include, for example, information on the gripping location of the workpiece 400 described above and a position in an image captured by the second imaging unit 210 (the posture of the arm 220 and the orientation in which the workpiece 400 is imaged by the second imaging unit 210, etc.).

[0044] The third control unit 313 controls the arm 220 of the robot 200 to perform a predetermined operation on the workpiece 400 based on the image of the workpiece 400 captured by the second imaging unit 210. As described above, when the second control unit 312 controls the robot 200, the second imaging unit 210 is able to image a specific workpiece 400 (for example, the workpiece 400 to be grasped). That is, when the third control unit 313 acquires an image (second image information) imaged by the second imaging unit 210, the third control unit 313 performs image recognition based on the second image information and recognizes the workpiece 400 recorded in the second image information. The third control unit 313 can use various known methods for image recognition, for example. However, since the robot 200 is movable, there is a possibility that the robot 200 is not positioned exactly at the reference position. In this case, the actual positional relationship between the first imaging unit 100 and the second imaging unit 210 (the positional relationship between the workpiece 400 and the second imaging unit 210) will be shifted compared to when the robot 200 is exactly at the reference position. For this reason, the third control unit 313 transmits to the robot 200 a control signal (for example, a signal including the three-dimensional coordinates of the second imaging unit 210 and the rotation (adjustment) directions of the three axes of the arm 220) for finely adjusting the posture of the arm 220 so that a predetermined operation can be performed on the workpiece 400 based on the image (second image information) of the workpiece 400 captured by the second imaging unit 210. That is, the third control unit 313 controls the posture of the arm 220 so that a predetermined operation can be performed at a predetermined location of the workpiece 400 based on, for example, the second image information and the above-mentioned posture information. In other words, the third control unit 313 controls the posture of the arm 220 so that the content of the image of the workpiece 400 (the position of the workpiece 400 in the image) based on the above-mentioned posture information becomes the same.

[0045] FIG. 4 is a diagram for explaining an example of a gripping position of the workpiece 400. In FIG.

[0046] The third control unit 313 may control the arm 220 to grip the workpiece 400 with the gripper 230 disposed at the tip of the arm 220 as a control of the arm 220 for performing a predetermined operation on the workpiece 400. For example, the third control unit 313 controls the posture of the arm 220 as described above, and when the position and posture of the workpiece 400 imaged by the second imaging unit 210 correspond to the image content of the workpiece 400 based on the posture information described above, transmits a control signal to the robot 200 to perform a predetermined operation (e.g., gripping, etc.) on the workpiece 400. For example, the third control unit 313 controls the arm 220 as described above to an angle at which the gripper 230 can grip the workpiece 400, that is, the arm 220 is adjusted so that the workpiece 400 can be gripped from either direction shown by the two dashed lines illustrated in FIG. 4, and by transmitting a control signal to grip the workpiece 400 to the robot 200, the gripper 230 can grip the workpiece 400. This enables the robot 200 to perform a predetermined operation (e.g., gripping) on ​​the workpiece 400. In addition, for example, when the robot 200 grips the workpiece 400, it becomes possible to transport the workpiece 400 to another location.

[0047] [Robot control method] Next, a robot control method according to an embodiment will be described.

[0048] First, a robot control method according to an embodiment, which is a preparation method for controlling the robot 200, will be described. FIG. 5 is a first flowchart illustrating a robot control method according to an embodiment.

[0049] In step ST101, the storage unit 330 stores the correspondence information. The correspondence information includes, for example, information on a first correspondence relationship, a second correspondence relationship, and a third correspondence relationship. The first correspondence relationship is a correspondence relationship that associates the position of the first imaging unit 100 with the position of the workpiece 400 imaged by the first imaging unit 100. The second correspondence relationship is a correspondence relationship that associates the position of the first imaging unit 100 with the reference position of the robot 200. The third correspondence relationship is a correspondence relationship that associates the position of the second imaging unit 210 with the position of the robot 200.

[0050] In step ST102, the storage unit 330 stores the identification information. The identification information is information for identifying the position and orientation of the workpiece 400.

[0051] In step ST103, the storage unit 330 stores the posture information. The posture information is information about the posture of the arm 220 of the robot 200 when the arm 220 performs a predetermined operation on the workpiece 400 when the robot 200 is in the reference position. The posture information may also include information about the relationship between the direction in which the gripper 230 grips the workpiece 400 and the position in the image captured by the second imaging unit 210 (the posture of the arm 220 and the direction in which the second imaging unit 210 captures the workpiece 400, etc.).

[0052] Next, a robot control method according to an embodiment will be described, which is a method for controlling a robot to perform a predetermined operation. FIG. 6 is a second flowchart illustrating a robot control method according to an embodiment.

[0053] In step ST201, the control unit 310 controls the robot 200 to enter the site. That is, when the robot 200 actually performs a predetermined operation on the workpiece 400, the first control unit 311 controls the robot 200 to move to a position (e.g., a reference position) where the predetermined operation on the workpiece 400 is performed.

[0054] In step ST202, the second control unit 312 controls the arm 220 of the robot 200 to assume a posture for imaging the workpiece 400 imaged by the first imaging unit 100 using the second imaging unit 210, based on the image of the workpiece 400 imaged by the first imaging unit 100 and the correspondence information, identification information, and posture information stored in steps ST101 to ST103 shown in FIG. 5.

[0055] In step ST203, the third control unit 313 controls the arm 220 of the robot 200 to perform a predetermined operation on the workpiece 400 based on the image of the workpiece 400 captured by the second imaging unit 210. That is, the third control unit 313 controls the arm 220 to finely adjust the posture of the arm 220 so that the predetermined operation can be performed on the workpiece 400 based on, for example, the image of the workpiece 400 captured by the second imaging unit 210 (second image information).

[0056] In step ST204, the third control unit 313 controls the arm 220 to perform a predetermined operation on the workpiece 400, by controlling the arm 220 (robot 200) to grip the workpiece 400 with the gripper 230 arranged at the tip of the arm 220.

[0057] Each of the above-mentioned units of the control device 300 may be realized as a function of an arithmetic processing device of a computer, etc. That is, the first control unit 311, the second control unit 312, and the third control unit 313 (control unit 310) of the control device 300 may be realized as a first control function, a second control function, and a third control function (control function), respectively, by an arithmetic processing device of a computer, etc. The control program can cause a computer to realize each of the above-mentioned functions. The control program may be recorded in a non-transitory computer-readable recording medium, such as an external memory or an optical disk. As described above, each unit of the control device 300 may be realized by an arithmetic processing device of a computer or the like. The arithmetic processing device or the like is configured by, for example, an integrated circuit or the like. Therefore, each unit of the control device 300 may be realized as a circuit that constitutes the arithmetic processing device or the like. That is, the first control unit 311, the second control unit 312, and the third control unit 313 (control unit 310) of the control device 300 may be realized as a first control circuit, a second control circuit, and a third control circuit (control circuits) that constitute the arithmetic processing device of a computer or the like. The communication unit 320 and the storage unit 330 of the control device 300 may be realized as a communication function and a storage function including the functions of an arithmetic processing device, for example. The communication unit 320 and the storage unit 330 of the control device 300 may be realized as a communication circuit and a storage circuit by being configured with an integrated circuit, for example. The communication unit 320 and the storage unit 330 of the control device 300 may be configured as a communication device and a storage device by being configured with a plurality of devices, for example.

[0058] In this specification, the term "information" is used, but the term "information" can be replaced with "data" and the term "data" can be replaced with "information".

[0059] [Aspects and Effects of the Present Embodiment] The aspects of this embodiment and the effects of each aspect will be described below.

[0060] (Aspect 1) A robot control method according to one aspect is a method executed in a robot control system including a movable robot having a first imaging unit disposed above a workpiece and a movable arm on which a second imaging unit is disposed, the robot performing a predetermined operation on the workpiece using the arm, and a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, the method including a first step in which the control device having a memory unit stores correspondence information relating to a predetermined correspondence relationship in the memory unit, and a second step in which the control device stores correspondence information relating to a predetermined correspondence relationship in the memory unit, and The method executes a second step of storing in a memory unit identification information that identifies the orientation of the workpiece, and a third step of storing in the memory unit posture information relating to the posture of the arm of the robot when the arm of the robot performs a predetermined operation on the workpiece when the robot is in a reference position. In the first step, information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to the reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot is stored in the memory unit as correspondence information. As a result, the robot control method can appropriately control the robot performing a predetermined operation, even if the robot is mobile, and can make the robot perform the predetermined operation efficiently. Also, the robot control method can store various information required for the robot to perform the predetermined operation in a storage unit, and can control the robot using the information. Furthermore, since the robot control method can control the robot to perform a specified operation multiple times using information stored in the memory unit, there is no need to perform calibration (the process of sharing the position coordinates (coordinate system) of the imaging unit, workpiece, and robot) every time a specified operation is performed, and the time required to start the specified operation can be shortened. Furthermore, since the robot control method utilizes the fixed first imaging unit (fixed camera), it is possible to reliably capture an image of the workpiece, and thereby to control the robot so as to reliably perform a predetermined operation. In addition, since the robot control method utilizes a second imaging unit (on-hand camera) arranged on the arm, even if the robot is mobile, the workpiece can be placed within the field of view of the second imaging unit based on the first image information, etc., thereby making it possible to control the robot to reliably perform the specified operation.

[0061] (Aspect 2) In one embodiment of the robot control method, the control device may execute a fourth step of controlling the robot to move to a position where the robot will actually perform a predetermined operation on the workpiece when it actually performs the predetermined operation on the workpiece; a fifth step of controlling the arm of the robot to assume a posture for imaging the workpiece imaged by the first imaging unit by the second imaging unit based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information and posture information stored in the memory unit; and a sixth step of controlling the arm of the robot to perform the predetermined operation on the workpiece based on the image of the workpiece imaged by the second imaging unit. As a result, the robot control method can provide appropriate control to a robot performing a predetermined operation, even if the robot is mobile, and can cause the robot to perform the predetermined operation efficiently.

[0062] (Aspect 3) In one embodiment of the robot control method, the control device may control the arm to grasp the workpiece using a gripping portion located at the tip of the arm in the sixth step to perform a specified operation on the workpiece. In other words, the robot control method can control the robot so that it reliably grips a workpiece, even if the robot is mobile.

[0063] (Aspect 4) A robot control system according to one embodiment is a system including a movable robot having a first imaging unit arranged above a workpiece and a movable arm on which a second imaging unit is arranged, the robot performing a predetermined operation on the workpiece using the arm, and a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, the control device storing correspondence information relating to the predetermined correspondence relationship, identification information for identifying the position and orientation of the workpiece, and posture information relating to the posture of the arm when the robot is in a reference position and the robot is in a reference position, and a control device that controls the robot to move to a position where the robot will perform the predetermined operation on the workpiece when the robot actually performs the predetermined operation on the workpiece. the workpiece imaged by the first imaging unit and the correspondence information, identification information, and posture information stored in the memory unit; and a third control unit that controls the robot's arm to perform a predetermined operation on the workpiece based on the image of the workpiece imaged by the second imaging unit. The memory unit stores, as the correspondence information, information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot. As a result, the robot control system can achieve the same effects as the robot control method of the above-described aspect.

[0064] (Aspect 5) A control device according to one embodiment is a control device for a robot in a system including a first imaging unit arranged above a workpiece, a movable robot equipped with a movable arm on which a second imaging unit is arranged, and performing a predetermined operation on the workpiece using the arm, and includes a memory unit that stores correspondence information relating to a predetermined correspondence relationship, identification information for identifying the position of the workpiece and the orientation of the workpiece, and posture information relating to the posture of the arm when the robot is in a reference position and performing a predetermined operation on the workpiece using the robot's arm, a first control unit that controls the robot to move to a position where the predetermined operation on the workpiece is performed when the robot actually performs the predetermined operation on the workpiece, and a work image captured by the first imaging unit. and a second control unit that controls the arm of the robot to assume a posture for capturing an image of the workpiece captured by the first imaging unit with the second imaging unit based on an image of the workpiece and the correspondence information, identification information and posture information stored in the memory unit, and a third control unit that controls the arm of the robot to perform a predetermined operation on the workpiece based on the image of the workpiece captured by the second imaging unit, wherein the memory unit stores, as the correspondence information, information including a first correspondence relationship that corresponds the position of the first imaging unit with the position of the workpiece captured by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit with a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit with the position of the robot. As a result, the control device can achieve the same effects as the robot control method according to the above aspect.

[0065] (Aspect 6) A control program of one embodiment is a control program for a robot of a system including a movable robot having a first imaging unit arranged above a workpiece and a movable arm on which a second imaging unit is arranged, the robot performing a predetermined operation on the workpiece using the arm, and a control device for controlling the robot, the control device for controlling the robot having a memory function for storing correspondence information regarding the predetermined correspondence relationship, identification information for identifying the position and orientation of the workpiece, and posture information regarding the posture of the arm when the robot is in a reference position and performing the predetermined operation on the workpiece using the robot's arm, and a first control function for controlling the robot to move to a position where the robot will perform the predetermined operation on the workpiece when the robot actually performs the predetermined operation on the workpiece, The system realizes a second control function that controls the arm of the robot to take a posture for imaging the workpiece imaged by the first imaging unit by the second imaging unit based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information and posture information stored in the memory function, and a third control function that controls the arm of the robot to perform a predetermined operation on the workpiece based on the image of the workpiece imaged by the second imaging unit, and the memory function stores information as correspondence information including a first correspondence relationship that corresponds the position of the first imaging unit to the position of the workpiece imaged by the first imaging unit, a second correspondence relationship that corresponds the position of the first imaging unit to a reference position of the robot, and a third correspondence relationship that corresponds the position of the second imaging unit to the position of the robot. As a result, the robot control program can achieve the same effects as the robot control method of the above-described aspect. [Explanation of symbols]

[0066] 1. Robot Control System 100 First imaging unit 200 Robots 210 Second imaging unit 220 Arm 230 Gripping part 300 Control device 310 Control section 311 First Imaging Unit 312 Second Imaging Unit 313 Third Imaging Unit 320 Communications Department 330 Storage section 400 Work

Claims

1. A first imaging unit disposed above the workpiece; A movable robot including a movable arm on which a second imaging unit is disposed, the movable robot performing a predetermined operation on a workpiece by the movable arm; A robot control method executed in a robot control system including a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, The control device including a storage unit, A first step of storing correspondence information relating to a predetermined correspondence relationship in the storage unit; A second step of storing, in the storage unit, identification information for identifying a position and an orientation of a workpiece based on an image of the workpiece captured by the first imaging unit; a third step of storing, in the storage unit, posture information regarding a posture of the arm when a predetermined operation is performed on a workpiece by the arm of the robot when the robot is in a reference position; In the first step, the correspondence information is A first correspondence relationship that associates a position of the first imaging unit with a position of a workpiece imaged by the first imaging unit; a second correspondence relationship that associates a position of the first imaging unit with a reference position of the robot; storing information including a third correspondence relationship that associates a position of the second imaging unit with a position of the robot in the storage unit; When the robot actually performs a predetermined operation on a workpiece, the control device executes a step of identifying a target workpiece to be subjected to the predetermined operation from among a plurality of overlapping workpieces by identifying an orientation of the workpiece according to an overlapping state of the workpieces based on the identification information. A method for controlling a robot.

2. The control device, A fourth step of controlling the robot to move to a position where the robot actually performs a predetermined operation on a workpiece when the robot actually performs a predetermined operation on the workpiece; a fifth step of controlling the arm of the robot so that the workpiece imaged by the first imaging unit is oriented so as to be imaged by the second imaging unit, based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information, and orientation information stored in the storage unit; A sixth step of controlling the arm of the robot so as to perform a predetermined operation on the workpiece based on an image of the workpiece captured by the second imaging unit; The robot control method according to claim 1, further comprising the steps of:

3. The control device controls the arm to grip the workpiece with a gripper disposed at a tip end of the arm as a control of the arm for performing a predetermined operation on the workpiece in the sixth step. The robot control method according to claim 2 .

4. A first imaging unit disposed above the workpiece; A movable robot including a movable arm on which a second imaging unit is disposed, the movable robot performing a predetermined operation on a workpiece by the movable arm; A robot control system including: a control device that controls the robot based on an image captured by the first imaging unit and an image captured by the second imaging unit, The control device includes: a storage unit that stores correspondence information regarding a predetermined correspondence relationship, identification information for identifying a position and an orientation of a workpiece, and posture information regarding a posture of the arm of the robot when a predetermined operation is performed on a workpiece by the arm of the robot when the robot is in a reference position; A first control unit controls the robot to move to a position where the robot actually performs a predetermined operation on a workpiece when the robot actually performs a predetermined operation on a workpiece; a second control unit that controls the arm of the robot so that the workpiece imaged by the first imaging unit is oriented so as to be imaged by the second imaging unit, based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information, and orientation information stored in the storage unit; A third control unit controls the arm of the robot to perform a predetermined operation on the workpiece based on an image of the workpiece captured by the second imaging unit, The storage unit, as the correspondence information, A first correspondence relationship that associates a position of the first imaging unit with a position of a workpiece imaged by the first imaging unit; a second correspondence relationship that associates a position of the first imaging unit with a reference position of the robot; storing information including a third correspondence relationship that associates a position of the second imaging unit with a position of the robot; When the robot actually performs a predetermined operation on a workpiece, the control device specifies the orientation of the workpiece according to the overlapping state of the workpieces based on the identification information, thereby specifying the target workpiece on which the predetermined operation is to be performed from among a plurality of overlapping workpieces. Robot control system.

5. A first imaging unit disposed above the workpiece; A movable robot including a movable arm on which a second imaging unit is disposed, the movable robot performing a predetermined operation on a workpiece by the movable arm; A control device for the robot of a system comprising: a storage unit that stores correspondence information regarding a predetermined correspondence relationship, identification information for identifying a position and an orientation of a workpiece, and posture information regarding a posture of the arm of the robot when a predetermined operation is performed on a workpiece by the arm of the robot when the robot is in a reference position; A first control unit controls the robot to move to a position where the robot actually performs a predetermined operation on a workpiece when the robot actually performs a predetermined operation on a workpiece; a second control unit that controls the arm of the robot so that the workpiece imaged by the first imaging unit is oriented so as to be imaged by the second imaging unit, based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information, and orientation information stored in the storage unit; A third control unit controls the arm of the robot to perform a predetermined operation on the workpiece based on an image of the workpiece captured by the second imaging unit, The storage unit, as the correspondence information, A first correspondence relationship that associates a position of the first imaging unit with a position of a workpiece imaged by the first imaging unit; a second correspondence relationship that associates a position of the first imaging unit with a reference position of the robot; storing information including a third correspondence relationship that associates the position of the second imaging unit with the position of the robot; When the robot actually performs a predetermined operation on a workpiece, the control device specifies the orientation of the workpiece according to the overlapping state of the workpieces based on the identification information, thereby specifying the target workpiece on which the predetermined operation is to be performed from among a plurality of overlapping workpieces. Control device.

6. A first imaging unit disposed above the workpiece; A movable robot including a movable arm on which a second imaging unit is disposed, the movable robot performing a predetermined operation on a workpiece by the movable arm; A control device for controlling the robot; A control program for the robot of a system comprising: The control device for controlling the robot includes: a memory function for storing correspondence information relating to a predetermined correspondence relationship, identification information for identifying a position and an orientation of a workpiece, and posture information relating to a posture of the arm of the robot when a predetermined operation is performed on a workpiece by the arm of the robot when the robot is in a reference position; A first control function that controls the robot to move to a position where the robot actually performs a predetermined operation on a workpiece when the robot actually performs a predetermined operation on a workpiece; a second control function that controls the arm of the robot so that the workpiece imaged by the first imaging unit is in a posture for imaging the workpiece by the second imaging unit, based on an image of the workpiece imaged by the first imaging unit and the correspondence information, identification information, and posture information stored in the storage function; and a third control function of controlling the arm of the robot so as to perform a predetermined operation on the workpiece based on an image of the workpiece captured by the second imaging unit. The storage function includes, as the corresponding information, A first correspondence relationship that associates a position of the first imaging unit with a position of a workpiece imaged by the first imaging unit; a second correspondence relationship that associates a position of the first imaging unit with a reference position of the robot; storing information including a third correspondence relationship that associates the position of the second imaging unit with the position of the robot; When the robot actually performs a predetermined operation on a workpiece, the control device specifies the orientation of the workpiece according to the overlapping state of the workpieces based on the identification information, thereby specifying the target workpiece on which the predetermined operation is to be performed from among a plurality of overlapping workpieces. Control program.

Images