Reference point determination system

Abstract

Determining the reference point for a controlled object in a robot equipped with an arbitrary controlled object. [Solution] The reference point determination system according to the present disclosure is a reference point determination system for determining a reference point of a controlled object whose movement is controlled by a robot, and comprises an acquisition unit that acquires an image of the controlled object captured by a camera, and a determination unit that determines a reference point based on the image.

Description

Technical Field

[0001] The present disclosure relates to a reference point determination system.

Background Art

[0002] In recent years, technological development related to calibration methods for coordinate systems equipped in robots has been advanced. For example, Patent Document 1 discloses a calibration method for associating the position relationship between the tool center point (TCP) of an end effector in a robotic arm and a control point provided at the tip of a force detection unit of the robotic arm. The robotic system according to Patent Document 1 mainly includes an imaging unit that images the end effector from below it. In the calibration method of Patent Document 1, the TCP is initially arranged at the imaging position of the imaging unit, and the robotic arm is rotated in a predetermined step and the position of the control point is moved, so that the robotic arm takes a plurality of poses. In each pose, since the TCP is located at the imaging center in the imaging unit, the position relationship between the control point and the TCP can be grasped by grasping the position of the control point in these poses.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In the method described in Patent Document 1, since the TCP is placed at the imaging position of the imaging unit, it is necessary to pre-define the TCP. Reference points in the robot's controlled object, such as TCP, can be determined by matching them with data from a CAD (Computer-Aided Design) model related to the design of the controlled object, such as an end effector. In order to determine the reference points of the controlled object in the robot described in Patent Document 1, it is necessary to create a CAD model corresponding to the controlled object. Therefore, for example, if the cost of creating a CAD model is high, it may be difficult to determine the reference points of the controlled object for a robot equipped with an arbitrary controlled object.

[0005] This disclosure is made to solve these problems and aims to provide a reference point determination system that can determine the reference point of a controlled object in a robot equipped with any controlled object. [Means for solving the problem]

[0006] The reference point determination system according to this disclosure is a reference point determination system for determining a reference point of a controlled object whose movement is controlled by a robot, and comprises an acquisition unit that acquires an image of the controlled object captured by a camera, and a determination unit that determines the reference point based on the image. As a result, the reference point determination system can acquire one or more images of the controlled object and determine a reference point by calculating an arbitrary point from the image. Therefore, the reference point determination system can determine a reference point for any controlled object without creating a CAD model. [Effects of the Invention]

[0007] This disclosure provides a reference point determination system that can determine the reference point of a controlled object in a robot equipped with any controlled object. [Brief explanation of the drawing]

[0008] [Figure 1]Figure 1 is a block diagram showing the configuration of the reference point determination system 1 related to this disclosure. [Figure 2] Figure 2 is a schematic diagram showing an example of an image captured by the acquisition unit 11. [Figure 3] Figure 3 is a schematic diagram showing an example of an image captured by the acquisition unit 11. [Figure 4] Figure 4 is a schematic diagram showing an example of an image captured by the acquisition unit 11. [Figure 5] Figure 5 shows an example of information related to the depth of the controlled object acquired by the acquisition unit 11. [Figure 6] Figure 6 shows an example of the hardware configuration of the reference point determination system 2 related to this disclosure. [Modes for carrying out the invention]

[0009] (Embodiment 1) Embodiments of the present disclosure will be described below with reference to the drawings. Figure 1 is a block diagram showing the configuration of the reference point determination system 1 according to the present disclosure. The reference point determination system 1 determines the reference point of a controlled object whose movement is controlled by a robot. The robot is a mechanical system that performs some automated work on a predetermined object. In this embodiment, the robot may be a robotic arm, but the robot is not limited to this and may be an automated forklift or a drone.

[0010] The robot according to this embodiment includes a control object whose movement is controlled. That is, a part of the robot's components can move in any direction, and can be moved by controlling that component. If the robot is a robot arm composed of multiple joints and links and equipped with an end effector at its tip, the end effector and individual joints or links can move in any direction. Since the robot arm moves these components by controlling them, the end effector, joints, and links can be said to be the control objects of the robot. In the case of an automated forklift, the forks are the control object of the automated forklift, and in the case of a drone equipped with a robot arm, the entire robot arm and each component within the robot arm are the control objects of the drone.

[0011] By controlling the movement of an object, the robot enables the object to perform predetermined tasks on the object being handled by the robot. These predetermined tasks include, for example, grasping, placing, polishing, cutting, suctioning, welding, and screwing into the object. The robot may perform one or more of these actions by controlling the object. In other words, a single robot may control one or more objects.

[0012] A robot uses a reference point for the controlled object to control its movement. When the controlled object of a robot arm is an end effector, the reference point of the controlled object may be called the TCP (Control Point). The reference point of the controlled object is the reference point used to determine the position and orientation of the controlled object in the controlled object's coordinate system relative to the robot coordinate system. In other words, the reference point of the controlled object is the reference point used to determine the movement of the controlled object.

[0013] The reference point determination system 1 comprises an acquisition unit 11 and a determination unit 12. The acquisition unit 11 acquires an image of the robot's controlled object captured by a camera. The captured image is typically an image captured by a two-dimensional camera using visible light, but is not limited to this. That is, as long as the contour of the robot's controlled object can be recognized, the captured image according to this embodiment may be captured by an infrared camera or a three-dimensional camera. The captured image may be an image that has been captured in advance, or an image that has been captured when determining the reference point. The acquisition unit 11 acquires the captured image from a server or PC (Personal Computer) where the captured image is stored. Here, the acquisition unit 11 can acquire the captured image by performing data communication with the server or PC using an internet connection or short-range wireless communication technology.

[0014] The image acquired by the acquisition unit 11 is, for example, an image of only the robot's controlled object. If the controlled object is the end effector of a robot arm, the image acquired by the acquisition unit 11 may be an image of only the end effector. Alternatively, the image may be an image of the entire robot, or an image of only the contact portion of the robot's controlled object that comes into contact with the object handled by the robot. Here, the image of the contact portion includes the contact area of ​​that contact portion. The contact area is the space in which the contact portion performs a predetermined operation on the object.

[0015] If the controlled object is an end effector, and the contact portion of the end effector has multiple claws, and the claws move in a predetermined direction to grip an object, the acquisition unit 11 acquires, for example, images of the movable claws and the arm to which the claws are connected. The acquired images include the contact area for the claws to grip the object. In this example, the contact area is the space surrounded by the claws. The acquisition unit 11 may acquire an image in which only the gripping portion is captured, or an image in which the entire end effector including the gripping portion is captured.

[0016] The captured image is an image of the control target captured from an arbitrary direction. For example, the captured image is an image of the control target captured from the side. Even if it is not the side, the captured image may be an image of the control target captured from the bottom or the top surface, or if the control target has a front and a back, it may be an image captured from the front or the back.

[0017] When the shape of the control target imaged depends on the viewing angle of the control target, the acquisition unit 11 acquires a plurality of captured images with different imaging directions. For example, when the control target is the end effector of a robotic arm, assume that the end effector has a gripping part that grips an object from the left and right. In this case, the shape of the end effector imaged in the captured image is different between the case where the end effector is imaged from the direction in which the end effector grips the object and the case where the end effector is imaged from a direction perpendicular to the gripping direction. In this case, the acquisition unit 11 acquires images of the end effector captured from the gripping direction and the perpendicular direction. Here, when the camera that images the control target is a 3D camera or an omnidirectional camera, and the shape of the control target can be captured by a single captured image, the acquisition unit 11 does not necessarily need to acquire a plurality of captured images.

[0018] Also, when the contact part where the control target contacts has mobility with respect to the object handled by the robot, the acquisition unit 11 can acquire a plurality of captured images before and after the operation of the contact part. Here, before and after the operation of the contact part means, for example, the start point and the end point of the operation, but it may be any two points during the operation. Also, the acquisition unit 11 may acquire three or more captured images during the operation of the contact part.

[0019] Here, an example of the captured image acquired by the acquisition unit 11 will be described using drawings. FIGS. 2 to 4 are schematic diagrams showing an example of the captured image acquired by the acquisition unit 11. FIGS. 2 to 4 are an example of an image that captures only the end effector of the robot arm that is the control target. In FIGS. 2 to 4, since the end effector has a function of gripping the object handled by the robot, it has two claws at its tip. The claws in the end effector can also be referred to as gripping parts. That is, the gripping part at the tip of the end effector has mobility, and by operating the gripping part, the object can be gripped.

[0020] FIGS. 2 and 3 are side views when the gripping parts of the end effector are arranged on the left and right of the image. FIG. 2 is a side view at the start point of the operation of the gripping part. Also, FIG. 3 is a side view at the end point of the operation of the gripping part. Thus, when the contact part of the control target with respect to the object has mobility, the acquisition unit 11 can acquire a plurality of captured images before and after the operation of the contact part.

[0021] FIG. 4 is a side view of the end effector imaged from the negative X-axis direction with respect to FIGS. 2 and 3. In other words, FIG. 4 is an image of the end effector taken from a direction perpendicular to the XZ plane with respect to FIGS. 2 and 3. Further in other words, FIG. 4 is a side view of the end effector imaged from the gripping direction of the end effector. As can be seen by comparing FIG. 2 or FIG. 3 with FIG. 4, the shape of the end effector shown in the drawing differs depending on the viewing angle. In such a case, the acquisition unit 11 acquires the captured image of FIG. 4 together with FIG. 2 or FIG. 3. Here, the acquisition unit 11 may further acquire, for example, an image of the end effector taken from below (negative Z-axis direction).

[0022] The explanation of the acquisition unit 11 shown in Figure 1 continues. In addition to the image of the controlled object captured by the camera, the acquisition unit 11 may also acquire information relating to the depth of the controlled object as viewed from approximately the same direction as the imaging direction of the captured image using a depth sensor. The acquisition unit 11 may separately acquire the depth information from the depth sensor in addition to the image captured by the 2D camera, or, if the captured image is captured by a 3D camera using a depth sensor, it may acquire the depth information from the depth sensor together with the captured image. The direction measured by the depth sensor includes not only the same direction as the imaging direction of the captured image, but also directions that are considered to be the same direction. The depth information may be an image in which the depth of the controlled object is represented by the intensity of color, or it may be data.

[0023] Here, an example of information related to the depth of the controlled object acquired by the acquisition unit 11 will be explained using the drawings. Figure 5 is an example of information related to the depth of the controlled object acquired by the acquisition unit 11. The controlled object in Figure 5 is the end effector of a robot arm, as in Figures 2 to 4. The direction of the depth acquired by the depth sensor in Figure 5 roughly coincides with the imaging direction in Figures 2 and 3. That is, Figure 5 is the depth information when the end effector is viewed from the side when the gripping part of the end effector is positioned on the left and right sides of the image. As shown in Figure 5, the depth information here is an image in which the depth of the controlled object, the end effector, is represented by the intensity of the colors.

[0024] Returning to Figure 1, let's continue the explanation of the configuration of the reference point determination system 1. The determination unit 12 determines the reference point of the controlled object based on the captured image acquired by the acquisition unit 11. In other words, the determination unit 12 can determine any point as the reference point based on the captured image acquired by the acquisition unit 11. Typically, the determination unit 12 determines a point in the contact area of ​​the contact portion where the robot's controlled object comes into contact with the object handled by the robot as the reference point. In this case, the acquisition unit 11 acquires an image of the contact area of ​​the contact portion.

[0025] The determination unit 12 determining the reference point of the controlled object means determining the coordinates of the reference point relative to the coordinate system of the controlled object. The determination unit 12 can set any point in the coordinate system of the controlled object as the origin. For example, the determination unit 12 can set any point at the connection point between the controlled object and the robot as the origin.

[0026] Here, we will describe an example of how the determination unit 12 determines a reference point, using as an example a case where the controlled object is an end effector, and the contact portion of the end effector has multiple claws, and the claws move in a predetermined direction to grip the object. In this example, it is assumed that the acquisition unit 11 acquires the captured images in Figures 2 to 4 and the depth information in Figure 5. Figures 2 to 5 will be referred to as appropriate in the explanation of this example.

[0027] First, the determination unit 12 identifies the type of end effector based on the captured image acquired by the acquisition unit 11. That is, the determination unit 12 identifies that the end effector has the function of gripping an object. The determination unit 12 may also identify the type of end effector using a trained model that has been trained using data on captured images and the types of end effectors corresponding to those images, as shown in Figures 2 to 4, as training data.

[0028] Next, the determination unit 12 identifies the contact portion, i.e., the gripping portion, of the end effector by using image segmentation. In other words, the determination unit 12 classifies the end effector into the contact portion and the non-contact portion based on the captured image. In Figures 2 to 4, the captured image includes not only the gripping portion but also the connection portion between the arm and the gripping portion in the robot arm. Therefore, the determination unit 12 identifies the gripping portion and the said connection portion by image segmentation. The determination unit 12 may also identify the gripping portion using a trained model. Here, the determination unit 12 may use either the image in Figure 2 or Figure 3 and the image in Figure 4 to identify the gripping portion. That is, the determination unit 12 does not have to use either the image in Figure 2 or Figure 3 to identify the gripping portion.

[0029] If the captured image includes only the gripping portion, the determination unit 12 does not need to perform image segmentation. That is, if the connection portion with the arm shown in Figures 2 to 4 is not included in the captured image, the gripping portion can be directly identified from the captured image, and in this case, the determination unit 12 may identify the gripping portion without performing image segmentation.

[0030] The determination unit 12 can determine the number of claws in the gripping part by image segmentation. That is, if two claws are identified by image segmentation, the determination unit 12 can determine that there are two claws in that image.

[0031] Furthermore, the determination unit 12 can determine the direction of movement of the gripping part by image segmentation. That is, the determination unit 12 determines the direction of movement of the gripping part by using the images in Figures 2 and 3. The determination unit 12 can determine the direction of movement of the gripping part by comparing the position of the gripping part in Figures 2 and 3. In Figures 2 and 3, the direction of movement of the gripping part is the X-axis direction.

[0032] Next, the determination unit 12 calculates the center point of the gripping portion. First, the determination unit 12 calculates the center point of each claw in the gripping portion. That is, using Figures 2 to 4, the determination unit 12 finds the center points on the X, Y, and Z axes of one claw identified by image segmentation. Then, the determination unit 12 calculates the point between the two calculated center points as the center point of the gripping portion. In other words, the determination unit 12 calculates the center point of the gripping portion based on the center points of each component in the gripping portion.

[0033] Here, the determination unit 12 determines the reference point of the gripping part by calculating the distance from an arbitrary origin defined for the coordinate system of the controlled object to the center point of the gripping part. The determination unit 12 may also transmit the coordinate data of the determined reference point to the control unit (not shown) of the robot.

[0034] The determination unit 12 may calculate the center point of the gripping part by calculating the depth in the Z-axis direction using the captured images from Figures 2 to 4, as well as the information relating to the depth of the controlled object in Figure 5. In this case, the determination unit 12 can calculate the center point in the Z-axis direction based on the distance in the Z-axis direction which can be determined based on the intensity of the colors in Figure 5. The determination unit 12 may calculate the depth in the Z-axis direction using Figure 4 and correct the said depth using Figure 5.

[0035] Furthermore, the determination unit 12 may calculate the depth using Figure 5 instead of Figure 4. That is, the determination unit 12 may calculate the center point of the gripping part by calculating the center point in the XY plane using the captured image in Figure 2 or Figure 3, and calculating the depth in the Z-axis direction using the depth information in Figure 5.

[0036] The determination unit 12 can determine the center point of the gripping portion calculated by the above method as the reference point of the end effector. In the above, the determination unit 12 determines a point in the contact area of ​​the contact portion where the end effector contacts the object as the reference point. In this case, the contact area is the space between the gripping portions of the end effector. Since the determination unit 12 determines the center point of the gripping portion as the reference point, the determination unit 12 determines a point in the contact area as the reference point. Specifically, the determination unit 12 determines the approximate center point of the contact area as the reference point.

[0037] The determination unit 12 may determine the reference point by other means. For example, in the above example, the determination unit 12 can determine the reference point as the center point between the lower ends of each component in the gripping part. That is, using Figures 2 to 4, the determination unit 12 can determine the reference point as the lower end in the Y-axis direction of each claw, which is between the center points in the X-axis and Z-axis directions. In this case as well, the determination unit 12 determines one point in the contact area by the gripping part as the reference point.

[0038] Furthermore, the determination unit 12 may determine the reference point using a method different from the above example based on the captured image acquired by the acquisition unit 11. For example, if the acquisition unit 11 acquires an image of an end effector equipped with a gripping portion taken from below (negative Z-axis direction), the determination unit 12 first identifies the contact area of ​​the gripping portion by performing segmentation on the image and calculates the center point of the contact area. Then, the determination unit 12 may calculate the center point of the gripping portion by calculating the center point in the Y-axis direction of the gripping portion from the side view of the end effector as shown in Figures 2 to 4.

[0039] The determination unit 12 does not use a single point in the contact area of ​​the contact portion as its reference point, but can use any point in the captured image acquired by the acquisition unit 11 as its reference point. For example, the determination unit 12 can determine the reference point by using the center point of the connection portion between the gripping portion and the arm in Figures 2 to 4. Furthermore, if the connection portion, the arm connected to the connection portion, and the end effector are controlled to move as a single unit, the determination unit 12 can determine the reference point by using a single point on the arm.

[0040] The determination unit 12 can determine any point as a reference point even when the contact point is not a gripping point. For example, if the end effector is a screwdriver that tightens screws onto an object, the determination unit 12 can use the center point at the tip of the screwdriver, which is the contact point, as the reference point based on the image captured by the acquisition unit 11, which includes the screwdriver. Also, if the robot is an automated forklift, the determination unit 12 can use the point between the center points at the ends of each fork as the reference point based on the image captured by the acquisition unit 11, which includes the forks.

[0041] The determination unit 12 can determine a reference point even if the contact part is not movable. For example, if the end effector is equipped with a suction mechanism that uses a motor inside the robot to attract an object, the determination unit 12 can identify the suction mechanism based on the image of the suction mechanism acquired by the acquisition unit 11, and set its center point as the reference point.

[0042] The determination unit 12 can determine a reference point even if the controlled object does not come into contact with the object. For example, consider a case where the end effector has an air blow mechanism that can clean the object without coming into contact with it. In this case, the determination unit 12 can identify the air blow mechanism based on the captured image of the gas discharge port of the air blow mechanism acquired by the acquisition unit 11, and set its center point as the reference point.

[0043] Thus, according to the reference point determination system 1, the reference point of a controlled object in a robot equipped with any controlled object can be determined based on captured images of the controlled object. When a robot attempts to move a controlled object, it needs to pre-determine the reference point of the controlled object. This is because the relative position and orientation of the controlled object to the robot changes as the controlled object moves in any direction relative to the robot. In other words, when a robot attempts to move a controlled object, it needs to know in advance the reference point for recognizing the relative position and orientation of the controlled object.

[0044] When attempting to determine the reference point of a robot's controlled object using related technologies, it is necessary to create a CAD model of the controlled object in advance. For example, if a single robot may have multiple controlled objects, it is impossible to define the reference point for each controlled object without creating a CAD model for each of them. Therefore, if the cost of creating CAD models is high, it may be difficult to determine the reference point of a controlled object using related technologies.

[0045] The reference point determination system 1 can determine reference points using captured images of the controlled object without using a CAD model. In other words, the reference point determination system 1 can determine reference points by acquiring one or more images of the controlled object and calculating arbitrary points from those images. Specifically, the reference point determination system 1 can identify the controlled object from the image using, for example, a trained model, and define any coordinate in the coordinate system of the controlled object as a reference point by calculating an arbitrary point for the identified controlled object. As a result, the reference point determination system 1 can determine reference points for any controlled object without creating a CAD model for each controlled object, even when there are multiple controlled objects for a single robot.

[0046] When creating a CAD model to define the reference point of a controlled object, the reference point is determined by designing the dimensions of the controlled object in detail. However, the reference point determination system 1 recognizes the size of each component of the controlled object from the captured image of the controlled object and can directly calculate the reference point from the image. As a result, the reference point determination system 1 can determine the reference point more easily compared to creating a CAD model.

[0047] The reference point determination system 1 acquires captured images of the contact area of ​​the contact point where the controlled object comes into contact with the object handled by the robot, and can determine one point in that contact area as the reference point. In other words, the reference point determination system 1 can determine the reference point based on captured images of the contact point and its contact area on the controlled object. As a result, the reference point determination system 1 can determine the reference point based on a small number of captured images.

[0048] Furthermore, the reference point determination system 1 can determine the approximate center point of the contact area of ​​the contact portion as the reference point. For example, if the controlled object is an end effector equipped with a gripping portion, the reference point determination system 1 can calculate the center point of each claw and set the center point between these center points as the reference point. This allows the reference point determination system 1 to easily calculate and define the reference point based on the acquired image.

[0049] Furthermore, if the contact part is movable, the reference point determination system 1 can acquire multiple images before and after the movement of the contact part and determine a reference point based on these images. As a result, the reference point determination system 1 can determine a reference point that takes into account the direction of movement of the contact part on the controlled object, thereby enabling it to determine a more appropriate point as a reference point when the robot controls the movement of the controlled object.

[0050] Furthermore, the reference point determination system 1 can acquire information related to the depth of the controlled object viewed from approximately the same direction as the imaging walk using a depth sensor, and determine the reference point by considering this depth information. As a result, the reference point determination system 1 can determine a point close to the ideal reference point as the reference point. In other words, by using a depth sensor, the accuracy of reference point calculation can be improved.

[0051] (Example hardware configuration) Figure 6 shows an example of the hardware configuration of the reference point determination system 2 according to this disclosure. In Figure 6, the reference point determination system 2 has a processor 21 and a memory 22. The processor 21 may be, for example, a microprocessor, an MPU (Micro Processing Unit), or a CPU (Central Processing Unit). The processor 21 may include multiple processors. The memory 22 is composed of a combination of volatile memory and non-volatile memory. The memory 22 may include storage located away from the processor 21. In this case, the processor 21 may access the memory 22 via an I / O (Input / Output) interface, which is not shown.

[0052] In the above example, the program can be stored and provided to the computer using various types of non-transitory computer-readable medium. Non-transitory computer-readable medium includes various types of tangible storage medium. Examples of non-transitory computer-readable medium include magnetic storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memory (e.g., mask ROMs, PROMs (Programmable ROMs), EPROMs (Erasable PROMs), flash ROMs, RAMs). Alternatively, the program may be provided to the computer using various types of transient computer-readable medium. Examples of transient computer-readable medium include electrical signals, optical signals, and electromagnetic waves. Transitory computer-readable medium can supply the program to the computer via wired communication channels such as electric wires and optical fibers, or via wireless communication channels. Computers include various information processing devices such as PCs, servers, CPUs, MPUs, FPGAs (Field Programmable Gate Arrays), and ASICs (Application Specific Integrated Circuits).

[0053] This disclosure is not limited to the embodiments described above, and can be modified as appropriate without departing from the spirit of the invention. For example, the reference point determination system 1 may determine the reference point using only images acquired by a depth sensor, without using a two-dimensional visible light camera, an infrared camera, or a three-dimensional camera. [Explanation of symbols]

[0054] 1 Reference point determination system, 2 Reference point determination system, 11 Acquisition unit, 12 Determination unit, 21 Processor, 22 Memory

Claims

1. A reference point determination system for determining the reference point of a controlled object whose movement is controlled by a robot, An acquisition unit that acquires the captured image of the controlled object captured by the camera, The system includes a determination unit that determines the reference point based on the captured image, Reference point determination system.

2. The acquisition unit acquires the captured image of the contact area of ​​the contact portion where the controlled object comes into contact with the object handled by the robot. The determination unit determines a point in the contact area as the reference point. The reference point determination system according to claim 1.

3. The determination unit determines the approximate center point of the contact area as the reference point. The reference point determination system according to claim 2.

4. The acquisition unit acquires multiple images of the contact portion before and after the movement of the contact portion when the contact portion of the controlled object is movable. The reference point determination system according to claim 2 or 3.

5. The acquisition unit further acquires information relating to the depth of the controlled object as viewed from substantially the same direction as the imaging direction of the captured image using a depth sensor. The determination unit determines the reference point based on the information relating to the depth. A reference point determination system according to any one of claims 1 to 3.

Images