robot

Abstract

To reduce the processing load on robot control. [Solution] Whenever the first camera acquires an image, the control unit determines the amount of rotational displacement of the first motor that rotates the first arm around the first rotation axis and the amount of rotational displacement of the second motor that rotates the second arm around the second rotation axis, based on the difference between the coordinate values ​​of the robot's control points determined from markers in the image and the coordinate values ​​of a designated target point, and controls the first motor and the second motor to rotate by the determined amount of displacement. When a preset condition is met, each time the first camera acquires an image, the control unit determines the amount of rotational displacement of the first motor and the second motor based on the difference between the coordinate values ​​of the robot's control points in the image and the coordinate values ​​of a return point set as the point to which the robot should return, and controls the first motor or the second motor to rotate by the determined amount of displacement.

Description

[[Technical Field]]

[0001] The present disclosure relates to a robot. [[Background Art]]

[0002] In the robot vision system disclosed in Patent Document 1, a computer executes three-dimensional measurement and recognition processing based on images acquired by a stereo camera, and a robot controller controls the robot based on target coordinates calculated by the computer. Further, in the technology described in Patent Document 1, calibration is executed prior to the operation of the robot. [[Prior Art Literature]] [[Patent Literature]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Publication No. 2010-172986 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] In the technology described in Patent Document 1, it is necessary to calculate the three-dimensional position of a target in a stereo camera coordinate system based on an image obtained by capturing the target, and convert the three-dimensional position in the stereo camera coordinate system into a three-dimensional position in a robot coordinate system, which results in a large processing load.

[0005] Furthermore, it is also necessary to execute calibration to calculate parameters for performing coordinate conversion from the stereo camera coordinate system to the robot coordinate system. For this reason, further improvement has been desired for a technology that reduces the processing load of robot control. [[Means for Solving the Problem]]

[0006] The present disclosure can be implemented in the following forms.

[0007] According to one embodiment of the present disclosure, a robot is provided. The robot includes a base, a first arm connected to the base and rotating around a first axis of rotation, a second arm connected to one end of the first arm and rotating around a second axis of rotation parallel to the first axis of rotation relative to the first arm, at least one physical marker attached to the second arm for defining the control point of the robot, a camera fixed in position and orientation relative to the base, positioned to capture images of the marker moving on a virtual plane, assuming a virtual plane perpendicular to the first axis of rotation, and capturing images of the first arm and the second arm at regular time intervals, and a control unit which, each time the first camera acquires an image, determines the robot from the marker in the image captured by the first camera. The robot comprises a control unit which determines the amount of rotational displacement of the first motor that rotates the first arm around the first rotation axis and the amount of rotational displacement of the second motor that rotates the second arm around the second rotation axis based on the difference between the coordinate value of the control point of the robot and the coordinate value of a designated target point, and controls the first motor and the second motor to rotate by the determined amount of displacement, and when a preset condition is met, each time the first camera acquires an image, it determines the amount of rotational displacement of the first motor and the amount of rotational displacement of the second motor based on the difference between the coordinate value of the control point of the robot in the image captured by the first camera and the coordinate value of a return point set as the point to which the robot should return, and controls the first motor or the second motor to rotate by the determined amount of displacement. [Brief explanation of the drawing]

[0008] [Figure 1] An explanatory diagram showing the schematic configuration of the robot system according to the first embodiment. [Figure 2] An explanatory diagram of the mechanism connecting the first arm and the second arm. [Figure 3] An explanatory diagram showing an example of an image captured by camera 510. [Figure 4]The first half of the flowchart showing the process for generating the arm's trajectory. [Figure 5] The latter half of the flowchart showing the process for generating the arm's trajectory. [Figure 6] An explanatory diagram showing an example of a rotation amount table for the X-axis. [Figure 7] An explanatory diagram showing how the robot's base is fixed to the wall. [Figure 8] An explanatory diagram showing images captured of a Cartesian robot. [Figure 9] An explanatory diagram showing the schematic configuration of the robot system according to the second embodiment. [Figure 10] An explanatory diagram showing an example of an image captured by camera 520. [Figure 11] An explanatory diagram showing the schematic configuration of a robot system of a modified example of the second embodiment. [Figure 12] An explanatory diagram showing an example of an image captured by camera 530. [Figure 13] An explanatory diagram of an image captured with guide light emitted. [Figure 14] An explanatory diagram of the sensor located at the tip of the arm. [Figure 15] An explanatory diagram showing the schematic configuration of the robot system according to the third embodiment. [Figure 16] An explanatory diagram showing the schematic configuration of the robot system according to the fourth embodiment. [Figure 17] An explanatory diagram showing the schematic configuration of a robot system according to a modified example of the fourth embodiment. [Figure 18] An explanatory diagram showing the schematic configuration of the robot system according to the fifth embodiment. [Figure 19] An explanatory diagram showing an example of image F[i] captured by camera 510. [Figure 20] An explanatory diagram of the algorithm for detecting the light-emitting part. [Figure 21] An explanatory diagram showing an example of an image taken of the robot according to the sixth embodiment. [Figure 22] A flowchart for the process of determining the initial positions of two marks. [Figure 23] Explanatory diagram showing an example of an image captured by the camera 510 according to the seventh embodiment. [Figure 24] Explanatory diagram showing a state of returning to a return point. [Figure 25] Explanatory diagram showing a schematic configuration of a robot system according to an eighth embodiment. [Figure 26] Explanatory diagram showing an example of an operation UI displayed on a terminal device. [Figure 27] Explanatory diagram showing another example of an operation UI displayed on a terminal device. DETAILED DESCRIPTION OF EMBODIMENTS

[0009] A. First Embodiment: A1. Basic Configuration: FIG. 1 is an explanatory diagram showing a schematic configuration of a robot system 10 according to a first embodiment. As shown in FIG. 1, the robot system 10 includes a robot 100, a control unit 700, a relay unit 750, and a terminal device 900.

[0010] The robot 100 includes a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 that supports the first arm 110 and the second arm 120, a mark 310, an end effector 400, and a camera 510. Although not illustrated in FIG. 1, the robot 100 further includes a drive mechanism D1, a drive mechanism D2, and a link mechanism L1. The mark 310 is also referred to as an indicator.

[0011] One end of the first arm 110 is connected to the base 210 via the joint 130. The other end of the first arm 110 is connected to one end of the second arm 120 via the joint 140. The first arm 110 and the second arm 120 are formed of, for example, resin rod-shaped members.

[0012] The joint 130 connects the base 210 and the first arm 110. The joint 130 is equipped with a rotation axis J1. The joint 130 can rotate the first arm 110 around the rotation axis J1. The rotation axis J1 is also called the "first rotation axis". As the rotation axis J1 rotates, the first arm 110 rotates relative to the base 210.

[0013] The joint 140 connects the first arm 110 and the second arm 120. The joint 140 is equipped with a rotation axis J2. The joint 140 can rotate the second arm around the rotation axis J2. The rotation axis J2 of the joint 140 is also called the "second rotation axis". As the rotation axis J2 rotates, the second arm 120 rotates relative to the first arm 110.

[0014] The rotation axis J1 of joint 130 and the rotation axis J2 of joint 140 are parallel. Preferably, the rotation axes J1 and J2 are parallel to the horizontal plane. Here, parallel means not only that the rotation axes J1 and J2 are strictly parallel to the horizontal plane, but also that they are approximately parallel to the horizontal plane. Approximately parallel to the horizontal plane means that when the rotation axes J1 and J2 are projected onto a plane perpendicular to the horizontal plane, the angle that the rotation axes J1 and J2 make with respect to the horizontal plane is in the range of -10 degrees to 10 degrees.

[0015] Mark 310 is attached near the end of the second arm 120 opposite to the end connected to the first arm 110. Mark 310 is provided to identify the position of the control point of the robot 100 in the image captured by the camera 510. For example, an arbitrary position is set on the end effector 400 as the control point. In this embodiment, Mark 310 is positioned close enough to the control point that its position can be considered equivalent to the position of the control point of the robot 100.

[0016] Mark 310 comprises a light-emitting unit LD and a light-shielding plate SP arranged around the light-emitting unit LD. The light-emitting unit LD comprises at least one point light source LED (Light Emitting Diode). The light-shielding plate SP is preferably made of a low-reflectivity resin plate-like material. The light-shielding plate SP is, for example, black. The light-shielding plate SP is provided to block light emitted from a light source on the side opposite to the camera 510 relative to the light-emitting unit LD. By arranging the light-shielding plate SP around the light-emitting unit LD, the control unit 700 is prevented from mistakenly identifying light behind the light-emitting unit LD in the captured image as light emitted by the light-emitting unit LD. Therefore, the control unit 700 can easily identify the position of Mark 310 in the image captured by the camera 510.

[0017] The end effector 400 is attached to the end of the second arm 120. The end effector 400 is, for example, a gripper that grasps a target. The gripper as the end effector 400 performs grasping or releasing actions according to the control of the control unit 700. The end effector 400 is also called the "gripping part".

[0018] Camera 510 is fixed to base 210 by fixing member FM. Therefore, the position and orientation of camera 510 are fixed relative to base 210. Camera 510 is positioned to capture images of at least mark 310 and the target point. The target point is the position that the control point of robot 100 should reach. The target point may be the final destination that the control point of robot 100 should reach, or it may be a waypoint that the control point of robot 100 should pass through before reaching the final destination. Camera 510 captures images of robot 100 at a constant frame rate. As camera 510, for example, a board camera with a lens directly mounted on an image sensor substrate can be used. Camera 510 outputs the captured images directly to the control unit 700. Camera 510 is also called the "first camera".

[0019] The control unit 700 controls the operation of the robot 100. Furthermore, the control unit 700 controls the operation of the end effector 400. The control unit 700 is composed of an FPGA (Field Programmable Gate Array). The control unit 700 may also be composed of an ASIC (application specific integrated circuit) or a PC (personal computer). The control unit 700 includes, for example, an image input circuit that receives images output from the camera 510, an image processing circuit that processes the images, a data relay circuit for communicating with other devices, and an image output circuit that outputs images to a terminal device 900, etc., via wireless communication such as Wi-Fi or an internet communication network. The data relay circuit is used for communication via wireless communication such as Wi-Fi (registered trademark) or an internet communication network.

[0020] In this embodiment, the control unit 700 controls the movement of the robot 100 using so-called visual servoing based on the image information acquired by the camera 510. Visual servoing is a technique that controls the robot so that the image acquired by the camera matches a target image. In this embodiment, the target image is the image in which the robot's control points in the image acquired by the camera coincide with a specified target point.

[0021] Here, we assume a virtual plane S1, which is a virtual plane to which the rotation axis J1 is orthogonal. Since the rotation axis J1 of joint 130 and the rotation axis J2 of joint 140 are parallel, the rotation axis J2 is also orthogonal to the virtual plane S1. To facilitate understanding of the technology, the virtual plane S1 is set to a limited range in which the mark 310 is expected to move. As at least one of joint 130 and joint 140 rotates, the mark 310 attached to the second arm 120 moves on the virtual plane S1. The position of the mark 310 can be expressed in coordinates on the virtual plane S1.

[0022] Figure 3 is an explanatory diagram showing an example of an image F[i] captured by camera 510. The subscript i is a positive integer that is added each time imaging is performed. For example, the coordinates of the pixel in the upper left corner of image F[i] are set as the coordinate origin of image F[i]. The right direction of image F[i] is set as the positive direction of the X axis, and the downward direction of image F[i] is set as the positive direction of the Y axis. In this embodiment, the position of mark 310 on the virtual plane S1 is identified by the position of mark 310 in the image coordinate system representing the position on image F[i] captured by camera 510. As shown in Figure 1, if the optical axis AX of camera 510 does not coincide with the perpendicular PL of the virtual plane S1, the coordinates in image F[i] captured by camera 510 do not exactly coincide with the coordinates in the virtual plane S1. However, since the position and orientation of camera 510 are fixed relative to base 210, it does not pose a major problem to identify the coordinates in the virtual plane S1 based on the coordinates in image F[i] captured by camera 510. However, it is preferable that the angle θ1 formed by the optical axis AX of the camera 510 with respect to the perpendicular PL of the virtual plane S1 is small. Specifically, it is preferable that the angle θ1 is 60 degrees or less.

[0023] The control unit 700 calculates the coordinates of the mark 310 on the image acquired by the camera 510 and controls the movement of the robot 100 based on the coordinates representing the current position of the mark 310 and the coordinates of the target point. In Figure 3, the coordinates of the target point are referred to as target coordinates. In order to bring the coordinates of the mark 310 closer to the coordinates of the target point in F[i] captured by the camera 510, the control unit 700 performs control such as repeatedly rotating the first arm to reduce the difference between the X coordinate of the mark 310 and the X coordinate of the target point, and rotating the second arm to reduce the difference between the Y coordinate of the mark 310 and the Y coordinate of the target point.

[0024] The relay unit 750 can communicate with the control unit 700 via wired or wireless communication. For example, a Wi-Fi router can be used as the relay unit 750. The relay unit 750 transmits the image received from the control unit 700 to the terminal device 900. Note that the use of the relay unit 750 is just one example, and the control unit 700 and the terminal device 900 may communicate directly via wireless communication. The same applies to subsequent embodiments.

[0025] The terminal device 900 is a smartphone or tablet equipped with a touch panel display. The terminal device 900 may also be a PC equipped with a mouse as an input device and a display as a display device. Hereinafter, the touch panel display and the display may simply be referred to as the display device. The terminal device 900 receives images from the control unit 700 via wireless communication such as Wi-Fi, through the relay unit 750. The terminal device 900 transmits signals indicating user P's operation instructions to the control unit 700 via the relay unit 750. The terminal device 900 is also referred to as the "operation unit".

[0026] Figure 2 is an explanatory diagram of the mechanism connecting the first arm 110 and the second arm 120. The left side of Figure 2 shows a side view of the robot 100, and the right side shows a rear view of the robot 100. The layout view shows the robot 100 as seen in the direction of arrow A2 in Figure 1.

[0027] The drive mechanism D1 rotates the joint 130. The drive mechanism D1 includes a stepping motor (not shown), a reduction gear, and a driver. The stepping motor in the drive mechanism D1 is also called the "first motor." Hereafter, the stepping motor may simply be referred to as the motor. The rotation shaft J1 rotates when the stepping motor in the drive mechanism D1 is driven. In this embodiment, the drive mechanism D1 does not have an encoder. Generally, when driving a stepping motor, a step-out phenomenon may occur where the rotation cannot keep up with the applied pulse rate, resulting in an error between the assumed axis angle and the actual axis angle. To correct this, an encoder is required to measure the actual axis angle. In this embodiment, the position of the arm is detected by the mark 310, and the arm is controlled based on this. Therefore, even if an error occurs between the assumed axis angle and the actual axis angle, the error will be corrected in the next control cycle. For this reason, there is no problem in not having an encoder. The same applies to the drive mechanism D2, which will be described later.

[0028] The drive mechanism D2 rotationally drives the joint 140. The drive mechanism D2 includes a stepping motor (not shown), a reduction gear, and a driver. The stepping motor of the drive mechanism D2 is also called the "second motor." The rotational force generated by the drive of the stepping motor of the drive mechanism D2 is transmitted to the joint 140 via the link mechanism L1. The link mechanism L1 includes a drive pulley PL1, a driven pulley PL2, and a belt B1. The drive pulley PL1 is attached to the rotating shaft of the stepping motor of the drive mechanism D2. The driven pulley PL2 is attached to the joint 140. The driven pulley PL2 is rotatable around the rotating shaft J2. The belt B1 is stretched between the drive pulley PL1 and the driven pulley PL2. It is desirable that the tooth ratio obtained by dividing the number of teeth of the drive pulley PL1 by the number of teeth of the driven pulley PL2 is a value of 1 or greater. As a result, when the arm is moving forward, if viewed from the side, the tip of the arm will trace an inverted arch shape. This allows the arm's movement to act in a direction that counteracts the possibility of it colliding with shelves or other objects.

[0029] As the stepping motor of the drive mechanism D2 is driven, the drive pulley PL1 rotates. The rotation of the drive pulley PL1 is transmitted to the driven pulley PL2 via belt B1. When the driven pulley PL2 rotates, the rotating shaft J2 rotates. As a result, the second arm 120 rotates relative to the first arm 110. This reduces the amount of change in the posture of the second arm 120 caused by the rotation of the first arm 110. More specifically, the link mechanism L1 is a mechanism that works to counteract the rotation of the second arm caused by the rotation of the first arm by mechanically transmitting at least a portion of the power output by the stepping motor that drives the rotating shaft J1 to the second arm.

[0030] The link mechanism L1 may be configured to move the second arm parallel to the rotation of the first arm 110. A link mechanism L1 configured in this way is called a parallel link mechanism. Moving the second arm 120 parallel means moving the second arm 120 while maintaining its posture. In this case, the angle of the second arm 120 with respect to the ground can always be kept constant. Compared to a configuration without the link mechanism L1, it is possible to suppress the control point from tracing a large trajectory. Alternatively, the link mechanism L1 may be configured to move the second arm 120 non-parallel to the rotation of the first arm 110. A link mechanism L1 configured in this way is called a non-parallel link mechanism. Moving the second arm 120 non-parallel means moving the second arm 120 without maintaining its posture. In this case, the effect of canceling out the change in the Y coordinate of the control point by the first arm 110 is greater than in a parallel link mechanism.

[0031] In a configuration without the link mechanism L1, for example, suppose only the rotation axis J1 is rotated. In this case, the posture of the second arm 120 changes significantly in conjunction with the rotation of the first arm 110. To prevent the posture of the second arm 120 from changing significantly in conjunction with the rotation of the first arm 110, it is possible to rotate the rotation axis J1 and rotation axis J2 simultaneously, or to rotate the rotation axis J1 and rotation axis J2 sequentially with a very short time interval between them. In this case, it is necessary to calculate the amount of rotation of the rotation axis J2 to offset the displacement of the posture of the second arm 120 caused by the rotation of the rotation axis J1.

[0032] The link mechanism L1 acts in a direction that counteracts the rotation of the second arm caused by the rotation of the first arm. By providing the link mechanism L1, the amount of change in the posture of the second arm 120, which is linked to the rotation of the first arm 110, can be reduced. Therefore, it is not necessary to calculate the amount of rotation of the rotation axis J2 to offset the displacement of the posture of the second arm 120 caused by the rotation of the rotation axis J1.

[0033] The robot according to the second and subsequent embodiments comprises a drive mechanism D1 for rotating the joint 130, a drive mechanism D2 for rotating the joint 140, and a link mechanism L1.

[0034] Figures 4 and 5 are flowcharts showing the process for generating the arm's trajectory. This process starts, for example, when an image is received from camera 510. When camera 510 is activated, for example by user operation, it performs imaging at regular intervals and outputs the captured image to control unit 700. The initial value of the subscript i of the image F[i] acquired by camera 510 is 0.

[0035] As shown in Figure 4, in step S101, the control unit 700 calculates the coordinates (x[0], y[0]) representing the position of the mark 310 from the image F[0] supplied from the camera 510. The control unit 700 calculates the coordinates of the mark 310 in image F[i] based on the brightness value of image F[i]. Since the mark 310 is equipped with a light-emitting unit LD, the coordinates of the mark 310 in image F[i] can be identified based on the brightness value of image F[i]. As shown in Figure 3, the calculated coordinates of the mark 310 are denoted as (x[i], y[i]). Note that if image F[i] is an RGB image rather than a grayscale image, the brightness value can be calculated based on the RGB values.

[0036] In this embodiment, the control point of the robot 100 is brought to the target TG, which is an apple, and then the robot 100 is made to grasp the target TG, which is an apple. Although Figure 3 shows an example of an image that shows the entire robot 100, the image captured by the camera 510 does not need to show the entire robot 100. It is sufficient for the image captured by the camera 510 to show at least the mark 310 and the target TG.

[0037] As shown in Figure 4, in step S102, the control unit 700 sets the coordinates (x[0], y[0]) to the target coordinates (Xg, Yg). As shown in Figure 3, the target coordinates (Xg, Yg) represent the target point that the control point should reach. By setting the initial value of the target coordinates to the coordinates of the current mark 310, it is possible to prevent the robot 100 from performing unexpected movements.

[0038] As shown in Figure 4, the subscript i is incremented in step S103.

[0039] In step S104, the control unit 700 determines whether a new target location has been specified. For example, each time the control unit 700 receives an image F[i] from the camera 510, it transmits the image F[i] to the terminal device 900. The terminal device 900 displays the received image F[i] on the display device. The user P can specify a new target location by pointing to any position on the image F[i] displayed on the terminal device 900. The terminal device 900 transmits information indicating the target location specified by the user to the control unit 700 via the relay unit 750.

[0040] If a new target location is specified (step S104; YES), the process in step S105 is executed. If no target coordinates are specified (step S104; NO), the process in step S106 is executed.

[0041] In step S105, the control unit 700 uses the position pointed to on the display device and the image F[i] to calculate the coordinates representing the specified target point within the image. The control unit 700 updates the target coordinates (Xg, Yg) with the calculated coordinates.

[0042] In step S106, the control unit 700 determines whether or not it has received image F[i] from the camera 510. If image F[i] is received from the camera 510 (step S106; YES), the process in step S108 (see Figure 5) is executed.

[0043] As shown in Figure 5, in step S108, the control unit 700 calculates the coordinates (x[i], y[i]) of the mark 310 in image F[i] based on the brightness value of image F[i].

[0044] In step S109, the control unit 700 calculates the differences ΔX and ΔY, which represent the difference between the target coordinates (Xg, Yg) in image F[i] and the coordinates (x[i], y[i]) representing the current position of mark 310. As shown in Figure 3, ΔX is the difference between the X coordinate of mark 310 on the X axis and the X coordinate of the target point, and ΔY is the difference between the Y coordinate of mark 310 on the Y axis and the Y coordinate of the target point. ΔX = Xg - x[i]. ΔY = Yg - y[i]. If ΔX and ΔY are real numbers, for example, the values ​​of ΔX and ΔY are rounded to integers.

[0045] In step S111, the control unit 700 rotates the rotation axis J1 at a pulse rate based on ΔX. First, the control unit 700 determines the rotation direction of the rotation axis J1. If ΔX is a positive value, the rotation direction of the rotation axis J1 is counterclockwise. If ΔX is a negative value, the rotation direction of the rotation axis J1 is clockwise. Note that the rotation direction of the rotation axis J1 determined according to the positive or negative value of ΔX is not limited to the above example. The rotation direction of the rotation axis J1 is set appropriately based on the mounting position of the base 210, the positional relationship between the robot 100 and the target TG, etc. Furthermore, the control unit 700 refers to the rotation amount table TBx in the X axis and determines the displacement amount per unit time of the rotation of the motor of the drive mechanism D1 that rotates the joint 130.

[0046] Figure 6 is an explanatory diagram showing an example of a rotation amount table TBx. The rotation amount table TBx defines the pulse rate, which represents the displacement of the rotation of the motor that rotates the joint 130, according to ΔX. Generally, the speed of a stepping motor is expressed in terms of the number of pulses (steps) per second. The number of pulses (steps) per second is called the pulse rate. The pulse rate is also called the pulse frequency. In the rotation amount table TBx, the pulse rate is defined according to ΔX such that the pulse rate increases as ΔX increases and decreases as ΔX decreases. Therefore, when the robot 100 is far from the target point, the rotation of the first arm 110 is large, and when the robot 100 is close to the target point, control is performed so that the rotation of the first arm 110 is small. By making the robot 100 move in small increments as it approaches the target point, it is suppressed that the control point of the robot 100 passes the target point.

[0047] Furthermore, in the rotation amount table TBx, the pulse rate is set to 0 when ΔX is in a predetermined range near 0. In the example shown in Figure 6, the pulse rate is set to 0 when ΔX is in the range of -2 to 2. As a result, the rotation axis J1 converges within the predetermined range near 0 for ΔX and eventually stops.

[0048] In the rotation amount table TBx, a negative pulse rate indicates that the stepping motor driving the joint 130 should rotate in the opposite direction to the determined rotation direction. For example, if the end-effector position of the robot 100 has moved beyond the target point, the stepping motor of the drive mechanism D1 must be rotated in the opposite direction to bring the end-effector position of the robot 100 closer to the target point. The rotation amount table TBx is set appropriately according to the mounting position of the base 210, the positional relationship between the robot 100 and the target TG, etc., as shown in Figure 3.

[0049] The control unit 700 drives the stepping motor of the drive mechanism D1 by controlling the driver of the drive mechanism D1 with the determined rotation direction and pulse rate. As a result, the first arm 110 rotates around the rotation axis J1.

[0050] In step S113, the control unit 700 rotates the rotation axis J2 at a pulse rate based on ΔY. First, the control unit 700 determines the rotation direction of the rotation axis J2. If ΔY is a positive value, the rotation direction of the rotation axis J2 is counterclockwise. If ΔY is a negative value, the rotation direction of the rotation axis J2 is clockwise. Note that when ΔY is 0, the absolute value of ΔY is less than or equal to the threshold Ymin, so the process in step S113 is not executed. Note that the rotation direction of the rotation axis J2 determined according to the positive or negative value of ΔY is not limited to the above example. Furthermore, the control unit 700 refers to the rotation amount table TBy on the Y axis and determines the displacement amount of the motor rotation corresponding to ΔY. The rotation amount table TBy is not shown in the diagram, but the rotation amount table TBy defines a pulse rate that represents the displacement amount of the motor rotation that rotates the joint 140, according to ΔY. Similar to the rotation amount table TBx, the rotation amount table TBy defines a pulse rate corresponding to ΔY such that the pulse rate increases as ΔY increases and decreases as ΔY decreases. Therefore, when the robot 100 is far from the target point, the rotation amount of the second arm 120 is large, and when the robot 100 is close to the target point, the rotation amount of the second arm 120 is small. By making the robot 100 move in small increments as it approaches the target point, it is prevented the control point of the robot 100 from passing the target point.

[0051] Furthermore, similar to the rotation amount table TBx, the rotation amount table TBy also has a pulse rate set to 0 within a predetermined range where ΔY is near 0. As a result, the rotation axis J2 converges within a predetermined range where ΔY is near 0 and eventually stops.

[0052] The control unit 700 drives the stepping motor of the drive mechanism D2 by controlling the driver of the drive mechanism D2 with the determined rotation direction and pulse rate. As a result, the second arm 120 rotates around the rotation axis J2.

[0053] According to this embodiment, the displacement amount of the motor's rotation is determined based on the difference between the coordinate values ​​of the mark 310 in the captured image F[i] and the specified target coordinates. With this configuration, it is not necessary to perform computationally intensive processes such as coordinate transformation to operate the robot, as in the conventional method. Therefore, the processing load on the robot control can be reduced. In addition, since the rotation around the rotation axis J1 (see step S111 in Figure 5) and the rotation around the rotation axis J2 (see step S113 in Figure 5) are controlled independently, the control of the robot 100 by the control unit 700 does not become complicated.

[0054] Furthermore, in this embodiment, the robot 100 is operated so that the distance between the mark 310 and the designated target point gradually decreases within the captured image F[i]. By using a visual servo, the process of converting the three-dimensional position of the target in the stereo camera coordinate system to the three-dimensional position in the robot coordinate system, as in the conventional method, is unnecessary, thereby reducing the processing load. In addition, the first arm 110 or the second arm 120 may bend due to the weight of the object grasped by the robot 100. Even in such cases, the pulse rate of the motor that rotates the joint 130 or joint 140 is determined based on the difference between the coordinates of the mark 310 and the coordinates of the target point in the captured image F[i]. This is because the coordinates of the mark 310 in the captured image F[i] are the coordinates when the first arm 110 or the second arm 120 is bent. With a visual servo, the information from the captured image becomes feedback information. Therefore, the control of the robot 100 by the control unit 700 can be simplified.

[0055] A2. Variation 1: In the example shown in Figure 3, as only the joint 130 rotates, the end of the first arm 110 connected to the second arm 120 moves mainly along the X-axis in the images captured continuously over time. The rotation of the first arm 110 has a significant impact on the movement of the robot 100's control point in the X-axis direction. As only the joint 140 rotates, the end of the second arm 120 connected to the end effector 400 moves mainly along the Y-axis in the images captured continuously over time. The rotation of the second arm 120 has a significant impact on the movement of the robot 100's control point in the Y-axis direction. However, these assumptions vary depending on the mounting position of the base 210, the positional relationship between the robot 100 and the target TG, etc.

[0056] Figure 7 is an explanatory diagram showing an example of an image F[i] captured by the camera 510 when the base 210 of the robot 100 is fixed to a wall. In the illustrated example, the first arm 110 extends along the X-axis direction in the image. Due to the rotation of only the joint 130, the end of the first arm 110 connected to the second arm 120 moves mainly along the Y-axis direction in the images captured continuously over time. In the illustrated example, the rotation of the first arm 110 has a significant effect on the movement of the robot 100's control point in the Y-axis direction. Due to the rotation of only the joint 140, the end of the second arm 120 connected to the end effector 400 moves mainly along the X-axis direction in the images captured continuously over time. The rotation of the second arm 120 has a significant effect on the movement of the robot 100's control point in the X-axis direction.

[0057] Therefore, in the example shown in Figure 7, the control unit 700 refers to the rotation amount table for the Y axis to determine the rotational displacement amount of the motor that rotates the joint 130 corresponding to ΔY. The control unit 700 also refers to the rotation amount table for the X axis to determine the rotational displacement amount of the motor that rotates the joint 140 corresponding to ΔX.

[0058] A3. Variation 2: Figure 8 is an explanatory diagram showing image F[i] of robot 100a with a different arm configuration. The robot 100a shown in Figure 8 is configured as a Cartesian robot. Cartesian robots are sometimes called gantry robots. Robot 100a includes a first slider 110a, a second slider 120a, a base 210a, a mark 310a, an end effector 400a, and a camera 510. Note that the camera 510 is not shown in the illustration. The position and orientation of the camera 510 are fixed relative to the base 210a. The camera 510 images the first slider 110a and the second slider 120a at a predetermined frame rate. Note that the camera 510 only needs to be positioned so that it can capture at least the mark 310 and the target point.

[0059] One end of the second slider 120a is connected to the base 210a. The second slider 120a extends vertically. The second slider 120a supports the first slider 110a so that it can move vertically. The first slider 110a can move linearly horizontally while supported by the second slider 120a. The end effector 400a is attached to the tip of the first slider 110a. The mark 310a is attached to the first slider 110a in the vicinity of the end effector 400a. As the first slider 110a moves linearly horizontally, the control point of the robot 100a moves linearly along the X-axis in the images captured in a series of images over time. As the second slider 120a moves the first slider 110a linearly vertically, the control point of the robot 100a moves linearly along the Y-axis in the images captured in a series of images over time.

[0060] Here, we assume a virtual plane parallel to the X and Y axes in the image captured by the camera 510. The control point is movable within this virtual plane. The control unit 700 calculates the coordinates of the mark 310a in the image captured by the camera 510, and uses the difference between the current coordinates of the mark and the target coordinates of the target point to control the amount of movement of the first slider 110a in the X axis direction and the Y axis direction.

[0061] B. Second Embodiment: B1.Basic configuration: Figure 9 is an explanatory diagram showing the schematic configuration of the robot system 10b according to the second embodiment. The following description will focus on the configurations that differ from the first embodiment, and the configurations that are the same as those in the first embodiment will not be described. As described above, in the first embodiment, the range in which the control points of the robot 100 can move is within the virtual plane S1, and the range in which the control points of the robot 100 can move is limited to two-dimensional space. In this embodiment, the range in which the control points of the robot 100 can move is extended to three-dimensional space.

[0062] The robot system 10b comprises a robot 100b, a control unit 700, and a terminal device 900. Note that the terminal device 900 is not shown in Figure 9. The robot 100b comprises a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 that supports the first arm 110 and the second arm 120, a rotation mechanism 220, a mark 310, an end effector 400, a camera 510, and a camera 520.

[0063] A characteristic configuration in this embodiment is that the robot 100b is equipped with a rotation mechanism 220. A base 210 is placed on top of the rotation mechanism 220. Together with the base 210, the first arm 110 and the second arm 120 are placed on top of the rotation mechanism 220. The rotation mechanism 220 rotates the base 210 around the rotation axis J3 according to the control of the control unit 700. The rotation axis J3 is also called the "third rotation axis". The rotation axis J3 is parallel to the vertical direction. Here, parallel includes not only the state in which the rotation axis J3 is strictly parallel to the vertical direction, but also the state in which the rotation axis J3 is approximately parallel to the vertical direction. The rotation axis J3 is perpendicular to the rotation axis J1. Here, perpendicular includes not only the state in which the rotation axis J3 is strictly perpendicular to the rotation axis J1, but also the state in which the rotation axis J3 is approximately perpendicular to the rotation axis J1. The rotation axis J3 being approximately perpendicular to the rotation axis J1 means that the angle of the rotation axis J3 with respect to the rotation axis J1 is in the range of 80 to 90 degrees.

[0064] As the rotation mechanism 220 rotates the base 210, the control point of the robot 100b rotates around the rotation axis J3. In this way, the range of motion of the control point of the robot 100b is extended into three-dimensional space. The control unit 700 does not rotate the joints 130 and 140 while the rotation mechanism 220 is rotating. Also, the control unit 700 does not rotate the base 210 with the rotation mechanism 220 while at least one of the joints 130 and 140 is rotating. Thus, the control of the robot 100b's movement by the control unit 700 is kept from becoming complicated.

[0065] The configuration and functions of camera 510 are the same as in the first embodiment.

[0066] Camera 520 is fixed to the second arm 120 in a position and orientation that allows it to capture images of the direction of movement of the second arm 120. Camera 520 captures images in the direction in which the second arm 120 extends at a constant frame rate. The direction in which the second arm 120 extends is also called the "direction of movement of the second arm". For example, a board camera can be used as camera 520. Camera 520 outputs the captured images directly to the control unit 700. Camera 520 may also be fixed to the first arm 110. Camera 520 is also called the "second camera".

[0067] Figure 10 is an explanatory diagram showing an example of an image F2[j] captured by camera 520. The image supplied by camera 520 is referred to as image F2[j]. The subscript j is a positive integer that is incremented each time imaging is performed. The coordinates of the pixel at the upper left corner of image F2[j] are set as the coordinate origin of image F2[j]. The rightward direction of image F2[j] is set as the positive X-axis direction, and the downward direction of image F2[j] is set as the positive Y-axis direction.

[0068] The control unit 700 transmits the image F2[j] received from the camera 520 to the terminal device 900 each time it receives an image from the camera 520. The terminal device 900 displays the received image F2[j] on its display device. The frame rate at which the camera 520 performs imaging is set to a relatively high level, and the images F[j] continuously displayed on the display device of the terminal device 900 constitute a video image. The user operates the rotation of the rotation mechanism 220 so that the direction of movement of the second arm 120, i.e., the control point, faces the target TG, while viewing the image F2 displayed on the terminal device 900. For example, the user points to a virtual button or similar displayed on the display device of the terminal device 900 to give an operation instruction to rotate the rotation axis J3 clockwise or counterclockwise. The terminal device 900 transmits a signal corresponding to the operation instruction to the control unit 700. The control unit 700 controls the rotation of the rotation mechanism 220 in response to the signal received from the terminal device 900. Alternatively, the user can also give instructions regarding rotation by operating a mouse or swiping on a touch panel display.

[0069] Since the camera 520 is fixed to the second arm 120, the X-coordinate of the tip of the second arm 120 on image F2 roughly coincides with the X-coordinate of the control point, regardless of the arm angle and the rotation mechanism 220 angle. Therefore, it is advisable to overlay a guideline GL, which coincides with the X-coordinate of the tip of the second arm 120 on image F2, onto image F2. The guideline GL indicates the center position of the end effector 400. In Figure 10, the end effector 400 is shown as a dashed line and overlaid on image F2 to represent its position. The user specifies the direction in which the tip of the second arm 120 should face by pointing to any position on image F2[j] displayed on the display device. In the upper part of Figure 10, the cross displayed overlaid on the target TG represents the position pointed to by the user. The user only needs to issue a rotation command for the rotation mechanism 220 with the goal of aligning the target TG and the guideline GL. Thus, operation becomes easier for the user.

[0070] The control unit 700 controls the rotation mechanism 220 to orient the tip of the second arm 120 to a specified position based on the touched position on image F2[j]. For ease of understanding the technology, in image F2[j], the X coordinate of the tip of the second arm 120 and the X coordinate of the control point of the robot 100b are assumed to coincide. A known algorithm may be used to control the rotation of the rotation mechanism 220.

[0071] For example, the control unit 700 may use an object tracking algorithm to control the rotation mechanism 220 based on the touched position on image F2[j]. Object tracking algorithms include the feature-point-based ORB-SLAM (Raul Mur-Artal, et al., “ORB-SLAM: A Versatile and Accurate Monocular SLAM System.”, IEEE Transactions on Robotics, vol. 31, no. 5, pp. 1147-1163, October 2015., [online], [Retrieved January 10, 2025], Internet URL: https: / / doi.org / 10.1109 / TRO.2015.2463671) and the direct method DTAM (Richard A. Newcombe, et al., “DTAM: Dense Tracking and Mapping in Real-Time”, IEEE Xplore, [online], [Retrieved January 10, 2025], Internet) Examples include URL: https: / / doi.org / 10.1109 / ICCV.2011.6126513). In this embodiment, the user only needs to point to any position on the image displayed on the display device to indicate the target location. The control unit 700 can use an object tracking algorithm to direct the direction of the second arm toward the direction of the indicated target location in real space in response to the user's operation instructions. The user can rotate the rotation mechanism 220 with more intuitive and simple operation instructions.

[0072] Since the rotation axes J1 and J2 are parallel, in an embodiment without the rotation mechanism 220, the operation of the robot system is limited to a two-dimensional plane. In this embodiment, the base 210 can be rotated around a rotation axis J3 perpendicular to rotation axis J1. Therefore, the range of motion of the first arm 110 and the second arm 120 can be extended to three-dimensional space.

[0073] B2. Variations: Figure 11 is an explanatory diagram showing the schematic configuration of robot 100b-2 according to a modified example of the second embodiment. The following description will focus on the configuration that differs from the basic configuration of the second embodiment described above. Robot 100b-2 includes a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 that supports the first arm 110 and the second arm 120, a rotation mechanism 220, a mark 310, an end effector 400, a camera 510, a camera 520, and in addition, a camera 530, a first guide light irradiation unit 610, and a second guide light irradiation unit 620. Note that robot 100b-2 does not necessarily have to include the rotation mechanism 220.

[0074] In a modified example of the second embodiment, for instance, the robot 100 is made to place the target TG, which is being held by the end effector 400, onto the mounting platform T1.

[0075] Camera 530 is fixed to the second arm 120 in a position and orientation that allows it to image the end effector 400. Camera 530 images the end effector 400 at a constant frame rate. As shown in Figure 11, camera 530 is positioned closer to the end effector 400 than cameras 510 and 520. For example, a board camera can be used as camera 530. Camera 530 outputs the captured image directly to the control unit 700. Camera 530 is also called the "third camera".

[0076] Figure 12 is an explanatory diagram showing an example of an image captured by camera 530. In Figure 12, the target TG held by the end effector 400 is not shown. The image captured by camera 530 is denoted as image F3[k]. The subscript k is a positive integer that is added each time imaging is performed. The control unit 700 displays image F3[k] captured by camera 530 on a display device (not shown). The user can confirm the end effector 400 from the image displayed on the display device. Camera 510 is positioned to capture mark 310, so camera 510 is located a certain distance from the robot. For this reason, it may be difficult to confirm the state of the end effector 400 in detail from the image captured by camera 510. In contrast, camera 530 is positioned close to the end effector 400, so the user can confirm the state of the end effector 400 and the direction of movement of the second arm 120 in detail from the image captured by camera 530.

[0077] As shown in Figure 11, the first guide light irradiator 610 is fixed to the second arm 120. For example, a line laser that projects a linear laser beam can be used as the first guide light irradiator 610. As shown in Figure 12, the first guide light irradiator 610 irradiates a linear guide light GL1 in the direction of travel of the second arm 120. Here, when the arm is viewed from the side, the guide light GL1 is positioned so as to lie on a virtual plane S1. More specifically, when the arm is viewed from the side, the guide light GL1 spreads to some extent in a direction intersecting the direction of light propagation on the virtual plane S1. At least a part of the virtual plane S1 can be visually perceived by the guide light GL1. The first guide light irradiator 610 is also called the "first irradiator".

[0078] As shown in Figure 11, the second guide light irradiator 620 is fixed near the end of the second arm 120 to which the end effector is connected. For example, a point light source LED can be used as the second guide light irradiator 620. The second guide light irradiator 620 irradiates light at a predetermined position relative to the end effector 400. In the illustrated example, the second guide light irradiator 620 irradiates light directly below the end effector 400. The second guide light irradiator 620 is also called the "second irradiator".

[0079] Figure 13 is an explanatory diagram showing an example of an image F[i] captured by the camera 510 when the first guide light emitting unit 610 emits guide light and the second guide light emitting unit 620 emits light. As shown in Figure 13, the guide light GL1 emitted from the first guide light emitting unit 610 and the guide light GL2 emitted from the second guide light emitting unit 620 are projected onto the mounting table T1. If the end effector 400 has not reached the mounting table T1, the light emitted from the second guide light emitting unit 620 is projected onto the floor instead of the mounting table T1.

[0080] The user can confirm the object onto which the guide light GL1 is projected by the image F[i] captured by the camera 510. The guide light GL1 allows the user to visually confirm the reachable range of the end effector 400's operating point. For example, the end effector 400, which is a gripper, may be gripping an object, and the robot may be operated to place the gripped object onto a platform. In this case, the user can use the light emitted from the first guide light irradiator 610 onto the platform as a guide, as shown by image F[i], to specify the position where the object should be placed. Furthermore, the user can easily determine whether the end effector 400 has reached the platform T1 by whether or not the light emitted from the second guide light irradiator 620 reaches the platform.

[0081] As shown in Figure 13, when the end effector 400 reaches the mounting platform T1, the user may, for example, specify the final target coordinates by pointing to any position on the image F[i] displayed on a display device (not shown). In the illustrated example, the currently specified target point is represented by a solid cross, and the newly specified final target point is represented by a dashed cross. In the illustrated example, the target point is specified considering the offset between mark 310 and target TG.

[0082] Figure 14 is an explanatory diagram relating to the sensor provided at the tip of the second arm 120. Figure 14 shows the tip of the end effector 400 and the second arm 120 in a state where the target TG, an apple, is being grasped. The left side of Figure 14 shows the state in which the target TG being grasped by the end effector 400 is not in contact with the mounting platform. The right side of Figure 14 shows the state in which the target TG being grasped by the end effector 400 is in contact with another object on the mounting platform.

[0083] The end effector 400 comprises an end-effector HN for gripping the target TG, a main body BD to which the end-effector HN is attached, a hinge joint HJ, and an extension part EX. The extension part EX extends from the main body BD along the direction in which the second arm 120 extends. The extension part EX is positioned opposite the lower surface of the tip of the second arm 120. The robot 100b-2 further includes a pressure sensor PS. The end effector 400 is connected to the tip of the second arm 120 by the hinge joint HJ. As the hinge joint HJ rotates, the end-effector HN, the main body BD, and the extension part EX rotate around the rotation axis J4. The rotation axis J4 is parallel to the rotation axis J1. The pressure sensor PS is positioned between the extension part EX and the lower surface of the tip of the second arm 120. The pressure sensor PS is connected to the control unit 700 by a signal line (not shown). The value detected by the pressure sensor PS is transmitted to the control unit 700.

[0084] As shown in the left part of Figure 5, when the target TG held by the end effector 400 is not in contact with the mounting table, the extension part EX does not rotate around the rotation axis J4. When the extension part EX is not rotating around the rotation axis J4, pressure is applied from the extension part EX to the pressure sensor PS. As shown in the right part of Figure 5, when the target TG held by the end effector 400 comes into contact with the mounting table, the extension part EX rotates around the rotation axis J4. When the extension part EX rotates around the rotation axis J4, the extension part EX moves away from the pressure sensor PS. Therefore, the pressure applied from the extension part EX to the pressure sensor PS decreases. The control unit 700 determines whether or not the target TG held by the end effector 400 has come into contact with the mounting table based on the change in the detected value detected by the pressure sensor PS. If the control unit 700 determines that the target TG being gripped by the end effector 400 has come into contact with the mounting table, it may instruct the end effector 400 to release the target TG. If the end effector 400 is instructed to release the target TG when the target TG being gripped by the end effector 400 is not in contact with the mounting table, the target TG will fall and be subjected to impact. By using a pressure sensor PS, the target TG can be prevented from falling. It can also prevent the target TG from being pressed too hard against the mounting table and being damaged.

[0085] C. Third Embodiment: C1.Basic configuration: Figure 15 is an explanatory diagram showing the schematic configuration of the robot system 10c according to the third embodiment. The following description will focus on the configurations that differ from the first embodiment, and the description of the configurations that are the same as those in the first embodiment will be omitted.

[0086] The robot system 10c comprises a robot 100c, a control unit 700, and a terminal device 900. Note that the terminal device 900 is not shown in Figure 17. The robot 100c comprises a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 that supports the first arm 110 and the second arm 120, a lifting mechanism 230, a mark 310, an end effector 400, and a camera 510.

[0087] A characteristic configuration in this embodiment is that the robot 100c is equipped with a lifting mechanism 230. A base 210 is positioned on the lifting mechanism 230. Together with the base 210, the first arm 110 and the second arm 120 are positioned on the lifting mechanism 230. For example, the control unit 700 displays a user interface for receiving instructions to raise or lower the base 210 on a display device (not shown). When the control unit 700 receives an instruction to raise or lower via the user interface, it outputs a signal to the lifting mechanism 230 to raise or lower it. In response, the lifting mechanism 230 raises or lowers the base 210.

[0088] The lifting mechanism 230 allows the first arm 110 and the second arm 120 to move vertically, expanding the vertical reach of the control point without changing the length of each arm. The camera 510 is fixed to the base 210 and moves up and down with the base 210. Even if the base 210 moves up or down, the position and orientation of the camera 510 with respect to the virtual plane S1 are maintained. Therefore, similar to the first embodiment, the control unit 700 can control the operation of the robot 100 using so-called visual servos based on the image information acquired by the camera 510.

[0089] C2. Variant: The robot 100c may also include the rotating mechanism 220 and camera 520 described in the second embodiment. In this case, the rotating mechanism 220 and base 210 may be arranged on top of the lifting mechanism 230, or the lifting mechanism 230 may be arranged on top of the rotating mechanism 220 and the base 210 may be arranged on top of the lifting mechanism 230. With this configuration, the range of motion of the first arm 110 and the second arm 120 can be extended into three-dimensional space, and by moving the first arm 110 and the second arm 120 in the vertical direction, the range of motion of the control points in the vertical direction can be expanded without changing the length of each arm.

[0090] Furthermore, the robot 100c may also include the first guide light irradiation unit 610, the second guide light irradiation unit 620, and the camera 530 described in the second embodiment.

[0091] D. Fourth Embodiment: D1.Basic configuration: Figure 16 is an explanatory diagram showing the schematic configuration of the robot system 10d according to the fourth embodiment. The following description will focus on the configurations that differ from the first embodiment, and the description of the configurations that are the same as those in the first embodiment will be omitted.

[0092] The robot system 10d comprises a robot 100d, a control unit 700, and a terminal device 900. Note that the terminal device 900 is not shown in Figure 16. The robot 100d comprises a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 supporting the first arm 110 and the second arm 120, a rotation mechanism 220, a lifting mechanism 230, a travel mechanism 240, a mark 310, an end effector 400, a camera 510, a camera 520, a camera 530, a first guide light emitting unit 610, and a second guide light emitting unit 620. The configurations of the rotation mechanism 220, camera 520, camera 530, first guide light emitting unit 610, and second guide light emitting unit 620 are as described in the second embodiment. The configuration of the lifting mechanism 230 is as described in the third embodiment.

[0093] A characteristic configuration in this embodiment is that the robot 100d is equipped with a travel mechanism 240. A lifting mechanism 230, a rotating mechanism 220, and a base 210 are arranged on the travel mechanism 240 in that order. The first arm 110 and the second arm 120 are positioned on the travel mechanism 240 together with the base 210. The travel mechanism 240 moves the base 210 according to the control of the control unit 700. The control unit 700 is also called the "travel control unit".

[0094] The driving mechanism 240 includes a pair of drive wheels 241, driven wheels 242, a pair of motors that individually drive the pair of drive wheels 241, and a camera 540. Note that the motors are not shown in Figure 16.

[0095] The travel mechanism 240 can move in a straight line and turn. The travel mechanism 240 may also have a function to perform a so-called pivot turn, which involves turning in place by making one of the pair of motors rotate in the opposite direction to the other. For example, when the front of the base 210 is facing the mounting platform, the travel mechanism 240 may perform a pivot turn. This has the advantage of minimizing the distance between the robot and the target TG. Furthermore, when operating the robot, rotation around the rotation axis J3 by the rotation mechanism 220 may be performed from the viewpoint of improving work efficiency. This is because it is expected that the execution of a pivot turn by the travel mechanism 240 will result in a waiting time compared to the execution of rotation by the rotation mechanism 220.

[0096] The driving mechanism 240 may have a function to perform a pivot turn, where it turns around the wheel on the side that is stopped by stopping one of the pair of motors and driving the other. The driving mechanism 240 may also have a function to perform a gentle turn, where it turns slowly by making the rotation speed of one of the pair of motors different from the rotation speed of the other. The configuration of the driving mechanism 240 is not limited to the configuration described above. The housing or base 210 of the driving mechanism 240 may be provided with a rear marker to identify the rear of the driving mechanism 240. The housing of the driving mechanism 240 may be provided with proximity sensors, tactile sensors, collision detection sensors, etc.

[0097] Camera 540 is fixed to the housing of the travel mechanism 240 in a position and orientation that allows it to capture images of the direction of travel of the travel mechanism 240. Camera 540 is also called the "forward camera". Camera 540 captures images of the direction of travel of the travel mechanism 240 at a constant frame rate. The mounting position of camera 540 is arbitrary as long as it can capture images of the direction of travel of the travel mechanism 240. Camera 540 transmits the images acquired by capturing images of the direction of travel of the travel mechanism 240 to the control unit 700. The image captured by camera 540 is denoted as image F4[l]. The subscript l is a positive integer that is added each time imaging is performed. The control unit 700 displays image F4[l] on a display device (not shown) located at a remote user via an internet communication network or the like. The user can move the robot 100d-a to a desired location by operating the travel mechanism 240, such as moving it forward, backward, or turning, while judging the situation by looking at image F4[l] displayed on the display device. Suitable methods for the user to control the driving mechanism 240 include virtual buttons displayed on a display device, mouse gestures, swiping and gestures on a touch panel, and the mouse wheel. Alternatively, the control unit 700 may be equipped with an object tracking algorithm to automatically control the driving mechanism 240 based on the image F4[l].

[0098] Object tracking algorithms include ORB-SLAM, a feature point-based method, and DTAM, a direct method. As the mobile mechanism 240 moves, the image F4[l] of the camera 540 changes rapidly, but by using an object tracking algorithm, the positions of various objects in the image F4[l] can be continuously tracked. Using this, the camera 540, i.e., the mobile mechanism 240, can be positioned directly in front of any desired location and autonomously controlled to approach a preset distance. This makes the operation more intuitive and convenient for the user, as they can simply point to any location on the display screen to indicate a target location for movement, and the robot system 10d will semi-automatically move to the indicated target location in real space.

[0099] D2. Variant: Figure 17 is an explanatory diagram showing the schematic configuration of a robot system 10g according to a modified example of the fourth embodiment. The following description will focus on configurations that differ from the basic configuration of the fourth embodiment described above, and will omit descriptions of configurations similar to the basic configuration.

[0100] The robot system 10g includes, in addition to the configuration described in the basic configuration above, a relay unit 750 and a plurality of cameras 550. The relay unit 750 has the configuration described in the first embodiment.

[0101] Multiple cameras 550 capture images of the robot 100d from the outside. Cameras 550 are also called "external cameras". The multiple cameras 550 are positioned to capture images within the range in which the robot 100d is expected to travel. For example, if the robot 100d travels inside a warehouse, the multiple cameras 550 are fixed to the ceiling or walls of the warehouse. Each camera 550 continuously captures images and transmits the captured images to the control unit 700 via wireless communication. The cameras 550 and the control unit 700 may communicate wirelessly directly or via a relay unit 750. The image captured by the camera 550 is denoted as image F5[n]. The subscript n is a positive integer that is added each time imaging is performed.

[0102] The control unit 700 displays images F5[n] received from multiple cameras 550 on the display device of the terminal device 900. The user can detect obstacles around the robot 100d and recognize the surrounding environment. In some cases, it may be difficult to fully recognize surrounding obstacles and conditions using only the image F4[l] acquired by the camera 540 installed on the travel mechanism 240. Even in this case, the user can use multiple cameras 550 to perform highly accurate monitoring of the robot 100d's movement. The user can judge the situation by looking at the image F5[n] displayed on the display device of the terminal device 900 and move the robot 100d to the desired location by operating the travel mechanism 240 to move forward, backward, or turn. Alternatively, position markers may be provided in the image captured by the camera 550 to identify the position of the travel mechanism 240. The travel markers are composed of, for example, point light source LEDs. It is also preferable that the position markers be provided in front of and behind the travel mechanism 240. This makes it easy for the user to identify the front of the travel mechanism 240 in the image F5[n] captured by the camera 550. The control unit 700 may have a function to detect position markers based on the image F5[n]. When the user specifies a destination point, the control unit 700 uses an object tracking algorithm to control the travel mechanism 240 to move toward the specified destination point. In this case, the user does not need to give sequential instructions regarding movement. The user can move the robot 100h to the desired location simply by specifying the target point by pointing at a desired position on the image. The control unit 700 uses a visual servo to control the movement of the travel mechanism 240 each time the camera 550 acquires image F5, so as to reduce the difference between the coordinates indicating the position markers in image F5 and the coordinates indicating the specified target point. In image F5, robot 100h reaches the target location when the coordinates indicating the position marker and the coordinates indicating the target location coincide.

[0103] Furthermore, the control unit 700 may, for example, estimate the three-dimensional shape of the robot 100d's surroundings based on the image F4[l] acquired by camera 540 and the images F5[n] acquired by each camera 550 using SfM (Structure from Motion). This makes it possible to move the driving mechanism 240 while avoiding surrounding obstacles.

[0104] E. Fifth Embodiment: E1.Basic configuration: Figure 18 is an explanatory diagram showing the schematic configuration of the robot system 10e according to the fifth embodiment. The following description will focus on the configurations that differ from the first embodiment, and the description of the configurations that are the same as those in the first embodiment will be omitted.

[0105] The robot system 10e comprises a robot 100e and a control unit 700. The robot 100e comprises a first arm 110, a second arm 120, a joint 130, a joint 140, a base 210 supporting the first and second arms 120, a mark 311, a mark 312, an end effector 400, and a camera 510.

[0106] A characteristic feature of this embodiment is that the robot 100e has two marks. Marks 311 and 312 are configured similarly to mark 310 in the first embodiment. Marks 311 and 312 are attached near the end of the first arm 110 on the side to which the end effector 400 is attached. Mark 312 is positioned further from the end effector 400 than mark 311 is positioned.

[0107] Figure 19 is an explanatory diagram showing an example of an image F[i] captured by camera 510. Marks 311 and 312 are provided to define the control point TCP of robot 100e. In this embodiment, the control point TCP of robot 100e is set at the position obtained by extending the straight line connecting the center of mark 311 and the center of mark 312. Marks 311 and 312 are positioned such that the control point TCP of robot 100e, the center of mark 311, and the center of mark 312 are all located on the same straight line. Let N be the distance between the center of the mark 311 closer to the control point TCP and the control point TCP of robot 100e, and let M be the distance between the center of mark 311 and the center of mark 312. Distances M and N are assumed to be known. The positional relationship between marks 311, 312, and the control point TCP and camera 510 in the depth direction as viewed from camera 510 is assumed to be adjusted as appropriate.

[0108] In this embodiment, the control unit 700 can easily calculate the coordinates (x[i], y[i]) of the control point TCP in image F[i] using the coordinates (x_m1, y_m1) of the center of mark 311 and the coordinates (x_m2, y_m2) of the center of mark 312 in image F[i] as follows.

[0109] x[i]=x_m1[i]+(x_m1[i]-x_m2[i])×N / M y[i]=y_m1[i]+(y_m1[i]-y_m2[i])×N / M

[0110] Thus, in this embodiment, the coordinates (x_m1[i], y_m1[i]) of the center of mark 311 and the coordinates (x_m2[i], y_m2[i]) of mark 312 in image F[i] can be easily obtained.

[0111] In the first embodiment, an example was described in which the mark 310 is positioned close to the control point so that its position can be considered equivalent to the position of the control point of the robot 100. However, because the light-shielding plate SP is provided, the mark 310 has a certain size. Therefore, when controlling the robot's movement, it is necessary to specify the target point by considering the distance between the control point and the center position of the mark 310 as an offset. To make the offset zero, it is conceivable to directly place the mark 310 on the end effector 400. However, if the mark 310 is directly attached to the end effector 400, the mark 310 becomes an obstruction and the end effector 400 cannot be seen. In this embodiment, the position of the control point is determined using two physical markers attached to the second arm and a known distance between one of the two markers and the control point. Therefore, it is possible to place the two marks at a distance from the end effector 400.

[0112] E2. Variation 1: Figure 20 is an explanatory diagram of an algorithm for detecting the coordinates of the light-emitting LD within image F[i]. The left side of Figure 20 shows the portion of the captured image F[i] that contains either mark 311 or mark 312. To facilitate understanding of the technology, the luminance values ​​of the respective light-emitting LD portions of marks 311 and 312 are assumed to be the highest within image F[i]. Based on the luminance value of each pixel in image F[i], fixed gaze regions R1 and R2, which include each light-emitting LD, are set. The gaze regions R1 and R2 have a size of, for example, 8x8 pixels and are movable within image F[i]. The right side of Figure 20 shows the gaze region R1 or R2 having a size of 8x8 pixels.

[0113] Here, it is assumed that each mark is initially located within its respective gaze area. The control unit 700 calculates the luminance centroid, which is the centroid of the luminance value of each pixel within gaze area R1. In the next received image F[i+1], the control unit 700 continues to capture mark 311 by shifting the coordinates of gaze area R1 so that the luminance centroid is located in the center of gaze area R1. The capture of mark 312 is done similarly. The coordinates of each mark are the sum of the coordinates of each gaze area and the coordinates of the luminance centroid within each gaze area. With the above method, the calculation target can be limited to gaze areas R1 and R2, thus reducing the processing load for capturing marks 311 and 312. However, if the apparent size of the marks is not constant with respect to the sizes of gaze areas R1 and R2, errors will occur in the calculation results, and capture will fail. Since the sizes of gaze areas R1 and R2 are constant, it is necessary to keep the apparent size on image F of the light-emitting unit LD constant. However, as the first arm 110 and the second arm 120 of the robot 100e move, the distance between the control point TCP and the camera 510 changes. Therefore, if the light emission intensity of each light-emitting part LD of mark 311 and mark 312 remains constant, the apparent size of mark 311 and mark 312 in the captured image F[i] will always fluctuate. The control unit 700 then adjusts the light emission intensity of each light-emitting LD so that the apparent size of marks 311 and 312 in the captured image F[i] falls within a predetermined range. The light emission intensity of the LEDs constituting the light-emitting LDs is proportional to the current flowing through the LEDs. The control unit 700 may increase the current flowing through each light-emitting LD of marks 311 and 312 as the distance between marks 311 and 312 and the camera 510 increases, thereby increasing the light emission intensity of each light-emitting LD. Also, the light emission intensity of an LED is proportional to the time the current flows. The control unit 700 may increase the time the current flows through each light-emitting LD of marks 311 and 312 as the distance between the camera 510 and marks 311 and 312 increases, thereby increasing the light emission intensity of each light-emitting LD. Alternatively, the exposure time of the camera 510 may be increased to increase the intensity of the light received by the image sensor of the camera 510. This control method ensures that the size of the light-emitting LD of mark 311 in the captured image remains constant, even when the distance between camera 510 and marks 311 and 312 increases.

[0114] E3. Modification 2: The robot 100e may also include the rotation mechanism 220 and camera 520 described in the second embodiment. With this configuration, the range of motion of the first arm 110 and the second arm 120 can be extended into three-dimensional space.

[0115] The robot 100e may include the lifting mechanism 230 described in the third embodiment. In this case, the rotation mechanism 220 and the base 210 may be arranged on top of the lifting mechanism 230, or the lifting mechanism 230 may be arranged on top of the rotation mechanism 220 and the base 210 may be arranged on top of the lifting mechanism 230. With such a configuration, the range of motion of the first arm 110 and the second arm 120 can be extended into three-dimensional space, and the first arm 110 and the second arm 120 can be moved in the vertical direction.

[0116] The robot 100e may also include a first guide light irradiator 610, a second guide light irradiator 620, and a camera 530, similar to the second embodiment.

[0117] The robot system 10e may also include a travel mechanism 240, a camera 540, and a plurality of cameras 550, similar to the fourth embodiment.

[0118] F. Sixth Embodiment: F1.Basic configuration: Figure 21 is an explanatory diagram showing an example of an image F[i] captured by camera 510 of robot 100f according to the sixth embodiment. Robot 100f has the same configuration as robot 100e according to the fifth embodiment. In the fifth embodiment, a method for calculating the coordinates of control points TCP in an image based on the coordinates of the centers of two marks in the image was described. In this disclosure, the movement of the robot is controlled based on the difference between the coordinates of the control points and the target coordinates. In E2. Modification 1 of the fifth embodiment, a method for calculating the coordinates of each mark was described in which gaze regions R1 and R2 were provided, and the marks were continuously tracked by applying feedback to the coordinates of each gaze region so that the luminance centroid within each gaze region was at the center of each gaze region, and the coordinates of each gaze region and the luminance centroid within each gaze region were added at that time. In this method, the luminance centroids of each gaze region and each mark must overlap at least in the initial state. Therefore, it is necessary to accurately determine the initial coordinate values ​​of marks 311 and 312 by some other method. The following describes a method for accurately determining the initial coordinate values ​​of marks 311 and 312.

[0119] Figure 22 is a flowchart of the process for determining the initial positions of marks 311 and 312. The process shown in Figure 22 is performed before starting the processes shown in Figures 4 and 5. Note that robot 100f must remain stopped while the following processes are being performed.

[0120] In step S201, the control unit 700 generates a first luminance image based on the image F[i] captured by the camera 510 with the light-emitting units LDs of marks 311 and 312 in the off state. Specifically, first, the control unit 700 controls the light-emitting units LDs of marks 311 and 312 so that they are in the off state. Furthermore, the control unit 700 causes the camera 510 to perform imaging. The control unit 700 generates a luminance image based on the image F[i] supplied from the camera 510. A luminance image is an image that contains only brightness information. If the image supplied from the camera 510 is an RGB image, the control unit 700 calculates the luminance value using the RGB values ​​of each pixel in image F[i] and a known calculation formula, and generates a first luminance image LI1 representing the luminance value for each pixel.

[0121] In step S202, the control unit 700 generates a second luminance image based on an image captured with the light-emitting units LDs of mark 311 and mark 312 turned on. Specifically, first, the control unit 700 controls the light-emitting units LDs of mark 311 and mark 312 so that they are turned on. Furthermore, the control unit 700 causes the camera 510 to perform imaging. The control unit 700 generates a luminance image based on the image F[i+1] supplied from the camera 510. Similar to step S201, the control unit 700 calculates the luminance value using the RGB values ​​of each pixel in image F[i+1] and a known calculation formula, and generates a second luminance image LI2 representing the luminance value for each pixel.

[0122] In step S203, the control unit 700 generates a difference luminance image LI3, which is the difference between the first luminance image LI1 and the second luminance image LI2. Specifically, the control unit 700 calculates the difference between the luminance value of the first luminance image LI1 and the luminance value of the second luminance image LI2 for each pixel, and generates a difference luminance image LI3 that represents the difference in luminance values ​​for each pixel. Furthermore, the control unit 700 performs image filtering on the difference luminance image LI3. Image filtering includes, for example, noise reduction filtering and binarization filtering.

[0123] In step S204, the control unit 700 creates an X-axis luminance profile and a Y-axis luminance profile based on the filtered differential luminance image LI3. The X-axis luminance profile is the sum of the luminance values ​​of each pixel with the same X coordinate along the Y axis of the differential luminance image LI3. The X-axis luminance profile represents a one-dimensional distribution of luminance along the X axis. The Y-axis luminance profile is the sum of the luminance values ​​of each pixel with the same Y coordinate along the X axis of the differential luminance image LI3. The Y-axis luminance profile represents a one-dimensional distribution of luminance along the Y axis.

[0124] In step S205, the control unit 700 determines whether there are exactly two peaks in the luminance profile in the X-axis direction and exactly two peaks in the luminance profile in the Y-axis direction. For example, the control unit 700 determines a peak in the luminance profile in the X-axis direction if the luminance is above a predetermined threshold. The same applies to the luminance profile in the Y-axis direction.

[0125] The control unit 700 executes the process in step S206 if there are exactly two peaks in the luminance profile in the X-axis direction and exactly two peaks in the luminance profile in the Y-axis direction (step S205; YES). On the other hand, if the above conditions are not met (step S205; NO), the control unit 700 executes the process from step S201 onwards again.

[0126] In step S206, the control unit 700 calculates the coordinate values ​​of mark 311 and mark 312. Specifically, of the two peaks included in the luminance profile in the X-axis direction, the control unit 700 sets the X coordinate of the peak with the smaller X coordinate value as the X coordinate of the center of mark 311, x_m1_In, and the X coordinate of the peak with the larger X coordinate value as the X coordinate of the center of mark 312, x_m2_In. Of the two peaks included in the luminance profile in the Y-axis direction, the control unit 700 sets the Y coordinate of the peak with the smaller Y coordinate value as the Y coordinate of mark 311, y_m1_In, and the Y coordinate of the peak with the larger Y coordinate value as the Y coordinate of mark 312, y_m2_In. It is assumed that the approximate positional relationship between mark 311 and mark 312 (see Figure 21) is known.

[0127] In step S207, the control unit 700 sets the X coordinate x_m1_In and Y coordinate y_m1_In of mark 311 as the center coordinates of the gaze region R1 for mark 311. Furthermore, the control unit 700 sets the X coordinate x_m2_In and Y coordinate y_m2_In of mark 312 as the center coordinates of the gaze region R2 for mark 312. The gaze regions are as described in the fifth embodiment (see Figure 20). After that, the process shown in Figure 22 is completed.

[0128] As described above, in this embodiment, the initial positions of the two marks can be easily identified within the image captured by the first camera. By accurately identifying the initial positions of the two marks, the two marks can be correctly captured in a series of images taken over time.

[0129] F2. Variation 1: In the example shown in Figure 22, assuming that the approximate positional relationship between marks 311 and 312 is known, images were acquired with both marks 311 and 312 turned on simultaneously.

[0130] However, the approximate positional relationship between marks 311 and 312 may be unknown. In such cases, the control unit 700 generates a first luminance image based on an image taken with the light-emitting LDs of marks 311 and 312 turned off. A fourth luminance image is generated based on an image taken with only mark 311 turned on. A fifth luminance image is generated based on an image taken with only mark 312 turned on. The control unit 700 can determine the coordinates of mark 311 based on the difference between the first luminance image and the fourth luminance image using the method described above. The control unit 700 can determine the coordinates of mark 312 based on the difference between the first luminance image and the fourth luminance image using the method described above.

[0131] F3. Variation 2: Robot 100f may also include a rotating mechanism 220 and a camera 520, similar to the second embodiment. Robot 100f may also include a first guide light irradiator 610, a second guide light irradiator 620, and a camera 530.

[0132] The robot 100f may also include a lifting mechanism 230, similar to the third embodiment. The robot system may also include a travel mechanism 240, a camera 540, and multiple cameras 550, similar to the fourth embodiment.

[0133] G. Seventh Embodiment: G1: Basic configuration: The robot system according to the seventh embodiment has the same configuration as the fifth embodiment (see Figure 18). A characteristic feature of this embodiment is that the control unit 700 automatically executes control to return the robot's control point TCP to a pre-set return point when a pre-set condition is met. Figure 23 is an explanatory diagram showing an example of the coordinates of the return point. Figure 24 is an explanatory diagram of the arm's operation. Here, we assume the operation of placing a target on a shelf with three shelves. The return point is set as the point to which the control point TCP should return. Examples of pre-set conditions include when the target TG is successfully grasped, when the grasped target TG is released, or when the user explicitly issues a return command. The control to return to the return point is to automatically specify the coordinates of the return point as a new target coordinate. The operation after the new target coordinate is specified is the same as in the first embodiment.

[0134] In this embodiment, control is performed to return the control point to its return point under pre-set conditions. In the mode where the control point is not returned to its return point, the control point will be moved from its current position, such as the previously specified target point, to a new target point. This can cause the robot to repeatedly assume unnatural postures and result in wasted movement. The mode in which the control point is automatically returned to its return point can suppress the occurrence of such problems.

[0135] H. Eighth Embodiment: Figure 25 is an explanatory diagram showing the schematic configuration of the robot system 10h according to the eighth embodiment. The following description will focus on the configurations that differ from the first embodiment, and the description of the configurations that are the same as those in the first embodiment will be omitted.

[0136] The robot system 10h comprises a robot 100h, multiple cameras 550, a control unit 700, a relay unit 750, a server 800, and a terminal device 900. The robot 100h has the same configuration as the robot 100d described in the fourth embodiment. The configuration similar to that of the fourth embodiment will not be described below.

[0137] The configuration of the multiple cameras 550 is the same as in the fourth embodiment. Let the image captured by the camera 550 be image F5[n]. The subscript n is a positive integer that is incremented each time imaging is performed.

[0138] The configuration of the control unit 700 is the same as in the fourth embodiment. The control unit 700 transmits captured images and various data indicating the status of the robot 100h to the server 800 via the relay unit 750. The images transmitted to the server 800 include images captured by cameras 510, 520, 530, 540, and 550.

[0139] The relay unit 750 transmits images F5[n] received from multiple cameras 550 to the control unit 700. The relay unit 750 is, for example, a Wi-Fi router that can access the Internet communication network. The relay unit 750 can communicate with the control unit 700 by wired or wireless communication. The relay unit 750 can communicate with the server 800 via the Internet communication network. Figure 25 illustrates an example in which the relay unit 750 is mounted on the robot 100h, but the relay unit 750 may be located in a different location from the robot 100h. The relay unit 750 transmits the captured images received from the control unit 700 to the server 800. The relay unit 750 transmits information such as operation instructions for the robot 100h and the coordinates of the target location received from the server 800 to the control unit 700.

[0140] Server 800 generates an image representing the operation UI using the captured image received from 700 via the relay unit 750. Server 800 transmits the generated image representing the operation UI to the terminal device 900. Server 800 transmits information such as operation instructions for robot 100h and the coordinates of the target point received from the terminal device 900 to the control unit 700.

[0141] The terminal device 900 can be anything that has at least an internet communication function and a pointing device and display. For example, a smartphone or tablet with a touch panel display, or a PC operated with a mouse. Alternatively, a mouse may be connected to a smartphone or tablet. The terminal device 900 displays an image representing the operation UI received from the server 800. The terminal device 900 and the server 800 are also called the "display unit". The terminal device 900 transmits to the server 800 information such as operation instructions for the robot 100h indicated by the operation instructions of user P1 to the operation UI, and the coordinates of the target point. The terminal device 900 and the server 800 are also called the "input receiving unit".

[0142] Figure 26 is an explanatory diagram showing an example of an operation UI displayed on the terminal device 900. As shown in Figure 26, the operation UI simultaneously displays images IM1 captured by camera 510, images IMs2 captured by camera 520, images IMs3 captured by camera 530, images IMs4 captured by camera 540, and images IMs5 and IMs6 captured by multiple cameras 550.

[0143] In the example shown in Figure 26, the image captured by camera 510 is displayed as the main image. The main image is the image displayed in a state where it can accept operation instructions. The main image is displayed larger than other images. In the illustrated example, the main image is displayed in the upper left corner of the screen. User P1 can operate robot 100h by pointing on the image selected as the main image. The control unit 700 controls the operation of robot 100h in response to pointing operations on image IM1. Operation instructions for image IM1 as the main image are also called "first screen operation instructions". Pointing operations are also called "operation instructions".

[0144] For example, a user can specify a new target coordinate by pointing to a desired position on the main screen. The robot 100h, as in the first embodiment, has a control unit that controls the movement of the arm to match the target coordinate.

[0145] As shown in the diagram, multiple virtual buttons may be displayed on the screen. For example, an upward arrow icon IC11, a downward arrow icon IC12, a leftward arrow icon IC13, and a rightward arrow icon IC14 may be displayed, and when user P1 points to any of icons IC11, IC12, IC13, or IC14, the target coordinates will move up, down, left, or right by a predetermined number of pixels, respectively. The user can easily fine-tune the target coordinates.

[0146] When the user presses the virtual "grab" button BT1, the control unit 700 instructs the end effector 400 to perform a grasping action in response to the user's command. When the user presses the virtual "release" button BT2, the control unit 700 instructs the end effector 400 to perform a release action in response to the user's command. The virtual buttons are particularly useful for users who are not familiar with the robot's operation.

[0147] In addition to virtual buttons, instructions can also be assigned to various operations such as clicking the secondary mouse button, mouse gestures, mouse wheel operations, or touch swipes, long presses, multi-finger touches, and multi-finger swipes on the screen. Examples of these instructions include slight movement of the target coordinates, rotation of the rotation axis J3, and opening and closing the gripping part. By using such mouse operations and touch operations on the screen, users skilled in robot operation can improve their work efficiency. Furthermore, the correspondence between each operation and the instructions for the robot can be customized by the user.

[0148] In the operation UI shown in Figure 26, the user can instruct the user to switch the main image by pointing to one of the sub-images IMs2 to IMs6. In this case, the terminal device 900 sends information to the server 800 indicating that the main image has been switched and identifying the selected image. The server 800 supplies the terminal device 900 with an image representing the operation UI with the selected image as the main image. Alternatively, the user may instruct the user to switch the main image by hovering the mouse pointer over one of the sub-images IMs2 to IMs6.

[0149] Figure 27 is an explanatory diagram showing another example of the operation UI displayed on the terminal device 900. In the example shown in Figure 27, the image IM2 captured by the camera 520 is displayed as the main image. For example, if the user performs a pointing operation to select image IMs2 in the state shown in Figure 26, the main image is switched and transitions to the state shown in Figure 27. The terminal device 900 and server 800, which function as input receiving units, can receive operation instructions for the main image and switching instructions to select one of the sub-images as the main image when an image captured by either camera is selected as the main image. Note that the manner in which the selected image is displayed as the main image is not limited to the examples shown in Figures 26 and 27. For example, the image selected as the main image may be enlarged and displayed at that position. That is, the selected image is displayed as an overlay on top of other images. In this case, if the mouse pointer is moved outside the display range of the main image, the state may be returned to, for example, the default image being displayed as the main image. Images IM1 to IMs6 do not necessarily have the same aspect ratio, but by displaying the selected main image as an overlay, no extra blank space appears on the screen.

[0150] As shown in Figure 27, when the image IM2 captured by the camera 520 is displayed as the main image, the user can rotate the rotation axis J3 to control the direction of the arm, as described in the second embodiment. Operation instructions for the image IM2 displayed as the main image are also called "second screen operation instructions".

[0151] The control unit 700 may execute processing in movement mode when the user issues a predetermined operation instruction. For example, the control unit 700 may start executing processing in movement mode in response to the user performing a pointing operation to select an image IMs4. In this case, although not shown, the selected image IMs4 may be displayed as the main image. The image IMs4, as the main image, has an upward arrow icon IC11, a downward arrow icon IC12, a left-pointing arrow icon IC13, and a right-pointing arrow icon IC14 superimposed on it. In movement mode, the movement of the travel mechanism 240 is controlled as described in the fourth embodiment. The user can make the travel mechanism 240 move forward, backward, turn left, or turn right by pointing to any of the icons IC11, IC12, IC13, or IC14, respectively. Alternatively, the travel mechanism 240 can be operated by mouse gestures, touch panel swiping, etc. When an operation command is issued to scroll the mouse tilt wheel horizontally, the control unit 700 can cause the travel mechanism 240 to perform a pivot turn. When an operation command is issued to scroll the mouse tilt wheel vertically, the control unit 700 can cause the travel mechanism 240 to move forward. Operation commands for image IMs4, which is displayed as the main image, are also called "fourth screen operation commands". Operation commands for image IMs3, which is displayed as the main image, are also called "third screen operation commands".

[0152] In response to a user performing a pointing operation to select image IMs5 or image IMs6, the control unit 700 may start executing processing in the movement mode. In this case, although not shown, the selected image IMs5 or image IMs6 may be displayed as the main image. The main image IMs5 or image IMs6 is overlaid with an upward arrow icon IC11, a downward arrow icon IC12, a leftward arrow icon IC13, and a rightward arrow icon IC14. In the movement mode, the movement of the travel mechanism 240 is controlled as described in the fourth embodiment. The user, while viewing image IMs5 or image IMs6 captured by the camera 550, points to, for example, icons IC11 to IC14 to move the travel mechanism 240 forward, backward, or turn to move the robot 100h to a desired location.

[0153] Furthermore, the travel mechanism 240 may be provided with position markers to identify its position. The control unit 700 shall have a function to detect the position markers based on images IMs5 or IMs6 captured by the camera 550. When the user specifies a destination point, the control unit 700 uses an object tracking algorithm to control the travel mechanism 240 to move toward the specified destination point. In this case, the user does not need to give sequential instructions regarding movement. The user can move the robot 100h to the desired location simply by pointing to the desired position on the image to specify the target point. The control unit 700 uses a visual servo to control the movement of the travel mechanism 240 so as to reduce the difference between the coordinates indicating the position marker in image F5 and the coordinates indicating the specified target point each time the camera 550 acquires image F5. When the coordinates indicating the position marker and the coordinates indicating the target point coincide in image F5, the robot 100h will reach the target point.

[0154] The operating UI may include a virtual button for emergency stopping the robot. When the virtual emergency stop button is pointed to, or when the mouse pointer is placed over the virtual emergency stop button, the power supplied to the robot is cut off. Alternatively, the power supplied to the robot may be cut off in response to the user tapping the operating UI multiple times with their finger.

[0155] In this embodiment, the terminal device 900 and the server 800 can receive operation instructions using one of the images captured by the camera. Therefore, the user can check the relationship between the control point and the target point while viewing different viewpoints and issue operation instructions. Furthermore, the terminal device 900 and the server 800 can receive operation instructions using one of the images captured by cameras 510 to 550. Therefore, the user can issue operation instructions while viewing different viewpoints.

[0156] This disclosure is not limited to the embodiments described above, and can be implemented in various configurations without departing from its spirit. For example, the technical features in the embodiments corresponding to the technical features in each form described in the summary of the invention can be replaced or combined as appropriate to solve some or all of the above-described problems, or to achieve some or all of the above-described effects. Furthermore, if a technical feature is not described as essential in this specification, it can be deleted as appropriate.

[0157] I. Other forms: (1) According to one embodiment of the present disclosure, a robot is provided. The robot includes a base, a first arm connected to the base and rotating around a first axis of rotation, a second arm connected to one end of the first arm and rotating around a second axis of rotation parallel to the first axis of rotation with respect to the first arm, at least one physical marker attached to the second arm for defining the control point of the robot, a camera fixed in position and orientation with respect to the base, positioned to capture images of the marker moving on a virtual plane, assuming a virtual plane perpendicular to the first axis of rotation, and capturing images of the first arm and the second arm at regular time intervals, and a control unit which, each time the first camera acquires an image, determines the robot from the marker in the image captured by the first camera. The robot comprises a control unit which determines the amount of rotational displacement of the first motor that rotates the first arm around the first rotation axis and the amount of rotational displacement of the second motor that rotates the second arm around the second rotation axis based on the difference between the coordinate value of the control point of the robot and the coordinate value of a designated target point, and controls the first motor and the second motor to rotate by the determined amount of displacement, and when a preset condition is met, each time the first camera acquires an image, it determines the amount of rotational displacement of the first motor and the amount of rotational displacement of the second motor based on the difference between the coordinate value of the control point of the robot in the image captured by the first camera and the coordinate value of a return point set as the point to which the robot should return, and controls the first motor or the second motor to rotate by the determined amount of displacement. In the above configuration, after the control point reaches the target point, control is performed to return the control point to its return point. In the configuration where the control point is not returned to its return point after reaching the target point, the control point will be moved from its current position, such as the previously specified target point, to a new target point. This can cause the robot to repeatedly assume unnatural postures and result in wasted movement. In the configuration where the control point is returned to its return point after reaching the target point, the occurrence of such problems can be suppressed.

[0158] (2) The robot of the above form further comprises a rotation mechanism that rotates the base around a third rotation axis perpendicular to the first rotation axis, according to the control of the control unit, and a second camera that captures the direction of travel of the second arm, and the second camera is fixed to the first arm or the second arm. The control unit may control at least the rotation mechanism to orient the direction of travel of the second arm toward a designated target point based on the image captured by the second camera, and may control the rotation mechanism so as not to rotate the first rotation axis and the second rotation axis while the rotation mechanism is rotating the base, and so as not to rotate the base while at least one of the first rotation axis and the second rotation axis is rotating. Since the first and second rotation axes are parallel, in embodiments without a rotation mechanism, the robot's movement is limited to a two-dimensional plane. According to the above embodiment, the base can be rotated around a third rotation axis perpendicular to the first and second rotation axes, thereby extending the range of motion of the first and second arms connected to the base into three-dimensional space.

[0159] (3) The robot of the above form further comprises a first illumination unit fixed to the second arm and irradiating guide light in the direction of movement of the second arm, the first camera being positioned to capture images of the area irradiated by the guide light, and the control unit may accept the designation of a target point by presenting the image captured by the first camera. According to the above configuration, the user can check the image captured where the guide light is illuminating, confirm the future direction of movement of the second arm represented by the guide light, and then specify the target point.

[0160] (4) In the robot of the above form, the robot further comprises a gripping part connected to the tip of the second arm and a second illumination part which is a light source that illuminates light at a predetermined positional relationship with the gripping part, and the first camera may be positioned in a location capable of capturing the light illuminated by the second illumination part. According to the above configuration, the user can easily determine whether or not there is a mounting platform at the location where the light is irradiated, based on an image captured of the light irradiated at a predetermined positional relationship with the gripping part.

[0161] (5) The robot of the above form may further include a third camera for imaging the gripping part.

[0162] (6) The robot in the above configuration may further include a lifting mechanism for raising and lowering the base. According to the above configuration, the lifting mechanism allows the first and second arms connected to the base to move vertically. Furthermore, since the first camera moves up and down together with the base, the position and orientation of the first camera with respect to a virtual plane perpendicular to the first rotation axis are maintained even if the base moves up or down.

[0163] (7) The robot of the above form further comprises a driving mechanism for driving the base. The driving mechanism may include a front camera that captures an image of the direction of travel of the driving mechanism, and a driving control unit that, upon receiving a driving instruction from the control unit, controls the driving of the driving mechanism using the image captured by the front camera. According to the above configuration, the first arm and the second arm connected to the base can be moved by the travel mechanism.

[0164] (8) In the robot of the above configuration, the travel control unit may control the movement of the travel mechanism using images captured by at least one external camera positioned to capture images within the range in which the robot is expected to travel. According to the above configuration, the robot's movement can be monitored with high accuracy using multiple external cameras.

[0165] (9) In the robot of the above embodiment, a link mechanism may be further provided that works in a direction to counteract the rotation of the second arm caused by the rotation of the first arm, by mechanically transmitting at least a portion of the power output by the motor that drives the first rotating shaft to the second arm. In a configuration without the linkage mechanism described above, for example, suppose only the first rotation axis is rotated without rotating the second rotation axis. In this case, the posture of the second arm changes significantly in conjunction with the rotation of the first arm. To prevent the posture of the second arm from changing significantly in conjunction with the rotation of the first arm, it is conceivable to rotate the first and second rotation axes simultaneously or with a very short time interval between them. However, in this case, calculations are required for the amount of rotation of the first rotation axis and the amount of rotation of the second rotation axis to offset the displacement of the posture of the second arm caused by the rotation of the first rotation axis. When the linkage mechanism described above is provided, the amount of change in the posture of the second arm in conjunction with the rotation of the first arm can be reduced. Therefore, in order to reduce the difference between the coordinate values ​​of the robot's control point and the coordinate values ​​of the target point in the image captured by the first camera, for example, it is only necessary to rotate the first arm to reduce the difference between the X coordinate of the robot's control point and the X coordinate of the target point, and to rotate the second arm to reduce the difference between the Y coordinate of the robot's control point and the Y coordinate of the target point, and repeat this process. Therefore, according to the above configuration, there is no need to perform calculations regarding the amount of rotation of the first rotation axis and the amount of rotation of the second rotation axis to offset the displacement of the attitude of the second arm caused by the rotation of the first rotation axis.

[0166] (10) In the robot of the above form, the link mechanism may be configured to move the second arm in parallel in conjunction with the rotation of the first arm.

[0167] (11) In the robot of the above form, the link mechanism may be configured to move the second arm in a non-parallel manner in conjunction with the rotation of the first arm. [Explanation of Symbols]

[0168] AX...Optical axis, B1...Belt, D1, D2...Drive mechanism, FM...Fixed member, IM1, IM2...Image, IMs1~IMs6...Image, J1, J2, J3...Rotation axis, L1...Link mechanism, LD...Light-emitting part, LI1...First brightness image, LI2...Second brightness image, LI3...Differential brightness image, PL...Perpendicular line, PL1...Drive pulley, PL2...Driven pulley, R1, R2...Observation area, S1...Virtual plane, SP...Shading plate, TBx, TBy...Rotation amount table, TG...Target, Xmin, Ymin...Threshold, θ1...Angle, 10, 10b~10e, 10h...Robot system, 100, 100a~100f, 100h...Robot, 110...First arm, 110a...First slider, 120...Second arm, 120a...Second Slider, 130, 140… Joint parts, 210, 210a… Base, 220… Rotation mechanism, 230… Lifting mechanism, 240… Travel mechanism, 241… Drive wheel, 242… Driven wheel, 243… Travel control unit, 310, 310a, 311, 312… Mark, 400, 400a… End effector, 510~550… Camera, 610… First guide light irradiation unit, 620… Second guide light irradiation unit, 700… Control unit, 750… Relay unit, 800… Server, 900… Terminal device, BD… Main unit, CL… Center line, EX… Extension unit, F, F2~F5… Image, GL1, GL2… Guide light, HJ… Hinge joint, HN… End effector, J4… Rotation axis, P1… User, PS… Pressure sensor, T1… Mounting platform, TCP… Control point

Claims

1. It is a robot, Bass and, A first arm connected to the base and rotating around a first rotation axis, A second arm connected to one end of the first arm, the second arm rotating around a second rotation axis parallel to the first rotation axis with respect to the first arm, A physical marker attached to the second arm for determining the control point of the robot, A camera whose position and orientation are fixed with respect to the base, and which is positioned to capture the marker moving on a virtual plane, assuming a virtual plane perpendicular to the first rotation axis, and which captures the first arm and the second arm at regular time intervals, A control unit, Each time the first camera acquires an image, the amount of rotational displacement of the first motor that rotates the first arm around the first rotation axis and the amount of rotational displacement of the second motor that rotates the second arm around the second rotation axis are determined based on the difference between the coordinate values ​​of the control point of the robot determined from the marker in the image captured by the first camera and the coordinate values ​​of the designated target point, and the first motor and the second motor are controlled to rotate by the determined amount of displacement. A control unit, which, when a pre-set condition is met, acquires an image from the first camera, determines the rotational displacement of the first motor and the rotational displacement of the second motor based on the difference between the coordinate values ​​of the robot's control point in the image captured by the first camera and the coordinate values ​​of the return point set as the point to which the robot should return, and controls the first motor or the second motor to rotate by the determined displacement, Equipped with, robot.

2. A robot according to claim 1, A rotation mechanism that rotates the base around a third rotation axis perpendicular to the first rotation axis, according to the control of the control unit, A second camera for imaging the direction of movement of the second arm, the second camera being fixed to the first arm or the second arm, Furthermore, The control unit, Based on the image captured by the second camera, at least the rotation mechanism is controlled to orient the direction of travel of the second arm toward a designated target point. While the rotating mechanism is rotating the base, the first and second rotating axes are not rotated. The rotation mechanism is controlled so as not to rotate the base while at least one of the first rotation axis and the second rotation axis is rotating. robot.

3. The robot according to claim 2, The system further comprises a first irradiation unit fixed to the second arm and irradiating guide light in the direction of travel of the second arm, The first camera is positioned to capture images of the area illuminated by the guide light. The control unit presents the image captured by the first camera and accepts the designation of a target location. robot.

4. The robot according to claim 3, A gripping portion connected to the tip of the second arm, A second irradiation unit is a light source that irradiates light to the gripping unit at a predetermined positional relationship, Furthermore, The first camera is positioned to capture the light emitted by the second illumination unit. robot.

5. The robot according to claim 4, The system further comprises a third camera for imaging the gripping portion. robot.

6. The robot according to claim 5, The system further comprises a lifting mechanism for raising and lowering the base. robot.

7. The robot according to claim 6, The vehicle further comprises a driving mechanism for moving the aforementioned base, The aforementioned travel mechanism is A forward-facing camera that captures images in the direction of travel of the aforementioned travel mechanism, When a driving instruction is received from the control unit, the driving control unit controls the driving of the driving mechanism using the image captured by the front camera, Equipped with, robot.

8. A robot comprising the robot described in claim 7, The aforementioned travel control unit controls the movement of the travel mechanism using images captured by at least one external camera positioned to capture images within the range in which the robot is expected to travel. robot.

9. A robot according to any one of claims 1 to 8, The invention further comprises a link mechanism that mechanically transmits at least a portion of the power output by the motor driving the first rotating shaft to the second arm, thereby counteracting the rotation of the second arm caused by the rotation of the first arm. robot.

10. A robot according to claim 9, The link mechanism is configured to move the second arm in parallel in conjunction with the rotation of the first arm. robot.

11. A robot according to claim 10, The link mechanism is configured to move the second arm non-parallel in conjunction with the rotation of the first arm. robot.

Images