Robot system and image processing method

Abstract

A robot system (100) is provided with: a robot body unit (11) that includes a robot arm unit (13); a plurality of imaging units (41, 42, and 43) that are disposed in mutually different positions in the robot arm unit (13) of the robot body unit (11), spaced apart from a base end of the robot arm unit (13); and a control unit (50) that generates a composite image as a bird's-eye image around the robot body unit (11) by combining a plurality of captured images captured by each of the plurality of imaging units (41, 42, and 43).

Description

ROBOT SYSTEM AND IMAGE PROCESSING METHOD

[0001] The present disclosure relates to a robot system and an image processing method.

[0002] Robots equipped with cameras have been known in the art. For example, Japanese Patent No. 7055883 describes a robot equipped with an arm having multiple joints and a camera attached to a support pole to which the arm is connected. The support pole is fixed to a base fixed to the installation surface and rotates around a rotation axis perpendicular to the installation surface relative to the base. Two cameras are attached to the left and right of the support pole. The robot described in Japanese Patent No. 7055883 performs coordinate transformation on each of images captured by the cameras attached to the left and right of the support pole, and synthesizes the transformed images to generate an image equivalent to a single plan view using around-view processing.

[0003] Patent No. 7055883

[0004] However, when a camera serving as an imaging unit is placed on a support pole to which the robot arm is connected, as in the robot described in Japanese Patent No. 7055883, the imaging unit is placed close to the installation surface of the robot body including the robot arm, which is considered to reduce the range of the display area displayed in the composite image generated from the captured images taken by the imaging unit. In this case, when checking the surroundings of the robot body using the composite image, it is difficult to easily check the surroundings of the robot body due to the small range of the display area of ​​the composite image.

[0005] This disclosure has been made to solve the above-mentioned problems, and one object of this disclosure is to provide a robot system and an image processing method that enable easy checking of the surroundings of the robot main body.

[0006] A robot system according to a first aspect of the present disclosure includes a robot main body including a robot arm, a plurality of imaging units arranged in the robot arm of the robot main body at different positions spaced apart from a base end of the robot arm, and a control unit that generates a composite image as a bird's-eye view of the periphery of the robot main body by combining a plurality of images captured by each of the plurality of imaging units. Note that the "base end" here refers to the end of the robot arm on the connection side to which the robot arm is connected.

[0007] As described above, a robot system according to a first aspect of this disclosure includes a robot arm of a robot main body, including multiple imaging units arranged at different positions spaced apart from the base end of the robot arm. The robot system also includes a control unit that generates a composite image as a bird's-eye view of the surroundings of the robot main body by combining multiple captured images captured by the multiple imaging units. Because the imaging units are arranged at a position spaced apart from the base end of the robot arm, the range of the area captured by the imaging units can be larger than when the imaging units are arranged closer to an installation surface, such as a support column to which the robot arm is connected or the base end of the robot arm. Therefore, when checking the surroundings of the robot main body using the composite image, the range of the display area of ​​the composite image can be larger than when the imaging units are arranged closer to the installation surface of the robot main body. As a result, the surroundings of the robot main body can be easily checked.

[0008] An image processing method according to a second aspect of this disclosure acquires, in a robot arm section, a plurality of captured images taken by each of a plurality of imaging units arranged at different positions spaced apart from the base end of the robot arm section, and generates a composite image as a bird's-eye view of the surroundings of a robot main body section including the robot arm section by synthesizing the acquired captured images.

[0009] As described above, an image processing method according to a second aspect of this disclosure acquires a plurality of captured images captured by a plurality of imaging units disposed at different positions away from the base end of the robot arm, and combines the acquired captured images to generate a composite image as a bird's-eye view of the surroundings of a robot main body including the robot arm. Because the imaging units are disposed at positions away from the base end of the robot arm, the range of the area captured by the imaging units can be made larger than when the imaging units are disposed closer to an installation surface, such as a support column to which the robot arm is connected or the base end of the robot arm. Therefore, when checking the surroundings of the robot main body using the composite image, the range of the display area of ​​the composite image can be made larger than when the imaging units are disposed closer to the installation surface of the robot main body. As a result, an image processing method can be provided that allows the surroundings of the robot main body to be easily checked.

[0010] According to the present disclosure, the surroundings of the robot main body can be easily checked.

[0011] FIG. 1 is a perspective view showing the overall configuration of a robot system according to an embodiment; FIG. 2 is a perspective view for explaining the configuration of a robot main body; FIG. 3 is a block diagram showing the configuration of a robot system; FIG. 4 is a diagram for schematically showing generation of a composite image based on captured images; FIG. 5 is a flowchart for explaining an image processing method by a robot system; FIG. 6 is a diagram for explaining viewpoint conversion of a captured image; and FIG. 7 is a diagram for explaining calibration processing using marker members. FIG. 8 is a perspective view showing a robot main body according to a modified example of an embodiment.

[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0013] The configuration of a robot system 100 according to this embodiment will be described with reference to FIGS. 1 to 4. FIG.

[0014] (Configuration of Robot System) As shown in FIG. 1 , the robot system 100 includes a robot 10, an operation device 20, and a display unit 30. The robot 10 includes a robot main body 11. The robot main body 11 includes a robot arm 13 attached to a base 12. The operation device 20 detects an operation performed by an operator. The robot main body 11 of the robot 10 operates in conjunction with the operation detected by the operation device 20 via remote control. That is, the robot system 100 is a master-slave system in which the robot 10 operates in conjunction with the operation performed by the operator detected by the operation device 20. In the robot system 100, the operation device 20 is the master device, and the robot 10 is the slave device. The robot 10 and the operation device 20 are connected to each other via a network. The robot 10 is an industrial robot that performs handling tasks such as processing, assembly, transportation, or inspection. The robot 10 also includes multiple image capture units 41, 42, and 43, and a control unit 50.

[0015] As shown in FIG. 2 , the robot arm unit 13 has a portion 14, a portion 15, and a portion 16. The portion 14 is disposed at the base end of the robot arm unit 13. The portions 15 and 16 are connected to the portion 14 and extend from the portion 14. Each of the portions 15 and 16 has a long, rod-like shape extending from the base end to the tip end. The portion 14 has two link members 14a and 14b. The portion 15 has a link member 15a. The portion 16 has three link members 16a, 16b, and 16c. The robot arm unit 13 is a multi-joint robot arm having multiple joints 17. For example, the robot arm unit 13 has six joints 17. That is, the robot 10 is a six-axis vertical multi-joint robot. The six joints 17 connect the link members 14a, 14b, 15a, 16a, 16b, and 16c to each other and to an end effector connected to the tip of the robot arm 13, and rotate around axes JT1, JT2, JT3, JT4, JT5, and JT6, respectively. Each of the joints 17 is provided with a drive unit 18 such as a motor and a position detection unit 19 such as an encoder, as shown in FIG. 3 . The portion 14 is an example of a base end portion. The portions 15 and 16 are examples of extension portions.

[0016] The robot arm unit 13 is connected to a base unit 12 that is placed on an installation surface, such as a floor. A portion 14 of the robot arm unit 13 is connected to the base unit 12. The base end of a portion 15 is connected to the portion 14. The tip end of the portion 15 is connected to the portion 16. The base end of the portion 16 is connected to the portion 15, and an end effector is attached to the tip end of the portion 16.

[0017] Specifically, link member 14a of portion 14 is fixed to base 12, and link member 14b rotates about the JT1 axis relative to link member 14a, causing portion 14 to rotate about the JT1 axis relative to base 12. The JT1 axis extends in a direction perpendicular to the installation surface on which robot body 11 is installed. That is, portion 14 is positioned at a constant distance from the installation surface of robot body 11 during operation of robot arm 13. In portion 15, link member 15a is connected to link member 14b of portion 14 so as to rotate about the JT2 axis. The JT2 axis extends in a direction perpendicular to the JT1 axis. Link member 15a is also positioned so as to extend in a direction perpendicular to the JT2 axis, and its tip is connected to link member 16a of portion 16 so as to rotate about the JT3 axis. The JT3 axis is parallel to the JT2 axis. Section 16 is disposed to extend in a direction perpendicular to the JT3 axis. In section 16, link member 16b is connected to link member 16a so as to rotate about the JT4 axis, and link member 16c is connected to link member 16b so as to rotate about the JT5 axis. The JT4 axis is perpendicular to the JT3 axis and is disposed along the extension direction of section 16. The JT5 axis is perpendicular to the JT4 axis. An end effector is attached to the tip side of section 16 so as to rotate about the JT6 axis relative to link member 16c. The JT6 axis is perpendicular to the JT5 axis. In other words, while section 14 has a constant distance from the installation surface, the distances of sections 15 and 16 from the installation surface change as the robot arm unit 13 moves. In other words, the height positions of sections 15 and 16 from the installation surface change as the robot arm unit 13 moves. Note that an end effector is not shown in Fig. 2. The end effector includes, for example, a hand for gripping a workpiece or the like, or a processing tool for performing processing.

[0018] In this embodiment, three imaging units 41, 42, and 43 are disposed in the robot arm 13 of the robot main body 11 at different positions spaced apart from the base end of the robot arm 13. Each of the imaging units 41, 42, and 43 has an imaging element such as a complementary metal oxide semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor, and captures images of the surroundings of the robot main body 11. The imaging units 41, 42, and 43 may be three-dimensional cameras. The imaging units 41, 42, and 43 are disposed in a link member 16a including a joint 17 of the JT3 axis connecting the portion 15 and the portion 16, at a portion 16 whose height position from the installation surface changes. That is, the imaging units 41, 42, and 43 are arranged on the common link member 16a. In other words, the imaging units 41, 42, and 43 are arranged in the robot arm 13, which has a plurality of joints 17, at positions via at least two or more joint axes from the base end. That is, the imaging units 41, 42, and 43 are arranged on the link member 16a of the link members 15a, 16a, 16b, and 16c, which is closer to the tip end than at least two or more joints 17, namely, the joint 17 on the JT1 axis and the joint 17 on the JT2 axis, from the base end of the robot arm 13. The imaging units 41, 42, and 43 are arranged inside the housing of the link member 16a. The imaging units 41, 42, and 43 capture images of the installation surface on which the robot main body 11 is placed through imaging holes arranged in the housing of the link member 16a.

[0019] The imaging units 41, 42, and 43 are fixed to a link member 16a of the portion 16 of the robot arm unit 13, which is operated by the drive unit 18, and move integrally in accordance with the movement of the robot arm unit 13. That is, the imaging directions of the imaging units 41, 42, and 43 are changed in accordance with changes in the posture of the robot arm unit 13. The robot system 100 does not include a drive mechanism that changes the orientation of the imaging units 41, 42, and 43 separately from the movement of the robot arm unit 13. Each of the imaging units 41, 42, and 43 has a wide-angle lens or a fisheye lens. The imaging ranges of the imaging units 41, 42, and 43 partially overlap each other. By combining their imaging ranges, the imaging units 41, 42, and 43 capture an image of the area around the portion 14 of the robot main body 11 that is located at the base end of the robot arm unit 13. For example, with a reference posture being a state in which part 15 is arranged perpendicular to the installation surface and part 16 is arranged in a direction along the installation surface, imaging units 41, 42, and 43 combine their imaging ranges with each other while robot arm unit 13 is in the reference posture, thereby capturing images of the entire periphery of part 14 of robot main body unit 11. Furthermore, by being disposed on part 16, imaging units 41, 42, and 43 are disposed at positions on robot arm unit 13 where the distance from the base end of robot arm unit 13 changes as robot arm unit 13 moves.

[0020] 3, the control unit 50 of the robot 10 includes a main control unit 51, a servo control unit 52, a drive circuit unit 53, and an image processing unit 54. The main control unit 51 and the servo control unit 52 each include a calculation device such as a CPU (Central Processing Unit). The control unit 50 is a robot controller that controls the operation of the robot 10. A storage device such as a hard disk drive is provided in the control unit 50, and the calculation device executes control processing based on programs, parameters, and the like stored in the storage device.

[0021] The main control unit 51 controls the overall operation of the robot 10. The servo control unit 52 controls the power supplied to the drive units 18 of the six joint units 17 based on commands from the main control unit 51. The drive circuit unit 53 supplies drive power to the drive units 18 of the six joint units 17. The drive units 18 include servo motors that serve as drive sources for the operation of the robot arm unit 13. The drive circuit unit 53 includes an inverter circuit that supplies AC power to the drive units 18. In the drive units 18, the rotation shafts of the servo motors are rotated by the three-phase AC power supplied from the drive circuit unit 53. The rotation of the rotation shafts of the servo motors of the drive units 18 rotates each of the multiple joint units 17, thereby operating the robot arm unit 13. A drive circuit unit 53 is provided for each of the six joint units 17. A position detection unit 19 is provided in each of the drive units 18 of the six joint units 17. The position detection unit 19 detects the rotation angle of the joint unit 17. The position detection unit 19 includes, for example, an encoder that detects the rotation angle of the servo motor of the drive unit 18 to detect the rotation angle of the joint unit 17. The position detection unit 19 outputs a detection value indicating the detected rotation angle of the servo motor of the drive unit 18 to the main control unit 51 and the servo control unit 52. The image processing unit 54 includes an FPGA (Field Programmable Gate Array) dedicated to image processing. The image processing unit 54 acquires signals from the imaging units 41, 42, and 43 to generate captured images 60 shown in FIG. 4 captured by each of the imaging units 41, 42, and 43. The image processing unit 54 also generates a composite image 70 based on the generated captured images 60. Details of the generation of the composite image 70 based on the captured images 60 by the image processing unit 54 of the control unit 50 will be described later.

[0022] As shown in FIG. 1 , the operating device 20 includes a gripper 21 and a control unit 22. The gripper 21 is held by an operator to control the operation of the robot 10. The gripper 21 is held, for example, with both hands of the operator, and its position and inclination are changed by the operator's operation. For example, the gripper 21 is supported by an arm member with a parallel link structure. The arm member supporting the gripper 21 is equipped with an encoder for detecting a rotation angle and a force sensor. The control unit 22 controls the operation of each component of the operating device 20. Similar to the control unit 50 of the robot 10, the control unit 22 includes a calculation device such as a CPU and a storage device, and executes control processing by the calculation device based on programs, parameters, and the like stored in the storage device. The control unit 22 also detects operation motions applied to the gripper 21 using the encoder and force sensor.

[0023] Each of the control unit 22 and the control unit 50 includes a communication module. The control unit 22 generates a control signal for operating the robot 10 based on the detected operating movement, and outputs the generated control signal to the control unit 50 of the robot 10 via the communication module. The control unit 50 also outputs the generated composite image 70 to the control unit 22 via the communication module, thereby displaying the composite image 70 on the display unit 30. The display unit 30 displays the composite image 70 generated by the control unit 50 to the operator performing an operating movement on the operation device 20. The display unit 30 includes, for example, a liquid crystal monitor. The display unit 30 is disposed in a position visible to the operator holding the grip unit 21 of the operation device 20.

[0024] The robot system 100 operates as a remotely controlled master-slave system by transmitting and receiving signals between the control unit 50 of the robot 10 and the control unit 22 of the operation device 20. The robot system 100 performs bilateral control, which causes the posture and force between the robot 10 and the operation device 20 to track each other. Based on an operation signal from the control unit 22, the robot 10 changes the posture of the robot arm unit 13 so as to follow the posture, including the position and angle, of the gripper 21, which has been changed by the operator's operation.

[0025] <Generation of Composite Image> As shown in FIG. 4 , in this embodiment, the control unit 50 generates a composite image 70 as a bird's-eye view image of the periphery of the robot main body 11 by combining multiple captured images 60. The composite image 70 is a bird's-eye view image including the installation surface surrounding the portion 14 of the robot main body 11. The control unit 50 performs around-view processing on the captured images 60 to generate the composite image 70 projected from a virtual viewpoint above the robot main body 11 onto the installation surface on which the robot main body 11 is installed. The virtual viewpoint in the composite image 70 is, for example, a position moved upward from the robot main body 11 in a direction perpendicular to the installation surface. In generating the composite image 70, the control unit 50 performs image conversion by performing image processing for viewpoint conversion on each of the three captured images 60 captured by the three imaging units 41, 42, and 43, and generates the composite image 70 by combining the three captured images 60 that have undergone image processing for viewpoint conversion.

[0026] The control unit 50 also superimposes a projected robot image 71, which represents the robot main body 11 projected onto the composite image 70 as a bird's-eye view image, on the display unit 30. In the projected robot image 71, the robot main body 11 is represented by a three-dimensional model that resembles the robot main body 11. The projected robot image 71 is an image of the three-dimensional model that resembles the robot main body 11 captured from a position corresponding to a virtual camera viewpoint Q, which is a virtual viewpoint from which the composite image 70 is generated. In other words, the projected robot image 71 is an image that represents the robot main body 11 when the actual robot main body 11 is projected onto an installation surface, which is a plane on which the composite image 70 is generated, from a virtual viewpoint above the robot main body 11. The control unit 50 also superimposes the projected robot image 71 on the composite image 70 and displays it on the display unit 30 with the operating state of the robot main body 11 projected in the projected robot image 71 changed to match the operating state of the actual robot main body 11. For example, the control unit 50 acquires the rotation angles of each of the plurality of joints 17 based on signals from the position detection unit 19. The control unit 50 then acquires the posture of the robot arm 13 of the robot main body 11 from the acquired rotation angles of each of the plurality of joints 17. The control unit 50 then changes the posture of the three-dimensional model simulating the robot main body 11 to correspond to the acquired posture of the actual robot arm 13, thereby causing the operating state of the robot main body 11 in the projected robot image 71 to follow the actual robot main body 11. In other words, the control unit 50 changes the posture of the three-dimensional model of the robot main body 11 in the projected robot image 71 in synchronization with the rotation angles of the joints 17 of the actual robot main body 11. For example, in the actual robot main body 11, when the joint 17 of the portion 14 is rotated around the JT1 axis by the drive of the drive unit 18, the control unit 50 causes the robot main body 11 in the projected robot image 71 displayed superimposed on the composite image 70 to move around the JT1 axis by the same angle as the actual robot main body 11, in accordance with the movement of the actual robot main body 11. The control unit 50 generates the composite image 70 as a moving image based on the captured images 60 as a moving image captured in real time.The control unit 50 then superimposes a projected robot image 71 that follows the actual movement of the robot main body 11 in real time on the composite image 70 and outputs the superimposed image to the control unit 22 so that it is displayed on the display unit 30 .

[0027] (Image Processing Method by Robot System) An image processing method by the robot system 100 in this embodiment will be described based on a flowchart with reference to Figures 5 to 7. Note that the control processing from step S1 to step S5 in Figure 5 is executed by the control unit 50 of the robot 10.

[0028] First, in step S1, captured images 60 are acquired by each of the multiple imaging units 41, 42, and 43. The control unit 50 generates three captured images 60 based on signals from each of the imaging units 41, 42, and 43. The control unit 50 generates a moving image from the three captured images 60 captured by each of the three imaging units 41, 42, and 43 at a common timing.

[0029] Next, in step S2, image transformation is performed on the captured images 60. Specifically, image processing for viewpoint transformation is performed on each of the three captured images 60. The control unit 50 acquires information corresponding to the three-dimensional position of each of the multiple image capturing units 41, 42, and 43 that move in conjunction with the operation of the robot arm unit 13, and performs image transformation on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional position of each of the multiple image capturing units 41, 42, and 43. The control unit 50 acquires the rotation angle of the joint unit 17 as information corresponding to the three-dimensional position of each of the multiple image capturing units 41, 42, and 43.

[0030] As shown in FIG. 6 , the control unit 50 performs image processing for viewpoint conversion on three captured images 60 so that each of the captured images 60 becomes an image captured from a virtual camera viewpoint Q, which is a virtual viewpoint, thereby generating a projected image 61, which is the captured image 60 with the converted viewpoint. Note that FIG. 6 illustrates only one of the three captured images 60. First, before performing the image processing for viewpoint conversion, the control unit 50 corrects distortion of the captured image 60 captured by a wide-angle lens or a fisheye lens. Then, the control unit 50 performs image processing for viewpoint conversion on the captured image 60, in which the distortion of the image capturing units 41, 42, and 43 has been corrected, so that the captured image 60 becomes an image captured from the virtual camera viewpoint Q. In the image processing for viewpoint conversion, when the actual camera viewpoints at which the image capturing units 41, 42, and 43 are located are P in FIG. 6 , and the virtual camera viewpoint after viewpoint conversion is Q, the camera coordinate system X of the camera viewpoint P is Ｐ , Y Ｐ , Z Ｐ From the virtual camera coordinate system X of the virtual camera viewpoint Q Ｑ , Y Ｑ , Z Ｑ In this case, a rotation matrix R representing a rotational movement that converts the orientation of the coordinate system is expressed by the following equation (1). Furthermore, a translation matrix T representing a translation from the position of the camera viewpoint P to the position of the virtual camera viewpoint Q is expressed as in the following equation (2). The transformation of the viewpoint from the actual camera viewpoint P to the virtual camera viewpoint Q is expressed by the product of a translation matrix T and a rotation matrix R. Therefore, from the above equations (1) and (2), the transformation matrix TR of the viewpoint is expressed as shown in the following equation (3). Here, n, o, and a are vector quantities expressed by the following equation (4). The position of the camera viewpoint P and the camera coordinate system X Ｐ , Y Ｐ , Z Ｐis acquired based on the rotation angle of the joint 17 of the robot arm 13 and the lengths of the plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c of the robot arm 13. The lengths of the plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c are stored in advance in the storage device of the control unit 50. The position of the virtual camera viewpoint Q and the virtual camera coordinate system X Ｑ , Y Ｑ , Z Ｑ are set and stored in advance. The position of the camera viewpoint P and the camera coordinate system X Ｐ , Y Ｐ , Z Ｐ and the position of the virtual camera viewpoint Q and the virtual camera coordinate system X Ｑ , Y Ｑ , Z Ｑ Based on this, n = (n ｘ , n ｙ , n ｚ ), o=(o ｘ , o ｙ , o ｚ ), a=(a ｘ , a ｙ , a ｚ ), and (x, y, z). The control unit 50 acquires the value of the rotation angle of the joint 17 in real time at each predetermined control period from a signal from the position detection unit 19, and performs arithmetic processing using the acquired rotation angle of the joint 17 and various parameters stored in advance, thereby acquiring the transformation matrix TR in real time in accordance with the operation of the robot main body 11.

[0031] The control unit 50 converts the viewpoint of the captured image 60 captured by each of the multiple image capturing units 41, 42, and 43 into a projection image 61 for synthesis by using the viewpoint transformation matrix TR expressed by equation (3). Ｐ , Y Ｐ , Z Ｐ Each pixel m of the captured image 60 Ｐ (x ｉ , y ｉ ) into the virtual camera coordinate system X using the viewpoint transformation matrix TR of the above equation (3). Ｑ , Y Ｑ , ZＱ By converting each pixel m of the projection image 61 for synthesis into Ｑ (x' ｉ , y′ ｉ ) is acquired. The control unit 50 converts three captured images 60 captured by the three imaging units 41, 42, and 43 arranged at different positions into three projected images 61, which are images from a virtual camera viewpoint Q, which is a virtual viewpoint different for each of the multiple imaging units 41, 42, and 43. In other words, the virtual viewpoint in the projected images 61 obtained by converting the captured images 60 is in a different position for each of the multiple imaging units 41, 42, and 43. For example, the projected images 61 are images obtained by projecting the captured images 60 onto an installation surface.

[0032] For example, as shown in Fig. 7, the control unit 50 determines three different positions spaced around the robot main body 11 when viewing the robot main body 11 from a direction perpendicular to the installation surface as positions of virtual camera viewpoints Q. The control unit 50 also determines that the distances from the installation surface of the three different virtual camera viewpoints Q are equal. The control unit 50 also determines that the viewpoint directions of the three different virtual camera viewpoints Q are the same so that the generated composite image 70 is an image projected onto a single plane. For example, in the virtual camera coordinate system X Ｑ , Y Ｑ , Z Ｑ In this case, Z Ｑ In this way, the control unit 50 acquires the rotation angle of the joint 17 as information corresponding to the three-dimensional position of each of the multiple image capturing units 41, 42, and 43, and performs arithmetic processing using the acquired rotation angle of the joint 17 and preset parameters to convert the captured image 60 into projected images 61 from different viewpoints. Note that by converting the captured images 60 into projected images 61 from different viewpoints, it is possible to increase the display area of ​​the generated composite image 70.

[0033] Next, in step S3, a composite image 70 is generated. The control unit 50 performs the image transformation in step S2 on the captured image 60 to generate the composite image 70. The control unit 50 generates the composite image 70 by combining projected images 61, which are the captured images 60 transformed by the image transformation in step S2. When combining the projected images 61, the control unit 50 generates the composite image 70 using correction values ​​as parameters previously set by a calibration process. Details of the calibration process will be described later. The control unit 50 arranges the projected images 61 side by side based on the position of the virtual camera viewpoint Q. The control unit 50 then corrects the projected images 61 based on the correction values ​​previously set by the calibration process, and combines the corrected projected images 61 to generate the composite image 70. In the correction based on the correction values, for example, the value of the transformation matrix TR performed in step S2 may be corrected, or affine transformation such as scaling, rotation, and translation may be performed on the projected images 61 generated in step S2. Note that the composite image 70 is generated by cutting out a portion of the projected image 61, which is the converted captured image 60. For example, a portion of the lower side of the projected image 61 is cut out so that the installation surface is included in the composite image 70, and the cut-out portions of the projected images 61 are superimposed on each other to generate the composite image 70. Furthermore, the composite image 70 is cut out so as to have a rectangular shape centered on the position of the robot main body 11, for example.

[0034] Next, in step S4, a projected robot image 71 is generated and superimposed on the composite image 70. The control unit 50 superimposes the projected robot image 71 on the composite image 70 generated in step S3. The control unit 50 acquires the motion state of the actual robot main body 11 based on the rotation angle of the joint 17 acquired in step S2, and changes the motion state of the robot main body 11 in the three-dimensional model to correspond to the acquired motion state, thereby generating the projected robot image 71.

[0035] Next, in step S5, the generated composite image 70 is displayed on the display unit 30. Specifically, the control unit 50 outputs the composite image 70 with the projected robot image 71 superimposed thereon to the control unit 22 of the operation device 20, thereby causing the control unit 22 to display the composite image 70 with the projected robot image 71 superimposed thereon on the display unit 30.

[0036] The control unit 50 executes the processes from step S1 to step S5 at predetermined control intervals, thereby displaying the continuously generated composite image 70 on the display unit 30. The predetermined control interval is, for example, any value between 10 milliseconds and 100 milliseconds. That is, the composite image 70 with the projected robot image 71 superimposed thereon is generated as a moving image, and the composite image 70 is displayed on the display unit 30 as a moving image that follows the movement of the robot main body 11 in real time. Note that the term "real time" used here is intended to encompass errors due to delays in control processing or communication.

[0037] 7 , the control unit 50 corrects parameters of the control process for generating the composite image 70 in step S3 by performing a calibration process in advance using the marker members 80. In the calibration process, the control unit 50 generates the composite image 70 based on the positions of the common marker members 80 in the multiple captured images 60.

[0038] The calibration process is performed, for example, before shipping the robot system 100 from a factory or during periodic maintenance of the robot system 100. In the calibration process, marker members 80 are placed at positions where the imaging ranges of the multiple imaging units 41, 42, and 43 overlap. In the robot system 100, the three imaging units 41, 42, and 43 are arranged to capture images of the periphery of the robot main body 11. Therefore, in the calibration process, marker members 80 are placed at three positions: a position where the imaging ranges of the imaging units 41 and 42 overlap; a position where the imaging ranges of the imaging units 42 and 43 overlap; and a position where the imaging ranges of the imaging units 41 and 43 overlap. The control unit 50 converts the captured image 60 in step S2 based on preset parameters and detection values ​​from the encoder of the position detection unit 19. The control unit 50 then generates a composite image 70 by combining projected images 61, which are the captured images 60 with their viewpoints converted. In the calibration process, the control unit 50 generates the composite image 70 based on the positions of the common marker members 80 in the multiple captured images 60. For example, in generating the composite image 70, the control unit 50 performs image conversion on each of the projected images 61, which are the converted captured images 60, so that the marker members 80 in each of the projected images 61 overlap with each other. The control unit 50 stores, as correction values ​​for generating the composite image 70, conversion parameters for the image conversion in which the marker members 80 overlap with each other.

[0039] Effect of the Present Embodiment The robot system 100 includes a plurality of imaging units 41, 42, and 43 in the robot arm unit 13 of the robot main body unit 11, the imaging units 41, 42, and 43 being arranged at different positions spaced apart from the base end of the robot arm unit 13. The robot system 100 also includes a control unit 50 that generates a composite image 70 as a bird's-eye view of the periphery of the robot main body unit 11 by combining a plurality of captured images 60 captured by each of the imaging units 41, 42, and 43. As a result, because the imaging units 41, 42, and 43 are arranged at positions spaced apart from the base end of the robot arm unit 13, the range of the area imaged by the imaging units 41, 42, and 43 can be made larger than when the imaging units 41, 42, and 43 are arranged at positions closer to an installation surface, such as a support column to which the robot arm unit 13 is connected, the base unit 12, or the base end of the robot arm unit 13. Therefore, when checking the surroundings of the robot main body 11 using the composite image 70, the display area of ​​the composite image 70 can be made larger than when the imaging units 41, 42, and 43 are located closer to the installation surface of the robot main body 11. As a result, the surroundings of the robot main body 11 can be checked more easily.

[0040] The robot arm unit 13 includes a portion 14, which is disposed at the base end of the robot arm unit 13 and serves as a base end portion at a fixed distance from the installation surface of the robot main body unit 11, and an extension portion, which is connected to the portion 14 and includes portions 15 and 16 extending from the portion 14. The imaging units 41, 42, and 43 are disposed in the extension portion of the robot arm unit 13, which includes portions 15 and 16. The control unit 50 generates a composite image 70 as a bird's-eye view image including the installation surface surrounding the portion 14. This generates a composite image 70 whose display range includes the installation surface surrounding the portion 14, which is disposed at the base end of the robot arm unit 13. Therefore, when the robot arm unit 13 is operated, physical interference between the robot arm unit 13 and an object disposed on the installation surface surrounding the portion 14 can be easily confirmed. This makes it easier to check the surroundings of the robot main body unit 11.

[0041] The control unit 50 acquires information corresponding to the three-dimensional position of each of the multiple imaging units 41, 42, and 43 that move in conjunction with the operation of the robot arm unit 13, and performs image conversion on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional position of each of the multiple imaging units 41, 42, and 43 to generate a composite image 70. As a result, even if the three-dimensional positions of the multiple imaging units 41, 42, and 43 change in conjunction with the operation of the robot arm unit 13, the control unit 50 can acquire information corresponding to the three-dimensional position of each of the multiple imaging units 41, 42, and 43, thereby generating a composite image 70 in accordance with the change in the positions of the multiple imaging units 41, 42, and 43. Therefore, when the robot arm unit 13, in which the multiple imaging units 41, 42, and 43 are arranged, is operated, the surroundings of the robot main body 11 can be checked effectively and easily.

[0042] The robot arm 13 has a plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c, and a joint 17 connecting the plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c to one another. The control unit 50 acquires the rotation angle of the joint 17 as information corresponding to the three-dimensional positions of each of the plurality of image capturing units 41, 42, and 43. By acquiring the rotation angle of the joint 17, the three-dimensional positions of the plurality of image capturing units 41, 42, and 43 can be easily acquired, making it easier to check the surroundings of the robot main body 11.

[0043] The robot system 100 includes a drive unit 18 that serves as a drive source for the operation of the robot arm unit 13. The multiple image capturing units 41, 42, and 43 are fixed to and disposed on the robot arm unit 13, which is operated by the drive unit 18. This allows the position and orientation of each of the multiple image capturing units 41, 42, and 43 to be changed by operating the robot arm unit 13 with the drive unit 18, without providing a drive mechanism for changing the orientation of each of the multiple image capturing units 41, 42, and 43. Therefore, by operating the robot arm unit 13 with the drive unit 18, a wider range can be captured by the multiple image capturing units 41, 42, and 43, making it possible to easily check a wider range around the robot main body unit 11.

[0044] The robot arm 13 includes multiple link members 14a, 14b, 15a, 16a, 16b, and 16c and a joint 17 connecting the multiple link members 14a, 14b, 15a, 16a, 16b, and 16c. The multiple image capture units 41, 42, and 43 are arranged on a common link member 16a. This allows the multiple image capture units 41, 42, and 43 to be aligned at the same height relative to the installation surface on which the robot main body 11 is installed. Therefore, when combining multiple captured images 60 captured by the multiple image capture units 41, 42, and 43, variations in the areas included in the multiple captured images 60 can be reduced, thereby preventing the generated composite image 70 from becoming unclear. Furthermore, because the multiple image capture units 41, 42, and 43 can be aligned at the same height relative to the installation surface on which the robot main body 11 is installed, the burden of control processing in generating the composite image 70 can be reduced compared to when the image capture units 41, 42, and 43 are aligned at different heights.

[0045] The robot system 100 includes a display unit 30 that displays a composite image 70 generated by the control unit 50. This allows the operator to easily check the surroundings of the robot main body 11 by visually recognizing the composite image 70 displayed on the display unit 30.

[0046] The robot system 100 includes an operation device 20 that detects an operation action performed by an operator. The robot main body 11 operates in conjunction with the operation action detected by the operation device 20 via remote control. The display unit 30 displays a composite image 70 to the operator performing the operation action via remote control. This allows the operator to visually recognize the composite image 70 displayed on the display unit 30 when remotely controlling the operation of the robot main body 11 using the operation device 20, thereby enabling the operator to effectively and easily check the surroundings of the robot main body 11.

[0047] The control unit 50 superimposes a projected robot image 71 representing the robot main body 11 projected onto the composite image 70 as a bird's-eye view image on the composite image 70 and displays the superimposed projected robot image 71 on the display unit 30. By visually recognizing the composite image 70 on which the projected robot image 71 is superimposed, the operator can easily recognize the positional relationship between the robot 10's surroundings and the robot main body 11.

[0048] The control unit 50 changes the operating state of the robot main body 11 projected in the projected robot image 71 to match the operating state of the actual robot main body 11, and then displays the projected robot image 71 on the composite image 70 on the display unit 30. As a result, the operating state of the projected robot image 71 superimposed on the composite image 70 is changed to match the operating state of the actual robot main body 11, and the operator can intuitively and easily recognize the positional relationship between the robot 10's surroundings and the robot main body 11 by visually recognizing the composite image 70 on which the projected robot image 71 is superimposed.

[0049] The control unit 50 generates a projected robot image 71 representing the robot main body 11 using a three-dimensional model that imitates the robot main body 11. As a result, in the projected robot image 71 superimposed on the composite image 70, the robot main body 11 is represented using a three-dimensional model that imitates the actual robot main body 11. Therefore, by visually recognizing the composite image 70 on which the projected robot image 71 is superimposed, the operator can more intuitively and easily recognize the positional relationship between the robot 10's surroundings and the robot main body 11.

[0050] The imaging ranges of the multiple imaging units 41, 42, and 43 partially overlap each other. The robot system 100 includes marker members 80 arranged at positions where the imaging ranges of the multiple imaging units 41, 42, and 43 overlap. The control unit 50 generates a composite image 70 based on the position of the common marker member 80 in the multiple captured images 60. By generating the composite image 70 based on the position of the marker member 80, it is possible to prevent misalignment in the generated composite image 70 when combining the multiple captured images 60 captured by the multiple imaging units 41, 42, and 43. This allows the multiple captured images 60 to be combined more accurately, and the generated composite image 70 makes it possible to easily and accurately check the surroundings of the robot main body 11.

[0051] The control unit 50 generates a composite image 70 by combining multiple captured images 60 captured at a common timing by the multiple imaging units 41, 42, and 43. This allows the state around the robot main body 11 to be confirmed more accurately than when combining captured images 60 captured at different timings by the multiple imaging units 41, 42, and 43.

[0052] (Effects of the Image Processing Method by the Robot System 100 of the Present Embodiment) The image processing method by the robot system 100 acquires a plurality of captured images 60 captured by each of a plurality of imaging units 41, 42, and 43 arranged in different positions in the robot arm unit 13 away from the base end of the robot arm unit 13, and combines the acquired captured images 60 to generate a composite image 70 as a bird's-eye view of the periphery of the robot main body unit 11 including the robot arm unit 13. As a result, because the imaging units 41, 42, and 43 are arranged in positions away from the base end of the robot arm unit 13, the range of the area captured by the imaging units 41, 42, and 43 can be made larger than when the imaging units 41, 42, and 43 are arranged in positions closer to an installation surface such as a support column to which the robot arm unit 13 is connected, the base unit 12, or the base end of the robot arm unit 13. Therefore, when checking the surroundings of the robot main body 11 using the composite image 70, the display area of ​​the composite image 70 can be made larger than when the imaging units 41, 42, and 43 are located closer to the installation surface of the robot main body 11. As a result, an image processing method can be provided that makes it possible to easily check the surroundings of the robot main body 11.

[0053] [Modifications] The embodiments disclosed herein should be considered to be illustrative and not restrictive in all respects. The scope of the present disclosure is defined by the claims, not by the description of the above-mentioned embodiments, and includes all modifications (modifications) within the meaning and scope of the claims.

[0054] For example, in the above embodiment, an example was shown in which the composite image 70 was generated as a bird's-eye view image including the installation surface surrounding the base end portion 14. However, the present disclosure is not limited to this. In the present disclosure, because the imaging ranges of each of the multiple imaging units change as the robot arm unit moves, the generated composite image does not need to include the installation surface surrounding the base end portion. For example, the robot main body may be installed on a ceiling or wall. In this case, since the installation surface on which the robot main body is installed is the ceiling or wall, the robot arm unit of the robot main body is also installed on the ceiling or wall. In this case, to check for physical interference between the robot arm unit and an object placed on the floor, a composite image may be generated as a bird's-eye view image including the floor surface within the range of the robot arm unit's movement range in the display area. This configuration allows the surroundings of the robot main body to be more effectively and easily checked not only when the robot main body unit is installed on the floor, but also when it is installed on a ceiling or wall.

[0055] Furthermore, in the above embodiment, the rotation angle of the joint 17 is acquired as information corresponding to the three-dimensional positions of each of the multiple image capture units 41, 42, and 43, and the composite image 70 is generated by performing image transformation on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional positions of each of the multiple image capture units 41, 42, and 43. However, the present disclosure is not limited to this. In the present disclosure, information corresponding to the three-dimensional positions of each of the multiple image capture units may not be acquired from a sensor or the like. For example, when capturing images for generating a composite image, the robot main body may be set to a predetermined posture, so that the composite image is generated based on predetermined parameters without acquiring information corresponding to the three-dimensional positions of each of the multiple image capture units. Furthermore, sensors may be provided to detect the positions of the image capture units rather than the rotation angles of the joints, so that information corresponding to the three-dimensional positions of each of the multiple image capture units is acquired. For example, a sensor may be provided to detect the position or posture of the robot arm, and the three-dimensional position of each of the multiple image capture units may be detected based on the detected position or posture of the robot arm. As a sensor for detecting the position or posture of the robot arm or a sensor for directly detecting the position of the imaging unit, a photoelectric sensor, a gyro sensor, a capacitance sensor, a magnetic sensor, a force sensor such as a piezoelectric element, or a mechanical switch may be provided. To acquire the three-dimensional position of each of the multiple imaging units, an imaging unit separate from the multiple imaging units that capture images for generating a composite image may be provided. Furthermore, when acquiring the rotation angle of a joint as information corresponding to the three-dimensional position of each of the multiple imaging units, the rotation angle of the joint may be acquired not by a signal from an encoder but by a command value for a drive unit or a signal from an operating device.

[0056] Furthermore, in the above embodiment, an example was shown in which the three imaging units 41, 42, and 43 were fixed and arranged inside the housing on the link member 16a of the portion 16 serving as the distal end of the common extension portion. However, the present disclosure is not limited to this. In the present disclosure, the multiple imaging units may be arranged on different link members. Furthermore, the multiple imaging units may be arranged not on the distal end portion of the extension portion of the robot arm, but on the proximal end portion of the extension portion. The multiple imaging units may be arranged separately in the distal end portion and the proximal end portion of the extension portion of the robot arm. Furthermore, the number of imaging units may be two or four or more. Furthermore, the imaging units may be arranged outside the housing of the robot arm. Furthermore, the imaging units do not need to have a wide-angle lens or a fisheye lens. Furthermore, a driving mechanism for changing the orientation of the imaging units may be provided.

[0057] Furthermore, in the above embodiment, an example has been described in which the robot arm unit 13, on which the imaging units 41, 42, and 43 are arranged, is connected to the base unit 12 arranged on an installation surface, but the present disclosure is not limited to this. In the present disclosure, as in the robot main body unit 211 according to a modified example shown in Fig. 8, the robot arm unit 213, on which the imaging units 41, 42, and 43 are arranged, may be attached to a cart 212 that moves by driving wheels. Furthermore, the robot arm unit on which the imaging units are arranged may be arranged directly on the installation surface.

[0058] In the above embodiment, the display unit 30 displays the composite image 70 to an operator who performs an operation by remote control using the operation device 20. However, the present disclosure is not limited to this. In the present disclosure, a composite image may be generated and displayed from images captured by an imaging unit disposed in a robot main body that is automatically operating based on preset parameters. Furthermore, a composite image may be generated to allow the operator to check the surroundings of the robot main body when performing a teaching operation to set the operation of the robot main body. Furthermore, a composite image may be generated and displayed for maintenance and inspection of the robot main body. Furthermore, a composite image may be displayed to an operator who directly views the robot main body rather than remotely controlling it. A control device that controls the operation of the robot main body may display the composite image on the display unit.

[0059] In the above embodiment, the projected robot image 71 is superimposed on the composite image 70, with the motion state of the robot main body 11 represented by a three-dimensional model in the projected robot image 71 being changed to match the actual motion of the robot main body 11. However, the present disclosure is not limited to this. In the present disclosure, a projected robot image in which the robot main body is represented by a two-dimensional image rather than a three-dimensional model may be displayed. The robot main body in the projected robot image may differ from the actual robot main body in terms of color scheme, shape, or other specifications. Furthermore, the projected robot image may be represented schematically by a relatively simple shape, such as a rectangle or circle. Furthermore, the motion state of the projected robot image may not be changed to match the actual motion of the robot main body. For example, in the projected robot image, only the orientation of the robot arm may be changed to match the actual motion of the robot main body, while the bending and stretching state of the robot arm may not be changed to match the actual motion of the robot main body. In other words, if the robot arm has multiple joints, only the motion states of some of the multiple joints of the robot arm in the projected robot image may be changed to match the actual motion of the robot arm. Furthermore, the operating state of the robot arm in the projected robot image may be changed to match the operating state of the actual robot arm, and the state of the end effector attached to the robot arm may be made different from the state of the actual end effector. Alternatively, the projected robot image may not be displayed, and only the composite image may be displayed on the display unit. Alternatively, a composite image with a superimposed projected robot image and a composite image without the superimposed projected robot image may be displayed on the display unit. Alternatively, the projected robot image may be displayed in a transparent state, superimposed on the composite image. Together with the composite image, a projected image, which is a captured image, a captured image with lens distortion corrected, or a captured image transformed by image processing for position transformation, may be displayed on the display unit.

[0060] Although the above embodiment illustrates an example in which calibration processing is performed using the marker members 80, the present disclosure is not limited to this. In the present disclosure, calibration processing may not be performed. For example, a composite image may be generated without calibration processing using marker members by performing synthesis processing based on information corresponding to the three-dimensional position of the imaging unit. Alternatively, calibration processing may be performed by overlaying a background image, such as the shape of the floor surface or an object placed on the floor surface, without placing marker members. Furthermore, instead of performing calibration processing to set parameters as correction values ​​in advance, a projected image, which is a captured image converted by image processing for position conversion using marker members, may be translated each time a composite image is generated from the captured images. In this case, a background image may also be overlaid without using marker members.

[0061] In the above embodiment, the composite image 70 is generated by combining the captured images 60 captured by the multiple image capturing units 41, 42, and 43 at a common timing. However, the present disclosure is not limited to this. In the present disclosure, the composite image may be generated by combining the captured images captured at different timings. For example, the composite image may be generated by combining a captured image stored in advance with a captured image captured in real time.

[0062] In the above embodiment, the captured images 60 captured from different viewpoints by the multiple image capture units 41, 42, and 43 are converted into projected images 61, which are images captured from different virtual viewpoints for each of the multiple image capture units 41, 42, and 43, and the composite image 70 is generated by combining the projected images 61 from the different virtual viewpoints. However, the present disclosure is not limited to this. In the present disclosure, the composite image may be generated by combining captured images that have been converted to form images from a common virtual viewpoint. For example, a position moved perpendicular to the installation surface from the robot main body may be used as the common virtual viewpoint. Furthermore, when converting multiple captured images into images from a common virtual viewpoint, the virtual viewpoint may be a position diagonally shifted from above the robot main body.

[0063] Furthermore, in the above embodiment, an example was described in which the robot arm 13 of the robot main body 11, which is an industrial robot, is a six-axis vertically articulated robot arm, and the operation device 20 includes an arm member with a parallel link structure. However, the present disclosure is not limited to this. In the present disclosure, the robot arm may include five or fewer joints or seven or more joints. Furthermore, the configuration of the joints of the robot arm of the robot main body is not limited to the above embodiment. For example, the first axis, which is the rotation axis of the joint at the base end, does not have to be perpendicular to the installation surface. When the robot arm is a six-axis vertically articulated robot, the orientation of the multiple joints may be different from that in the above embodiment. Furthermore, the robot arm may be horizontally articulated. Furthermore, multiple robot arms may be arranged. In this case, the imaging unit may be arranged in each of the multiple robot arms, or in one of the multiple robot arms. Furthermore, the operation device may include link members with a serial link structure rather than a parallel link structure. That is, the operation device may include a vertically articulated or horizontally articulated robot. The operating device may include an input device such as an operating unit having a plurality of switches or a joystick. The operating device may also be a keyboard, a pointing device such as a mouse, or a touch panel that also serves as a display unit such as a display. The robot main body may be a medical robot.

[0064] In the above embodiment, the control unit 50 includes the main control unit 51, the servo control unit 52, the drive circuit unit 53, and the image processing unit 54, and the control unit 22 that controls the operation of the operation device 20 is disposed separately from the control unit 50. However, the present disclosure is not limited to this. In the present disclosure, the operation of the robot main body may be controlled by a single control unit rather than two control units, namely, the main control unit and the servo control unit. Furthermore, the control of the robot's operation and the control of the generation of the composite image may be controlled by different control units. Furthermore, the control unit that controls the robot's operation and the control unit that generates the composite image may be disposed in different housings. For example, the control process for generating the composite image may be performed by an image processing control device that is disposed separately from the robot controller that controls the robot's operation. Furthermore, the operation of the robot and the operation of the operation device may be controlled by a common control unit. Furthermore, instead of disposing a drive circuit unit for each of the multiple joints, a common drive circuit unit may be disposed for multiple joints.

[0065] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general-purpose processors, special-purpose processors, integrated circuits, ASICs (Application Specific Integrated Circuits), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. A processor is considered a processing circuit or circuit because it includes transistors and other circuitry. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or processor.

[0066] Aspects It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0067] (Aspect 1) A robot system comprising: a robot main body including a robot arm; a plurality of imaging units arranged in the robot arm of the robot main body at different positions spaced apart from a base end of the robot arm; and a control unit that generates a composite image as a bird's-eye view of the surroundings of the robot main body by combining a plurality of captured images captured by each of the plurality of imaging units.

[0068] (Aspect 2) The robot system according to Aspect 1, wherein the robot arm includes a base end portion disposed at a base end of the robot arm portion and having a fixed distance from an installation surface of the robot main body portion, and an extension portion connected to the base end portion and extending from the base end portion, the imaging unit is disposed at the extension portion of the robot arm portion, and the control unit generates the composite image as a bird's-eye view image including the installation surface around the base end portion.

[0069] (Aspect 3) The robot system according to Aspect 1 or Aspect 2, wherein the control unit acquires information corresponding to a three-dimensional position of each of the plurality of image capturing units that moves in accordance with the operation of the robot arm unit, and performs image transformation on the plurality of captured images based on the acquired information corresponding to the three-dimensional position of each of the plurality of image capturing units to generate the composite image.

[0070] (Aspect 4) The robot system according to Aspect 3, wherein the robot arm unit has a plurality of link members and a joint unit connecting the plurality of link members, and the control unit acquires a rotation angle of the joint unit as information corresponding to a three-dimensional position of each of the plurality of image capturing units.

[0071] (Aspect 5) The robot system according to any one of Aspects 1 to 4, further comprising a drive unit that serves as a drive source for the operation of the robot arm unit, wherein the plurality of image capturing units are fixed to and disposed on the robot arm unit that is operated by the drive unit.

[0072] (Aspect 6) The robot system according to any one of Aspects 1 to 5, wherein the robot arm unit has a plurality of link members and a joint unit connecting the plurality of link members, and the plurality of imaging units are disposed on a common link member.

[0073] (Aspect 7) The robot system according to any one of Aspects 1 to 6, further comprising a display unit that displays the composite image generated by the control unit.

[0074] (Aspect 8) The robot system according to Aspect 7, further comprising an operation device that detects an operation action by an operator, wherein the robot main body operates in conjunction with the operation action detected by the operation device by remote control, and the display unit displays the composite image to the operator performing the operation action by remote control.

[0075] (Aspect 9) In the robot system according to aspect 7 or aspect 8, the control unit is configured to superimpose a projected robot image representing the robot main body projected onto the composite image as a bird's-eye view image on the display unit.

[0076] (Aspect 10) The robot system according to Aspect 9, wherein the control unit superimposes the projected robot image on the composite image and displays the projected robot image on the display unit, with the motion state of the robot main body unit projected in the projected robot image changed to match the actual motion of the robot main body unit.

[0077] (Aspect 11) The robot system according to aspect 9 or aspect 10, wherein the control unit generates the projected robot image representing the robot main body using a three-dimensional model that imitates the robot main body.

[0078] (Aspect 12) A robot system according to any one of Aspects 1 to 11, wherein each of the plurality of imaging units has an imaging range that partially overlaps with each other, and further includes a marker element that is placed at a position where the imaging ranges of the plurality of imaging units overlap, and the control unit generates the composite image based on the position of the common marker element in the plurality of captured images.

[0079] (Aspect 13) The robot system according to any one of Aspects 1 to 12, wherein the control unit generates the composite image by combining the plurality of captured images captured by the plurality of imaging units at a common timing.

[0080] (Aspect 14) An image processing method comprising: acquiring, in a robot arm section, a plurality of captured images taken by a plurality of imaging units arranged at different positions spaced apart from a base end of the robot arm section; and synthesizing the acquired captured images to generate a composite image as a bird's-eye view of the surroundings of a robot main body section including the robot arm section.

Claims

1. A robot system comprising: a robot main body including a robot arm; a plurality of imaging units arranged in the robot arm of the robot main body at different positions spaced apart from a base end of the robot arm; and a control unit that generates a composite image as a bird's-eye view of the surroundings of the robot main body by combining a plurality of images captured by each of the plurality of imaging units.

2. The robot system of claim 1, wherein the robot arm section includes a base end portion disposed at a base end of the robot arm section and having a fixed distance from an installation surface of the robot main body section, and an extension portion connected to the base end portion and extending from the base end portion, the imaging unit is disposed at the extension portion of the robot arm section, and the control unit generates the composite image as a bird's-eye view image including the installation surface around the base end portion.

3. The robot system of claim 1, wherein the control unit acquires information corresponding to the three-dimensional position of each of the multiple imaging units which move in conjunction with the operation of the robot arm unit, and performs image conversion on the multiple captured images based on the acquired information corresponding to the three-dimensional position of each of the multiple imaging units to generate the composite image.

4. A robot system as described in claim 3, wherein the robot arm section has a plurality of link members and a joint section connecting the plurality of link members together, and the control section acquires the rotation angle of the joint section as information corresponding to the three-dimensional position of each of the plurality of imaging sections.

5. The robot system according to claim 1, further comprising a drive unit that serves as a drive source for the operation of the robot arm unit, wherein the multiple imaging units are fixed to and positioned on the robot arm unit which is operated by the drive unit.

6. A robot system according to claim 1, wherein the robot arm section has a plurality of link members and a joint section connecting the plurality of link members, and the plurality of imaging units are arranged on a common link member.

7. The robot system according to claim 1, further comprising a display unit for displaying the composite image generated by the control unit.

8. A robot system as described in claim 7, further comprising an operating device that detects an operating action by an operator, wherein the robot main body operates in conjunction with the operating action detected by the operating device via remote control, and the display unit displays the composite image to the operator performing the operating action via remote control.

9. The robot system according to claim 7, wherein the control unit displays on the display unit a projected robot image representing the robot main body projected onto the composite image as a bird's-eye view image, superimposed on the composite image.

10. A robot system as described in claim 9, wherein the control unit superimposes the projected robot image on the composite image and displays it on the display unit, with the motion state of the robot main body unit projected in the projected robot image being changed to match the actual motion of the robot main body unit.

11. The robot system according to claim 9, wherein the control unit generates the projected robot image representing the robot main body using a three-dimensional model that mimics the robot main body.

12. The robot system of claim 1, wherein each of the multiple imaging units has an imaging range that partially overlaps with each other, and further comprises a marker member that is arranged at a position where the imaging ranges of each of the multiple imaging units overlap, and the control unit generates the composite image based on the position of the common marker member in the multiple captured images.

13. The robot system according to claim 1, wherein the control unit generates the composite image by combining the multiple captured images captured by each of the multiple imaging units at a common timing.

14. An image processing method comprising the steps of: acquiring, in a robot arm section, a plurality of captured images captured by a plurality of imaging units arranged at different positions spaced apart from a base end of the robot arm section; and synthesizing the acquired captured images to generate a composite image as a bird's-eye view of the periphery of a robot main body section including the robot arm section.

Images