Robot system and image processing method

Abstract

To provide a robot system and an image processing method that enable the periphery of a robot body part to be easily checked.SOLUTION: A robot system 100 includes: a robot body part 11 including a robot arm part 13; a plurality of imaging parts 41, 42, and 43 arranged at different positions spaced from a base end of the robot arm part 13 in the robot arm part 13 of the robot body part 11; and a control part 50 for generating a synthesis image as a birds-eye-view image of the periphery of the robot body part 11 by synthesizing a plurality of captured images imaged by each of the plurality of imaging parts 41, 42, and 43.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to a robot system and an image processing method. [Background technology]

[0002] Conventionally, robots equipped with cameras are known. For example, Patent Document 1 describes a robot equipped with an arm having a plurality of joints and a camera arranged on a support to which the arm is connected. The support is fixed to a base fixed to an installation surface, and rotates around a rotation axis perpendicular to the installation surface relative to the base. Two cameras are arranged on the left and right of the support. The robot described in Patent Document 1 performs coordinate conversion on each of images taken by the cameras arranged on the left and right of the support, and generates an image equivalent to one plan view by around-view processing by synthesizing the converted images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7055883 Summary of the Invention [Problem to be solved by the invention]

[0004] However, when a camera serving as an imaging unit is disposed on a support to which a robot arm is connected, as in the robot described in the above Patent Document 1, the imaging unit is disposed close to the installation surface of the robot body including the robot arm, and therefore the range of the display area displayed in the composite image generated from the captured images captured by the imaging unit is considered to be small. In that case, when checking the surroundings of the robot body using the composite image, it is difficult to check the surroundings of the robot body due to the small range of the display area of ​​the composite image.

[0005] This disclosure has been made to solve the above-mentioned problems, and one objective of this disclosure is to provide a robot system and an image processing method that enable easy checking of the surroundings of the robot main body. [Means for solving the problem]

[0006] A robot system according to a first aspect of the disclosure includes a robot main body including a robot arm, a plurality of imaging units arranged in the robot arm of the robot main body at different positions spaced apart from a base end of the robot arm, and a control unit that generates a composite image as a bird's-eye view image of the periphery of the robot main body by combining a plurality of captured images captured by each of the plurality of imaging units. Note that the "base end" here refers to the end of the robot arm on the side of a connection portion to which the robot arm is connected.

[0007] As described above, the robot system according to the first aspect of the disclosure includes a plurality of imaging units arranged at different positions apart from the base end of the robot arm unit in the robot main body unit. The robot system according to the first aspect also includes a control unit that generates a composite image as a bird's-eye view image of the periphery of the robot main body unit by combining a plurality of captured images captured by each of the plurality of imaging units. As a result, since the imaging units are arranged at a position apart from the base end of the robot arm unit, the range of the area captured by the imaging units can be made larger than when the imaging units are arranged at a position closer to the installation surface such as a support column to which the robot arm unit is connected or the base end of the robot arm unit. Therefore, when the periphery of the robot main body unit is checked using the composite image, the range of the display area of ​​the composite image can be made larger than when the imaging units are arranged at a position closer to the installation surface of the robot main body unit. As a result, the periphery of the robot main body unit can be easily checked.

[0008] An image processing method according to a second aspect of this disclosure acquires, in a robot arm section, a plurality of captured images captured by a plurality of imaging units arranged at different positions spaced apart from a base end of the robot arm section, and generates a composite image as a bird's-eye view of the surroundings of a robot main body section including the robot arm section by combining the acquired captured images.

[0009] As described above, the image processing method according to the second aspect of the disclosure acquires a plurality of captured images captured by a plurality of imaging units arranged at different positions away from the base end of the robot arm unit in the robot arm unit, and synthesizes the acquired captured images to generate a composite image as a bird's-eye view image of the surroundings of the robot main body unit including the robot arm unit. As a result, since the imaging units are arranged at positions away from the base end of the robot arm unit, the range of the area captured by the imaging units can be made larger than when the imaging units are arranged at positions closer to the installation surface such as the support column to which the robot arm unit is connected or the base end of the robot arm unit. Therefore, when the surroundings of the robot main body unit are checked using the composite image, the range of the display area of ​​the composite image can be made larger than when the imaging units are arranged at positions closer to the installation surface of the robot main body unit. As a result, an image processing method that allows the surroundings of the robot main body unit to be easily checked can be provided. Effect of the Invention

[0010] According to the present disclosure, the surroundings of the robot main body can be easily checked. [Brief description of the drawings]

[0011] [Figure 1] 1 is a perspective view showing an overall configuration of a robot system according to an embodiment. [Diagram 2] FIG. 2 is a perspective view for explaining the configuration of a robot main body. [Diagram 3] FIG. 1 is a block diagram showing a configuration of a robot system. [Figure 4] 1A to 1C are diagrams illustrating the generation of a composite image based on captured images. [Diagram 5] 11 is a flowchart illustrating an image processing method performed by the robot system. [Figure 6] FIG. 2 is a diagram for explaining viewpoint conversion of a captured image. [Figure 7] 11A and 11B are diagrams for explaining a calibration process using a marker member. [Figure 8] FIG. 13 is a perspective view showing a robot main body according to a modified example of the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings.

[0013] The configuration of a robot system 100 according to the present embodiment will be described with reference to FIGS.

[0014] (Robot system configuration) As shown in FIG. 1, the robot system 100 includes a robot 10, an operating device 20, and a display unit 30. The robot 10 includes a robot main body 11. The robot main body 11 includes a robot arm 13 attached to a base 12. The operating device 20 detects an operating action by an operator. The robot main body 11 of the robot 10 operates in conjunction with an operating action detected by the operating device 20 by remote control. That is, the robot system 100 is a master-slave system in which the robot 10 operates in conjunction with an operating action by an operator detected by the operating device 20. In the robot system 100, the operating device 20 is a master device, and the robot 10 is a slave device. The robot 10 and the operating device 20 are connected to each other via a network. The robot 10 is an industrial robot that performs handling such as processing, assembly, transport, or inspection. The robot 10 also includes a plurality of imaging units 41, 42, and 43, and a control unit 50.

[0015] As shown in FIG. 2, the robot arm section 13 has a portion 14, a portion 15, and a portion 16. The portion 14 is disposed at the base end of the robot arm section 13. The portions 15 and 16 are connected to the portion 14 and disposed so as to extend from the portion 14. Each of the portions 15 and 16 has a long rod-like shape extending from the base end side to the tip side. The portion 14 has two link members 14a and 14b. The portion 15 has a link member 15a. The portion 16 has three link members 16a, 16b, and 16c. The robot arm section 13 is a multi-joint robot arm having a plurality of joints 17. For example, six joints 17 are disposed in the robot arm section 13. That is, the robot 10 is a six-axis vertical multi-joint robot. The six joints 17 connect the link members 14a, 14b, 15a, 16a, 16b, and 16c to each other and to an end effector connected to the tip of the robot arm unit 13, and rotate around the JT1, JT2, JT3, JT4, JT5, and JT6 axes, respectively. A driving unit 18 such as a motor and a position detection unit 19 such as an encoder, shown in FIG. 3, are disposed in each of the joints 17. The portion 14 is an example of a base end portion. The portions 15 and 16 are examples of extension portions.

[0016] The robot arm unit 13 is connected to a base unit 12 that is disposed on an installation surface, such as a floor surface. A portion 14 of the robot arm unit 13 is connected to the base unit 12. A base end side of a portion 15 is connected to the portion 14. A tip end side of the portion 15 is connected to a portion 16. A base end side of the portion 16 is connected to the portion 15, and an end effector is attached to the tip end side of the portion 16.

[0017] Specifically, the link member 14a of the part 14 is fixed to the base 12, and the link member 14b rotates about the JT1 axis relative to the link member 14a, whereby the part 14 rotates about the JT1 axis relative to the base 12. The JT1 axis extends in a direction perpendicular to the installation surface on which the robot body 11 is installed. That is, the part 14 is disposed at a constant distance from the installation surface of the robot body 11 during the operation of the robot arm 13. In the part 15, the link member 15a is connected to the link member 14b of the part 14 so as to rotate about the JT2 axis. The JT2 axis extends in a direction perpendicular to the JT1 axis. The link member 15a is disposed so as to extend along a direction perpendicular to the JT2 axis, and is connected at its tip end to the link member 16a of the part 16 so as to rotate about the JT3 axis. The JT3 axis is parallel to the JT2 axis. The portion 16 is disposed so as to extend along a direction perpendicular to the JT3 axis. In the portion 16, the link member 16b is connected to the link member 16a so as to rotate around the JT4 axis, and the link member 16c is connected to the link member 16b so as to rotate around the JT5 axis. The JT4 axis is perpendicular to the JT3 axis and is disposed along the direction in which the portion 16 extends. The JT5 axis is perpendicular to the JT4 axis. The end effector is attached to the tip of the portion 16 so as to rotate around the JT6 axis with respect to the link member 16c. The JT6 axis is perpendicular to the JT5 axis. That is, the distance from the installation surface of the portion 15 and the portion 16 changes with the movement of the robot arm unit 13, while the distance from the installation surface of the portion 14 is constant. In other words, the height positions of the portions 15 and 16 from the installation surface change with the movement of the robot arm unit 13. It should be noted that an end effector is omitted in Fig. 2. The end effector includes, for example, a hand for gripping a workpiece or the like, or a processing tool for performing processing.

[0018] In this embodiment, three imaging units 41, 42, and 43 are arranged in the robot arm unit 13 of the robot main body unit 11 at different positions spaced apart from the base end of the robot arm unit 13. Each of the imaging units 41, 42, and 43 has an imaging element such as a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures an image of the surroundings of the robot main body unit 11. The imaging units 41, 42, and 43 may be three-dimensional cameras. The imaging units 41, 42, and 43 are arranged in the link member 16a including the joint portion 17 of the JT3 axis where the portion 15 and the portion 16 are connected to each other in the portion 16 whose height position from the installation surface changes. That is, the imaging units 41, 42, and 43 are arranged in a common link member 16a. In other words, the imaging units 41, 42, and 43 are disposed at positions through at least two or more joint axes from the base end in the robot arm unit 13 having a plurality of joints 17. That is, the imaging units 41, 42, and 43 are disposed in the link member 16a of the link members 15a, 16a, 16b, and 16c, which is located on the tip side of at least two or more joints 17 between the base end and the joints 17 on the JT1 axis and the JT2 axis in the robot arm unit 13. The imaging units 41, 42, and 43 are disposed inside the housing of the link member 16a. The imaging units 41, 42, and 43 capture images of the installation surface on which the robot main body unit 11 is disposed through imaging holes disposed in the housing of the link member 16a.

[0019] The imaging units 41, 42, and 43 are fixed to the link member 16a of the part 16 of the robot arm unit 13 operated by the drive unit 18, and move together with the robot arm unit 13 in accordance with the operation of the robot arm unit 13. That is, the imaging directions of the imaging units 41, 42, and 43 are changed in accordance with the change in the posture of the robot arm unit 13. In the robot system 100, no drive mechanism is provided for changing the orientation of the imaging units 41, 42, and 43 separately from the operation of the robot arm unit 13. Each of the imaging units 41, 42, and 43 has a wide-angle lens or a fisheye lens. The imaging ranges of the imaging units 41, 42, and 43 overlap with each other in part. Each of the imaging units 41, 42, and 43 captures an image of the periphery of the part 14 arranged at the base end of the robot arm unit 13 in the robot main body unit 11 by combining the imaging ranges of each other. For example, with a state in which part 15 is arranged perpendicular to the installation surface and part 16 is arranged in a direction along the installation surface as a reference posture, imaging units 41, 42, and 43 combine their imaging ranges with robot arm unit 13 in the reference posture to capture images of the entire periphery of part 14 in robot main body unit 11. Furthermore, by being disposed on part 16, imaging units 41, 42, and 43 are disposed at positions on robot arm unit 13 where the distance from the base end of robot arm unit 13 changes as robot arm unit 13 moves.

[0020] 3, the control unit 50 of the robot 10 includes a main control unit 51, a servo control unit 52, a drive circuit unit 53, and an image processing unit 54. Each of the main control unit 51 and the servo control unit 52 includes a calculation device such as a CPU (Central Processing Unit). The control unit 50 is a robot controller that controls the operation of the robot 10. A storage device such as a hard disk drive is arranged in the control unit 50, and the calculation device executes control processing based on programs and parameters stored in the storage device.

[0021] The main control unit 51 controls the overall operation of the robot 10. The servo control unit 52 controls the power supplied to each of the drive units 18 of the six joint units 17 based on a command from the main control unit 51. The drive circuit unit 53 supplies drive power to the drive units 18 of the six joint units 17. The drive units 18 include servo motors that serve as drive sources for the operation of the robot arm unit 13. The drive circuit unit 53 includes an inverter circuit that supplies AC power to the drive units 18. In the drive units 18, the rotation shaft of the servo motor rotates due to the three-phase AC power supplied from the drive circuit unit 53. The rotation of the rotation shaft of the servo motor of the drive unit 18 rotates each of the multiple joint units 17, and the robot arm unit 13 operates. The drive circuit unit 53 is disposed for each of the six joint units 17. In addition, a position detection unit 19 is disposed in each of the drive units 18 of the six joint units 17. The position detection unit 19 detects the rotation angle of the joint unit 17. The position detection unit 19 includes an encoder that detects the rotation angle of the joint unit 17 by detecting the rotation angle of the servo motor of the drive unit 18, for example. The position detection unit 19 outputs a detection value indicating the detected rotation angle of the servo motor of the drive unit 18 to the main control unit 51 and the servo control unit 52. The image processing unit 54 includes an FPGA (Field Programmable Gate Array) dedicated to image processing. The image processing unit 54 generates a captured image 60 shown in FIG. 4 captured by each of the image capturing units 41, 42, and 43 by acquiring signals from the image capturing units 41, 42, and 43. The image processing unit 54 also generates a composite image 70 based on the generated captured image 60. The generation of the composite image 70 based on the captured image 60 by the image processing unit 54 of the control unit 50 will be described in detail later.

[0022] As shown in FIG. 1, the operation device 20 includes a gripping unit 21 and a control unit 22. The gripping unit 21 is gripped by an operator to control the operation of the robot 10. The gripping unit 21 is changed in position and inclination by an operation operation by the operator while being gripped by both hands of the operator, for example. For example, the gripping unit 21 is supported by an arm member having a parallel link structure. An encoder for detecting a rotation angle and a force sensor are disposed on the arm member supporting the gripping unit 21. The control unit 22 controls the operation of each part of the operation device 20. The control unit 22 includes a calculation device such as a CPU and a storage device, similar to the control unit 50 of the robot 10, and executes control processing by the calculation device based on a program, parameters, and the like stored in the storage device. The control unit 22 also detects an operation operation applied to the gripping unit 21 by the encoder and the force sensor.

[0023] Each of the control unit 22 and the control unit 50 includes a communication module. The control unit 22 generates a control signal for operating the robot 10 based on the detected operating action, and outputs the generated control signal to the control unit 50 of the robot 10 via the communication module. The control unit 50 outputs the generated composite image 70 to the control unit 22 via the communication module, thereby causing the display unit 30 to display the composite image 70. The display unit 30 displays the composite image 70 generated by the control unit 50 to an operator who performs an operating action on the operating device 20. The display unit 30 includes, for example, a liquid crystal monitor. The display unit 30 is disposed at a position where it is visible to an operator who holds the grip unit 21 of the operating device 20.

[0024] The robot system 100 operates as a remotely controlled master-slave system by transmitting and receiving signals between the control unit 50 of the robot 10 and the control unit 22 of the operation device 20. The robot system 100 performs bilateral control in which the posture and force between the robot 10 and the operation device 20 follow each other. Based on an operation signal from the control unit 22, the robot 10 changes the posture of the robot arm unit 13 so as to follow the posture, including the position and angle, of the gripper 21 changed by the operation action of the operator.

[0025] <Generation of synthetic images> As shown in FIG. 4, in this embodiment, the control unit 50 generates a composite image 70 as a bird's-eye view image of the periphery of the robot main body unit 11 by combining a plurality of captured images 60. The composite image 70 is a bird's-eye view image including the installation surface around the portion 14 of the robot main body unit 11. The control unit 50 generates the composite image 70 projected from a virtual viewpoint above the robot main body unit 11 onto the installation surface on which the robot main body unit 11 is installed, by performing around-view processing on the captured images 60. The virtual viewpoint in the composite image 70 is, for example, a position moved upward from the robot main body unit 11 along a direction perpendicular to the installation surface. In generating the composite image 70, the control unit 50 performs image conversion by performing image processing of viewpoint conversion on each of the three captured images 60 captured by each of the three imaging units 41, 42, and 43, and generates the composite image 70 by combining the three captured images 60 that have been subjected to image processing of viewpoint conversion.

[0026] The control unit 50 also displays on the display unit 30 a projected robot image 71 representing the robot main body 11 projected onto the composite image 70 as a bird's-eye view image, superimposed on the composite image 70. In the projected robot image 71, the robot main body 11 is represented by a three-dimensional model simulating the robot main body 11. The projected robot image 71 is an image obtained by capturing the three-dimensional model simulating the robot main body 11 from a position corresponding to a virtual camera viewpoint Q, which is a virtual viewpoint from which the composite image 70 is generated. That is, the projected robot image 71 is an image representing the robot main body 11 when the actual robot main body 11 is projected onto an installation surface, which is a plane on which the composite image 70 is generated, from a virtual viewpoint above the robot main body 11. The control unit 50 also displays on the display unit 30 the projected robot image 71 superimposed on the composite image 70, with the operating state of the robot main body 11 projected in the projected robot image 71 changed to match the operating state of the actual robot main body 11. For example, the control unit 50 acquires the rotation angle of each of the joints 17 based on a signal from the position detection unit 19. Then, the control unit 50 acquires the posture of the robot arm unit 13 of the robot main body unit 11 from the acquired rotation angle of each of the joints 17. Then, the control unit 50 changes the posture of the three-dimensional model simulating the robot main body unit 11 so as to correspond to the acquired posture of the actual robot arm unit 13, thereby making the operating state of the robot main body unit 11 in the projected robot image 71 follow the actual robot main body unit 11. In other words, the control unit 50 changes the posture of the robot main body unit 11 of the three-dimensional model in the projected robot image 71 in synchronization with the rotation angle of the joints 17 of the actual robot main body unit 11. For example, in the actual robot main body 11, when the joint 17 of the portion 14 is rotated around the JT1 axis by the drive of the drive unit 18, the control unit 50 moves the robot main body 11 in the projected robot image 71 displayed superimposed on the composite image 70 around the JT1 axis by the same angle as the actual robot main body 11, in accordance with the movement of the actual robot main body 11. The control unit 50 generates the composite image 70 as a moving image based on the captured images 60 as a moving image captured in real time.Then, the control unit 50 superimposes a projected robot image 71 that follows the actual movement of the robot main body 11 in real time on the composite image 70 and outputs the superimposed image to the control unit 22 so as to display the image on the display unit 30 .

[0027] (Image processing method by robot system) An image processing method by the robot system 100 in this embodiment will be described based on a flowchart with reference to Figures 5 to 7. Note that the control process from step S1 to step S5 in Figure 5 is executed by the control unit 50 of the robot 10.

[0028] First, in step S1, captured images 60 are acquired by each of the multiple imaging units 41, 42, and 43. The control unit 50 generates three captured images 60 based on signals from each of the imaging units 41, 42, and 43. The control unit 50 generates the three captured images 60 captured by each of the three imaging units 41, 42, and 43 at a common timing as a moving image.

[0029] Next, in step S2, image conversion is performed on the captured images 60. Specifically, image processing for viewpoint conversion is performed on each of the three captured images 60. The control unit 50 acquires information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43 that move with the operation of the robot arm unit 13, and performs image conversion on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43. The control unit 50 acquires the rotation angle of the joint unit 17 as information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43.

[0030] As shown in FIG. 6, the control unit 50 performs image processing of viewpoint conversion so that each of the three captured images 60 becomes an image captured from a virtual camera viewpoint Q, which is a virtual viewpoint, to generate a projected image 61, which is a captured image 60 with a converted viewpoint. Note that FIG. 6 illustrates one of the three captured images 60. First, the control unit 50 corrects distortion of the captured image 60 captured by a wide-angle lens or a fisheye lens before performing image processing of viewpoint conversion. Then, the control unit 50 performs image processing of viewpoint conversion on the captured image 60 in which the distortion of the imaging units 41, 42, and 43 has been corrected so that the captured image 60 becomes an image captured from a virtual camera viewpoint Q. In the image processing of viewpoint conversion, in FIG. 6, when the actual camera viewpoint where each of the imaging units 41, 42, and 43 is located is P, and the virtual virtual camera viewpoint after the viewpoint conversion is Q, the camera coordinate system X of the camera viewpoint P is P ,Y P ,Z P From the virtual camera coordinate system X of the virtual camera viewpoint Q Q ,Y Q ,Z Q In this case, the rotation matrix R that represents the rotational movement that transforms the orientation of the coordinate system is expressed by the following formula (1).

number

number

number

number

[0031] The control unit 50 converts the viewpoint of the captured image 60 captured by each of the multiple image capturing units 41, 42, and 43 into a projection image 61 for synthesis by using the viewpoint transformation matrix TR expressed by the formula (3). P ,Y P ,Z P Each pixel m of the captured image 60 P(x i ,y i ) into the virtual camera coordinate system X Q ,Y Q ,Z Q By converting each pixel m of the projection image 61 for synthesis into Q (x' i ,y' i ) is acquired. The control unit 50 converts three captured images 60 captured by the three imaging units 41, 42, and 43 arranged at different positions into three projected images 61, which are images from a virtual camera viewpoint Q, which is a virtual viewpoint different for each of the multiple imaging units 41, 42, and 43. That is, the virtual viewpoint in the projected image 61 obtained by image-converting the captured image 60 is in a different position for each of the multiple imaging units 41, 42, and 43. For example, the projected image 61 is an image obtained by projecting the captured image 60 onto an installation surface.

[0032] For example, as shown in FIG. 7, the control unit 50 determines three different positions spaced from the robot main body 11 around the installation surface as positions of virtual camera viewpoints Q when the robot main body 11 is viewed from a direction perpendicular to the installation surface. The control unit 50 also determines the distances from the installation surface of the three different virtual camera viewpoints Q to be equal. The control unit 50 also determines the orientation of the viewpoints of the three different virtual camera viewpoints Q to be a common orientation so that the generated composite image 70 becomes an image projected onto a single plane. For example, in the virtual camera coordinate system X Q ,Y Q ,Z Q In Z Q are common to all of the imaging units 41, 42, and 43. In this manner, the control unit 50 acquires the rotation angle of the joint 17 as information corresponding to the three-dimensional positions of each of the imaging units 41, 42, and 43, and performs arithmetic processing using the acquired rotation angle of the joint 17 and preset parameters to convert the captured images 60 into projected images 61 from different viewpoints. By converting the captured images 60 into projected images 61 from different viewpoints, it is possible to enlarge the display area of ​​the generated composite image 70.

[0033] Next, in step S3, a composite image 70 is generated. The control unit 50 performs the image conversion in step S2 on the captured image 60 to generate the composite image 70. The control unit 50 generates the composite image 70 by combining the projected images 61, which are the captured images 60 converted by the image conversion in step S2. In combining the projected images 61, the control unit 50 generates the composite image 70 using a correction value as a parameter previously set by a calibration process. The calibration process will be described in detail later. The control unit 50 arranges the projected images 61 side by side based on the position of the virtual camera viewpoint Q. Then, the control unit 50 corrects the projected images 61 based on the correction value previously set by the calibration process, and generates the composite image 70 by combining the corrected projected images 61. In the correction based on the correction value, for example, the value of the transformation matrix TR in step S2 may be corrected, or affine transformation such as enlargement / reduction, rotation, and translation may be performed on the projected images 61 generated in step S2. In generating the composite image 70, a part of the projected image 61, which is the converted captured image 60, is cut out and then composited. For example, a part of the lower side of the projected image 61 is cut out so that the installation surface is included in the composite image 70, and the cut out parts of the projected images 61 are superimposed on each other to generate the composite image 70. In addition, the composite image 70 is cut out so as to have a rectangle centered on the position of the robot main body 11.

[0034] Next, in step S4, a projected robot image 71 is generated, and the generated projected robot image 71 is superimposed on the composite image 70. The control unit 50 superimposes the projected robot image 71 on the composite image 70 generated in step S3. The control unit 50 acquires the motion state of the actual robot main body unit 11 based on the rotation angle of the joint unit 17 acquired in step S2, and generates the projected robot image 71 by changing the motion state of the robot main body unit 11 in the three-dimensional model so as to correspond to the acquired motion state.

[0035] Next, in step S5, the generated composite image 70 is displayed on the display unit 30. Specifically, the control unit 50 outputs the composite image 70 with the projected robot image 71 superimposed thereon to the control unit 22 of the operation device 20, thereby causing the control unit 22 to display the composite image 70 with the projected robot image 71 superimposed thereon on the display unit 30.

[0036] The control unit 50 executes the processes from step S1 to step S5 at every predetermined control period, thereby displaying the continuously generated composite image 70 on the display unit 30. The predetermined control period is, for example, any value between 10 milliseconds and 100 milliseconds. That is, the composite image 70 with the projected robot image 71 superimposed thereon is generated as a moving image, and the composite image 70 as a moving image that follows the operation of the robot main body 11 in real time is displayed on the display unit 30. Note that the term "real time" used here is described as a concept that includes errors due to delays in control processing or communication.

[0037] Calibration Processing 7, the control unit 50 corrects parameters of the control process in generating the composite image 70 in step S3 by performing in advance a calibration process using the marker members 80. In the calibration process, the control unit 50 generates the composite image 70 based on the positions of the common marker members 80 in the multiple captured images 60.

[0038] The calibration process is performed, for example, at the time of shipment from a factory or at the time of periodic maintenance of the robot system 100. In the calibration process, the marker members 80 are arranged at positions where the imaging ranges of the imaging units 41, 42, and 43 overlap. In the robot system 100, the three imaging units 41, 42, and 43 are arranged to capture images of the periphery of the robot main body unit 11. Therefore, in the calibration process, the marker members 80 are arranged at three positions: a position where the imaging ranges of the imaging units 41 and 42 overlap, a position where the imaging ranges of the imaging units 42 and 43 overlap, and a position where the imaging ranges of the imaging units 41 and 43 overlap. The control unit 50 converts the captured image 60 in step S2 based on preset parameters and a detection value from the encoder of the position detection unit 19. Then, the control unit 50 generates a composite image 70 by combining the projected images 61, which are the captured images 60 whose viewpoints have been converted. In the calibration process, the control unit 50 generates a composite image 70 based on the positions of the common marker members 80 in the multiple captured images 60. For example, in generating the composite image 70, the control unit 50 performs image conversion on each of the projected images 61, which are the converted captured images 60, so that the marker members 80 in each of the projected images 61 overlap with each other. The control unit 50 stores parameters of the image conversion in which the marker members 80 overlap with each other as correction values ​​in generating the composite image 70.

[0039] [Effects of this embodiment] The robot system 100 includes a plurality of imaging units 41, 42, and 43 arranged at different positions in the robot arm unit 13 of the robot main body unit 11, the imaging units 41, 42, and 43 being spaced apart from the base end of the robot arm unit 13. The robot system 100 also includes a control unit 50 that generates a composite image 70 as a bird's-eye view image of the periphery of the robot main body unit 11 by combining a plurality of captured images 60 captured by each of the plurality of imaging units 41, 42, and 43. As a result, since the imaging units 41, 42, and 43 are arranged at positions spaced apart from the base end of the robot arm unit 13, the range of the area captured by the imaging units 41, 42, and 43 can be made larger than when the imaging units 41, 42, and 43 are arranged at positions close to the installation surface, such as the support column to which the robot arm unit 13 is connected, the base unit 12, or the base end of the robot arm unit 13. Therefore, when checking the surroundings of the robot main body unit 11 using the composite image 70, the range of the display area of ​​the composite image 70 can be made larger than when the imaging units 41, 42, and 43 are disposed in positions closer to the installation surface of the robot main body unit 11. As a result, the surroundings of the robot main body unit 11 can be easily checked.

[0040] The robot arm unit 13 includes a part 14 as a base end part arranged at a fixed distance from the installation surface of the robot main body unit 11 at the base end of the robot arm unit 13, and an extension part consisting of parts 15 and 16 connected to the part 14 and extending from the part 14. The imaging units 41, 42, and 43 are arranged in the extension part consisting of the parts 15 and 16 in the robot arm unit 13. The control unit 50 generates a composite image 70 as a bird's-eye view image including the installation surface around the part 14. As a result, a composite image 70 is generated in which the installation surface around the part 14 arranged at the base end of the robot arm unit 13 is included in the display range, so that when the robot arm unit 13 is operated, physical interference between the robot arm unit 13 and an object arranged on the installation surface around the part 14 can be easily confirmed. Therefore, the surroundings of the robot main body unit 11 can be more easily confirmed.

[0041] The control unit 50 acquires information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43 that move with the operation of the robot arm unit 13, and performs image conversion on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43 to generate a composite image 70. As a result, even if the three-dimensional positions of the multiple imaging units 41, 42, and 43 are changed with the operation of the robot arm unit 13, the control unit 50 can acquire information corresponding to the three-dimensional positions of each of the multiple imaging units 41, 42, and 43, thereby generating a composite image 70 corresponding to the change in the positions of the multiple imaging units 41, 42, and 43. Therefore, when the robot arm unit 13 in which the multiple imaging units 41, 42, and 43 are arranged is operated, the surroundings of the robot main body unit 11 can be effectively and easily confirmed.

[0042] The robot arm unit 13 has a plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c, and a joint unit 17 connecting the plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c to each other. The control unit 50 acquires a rotation angle of the joint unit 17 as information corresponding to the three-dimensional positions of each of the plurality of imaging units 41, 42, and 43. As a result, by acquiring the rotation angle of the joint unit 17, the three-dimensional positions of the plurality of imaging units 41, 42, and 43 can be easily acquired, so that the surroundings of the robot main body unit 11 can be more easily confirmed.

[0043] The robot system 100 includes a driving unit 18 that serves as a driving source for the operation of the robot arm unit 13. The multiple imaging units 41, 42, and 43 are fixedly disposed on the robot arm unit 13 that is operated by the driving unit 18. This allows the position and orientation of each of the multiple imaging units 41, 42, and 43 to be changed by operating the robot arm unit 13 with the driving unit 18 without providing a driving mechanism for changing the orientation of each of the multiple imaging units 41, 42, and 43. Therefore, by operating the robot arm unit 13 with the driving unit 18, a wider range can be captured by the multiple imaging units 41, 42, and 43, so that a wider range around the robot main body unit 11 can be easily confirmed.

[0044] The robot arm 13 has a plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c, and a joint 17 connecting the plurality of link members 14a, 14b, 15a, 16a, 16b, and 16c. The plurality of imaging units 41, 42, and 43 are arranged on a common link member 16a. This allows the height positions of the plurality of imaging units 41, 42, and 43 to be aligned with respect to the installation surface on which the robot main body 11 is installed. Therefore, when the plurality of captured images 60 captured by the plurality of imaging units 41, 42, and 43 are synthesized, the variation in the areas included in the plurality of captured images 60 can be suppressed, and the generated composite image 70 can be prevented from becoming unclear. In addition, since the height positions of the plurality of imaging units 41, 42, and 43 can be aligned with respect to the installation surface on which the robot main body 11 is installed, the burden of the control process in generating the composite image 70 can be reduced compared to the case where the height positions are different.

[0045] The robot system 100 includes a display unit 30 that displays a composite image 70 generated by a control unit 50. This allows the operator to easily check the surroundings of the robot main body unit 11 by visually recognizing the composite image 70 displayed on the display unit 30.

[0046] The robot system 100 includes an operating device 20 that detects an operating action by an operator. The robot main body 11 operates in conjunction with the operating action detected by the operating device 20 by remote control. The display unit 30 displays a composite image 70 to the operator performing the operating action by remote control. In this way, when remotely controlling the operation of the robot main body 11 by remote control using the operating device 20, the operator can visually recognize the composite image 70 displayed on the display unit 30, thereby effectively and easily checking the surroundings of the robot main body 11.

[0047] The control unit 50 superimposes a projected robot image 71, which represents the robot main body 11 projected onto the composite image 70 as a bird's-eye view image, on the composite image 70 and displays it on the display unit 30. This allows the operator to easily recognize the positional relationship between the robot 10's surroundings and the robot main body 11 by visually recognizing the composite image 70 on which the projected robot image 71 is superimposed.

[0048] The control unit 50 changes the motion state of the robot main body unit 11 projected in the projected robot image 71 to match the motion of the actual robot main body unit 11, and displays the projected robot image 71 on the composite image 70 on the display unit 30. As a result, the motion state of the projected robot image 71 superimposed on the composite image 70 is changed to match the motion of the actual robot main body unit 11, so that the operator can intuitively and easily recognize the positional relationship between the surroundings of the robot 10 and the robot main body unit 11 by visually recognizing the composite image 70 on which the projected robot image 71 is superimposed.

[0049] The control unit 50 generates a projected robot image 71 that represents the robot main body unit 11 using a three-dimensional model that imitates the robot main body unit 11. As a result, in the projected robot image 71 that is superimposed on the composite image 70, the robot main body unit 11 is represented using a three-dimensional model that imitates the actual robot main body unit 11, so that the operator can more intuitively and easily recognize the positional relationship between the surroundings of the robot 10 and the robot main body unit 11 by visually recognizing the composite image 70 on which the projected robot image 71 is superimposed.

[0050] The imaging ranges of the multiple imaging units 41, 42, and 43 overlap with each other in part. The robot system 100 includes a marker member 80 disposed at a position where the imaging ranges of the multiple imaging units 41, 42, and 43 overlap with each other. The control unit 50 generates a composite image 70 based on the position of the common marker member 80 in the multiple captured images 60. By generating the composite image 70 based on the position of the marker member 80, it is possible to suppress the occurrence of a shift in the generated composite image 70 when the multiple captured images 60 captured by the multiple imaging units 41, 42, and 43 are combined. Therefore, the multiple captured images 60 can be combined with higher accuracy, and the surroundings of the robot main body 11 can be easily and accurately confirmed by the generated composite image 70.

[0051] The control unit 50 generates a composite image 70 by combining a plurality of captured images 60 captured at a common timing by the plurality of imaging units 41, 42, and 43. This makes it possible to check the state of the surroundings of the robot main body unit 11 more accurately than when combining the captured images 60 captured at different timings by the plurality of imaging units 41, 42, and 43.

[0052] (Effects of the image processing method by the robot system 100 of this embodiment) The image processing method by the robot system 100 acquires a plurality of captured images 60 captured by each of the plurality of imaging units 41, 42, and 43 arranged at different positions away from the base end of the robot arm unit 13 in the robot arm unit 13, and synthesizes the acquired captured images 60 to generate a composite image 70 as a bird's-eye view image of the surroundings of the robot main body unit 11 including the robot arm unit 13. As a result, since the imaging units 41, 42, and 43 are arranged at positions away from the base end of the robot arm unit 13, the range of the area captured by the imaging units 41, 42, and 43 can be made larger than when the imaging units 41, 42, and 43 are arranged at positions closer to the installation surface such as the support column to which the robot arm unit 13 is connected, the base unit 12, or the base end of the robot arm unit 13. Therefore, when the surroundings of the robot main body unit 11 are checked using the composite image 70, the range of the display area of ​​the composite image 70 can be made larger than when the imaging units 41, 42, and 43 are arranged at positions closer to the installation surface of the robot main body unit 11. As a result, an image processing method that allows the surroundings of the robot main body 11 to be easily confirmed can be provided.

[0053] [Variations] It should be noted that the embodiments disclosed herein are illustrative and not restrictive in all respects. The scope of the present disclosure is indicated by the claims, not by the description of the embodiments above, and further includes all modifications (variations) within the meaning and scope equivalent to the claims.

[0054] For example, in the above embodiment, an example was shown in which the composite image 70 was generated as a bird's-eye view image including the installation surface around the portion 14 as the base end portion, but the present disclosure is not limited to this. In the present disclosure, the imaging range of each of the multiple imaging units is changed with the movement of the robot arm unit, so the installation surface around the base end portion may not be included in the generated composite image. For example, the robot main body may be installed on a ceiling surface or a wall surface. In that case, since the installation surface on which the robot main body is installed is the ceiling surface or a wall surface, the robot arm unit in the robot main body is also placed on the ceiling surface or a wall surface. In that case, in order to check physical interference between the robot arm unit and an object placed on the floor surface, a composite image may be generated as a bird's-eye view image including the floor surface in the operating range of the robot arm unit in the range of the display area. With this configuration, the surroundings of the robot main body can be checked more effectively and easily not only when the robot main body is installed on the floor surface, but also when it is installed on the ceiling surface or a wall surface.

[0055] In the above embodiment, the rotation angle of the joint 17 is acquired as information corresponding to the three-dimensional position of each of the multiple imaging units 41, 42, and 43, and the composite image 70 is generated by performing image conversion on the multiple captured images 60 based on the acquired information corresponding to the three-dimensional position of each of the multiple imaging units 41, 42, and 43. However, the present disclosure is not limited to this. In the present disclosure, information corresponding to the three-dimensional position of each of the multiple imaging units may not be acquired from a sensor or the like. For example, when capturing an image for generating a composite image, the robot main body may be set to a preset posture, so that the composite image is generated by preset parameters without acquiring information corresponding to the three-dimensional position of each of the multiple imaging units. Also, information corresponding to the three-dimensional position of each of the multiple imaging units may be acquired by arranging a sensor that detects the position of the imaging unit instead of the rotation angle of the joint. For example, a sensor that detects the position or posture of the robot arm unit may be arranged to detect the three-dimensional position of each of the multiple imaging units based on the detected position or posture of the robot arm unit. As a sensor for detecting the position or posture of the robot arm, or a sensor for directly detecting the position of the imaging unit, a photoelectric sensor, a gyro sensor, a capacitance sensor, a magnetic sensor, a force sensor such as a piezoelectric element, or a mechanical switch may be arranged. In order to obtain the three-dimensional position of each of the multiple imaging units, an imaging unit separate from the multiple imaging units that capture images for generating a composite image may be arranged. In addition, when obtaining the rotation angle of the joint as information corresponding to the three-dimensional position of each of the multiple imaging units, the rotation angle of the joint may be obtained by a command value for the drive unit or a signal from an operating device, instead of a signal from an encoder.

[0056] In the above embodiment, the three imaging units 41, 42, and 43 are fixed and arranged inside the housing in the link member 16a of the part 16 as the tip side of the common extension part, but the present disclosure is not limited to this. In the present disclosure, the multiple imaging units may be arranged in different link members. Also, the multiple imaging units may be arranged not in the tip side part of the extension part of the robot arm part, but in the base end side part of the extension part. The multiple imaging units may be arranged separately in the tip side part and the base end side part of the extension part of the robot arm part. Also, the number of imaging units may be two or four or more. Also, the imaging units may be arranged outside the housing of the robot arm part. Also, the imaging units may not have a wide-angle lens and a fisheye lens. Also, a driving mechanism for changing the orientation of the imaging units may be arranged.

[0057] In the above embodiment, an example has been shown in which the robot arm unit 13 on which the imaging units 41, 42, and 43 are arranged is connected to the base unit 12 arranged on the installation surface, but the present disclosure is not limited to this. In the present disclosure, as in a robot main body unit 211 according to a modified example shown in Fig. 8, the robot arm unit 213 on which the imaging units 41, 42, and 43 are arranged may be attached to a cart 212 that moves by driving wheels. Also, the robot arm unit on which the imaging units are arranged may be directly arranged on the installation surface.

[0058] In the above embodiment, the composite image 70 is displayed by the display unit 30 to the operator who performs the operation operation by remote control using the operation device 20, but the present disclosure is not limited to this. In the present disclosure, a composite image may be generated and displayed from an image captured by an imaging unit disposed in the robot main body unit that is automatically operating based on preset parameters. In addition, a composite image may be generated to check the surroundings of the robot main body unit when performing a teaching operation to set the operation of the robot main body unit. In addition, a composite image may be generated and displayed to perform maintenance and inspection of the robot main body unit. In addition, a composite image may be displayed to a worker who directly views the robot main body unit, not by remote control. A composite image may be displayed on the display unit by a control device that controls the operation of the robot main body unit.

[0059] In the above embodiment, the projected robot image 71 is displayed superimposed on the composite image 70 in a state in which the operating state of the robot main body 11 according to the three-dimensional model in the projected robot image 71 is changed according to the operating state of the actual robot main body 11, but the present disclosure is not limited to this. In the present disclosure, a projected robot image in which the robot main body is represented by a two-dimensional image, rather than a three-dimensional model, may be displayed. The robot main body in the projected robot image may have specifications such as a color scheme or a shape different from those of the actual robot main body. In addition, the robot main body may be represented in a schematic manner by a relatively simple figure such as a rectangle or a circle in the projected robot image. In addition, the operating state of the projected robot image may not be changed according to the operating state of the actual robot main body. For example, in the projected robot image, only the orientation of the robot arm may be changed according to the operating state of the actual robot main body, while the bending and stretching state of the robot arm may not be changed according to the operating state of the actual robot main body. In other words, when the robot arm has a plurality of joints, only the operating state of some of the joints of the robot arm in the projected robot image may be changed according to the operating state of the actual robot arm. Also, the operating state of the robot arm part in the projected robot image may be changed according to the operation of the actual robot arm part, and the state of the end effector attached to the robot arm part may be made different from the state of the actual end effector. Also, the projected robot image may not be displayed, and only the composite image may be displayed on the display unit. Also, a composite image with the projected robot image superimposed and a composite image without the superimposed may be displayed on the display unit. Also, the projected robot image may be displayed superimposed on the composite image in a transparent state. Together with the composite image, a projected image which is a captured image, a captured image with distortion due to a lens corrected, or a captured image converted by image processing for position conversion may be displayed on the display unit.

[0060] In the above embodiment, an example of performing the calibration process using the marker member 80 has been shown, but the present disclosure is not limited to this. In the present disclosure, the calibration process may not be performed. For example, a composite image may be generated without performing the calibration process using the marker member by performing the synthesis process based on information corresponding to the three-dimensional position of the imaging unit. Also, the calibration process may be performed by superimposing a background image such as the shape of the floor surface or an object placed on the floor surface without placing a marker member. Also, when generating a composite image from a captured image, a projection image that is a captured image converted by image processing of position conversion using a marker member may be translated each time without performing the calibration process in order to set parameters as correction values ​​in advance. In that case, the background image may be superimposed without using a marker member.

[0061] In the above embodiment, the composite image 70 is generated by synthesizing the captured images 60 captured by the imaging units 41, 42, and 43 at a common timing, but the present disclosure is not limited to this. In the present disclosure, the composite image may be generated by synthesizing the captured images captured at different timings. For example, the composite image may be generated by combining a captured image stored in advance with a captured image captured in real time.

[0062] In the above embodiment, the captured images 60 captured from different viewpoints by the multiple imaging units 41, 42, and 43 are converted into projected images 61 that are images from different virtual viewpoints for each of the multiple imaging units 41, 42, and 43, and the composite image 70 is generated by synthesizing the projected images 61 from the different virtual viewpoints. However, the present disclosure is not limited to this. In the present disclosure, a composite image may be generated by synthesizing captured images that have been converted to be images from a common virtual viewpoint. For example, a position moved vertically from the robot main body to the installation surface may be set as the common virtual viewpoint. In addition, when converting a plurality of captured images into images from a common virtual viewpoint, a position diagonally shifted from above the robot main body may be set as the virtual viewpoint.

[0063] In the above embodiment, the robot arm unit 13 of the robot main body unit 11, which is an industrial robot, is a six-axis vertical multi-joint robot arm, and the operation device 20 includes an arm member with a parallel link structure, but the present disclosure is not limited to this. In the present disclosure, the robot arm unit may include joints with five or fewer axes or seven or more axes. In addition, the configuration of the joint unit of the robot arm unit of the robot main body unit is not limited to the above embodiment. For example, the first axis, which is the rotation axis of the joint unit at the base end portion, does not have to be perpendicular to the installation surface. When the robot arm unit is a six-axis vertical multi-joint robot, the direction of the multiple joint units may be different from that of the above embodiment. In addition, the robot arm unit may be a horizontal multi-joint robot. In addition, multiple robot arm units may be arranged. In that case, the imaging unit may be arranged in each of the multiple robot arm units, or may be arranged in one of the multiple robot arm units. In addition, the operation device may include a link member with a serial link structure rather than a parallel link structure. That is, the operation device may include a vertical multi-joint robot or a horizontal multi-joint robot. The operation device may include an input device such as an operation unit having a plurality of switches or a joystick. The operation device may be a keyboard, a pointing device such as a mouse, or a touch panel that also serves as a display unit such as a display. The robot main body may be a medical robot.

[0064] In the above embodiment, the control unit 50 includes the main control unit 51, the servo control unit 52, the drive circuit unit 53, and the image processing unit 54, and the control unit 22 that controls the operation of the operation device 20 is arranged separately from the control unit 50. However, the present disclosure is not limited to this. In the present disclosure, the operation of the robot main body may be controlled by one control unit, not by two control units, the main control unit and the servo control unit. The control of the robot's operation and the control of the generation of the composite image may be performed by different control units. The control unit that controls the robot's operation and the control unit that generates the composite image may be arranged in different housings. For example, the control process of the generation of the composite image may be performed by a control device for image processing that is arranged separately from the robot controller that controls the robot's operation. The operation of the robot and the operation of the operation device may be controlled by a common control unit. A common drive circuit unit may be arranged for multiple joints, instead of arranging a drive circuit unit for each of multiple joints.

[0065] The functions of the elements disclosed herein can be performed using circuits or processing circuits, including general purpose processors, special purpose processors, integrated circuits, Application Specific Integrated Circuits (ASICs), conventional circuits, and / or combinations thereof, configured or programmed to perform the disclosed functions. Processors are considered processing circuits or circuits because they include transistors and other circuits. In this disclosure, a circuit, unit, or means is hardware that performs the recited functions or hardware that is programmed to perform the recited functions. The hardware may be hardware disclosed herein or other known hardware that is programmed or configured to perform the recited functions. Where the hardware is a processor, which is considered a type of circuit, the circuit, means, or unit is a combination of hardware and software, and the software is used to configure the hardware and / or the processor.

[0066] [Aspects] It will be appreciated by those skilled in the art that the exemplary embodiments described above are examples of the following aspects.

[0067] (Aspect 1) a robot main body including a robot arm; a plurality of imaging units disposed at different positions in the robot arm unit of the robot main body unit and spaced apart from a base end of the robot arm unit; a control unit that generates a composite image as a bird's-eye view image of the surroundings of the robot main body by combining a plurality of captured images captured by each of the plurality of imaging units.

[0068] (Aspect 2) the robot arm unit includes a base end portion that is disposed at a base end of the robot arm unit and has a constant distance from an installation surface of the robot main body unit, and an extension portion that is connected to the base end portion and extends from the base end portion, the imaging unit is disposed on the extension portion of the robot arm unit, The robot system of aspect 1, wherein the control unit generates the composite image as a bird's-eye view image including the installation surface around the base end portion.

[0069] (Aspect 3) The robot system of aspect 1 or aspect 2, wherein the control unit acquires information corresponding to the three-dimensional position of each of the multiple imaging units that move in conjunction with the operation of the robot arm unit, and performs image conversion on the multiple captured images based on the acquired information corresponding to the three-dimensional position of each of the multiple imaging units to generate the composite image.

[0070] (Aspect 4) the robot arm unit has a plurality of link members and joints connecting the plurality of link members to each other, 4. The robot system according to aspect 3, wherein the control unit acquires a rotation angle of the joint unit as information corresponding to a three-dimensional position of each of the plurality of imaging units.

[0071] (Aspect 5) The robot arm further includes a drive unit that serves as a drive source for the operation of the robot arm, The robot system according to any one of aspects 1 to 4, wherein the plurality of imaging units are fixed to and disposed on the robot arm unit that is operated by the driving unit.

[0072] (Aspect 6) the robot arm unit has a plurality of link members and joints connecting the plurality of link members to each other, The robot system according to any one of aspects 1 to 5, wherein the multiple imaging units are arranged on a common link member.

[0073] (Aspect 7) The robot system according to any one of aspects 1 to 6, further comprising a display unit that displays the composite image generated by the control unit.

[0074] (Aspect 8) Further comprising an operation device for detecting an operation action by an operator, the robot main body operates in conjunction with an operation action detected by the operation device through remote control, 8. The robot system of claim 7, wherein the display unit displays the composite image to the operator performing an operation action by remote control.

[0075] (Aspect 9) The robot system according to aspect 7 or aspect 8, wherein the control unit superimposes a projected robot image representing the robot main body projected onto the composite image as a bird's-eye view image on the display unit.

[0076] (Aspect 10) A robot system as described in aspect 9, wherein the control unit superimposes the projected robot image on the composite image and displays it on the display unit, with the motion state of the robot main body part projected in the projected robot image being changed to match the actual motion of the robot main body part.

[0077] (Aspect 11) The robot system according to aspect 9 or aspect 10, wherein the control unit generates the projected robot image representing the robot main body using a three-dimensional model that mimics the robot main body.

[0078] (Aspect 12) The imaging ranges of the plurality of imaging units are partially overlapped with each other, The imaging device further includes a marker member that is arranged at a position where the imaging ranges of the plurality of imaging units overlap each other, The robot system according to any one of aspects 1 to 11, wherein the control unit generates the composite image based on a position of the marker member that is common to the plurality of captured images.

[0079] (Aspect 13) A robot system according to any one of aspects 1 to 12, wherein the control unit generates the composite image by combining the multiple captured images captured by each of the multiple imaging units at a common timing.

[0080] (Aspect 14) acquiring a plurality of captured images captured by a plurality of imaging units arranged in different positions apart from a base end of the robot arm unit in the robot arm unit; An image processing method for generating a composite image as a bird's-eye view image of the periphery of a robot main body including the robot arm by combining the plurality of captured images. [Explanation of symbols]

[0081] 10. Robot 11, 211 Robot body 13, 213 Robot arm section 14 parts (proximal part) 14a, 14b, 15a, 16a, 16b, 16c Link members 15 parts (extension part) 16 parts (extension part) 17 Joints 18 Drive unit 20 Operating device 30 Display section 41, 42, 43 Imaging unit 50 Control section 60 Captured images 70 Composite Images 71 Projected Robot Image 80 Marker material 100 Robot System

Claims

1. a robot main body including a robot arm; a plurality of imaging units disposed at different positions in the robot arm unit of the robot main body unit and spaced apart from a base end of the robot arm unit; a control unit that generates a composite image as a bird's-eye view image of the surroundings of the robot main body by combining a plurality of captured images captured by each of the plurality of imaging units.

2. the robot arm unit includes a base end portion that is disposed at a base end of the robot arm unit and has a constant distance from an installation surface of the robot main body unit, and an extension portion that is connected to the base end portion and extends from the base end portion, the imaging unit is disposed on the extension portion of the robot arm unit, The robot system according to claim 1 , wherein the control unit generates the composite image as a bird's-eye view image including the installation surface around the base end portion.

3. 3. The robot system according to claim 1, wherein the control unit acquires information corresponding to a three-dimensional position of each of the plurality of imaging units that moves in conjunction with the operation of the robot arm unit, and performs image conversion on the plurality of captured images based on the acquired information corresponding to the three-dimensional position of each of the plurality of imaging units to generate the composite image.

4. the robot arm unit has a plurality of link members and joints connecting the plurality of link members to each other, The robot system according to claim 3 , wherein the control unit acquires a rotation angle of the joint unit as information corresponding to a three-dimensional position of each of the plurality of imaging units.

5. The robot arm further includes a drive unit that serves as a drive source for the operation of the robot arm, The robot system according to claim 1 , wherein the plurality of image capturing units are fixed to the robot arm unit that is operated by the drive unit.

6. the robot arm unit has a plurality of link members and joints connecting the plurality of link members to each other, The robot system according to claim 1 , wherein the plurality of imaging units are disposed on a common link member.

7. The robot system according to claim 1 , further comprising a display unit that displays the composite image generated by the control unit.

8. Further comprising an operation device for detecting an operation action by an operator, the robot main body operates in conjunction with an operation action detected by the operation device through remote control, The robot system according to claim 7 , wherein the display unit displays the composite image to the operator performing an operation action by remote control.

9. The robot system according to claim 7 , wherein the control unit displays, on the display unit, a projected robot image representing the robot main body projected onto the composite image as a bird's-eye view image, superimposed on the composite image.

10. The robot system according to claim 9, wherein the control unit superimposes the projected robot image on the composite image and displays it on the display unit, with the motion state of the robot main body part projected in the projected robot image being changed to match the actual motion of the robot main body part.

11. The robot system according to claim 9 , wherein the control unit generates the projected robot image representing the robot main body using a three-dimensional model that imitates the robot main body.

12. The imaging ranges of the plurality of imaging units are partially overlapped with each other, The imaging device further includes a marker member that is arranged at a position where the imaging ranges of the plurality of imaging units overlap each other, The robot system according to claim 1 , wherein the control unit generates the composite image based on a position of the marker member that is common to the plurality of captured images.

13. The robot system according to claim 1 , wherein the control unit generates the composite image by combining the plurality of captured images captured by the plurality of imaging units at a common timing.

14. acquiring a plurality of captured images captured by a plurality of imaging units arranged in different positions apart from a base end of the robot arm unit in the robot arm unit; An image processing method for generating a composite image as a bird's-eye view image of the periphery of a robot main body including the robot arm by combining the plurality of captured images.

Images