Movable region display device

Abstract

Provided is a movable region display device that can display a movable region of a robot arm in a robot device that has the robot arm. Said movable region display device comprises: a device main body; a display unit; an imaging unit which can capture an image of an object; a distance measurement unit which measures a distance from the device main body to the object; a relative position recognition unit which recognizes the relative position of the robot device with respect to the device main body on the basis of the measured distance from the device main body to the robot device; an orientation recognition unit which can recognize the orientation of the object with respect to the device main body; an acquisition unit which acquires a coordinate value of the movable region of the robot arm in a robot coordinate system; and a display control unit which converts, on the basis of the relative position of the robot device recognized by the relative position recognition unit and the orientation of the robot device recognized by the orientation recognition unit, the coordinate value of the movable region of the robot arm in the robot coordinate system acquired by the acquisition unit into a coordinate value of the movable region of the robot arm in an image coordinate system, and which causes the display unit to perform display with superimposition on a robot device image.

Description

Movable area display device

[0001] Disclosed herein is a movable area display device.

[0002] Conventionally, robot systems have been proposed that include a display device capable of displaying a motion space in which a movable robot operates. For example, Patent Document 1 discloses a system in which the motion space of a movable robot, whose position relative to a workspace is known, is displayed on a display unit of a motion space presentation device (e.g., a terminal such as a mobile phone or PDA). In this robot system, a worker first captures an image of the movable robot using a camera of the motion space presentation device. The motion space presentation device then calculates the position of the movable robot relative to the workspace by wirelessly communicating with sensing devices provided on the ceiling and floor of the workspace, and calculates the orientation of the movable robot in the workspace using the sensing devices provided on the motion space presentation device. The motion space presentation device then calculates the position and orientation of the movable robot relative to the movable robot based on the position and orientation of the movable robot relative to the workspace. The motion space presentation device then displays the motion space of the movable robot based on the calculated position and orientation on an image captured by the camera and superimposed on the display unit.

[0003] International Publication No. 2011 / 080882

[0004] However, in the above-mentioned robot system, the position and orientation of the motion space presentation device relative to the movable robot are calculated on the assumption that the position of the movable robot relative to the workspace is known, and the motion space of the movable robot based on the calculated position and orientation is displayed on the display unit of the motion space presentation device. This requires that the position of the movable robot relative to the workspace be measured in advance, which is time-consuming. In addition, sensing devices must be installed on the ceiling and floor of the workspace, which is costly.

[0005] A primary object of the present disclosure is to reduce the effort and cost required to display the movable range of a robot arm.

[0006] The present disclosure has adopted the following means to achieve the above-mentioned main object.

[0007] a display control unit that converts the coordinate values ​​of the movable area of ​​the robot arm in the robot coordinate system acquired by the acquisition unit based on the relative position of the robot device recognized by the relative position recognition unit into coordinate values ​​of the movable area of ​​the robot arm in an image coordinate system of an image of the robot device captured by the imaging unit, and displays the converted movable area of ​​the robot arm on the display unit, superimposed on the image of the robot device.

[0008] The movable area display device of the present disclosure does not require measuring the position of the movable robot in advance or providing an external sensing device.

[0009] 1 is a schematic configuration diagram of a robot system 10. FIG. 2 is a side view of a robot device 20. FIG. 3 is a top view of a housing 60. FIG. 4 is a block diagram showing electrical connections of the robot system 10. FIG. 5 is a flowchart showing an example of a movable area display process. FIG. 6 is an explanatory diagram showing an example of a captured image Im obtained by capturing an image of the housing 60 of the robot device 20 from the left rear. FIG. 7 is an explanatory diagram showing an example of a captured image Im obtained by capturing an image of the housing 60 of the robot device 20 from the right rear. FIG. 8 is an explanatory diagram showing a state in which a first movable area A1 and a second movable area A2 of a robot arm 21 are displayed on a display operation device 84. FIG. 9 is a flowchart showing a movable area display process of a first modified example. FIG. 10 is a flowchart showing a movable area display process of a second modified example.

[0010] Next, an embodiment of the present disclosure will be described with reference to the drawings. Fig. 1 is a schematic configuration diagram of a robot system 10. Fig. 2 is a side view of a robot device 20. Fig. 3 is a top view of a housing 60. Fig. 4 is a block diagram showing the electrical connections of the robot system 10. In this embodiment, the left-right direction shown in Figs. 2 and 3 is the X-axis direction (a direction perpendicular to the paper surface in Fig. 2), the front-rear direction is the Y-axis direction, and the up-down direction is the Z-axis direction (a direction perpendicular to the paper surface in Fig. 3).

[0011] As shown in FIG. 1 , the robot system 10 of this embodiment includes a robot device 20 having an articulated robot arm 21 and a tablet terminal 80 .

[0012] As shown in FIG. 1 , the robot system 10 holds an ultrasound probe 101 of an ultrasound diagnostic device 100 at the tip of a robot arm 21, and controls the robot device 20 to move while pressing the ultrasound probe 101 against the surface of a human body, thereby causing the ultrasound diagnostic device 100 to acquire ultrasound echo images of the human body. The robot system 10 is used as an ultrasound echo guide during surgery, such as catheterization. An operator (surgeon) operating a catheter guidewire instructs the robot device 20 to press the ultrasound probe 101 against the surface of the human body (patient). The operator advances the guidewire while recognizing the positional relationship between the tip of the guidewire and the blood vessel from the obtained ultrasound echo image, thereby accurately passing the guidewire through the center of an occluded or narrowed portion of the blood vessel. As a preliminary step, the operator manually operates the robot arm 21, applies the ultrasound probe 101 held by the robot arm 21 to the patient, and while checking the obtained ultrasound echo image, determines points (images) to be reproduced during surgery and performs direct teaching, registering the points (images) in the robot device 20 (robot control device 70).

[0013] 1, the ultrasound diagnostic device 100 includes an ultrasound probe 101 and an ultrasound diagnostic device main body 110 connected to the ultrasound probe 101 via a cable 102. As shown in FIG. 4, the ultrasound diagnostic device main body 110 includes an ultrasound diagnosis control unit 111 that controls the entire device, an image processing unit 112 that processes signals received from the ultrasound probe 101 to generate ultrasound echo images, an image display unit 113 that displays the ultrasound echo images, and various operation switches (not shown).

[0014] As shown in Figures 1 and 2, the robot device 20 is a medical robot that includes a base 25, a robot arm 21 installed on the base 25, a housing 60, an operation panel 76, and a robot control device 70.

[0015] 1 and 2, casters 26 with stoppers are attached to the four corners of the back surface of the base 25. The robot device 20 can be freely moved by the casters 26. In addition, locking portions 28 are provided at multiple locations (e.g., three locations) on the back surface of the base 25. These locking portions 28 protrude vertically downward when a lever 27 is pressed down, thereby locking (fixing) the robot device 20 so that it cannot move.

[0016] As shown in FIG. 1 , the robot arm 21 has a first arm 22, a first arm driving device 35, a second arm 23, a second arm driving device 36, a base 24, a three-axis rotation mechanism 50, and an attitude holding device 37.

[0017] The base end of the first arm 22 is connected to the base 24 via a first joint shaft 31 extending in the vertical direction (Z-axis direction). The first arm driving device 35 includes a motor 35a (see FIG. 4) and an encoder 35b (see FIG. 4). The rotation shaft of the motor 35a is connected to the first joint shaft 31 via a reducer (not shown). The first arm driving device 35 rotates (pivots) the first arm 22 along a horizontal plane (XY plane) around the first joint shaft 31 as a fulcrum by driving the first joint shaft 31 to rotate using the motor 35a. The encoder 35b is attached to the rotation shaft of the motor 35a and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 35a.

[0018] The base end of the second arm 23 is connected to the tip end of the first arm 22 via a second joint shaft 32 extending in the vertical direction. The second arm driving device 36 includes a motor 36a (see FIG. 4) and an encoder 36b (see FIG. 4). The rotation shaft of the motor 36a is connected to the second joint shaft 32 via a reducer (not shown). The second arm driving device 36 rotates (pivots) the second arm 23 along a horizontal plane around the second joint shaft 32 as a fulcrum by driving the second joint shaft 32 to rotate using the motor 36a. The encoder 36b is attached to the rotation shaft of the motor 36a and is configured as a rotary encoder that detects the amount of rotational displacement of the motor 36a.

[0019] The base 24 is movable up and down relative to the base 25 by an elevator device 40 installed on the base 25. As shown in FIG. 2 , the elevator device 40 includes a slider 41 fixed to the base 24, a guide member 42 extending in the vertical direction to guide the movement of the slider 41, a ball screw shaft 43 (elevation shaft) extending in the vertical direction and threadedly engaged with a ball screw nut (not shown) fixed to the slider 41, a motor 44 a (see FIG. 4 ) that rotates the ball screw shaft 43, and an encoder 44 b (see FIG. 4 ). The elevator device 40 moves the base 24 fixed to the slider 41 up and down along the guide member 42 by rotating the ball screw shaft 43 using the motor 44 a. The encoder 44 b is configured as a linear encoder that detects the vertical position (elevation position) of the slider 41 (base 24).

[0020] As shown in FIG. 2 , the three-axis rotating mechanism 50 is connected to the tip of the second arm 23 via an attitude maintaining shaft 33 extending in the vertical direction. The three-axis rotating mechanism 50 includes a first rotation shaft 51, a second rotation shaft 52, and a third rotation shaft 53 that are perpendicular to one another, a first rotation device 55 that rotates the first rotation shaft 51, a second rotation device 56 that rotates the second rotation shaft 52, and a third rotation device 57 that rotates the third rotation shaft 53. The first rotation shaft 51 is supported in an orthogonal orientation with respect to the attitude maintaining shaft 33. The second rotation shaft 52 is supported in an orthogonal orientation with respect to the first rotation shaft 51. The third rotation shaft 53 is supported in an orthogonal orientation with respect to the second rotation shaft 52. The third rotation shaft 53 is provided with a holder 58 for holding the ultrasonic probe 101. The holder 58 holds the ultrasonic probe 101 so that it is positioned coaxially with the third rotation shaft 53. The holding portion 58 is also provided with a direct teaching switch 59 for allowing the operator to manually operate the robot arm 21 in direct teaching.

[0021] The attitude holding device 37 holds the attitude of the three-axis rotation mechanism 50 (the orientation of the first rotation shaft 51) in a constant direction regardless of the attitudes of the first arm 22 and the second arm 23. The attitude holding device 37 includes a motor 37a (see FIG. 4) and an encoder 37b (see FIG. 4). The rotation shaft of the motor is connected to the attitude holding shaft 33 via a reducer (not shown). The attitude holding device 37 sets a target rotation angle of the attitude holding shaft 33 based on the rotation angles of the first joint shaft 31 and the second joint shaft 32 so that the axial direction of the first rotation shaft 51 is always in the left-right direction (X-axis direction), and drives and controls the motor so that the attitude holding shaft 33 reaches the target rotation angle. This enables the translational motion in three directions and the rotational motion in three directions to be controlled independently, making control easier.

[0022] The robot device 20 of this embodiment can move the ultrasonic probe 101 to any position in any orientation within its movable range by combining translational motion in three directions, namely, the X-axis direction, the Y-axis direction, and the Z-axis direction, by the first arm driving device 35, the second arm driving device 36, and the lifting device 40, with rotational motion in three directions, namely, around the X-axis (pitching), around the Y-axis (rolling), and around the Z-axis (yawing), by the three-axis rotation mechanism 50.

[0023] The housing 60 houses the lifting device 40, the robot control device 70 (see FIG. 4), etc. As shown in FIGS. 1 and 2, the housing 60 is fixed to the base 25. As shown in FIG. 3, AR markers M1 and M2 are attached to the top surface of the housing 60 at a predetermined interval in the left-right direction (X-axis direction). The AR markers M1 and M2 are, for example, two-dimensional patterns and colors printed on paper. The AR markers M1 and M2 have different patterns and colors. The AR markers M1 and M2 have the same dimensions.

[0024] The robot controller 70 is a computer having a CPU, ROM, RAM, storage (e.g., HDD or SSD), etc. The robot controller 70 is responsible for overall control of the robot device 20. The robot controller 70 outputs image signals to an operation panel 76 and receives operations from the operator via the operation panel 76. The robot controller 70 receives inputs such as displacement amount signals from the encoder 44b of the lifting device 40, the encoder 35b of the first arm driver 35, and the encoder 36b of the second arm driver 36, and an ON signal from the direct teaching switch 59. The robot controller 70 also outputs control signals to the motor 35a of the first arm driver 35, the motor 36a of the second arm driver 36, the motor 44a of the lifting device 40, the three-axis rotation mechanism 50, the operation panel 76, etc.

[0025] The operation panel 76 is a touch panel display that displays various information related to the robot system 10 and allows various instructions to be input to the robot system 10. In this embodiment, the operation panel 76 is installed on the top surface of the housing 60.

[0026] As shown in Figure 4, the tablet terminal 80 includes a terminal body 81 (see Figure 1), a camera 82, a three-dimensional sensor 83, a display operation device 84, an acceleration sensor 85, a gyro sensor 86, and a control device 90.

[0027] The terminal body 81 is a member having a rectangular shape in a plan view. Various members are attached to or built into the terminal body 81.

[0028] The camera 82 is provided on the rear surface of the terminal main body 81. The camera 82 is, for example, a color camera capable of capturing a color image of an object. The camera 82 outputs an image signal to the control device 90.

[0029] The three-dimensional sensor 83 is built into the terminal main body 81. The three-dimensional sensor 83 is capable of detecting the distance to an object present around the terminal main body 81. The three-dimensional sensor 83 is configured as, for example, a LiDAR (Light Detection and Ranging) that is capable of detecting point cloud data, which is distance data for each scanning angle. The three-dimensional sensor 83 outputs the point cloud data to the control device 90. The control device 90 recognizes the direction and distance to the object and the shape of the object based on the point cloud data.

[0030] The display operation device 84 is provided on the front of the terminal main body 81. The display operation device 84 displays various information to the worker and receives various operation inputs from the worker. The display operation device 84 is configured as, for example, a touch panel display. The display operation device 84 outputs operation signals input by the worker to the control device 90.

[0031] The acceleration sensor 85 is built into the terminal main body 81. The acceleration sensor 85 can detect acceleration in each of the axial directions, ie, the left-right direction (X-axis direction), the front-back direction (Y-axis direction), and the up-down direction (Z-axis direction) of the tablet terminal 80. The acceleration sensor 85 outputs an acceleration signal in each axial direction to the control device 90 at predetermined time intervals (for example, every few milliseconds).

[0032] The gyro sensor 86 is built into the terminal main body 81. The gyro sensor 86 can detect the angle and angular velocity of each axis, i.e., rotation around the X axis, rotation around the Y axis, and rotation around the Z axis, of the tablet terminal 80. The gyro sensor 86 outputs an angular velocity signal around each axis to the control device 90 at predetermined time intervals (for example, every few ms).

[0033] The control device 90 is built into the terminal main body 81. The control device 90 is a computer having a CPU 91, a ROM 92, a RAM 93, a storage 94, and a communication unit 95. The control device 90 receives inputs such as an image signal from the camera 82, a distance signal from the three-dimensional sensor 83, an operation signal from the display operation device 84, an acceleration signal from the acceleration sensor 85, and an angular velocity signal from the gyro sensor 86. The control device 90 also outputs control signals to the camera 82, the display operation device 84, etc. The communication unit 95 communicates wirelessly with external devices (e.g., the robot control device 70) to exchange various types of data.

[0034] The storage 94 stores the movable area of ​​the robot arm 21 as coordinate values ​​in a robot coordinate system. In this embodiment, the movable area includes a first movable area A1 (first movable area data 94a) according to specifications and a second movable area A2 (second movable area data 94b) that is slightly narrower than the first movable area. Here, the robot coordinate system is an orthogonal coordinate system based on a predetermined position of the robot device 20 (for example, the base 25).

[0035] Next, an example of how the robot system 10 configured as described above is described. Here, the procedure for an operator to perform an ultrasound diagnosis on a patient is described. Note that the patient is lying on a bed (not shown) when the operator performs this procedure. To begin this procedure, the operator first locks the stoppers of the casters 26. Next, the operator presses down the lever 27. This causes the locking portion 28 to protrude vertically downward from the base 25, locking the robot device 20 so that it cannot move. Next, the operator rotates the operating handle 49 to adjust the height of the robot arm 21 to correspond to the height of the bed. The operator then performs direct teaching. Direct teaching is a process in which the operator manually operates the robot arm 21 to register the movement trajectory of the ultrasound probe 101 in the robot control device 70 of the robot device 20. Direct teaching allows the operator to directly operate the robot arm 21 by applying force to the robot arm 21. This function generates an assist force from the motors of each axis in the direction of the applied force, enabling the operator to operate the robot arm 21 with minimal force. Direct teaching is performed only while the direct teaching switch 59 is on. When the direct teaching switch 59 is turned off, the motor assistance stops, making it difficult for the operator to manually operate the robot arm 21. The operator first checks the movable area of ​​the robot arm 21 displayed on the display / operation device 84 of the tablet terminal 80 before performing direct teaching. The process of displaying the movable area of ​​the robot arm 21 on the display / operation device 84 will be described later. The operator then begins ultrasound diagnosis. When ultrasound diagnosis begins, the robot control device 70 controls various components in response to instructions input by the operator so that the ultrasound probe 101 moves along the registered movement trajectory. The image processing unit 112 of the ultrasound diagnostic device main body 110 processes the received signal from the ultrasound probe 101 to generate an ultrasound echo image. The ultrasound diagnosis control unit 111 of the ultrasound diagnostic device main body 110 controls the image display unit 113 to display the ultrasound echo image. The operator diagnoses the patient based on the echo image displayed on the image display unit 113.

[0036] In the robot system 10, the operator performs direct teaching after checking the movable area of ​​the robot arm 21 displayed on the display operation device 84 of the tablet terminal 80. Therefore, the operator can perform direct teaching while knowing the movable area of ​​the ultrasonic probe 101 in advance.

[0037] Next, a movable area display process to be displayed on the display / operation device 84 of the tablet terminal 80 will be described. FIG. 5 is a flowchart illustrating an example of the movable area display process. This process is repeatedly executed by the CPU 91 of the control device 90 at predetermined time intervals (e.g., every few milliseconds) after an operator inputs a process start instruction via the display / operation device 84. In this process, the CPU 91 reads the movable area of ​​the robot arm 21 stored in the storage 94 as coordinate values ​​in the robot coordinate system and converts the coordinate values ​​into coordinate values ​​in the image coordinate system of the display / operation device 84 using a transformation matrix. The movable area of ​​the robot arm 21 represented by the transformed coordinate values ​​is superimposed on an image of the robot device 20 captured by the camera 82 and displayed on the display / operation device 84. Here, the image coordinate system is a two-dimensional Cartesian coordinate system with the center of the image as the origin. The transformation matrix is ​​determined based on the relative position and orientation of the robot device 20 with respect to the tablet terminal 80. The process of FIG. 5 recognizes the relative position and orientation based on the AR markers M1 and M2 captured in the captured image input from the camera 82.

[0038] When this process starts, the CPU 91 inputs an image captured by the camera 82 (hereinafter referred to as the captured image Im) (S100). Next, the CPU 91 processes the captured image Im to recognize the AR markers M1 and M2 (S102). Next, the CPU 91 determines whether the AR markers M1 and M2 have been successfully recognized from the captured image Im (S104). If it is determined that the AR markers M1 and M2 have been successfully recognized, the CPU 91 acquires the size and position within the image of the recognized AR markers M1 and M2 (S106). Specifically, the CPU 91 acquires the number of pixels contained in the AR markers M1 and M2 as the size of the AR markers M1 and M2 on the image. The CPU 91 also acquires, as the in-image positions of the AR markers M1 and M2, the orthogonal coordinate values ​​of the center positions (hereinafter referred to as the marker centers) of the AR markers M1 and M2 relative to the image center (the origin of the image coordinate system) of the captured image Im. The CPU 91 then estimates the distance from the terminal main body 81 to the robot device 20 based on the sizes of the AR markers M1 and M2 in the image (S108). The CPU 91 executes this process as follows. Specifically, the CPU 91 stores, in advance in the storage 94, as reference sizes, the sizes of the AR markers M1 and M2 in the image when the AR markers M1 and M2 are captured from a position a predetermined distance (hereinafter referred to as the reference distance). Then, the CPU 91 measures the distance from the terminal main body 81 to the AR marker M1 and the AR marker M2 by multiplying the reference distance by the ratio of the size of the recognized AR marker M1 and the AR marker M2 on the image to the reference size, and measures the distance from the terminal main body 81 to the robot device 20 by taking the average of both.

[0039] Next, the CPU 91 estimates the direction of the robot device 20 relative to the terminal main body 81 based on the positions of the AR markers M1 and M2 in the image and their sizes on the image (S110). Here, the direction of the robot device 20 is given as the angle between the Z axis and a line segment connecting the origin and the robot device 20 in the XZ plane view in a terminal coordinate system, which is a three-dimensional Cartesian coordinate system with a predetermined position on the terminal main body 81 as the origin, the optical axis of the camera 82 as the Z axis, the longitudinal direction of the terminal main body 81 as the X axis, and the lateral direction of the terminal main body 81 as the Y axis. The CPU 91 executes this process as follows. That is, first, the CPU 91 calculates the length per pixel of the captured image Im (hereinafter referred to as the reference length) based on the number of pixels of the AR markers M1 and M2 on the image Im recognized in S106 and the actual sizes of the AR markers M1 and M2. Next, the CPU 91 calculates the number of pixels to the marker centers of the AR markers M1 and M2 in the image coordinate system. Then, the calculated number of pixels is multiplied by the reference length. This calculates the length of a perpendicular line drawn from the AR markers M1 and M2 to the Z axis in the XZ plane view of the device coordinate system. Next, the CPU 91 calculates the ratio of the length of a perpendicular line drawn from the AR markers M1 and M2 to the Z axis in the XZ plane view of the device coordinate system to the distance to the AR markers M1 and M2 estimated in S108. Next, the CPU 91 uses the calculated ratio and an inverse trigonometric function to calculate the angle formed by the Z axis and a line segment connecting the origin and the AR markers M1 and M2 in the XZ plane view of the device coordinate system. The CPU 91 then calculates the average of the calculated angles as the direction of the robot device 20.

[0040] Next, the relative position of the robot device 20 with respect to the terminal body 81 is recognized based on the distance and direction estimated in S108 and S110 (S112). Note that the relative position is, for example, the position of the center of the AR marker M1 and the AR marker M2 with respect to the terminal body 81, and is given as the coordinate values ​​of each axis in the terminal coordinate system described above.

[0041] The CPU 91 then recognizes the orientation of the robot device 20 relative to the terminal body 81 based on the difference in size between the AR marker M1 and the AR marker M2 on the image (S114). Here, the orientation is, for example, the tilt of the robot device 20 relative to the terminal body 81, and is given as the tilt of each axis of the robot coordinate system relative to each axis of the terminal coordinate system described above. Here, the closer the subject image is to the camera 82, the larger it becomes. Therefore, as shown in FIG. 6A , if the image of the AR marker M1 on the left side as viewed from the camera 82 is larger than the image of the AR marker M2 on the right side as viewed from the camera 82, the larger the image of the AR marker M1, the more the orientation of the robot device 20 is tilted counterclockwise around the Z axis in top view. On the other hand, as shown in FIG. 6B , if the image of the AR marker M2 on the right side as viewed from the camera 82 is larger than the image of the AR marker M1 on the left side as viewed from the camera 82, the larger the image of the AR marker M2, the more the orientation of the robot device 20 is tilted clockwise around the Z axis in top view. Based on this tendency, the CPU 91 calculates the inclination of each axis in the robot coordinate system relative to each axis in the terminal coordinate system.

[0042] Next, the CPU 91 reads the coordinate values ​​of the first movable area A1 and the second movable area A2 in the robot coordinate system from the first movable area data 94a and the second movable area data 94b stored in the storage 94 (S116). The first movable area A1 and the second movable area A2 are, for example, rectangular parallelepipeds, and the coordinate values ​​include the coordinate values ​​of each vertex. The CPU 91 reads the coordinate values ​​of each vertex of the first movable area A1 and the second movable area A2. Next, the CPU 91 uses a transformation matrix to convert the coordinate values ​​of the first movable area A1 and the second movable area A2 in the robot coordinate system into coordinate values ​​of the first movable area A1 and the second movable area A2 in the image coordinate system, which is an orthogonal coordinate system within the virtual space of the image (S118). The transformation matrix is ​​determined by the relative position of the robot device 20 with respect to the terminal main body 81 (camera 82) and the attitude of the robot device 20 (tilt with respect to each axis). 7, the CPU 91 displays the first movable area A1 and the second movable area A2 in the image coordinate system superimposed on the captured image Im (S120). In the captured image Im, the CPU 91 displays the first movable area A1 and the second movable area A2 as semi-transparent areas of different colors.

[0043] The CPU 91 displays the first movable area A1 and the second movable area A2 of the robot arm 21 as semi-transparent areas of different colors in the captured image Im, allowing the worker to easily recognize the boundary between the first movable area A1 and the second movable area A2 before performing direct teaching.

[0044] Next, the CPU 91 sets the value of the recognition flag F to 1 (S122) and ends this process. The recognition flag F indicates whether or not the relative position and orientation have already been recognized at least once based on the captured image Im. If the value of the recognition flag F is 0, this indicates that the relative position and orientation have not yet been recognized from the captured image Im, and if the value of the recognition flag F is 1, this indicates that the relative position and orientation have been recognized at least once up to that point.

[0045] If it is determined in S102 that at least one of the AR marker M1 and the AR marker M2 could not be recognized from the captured image Im, the CPU 91 determines whether the value of the recognition flag F is 1 (S124). If it is determined that the value of the recognition flag F is 0, the CPU 91 terminates this process. If it is determined that the value of the recognition flag F is 1, the CPU 91 determines that the relative position and orientation can be estimated based on the relative position and orientation recognized in S110 and S112 of the most recently executed process or the relative position and orientation estimated in S126 (described later) of the most recently executed process, and the measurement values ​​of the acceleration sensor 85 and the gyro sensor 86. That is, the CPU 91 calculates the amount of change in the relative position (position in each axial direction in the device coordinate system) and the orientation (tilt around each axis in the device coordinate system) based on the measurement values ​​of the acceleration sensor 85 and the gyro sensor 86 (S126). Specifically, the CPU 91 calculates the amount of change in relative position by double integrating, over the execution time interval of this process, the accelerations in the X-axis, Y-axis, and Z-axis directions input from the acceleration sensor 85. The CPU 91 also calculates the amount of change in attitude by integrating, over the execution time interval of this process, the angular velocities around the X-axis, Y-axis, and Z-axis input from the gyro sensor 86.

[0046] Next, the CPU 91 estimates the current relative position and orientation of the robot device 20 based on the most recently recognized or estimated relative position and orientation and the calculated change in relative position and orientation (S130). Specifically, the CPU 91 estimates the current relative position by adding the calculated change in relative position to the most recently recognized or estimated relative position. The CPU 91 also estimates the current orientation by adding the calculated change in orientation to the most recently recognized or estimated orientation. Next, the CPU 91 recalculates the coordinate values ​​of the first movable area A1 and the second movable area A2 in the image coordinate system based on the estimated relative position and orientation (S130). Next, the CPU 91 updates the display of the first movable area A1 and the second movable area A2 on the display / operation device 84 (S132) and ends this process.

[0047] In this way, even if the CPU 91 is no longer able to recognize the AR markers M1 and M2 from the captured image Im, the CPU 91 estimates the current relative position and orientation from the relative position and orientation at the time when the relative position and orientation were last recognized or estimated and the amount of change in the relative position and orientation calculated from the measurement values ​​of the acceleration sensor 85 and the gyro sensor 86, and updates the display of the first movable area A1 and the second movable area A2. Therefore, the worker can grasp the movable area of ​​the robot arm 21 using the tablet terminal 80.

[0048] Here, the correspondence between the main elements of the embodiment and the main elements of the present disclosure described in the claims will be described. Specifically, the tablet terminal 80 of the present embodiment corresponds to the movable area display device of the present disclosure, the robot arm 21 corresponds to the robot arm of the present disclosure, the robot device 20 corresponds to the robot device, the terminal main body 81 corresponds to the device main body, the display / operation device 84 corresponds to the display unit, the camera 82 corresponds to the imaging unit, the CPU 91 that executes the process of S108 of the movable area display process corresponds to the distance measurement unit, the CPU 91 that executes the process of S112 of the movable area display process corresponds to the relative position recognition unit, the CPU 91 that executes the process of S114 of the movable area display process corresponds to the attitude recognition unit, the CPU 91 that executes the process of S116 of the movable area display process corresponds to the acquisition unit, and the CPU 91 that executes the processes of S118 and S120 of the movable area display process corresponds to the display control unit. Furthermore, the three-dimensional sensor 83 corresponds to the shape acquisition unit. Furthermore, the AR marker M1 and the AR marker M2 correspond to the recognized unit. The acceleration sensor 85 corresponds to an acceleration sensor, the gyro sensor 86 corresponds to an angular velocity sensor, and the CPU 91 that executes the processes of S126 to S132 of the movable area display process corresponds to an estimation unit.

[0049] It goes without saying that the present disclosure is not limited to the above-described embodiments, and can be embodied in various forms as long as they fall within the technical scope of the present disclosure.

[0050] For example, in the above-described embodiment, the robot device 20 is configured as a seven-axis articulated robot capable of translational movement in three directions and rotational movement in three directions. However, any number of axes may be used. Furthermore, the robot device 20 may be configured as a so-called vertical articulated robot, horizontal articulated robot, or the like.

[0051] In the above-described embodiment, the recognized portions of the present disclosure have been described as the AR markers M1 and M2. However, the recognized portions of the present disclosure may be two-dimensional codes or distinctive color patterns.

[0052] In the above-described embodiment, the robot device 20 is provided with the AR marker M1 and the AR marker M2. However, the robot device 20 may be provided with only one of the AR marker M1 and the AR marker M2. In this case, the CPU 91 may recognize the posture of the robot device 20 by processing similar to the movable area display processing of the first modified example or the movable area display processing of the second modified example, which will be described later.

[0053] In the above-described embodiment, the CPU 91 may execute the movable area display process without recognizing the posture of the robot device 20. In this case, the image capturing direction may be specified by the worker to capture an image of the robot device 20. In this case, the CPU 91 may change the sizes of the first movable area A1 and the second movable area A2 to be superimposed on the captured image of the robot device 20 in accordance with the distance to the robot device 20.

[0054] In the above-described embodiment, the AR marker M1 and the AR marker M2 are attached to the top surface of the housing 60. However, the AR marker M1 and the AR marker M2 may be attached to the side surface of the housing 60, the base 25, or the like, as long as they are not a movable part such as the robot arm 21 or the base 24.

[0055] In the above-described embodiment, the movable area of ​​the robot arm 21 is divided into a first movable area A1 and a second movable area A2, each of which is displayed in a different manner. However, the movable area of ​​the robot arm 21 may be divided into three or more movable areas, each of which may be displayed in a different manner, or the movable area of ​​the robot arm 21 may not be divided and may be displayed in a single display manner.

[0056] In the above-described embodiment, the CPU 91 recognizes the orientation of the robot device 20 relative to the terminal main body 81 based on the difference in size between the AR marker M1 and the AR marker M2. However, the CPU 91 may recognize the orientation of the robot device 20 based on the positional relationship between the AR marker M1 and the AR marker M2. Here, the positional relationship may be, for example, the relative position of the AR marker M2 with respect to the AR marker M1 (coordinate values ​​of each axis of the AR marker M2 with respect to the AR marker M1 in the image coordinate system). In this case, the CPU 91 recognizes the orientation as follows, instead of the processing of S114, in the movable area display processing shown in FIG. 5 . That is, after recognizing the relative position of the robot device 20 in S112 of the movable area display processing, the CPU 91 subtracts the in-image position of the AR marker M1 recognized in S106 from the in-image position of the AR marker M2 recognized in S106 (coordinate values ​​in the image coordinate system). This calculates an in-image relative position, which is the position of one AR marker M2 relative to the other AR marker M1 in the image coordinate system. Next, the CPU 91 estimates the posture of the robot device 20 based on the calculated in-image relative positions. This process is performed as follows. That is, the CPU 91 registers in advance in the storage 94 the in-image relative positions for each of a plurality of postures when an image of the robot device 20 is captured from a position away from the robot device 20 by a reference distance. Next, the CPU 91 corrects the calculated in-image relative position by multiplying it by the ratio of the distance estimated in S106 to the reference distance. Then, the CPU 91 recognizes the posture corresponding to the in-image relative position closest to the recognized in-image relative position among the corrected in-image relative positions as the posture of the robot device 20.

[0057] In the above-described embodiment, in the movable area display process, after the CPU 91 determines in S102 that it was unable to recognize the AR marker M1 and the AR marker M2, it estimates the relative position and orientation based on the measurement values ​​of the acceleration sensor 85 and the gyro sensor 86, and based on these, updates the display of the first movable area A1 and the second movable area A2 on the display operation device 84. However, the CPU 91 may end the movable area display process without updating the display of the first movable area A1 and the second movable area A2.

[0058] In the above-described embodiment, the robot device 20 has the AR marker M1 and the AR marker M2. However, the robot device 20 may not have the AR marker M1 and the AR marker M2. In this case, instead of the movable area display process shown in FIG. 5 , the CPU 91 may execute a movable area display process as shown in FIG. 8 . FIG. 8 is a flowchart showing the movable area display process of a first modified example. In the process shown in FIG. 8 , the CPU 91 recognizes the relative position and orientation used to determine the transformation matrix based on point cloud data (distance and direction of each measurement point) input from the three-dimensional sensor 83. Note that in FIG. 8 , the same steps as those in FIG. 5 are assigned the same step numbers, and detailed descriptions thereof will be omitted below.

[0059] When this process starts, the CPU 91 first inputs the captured image Im from the camera 82 (S100). Next, the CPU 91 acquires point cloud data from the three-dimensional sensor 83 (S200). Next, the CPU 91 detects edges based on changes in distance between each measurement point based on the point cloud data, and recognizes the area of ​​the robot device 20 (S202). The CPU 91 then determines whether or not the recognition of the area of ​​the robot device 20 was successful (S204). Specifically, the CPU 91 determines whether or not the recognition was successful based on whether or not the size of the area surrounded by the detected edges is equal to or larger than a predetermined size.

[0060] If the recognition is successful, the CPU 91 determines that a robot device area exists, which is the area of ​​the robot device 20, and recognizes the relative position of the robot device 20 with respect to the terminal body 81 based on the distance and direction of the measurement point at the center of gravity of the robot device area (S206). The CPU 91 may recognize the relative position based on the average distance and average direction of the entire robot area, rather than the center of gravity of the robot area. Next, the CPU 91 recognizes the posture of the robot device 20 by shape pattern matching of the robot device area according to the distance to the center of gravity (S208). This process is performed as follows. Prior to this process, the CPU 91 registers in advance in the storage 94 the outer shape of the robot device area for each posture when point cloud data is acquired from a position a reference distance away. The CPU 91 then reduces or enlarges the shape registered in the storage 94 according to the ratio of the reference distance to the distance to the center of gravity, and recognizes the posture of the robot device 20 by performing shape pattern matching. The CPU 91 recognizes the posture of the robot device 20 that corresponds to the shape of the robot device 20 after reduction or enlargement that has the smallest error from the outer shape of the recognized robot device area. The CPU 91 then executes the processes of S116 to S122 of the movable area display process in the above-described embodiment, and ends this process. Furthermore, if it is determined in S204 that recognition of the area of ​​the robot device 20 has failed, the CPU 91 executes the processes of S124 to S132 of the movable area display process in the above-described embodiment, and ends this process.

[0061] If the robot device 20 does not have the AR marker M1 and the AR marker M2, the CPU 91 may execute a movable area display process as shown in Fig. 9 instead of the movable area display process shown in Fig. 5. Fig. 9 is a flowchart showing the movable area display process of a second modified example. In the process shown in Fig. 9, the CPU 91 recognizes the relative position and orientation used to determine the transformation matrix based on the captured image Im captured by the camera 82. Note that in Fig. 9, the same steps as those in Fig. 5 or 8 are assigned the same step numbers, and detailed descriptions thereof will be omitted below.

[0062] When this process starts, the CPU 91 first inputs the captured image Im from the camera 82 (S100). Next, the CPU 91 detects edges in the captured image Im based on changes in brightness, and recognizes the area of ​​the robot device 20 (S300). The CPU 91 then determines whether or not the recognition of the area of ​​the robot device 20 was successful (S302). Specifically, the CPU 91 determines whether or not the recognition was successful based on whether or not the size of the area surrounded by the detected edges is equal to or larger than a predetermined size.

[0063] If the CPU 91 determines that the recognition is successful, it determines that a robot device area, which is the area of ​​the robot device 20, is present and recognizes the size of the robot device 20 in the image based on the number of pixels in the robot device area (S304). Next, the CPU 91 recognizes the distance to the robot device 20 based on the size of the robot device 20 in the image (S306). This process can be performed similarly to S108 in the above-described embodiment by replacing the size of the AR markers M1 and M2 in the image with the size of the robot device area in the image. Next, the CPU 91 recognizes the center of gravity of the robot device area in the image coordinate system as the position of the robot device 20 in the image (S308). Then, the CPU 91 recognizes the direction of the robot device 20 based on the distance to the robot device 20 and the position of the robot device 20 in the image (S310). This process can be performed similarly to S110 in the above-described embodiment by replacing the positions of the AR markers M1 and M2 in the image with the position of the robot device 20 in the image.

[0064] Next, the CPU 91 recognizes the relative position of the robot device 20 with respect to the robot itself based on the distance and direction recognized in S306 and S310 (S114). Subsequently, the CPU 91 recognizes the posture of the robot device 20 by shape pattern matching of the robot device area according to the distance to the robot device 20 (S208). The CPU 91 then executes the processes of S116 to S122 of the movable area display process in the above-described embodiment, and ends this process. Furthermore, if it is determined in S302 that recognition of the area of ​​the robot device 20 has failed, the CPU 91 executes the processes of S124 to S132 of the movable area display process in the above-described embodiment, and ends this process.

[0065] In the movable area display process of the above-described embodiment, the CPU 91 may recognize the relative position and orientation by appropriately combining the processes described in the present embodiment, the first modified example, and the second modified example. That is, the CPU 91 may recognize the relative position based on point cloud data from the three-dimensional sensor 83 or the relative position based on the captured image Im from the camera 82, and then recognize the orientation based on the difference in size of the AR marker M1 and the AR marker M2 on the image. The CPU 91 may also recognize the relative position based on the size and position of the AR marker M1 and the AR marker M2 on the image or the captured image Im from the camera 82, and then recognize the orientation based on the point cloud data from the three-dimensional sensor 83. The CPU 91 may also recognize the relative position based on the size and position of the AR marker M1 and the AR marker M2 on the image or the point cloud data from the three-dimensional sensor 83, and then recognize the orientation based on the captured image Im from the camera 82.

[0066] In the above-described embodiment, the tablet terminal 80 may be provided with a three-dimensional camera as the camera 82. In that case, the three-dimensional sensor 83 may be omitted. The CPU 91 may perform processing similar to the movable area display processing of the first modified example based on the distance data for each pixel input from the three-dimensional camera, and may display the movable area of ​​the robot arm 21 on the display operation device.

[0067] The movable area display device of the present disclosure described above does not require measuring the position of the movable robot in advance or providing an external sensing device.

[0068] Furthermore, in the movable area display device of the present disclosure, the distance measurement unit may measure the direction and distance from the device main body to the target object, and the relative position recognition unit may recognize the relative position of the robot device with respect to the device main body based on the direction and distance from the device main body to the robot device measured by the distance measurement unit.

[0069] In the movable area display device of the present disclosure, the distance measurement unit may recognize the size of the recognized part from an image of the robot device captured by the imaging unit while the recognized part is included in the image of the robot device, and measure the distance from the device main body to the robot device based on the recognized size of the recognized part. In this way, the relative position of the robot device can be recognized with relatively high accuracy.

[0070] The movable area display device of the present disclosure may further include an acceleration sensor capable of measuring acceleration, and an estimation unit that estimates a change in the relative position based on a measurement value of the acceleration sensor after the relative position recognition unit recognizes the relative position, and the display control unit may update the display of the movable area of ​​the robot arm on the display unit based on the estimated change in the relative position. This makes it possible to continue displaying the movable area of ​​the robot arm based on the estimated relative position.

[0071] In the movable area display device of the present disclosure, the movable area of ​​the robot arm may include a plurality of areas, and the display control unit may display each of the plurality of areas in a different manner, thereby enabling a worker to easily recognize the boundaries of each movable area.

[0072] The movable area display device of the present disclosure may include a relative position recognition unit that recognizes the relative position of the robot device with respect to the device main body based on the distance from the device main body to the robot device measured by the distance measurement unit, and the display control unit may convert coordinate values ​​of the movable area of ​​the robot arm in the robot coordinate system acquired by the acquisition unit based on the relative position of the robot device recognized by the relative position recognition unit and the attitude of the robot device recognized by the attitude recognition unit into coordinate values ​​of the movable area of ​​the robot arm in an image coordinate system of an image of the robot device captured by the imaging unit. In this way, the coordinate values ​​of the movable area of ​​the robot arm in the robot coordinate system can be converted into more accurate coordinate values ​​of the movable area of ​​the robot arm in the image coordinate system.

[0073] The movable area display device of the present disclosure may further include a shape acquisition unit including at least one of a three-dimensional sensor and a three-dimensional camera capable of acquiring shape data of the object, and the posture recognition unit may recognize the posture of the robot device relative to the device body based on the shape data of the robot device acquired by the shape acquisition unit. In this way, the posture of the robot device can be recognized with relatively high accuracy.

[0074] In the movable area display device of the present disclosure, the robot device may have a plurality of recognized parts recognized by the relative position recognition unit, and the posture recognition unit may recognize the posture based on the positional relationship or size difference of the plurality of recognized parts included in the image of the robot device. In this way, the relative position and posture of the robot device can be recognized relatively easily.

[0075] This specification discloses the technical idea of ​​changing "a movable area display device according to claim 1 or 2" in claim 4 originally filed to "a movable area display device according to any one of claims 1 to 3." Also, this specification discloses the technical idea of ​​changing "a movable area display device according to claim 1 or 2" in claim 5 originally filed to "a movable area display device according to any one of claims 1 to 4."

[0076] The present disclosure is applicable to industries such as the manufacturing of robot systems used in ultrasonic echo guidance.

[0077] 10 Robot system, 20 Robot device, 21 Robot arm, 22 First arm, 23 Second arm, 24 Base, 25 Base, 26 Caster, 27 Lever, 28 Locking unit, 30 Three-dimensional sensor, 31 First joint axis, 32 Second joint axis, 33 Attitude holding axis, 35 First arm driving device, 35a Motor, 35b Encoder, 36 Second arm driving device, 36a Motor, 36b Encoder, 37 Attitude holding device, 37a Motor, 37b Encoder, 40 Lifting device, 41 Slider, 42 Guide member, 43 Ball screw shaft, 44a Motor, 44b Encoder, 50 Rotation three-axis mechanism, 51 First rotating axis, 52 Second rotating axis, 53 Third rotating axis, 55 First rotating device, 56 Second rotating device, 57 Third rotating device, 58 Holding unit, 59 Direct teaching switch, 60 housing, 70 robot control device, 76 operation panel, 80 tablet terminal, 81 terminal main body, 82 camera, 83 three-dimensional sensor, 84 display operation device, 85 acceleration sensor, 86 gyro sensor, 90 control device, 91 CPU, 92 ROM, 93 RAM, 94 storage, 94a first movable area data, 94b second movable area data, 95 communication unit, 100 ultrasound diagnostic device, 101 ultrasound probe, 102 cable, 110 ultrasound diagnostic device main body, 111 ultrasound diagnosis control unit, 112 image processing unit, 113 image display unit, A1 first movable area, A2 second movable area, F1 recognition flag, Im captured image, M1, M2 AR markers.

Claims

1. A movable area display device capable of displaying a movable area of ​​a robot arm of a robot device having a robot arm, comprising: a device main body; a display unit provided on the device main body; an imaging unit provided on the device main body and capable of capturing an image of an object; a distance measurement unit provided on the device main body and measuring a distance from the device main body to the object; a relative position recognition unit that recognizes a relative position of the robot device with respect to the device main body based on the distance from the device main body to the robot device measured by the distance measurement unit; an acquisition unit that acquires coordinate values ​​of the movable area of ​​the robot arm in a robot coordinate system based on a predetermined position of the robot device; and a display control unit that converts the coordinate values ​​of the movable area of ​​the robot arm in the robot coordinate system acquired by the acquisition unit based on the relative position of the robot device recognized by the relative position recognition unit into coordinate values ​​of the movable area of ​​the robot arm in an image coordinate system of an image of the robot device captured by the imaging unit, and displays the converted movable area of ​​the robot arm on the display unit while superimposing it on the image of the robot device.

2. A movable area display device as described in claim 1, wherein the robot device has a recognized part that is recognized by the relative position recognition unit, and the distance measurement unit recognizes the size of the recognized part from an image of the robot device captured by the imaging unit while including the recognized part, and measures the distance from the device body to the robot device based on the recognized size of the recognized part.

3. A movable area display device as described in claim 1 or 2, comprising: an acceleration sensor capable of measuring acceleration; and an estimation unit that estimates a change in the relative position based on the measured acceleration value after the relative position recognition unit recognizes the relative position, wherein the display control unit updates the display of the movable area of ​​the robot arm on the display unit based on the estimated change in the relative position.

4. A movable area display device according to claim 1 or 2, wherein the movable area of ​​the robot arm has a plurality of areas, and the display control unit displays each of the plurality of areas in a different manner.

5. A movable area display device as claimed in claim 1 or 2, comprising an attitude recognition unit capable of recognizing the attitude of the object relative to the device main body, and the display control unit converts the coordinate values ​​of the movable area of ​​the robot arm in the robot coordinate system acquired by the acquisition unit based on the relative position of the robot device recognized by the relative position recognition unit and the attitude of the robot device recognized by the attitude recognition unit, into coordinate values ​​of the movable area of ​​the robot arm in the image coordinate system of the image of the robot device captured by the imaging unit.

6. A movable area display device as described in claim 5, comprising a shape acquisition unit including at least one of a three-dimensional sensor and a three-dimensional camera capable of acquiring shape data of the object, and the posture recognition unit recognizes the posture of the robot device relative to the device main body based on the shape data of the robot device acquired by the shape acquisition unit.

7. A movable area display device as described in claim 5, wherein the robot device has a plurality of recognized parts that are recognized by the relative position recognition unit, and the posture recognition unit recognizes the posture based on the positional relationship or the difference in size of the plurality of recognized parts included in an image of the robot device.

Images