Image processing device, robot hand operation system, image processing method, and image processing program

Abstract

To provide an image processing device, etc. which can easily grasp distance feeling between a robot hand and an object, without using a special display.SOLUTION: An image processing device comprises: robot hand coordinates acquisition means which acquires robot hand coordinates; robot hand travel direction acquisition means which acquires a travel direction of the robot hand; depth image acquisition means which acquires a depth image of the robot hand from a depth camera; color image acquisition means which acquires a color image; pixel coordinate calculation means which calculates a pixel coordinate of each pixel of the color image, on the basis of a position of the depth camera and the depth image; virtual plane calculation means which calculates a virtual plane perpendicular to the travel direction of the robot hand and located at a predetermined distance, by using the coordinates and the travel direction of the robot hand; virtual plane image generation means which creates a virtual plane image by using a predetermined part on the virtual plane; and composite image generation means which creates a composite image of the virtual plane image and the color image.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device, a robot hand operation system, an image processing method, and an image processing program. [Background technology]

[0002] In recent years, there has been an increasing demand for remote control of robots. In remote control of such robots, a robot controller and a display installed at a location away from the robot are used. In operating a robot equipped with a robot hand, for example, an image including the robot hand and an object is displayed on the display. An operator then operates the robot hand while looking at this image. In this case, a challenge is for the operator to grasp the sense of distance between the robot hand and the object.

[0003] A technique for solving the above problem is disclosed in, for example, Patent Document 1. In this technique, a driver's seat similar to the cockpit of the construction machine, an operating stick, and a display unit that displays the scenery captured by the imaging unit of the construction machine are provided at a location away from the construction machine. For example, a hydraulic excavator (backhoe) equipped with a bucket (corresponding to a robot hand) is used as the construction machine. The construction machine is remotely controlled by operating the operating stick. And, as the above display unit, for example, three display devices are arranged in a horizontal row. By displaying such an image, it becomes easy to recognize the positional relationship between the bucket and the construction object. Patent Document 1 also claims that the same effect can be obtained by using a single horizontally long display device or a display device curved in a circular ring shape. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-214892 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technology of Patent Document 1 requires arranging multiple display devices horizontally, using a horizontally long display device, or using a display device curved in a circular ring shape, which means that a special display needs to be configured.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide an image processing device, etc. that makes it easy to grasp the distance between a robot hand and an object without using a special display. [Means for solving the problem]

[0007] In order to solve the above problems, the image processing device of the present disclosure has a robot hand coordinate acquisition means for acquiring robot hand coordinates of a robot hand that can move relative to a reference point, a robot hand movement direction acquisition means for acquiring the movement direction of the robot hand, a depth image acquisition means for acquiring a depth image of the robot hand from a depth camera that photographs the robot hand, a color image acquisition means for acquiring a color image of the robot hand from the depth camera, a pixel coordinate calculation means for calculating pixel coordinates of each pixel that constitutes the color image based on the position of the depth camera and the depth image, a virtual plane calculation means for calculating a virtual plane that is perpendicular to the movement direction of the robot hand and is a predetermined distance from the robot hand using the coordinates of the robot hand and the movement direction of the robot hand, a virtual plane image generation means for generating a virtual plane image using a predetermined portion on the virtual plane, and a composite image generation means for generating a composite image by combining the virtual plane image and the color image.

[0008] In addition, the robot hand operating system disclosed herein includes the above-mentioned image processing device, the robot hand, and a robot hand operating unit that is provided at a location separate from the robot hand and includes a display device that displays the composite image and a robot hand controller that operates the robot hand.

[0009] In addition, the image processing method disclosed herein involves a computer acquiring robot hand coordinates of a robot hand that can move relative to a reference point, acquiring the movement direction of the robot hand, acquiring a depth image of the robot hand from a depth camera that photographs the robot hand, acquiring a color image of the robot hand from the depth camera, calculating pixel coordinates of each pixel that makes up the color image based on the position of the depth camera and the depth image, calculating a virtual plane that is perpendicular to the movement direction of the robot hand and at a predetermined distance from the robot hand using the coordinates of the robot hand and the movement direction of the robot hand, generating a virtual plane image using a predetermined portion on the virtual plane, and generating a composite image by combining the virtual plane image and the color image.

[0010] In addition, the image processing program of the present disclosure causes a computer to execute the following processes: acquiring robot hand coordinates of a robot hand that can move relative to a reference point; acquiring the movement direction of the robot hand; acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; acquiring a color image of the robot hand from the depth camera; calculating pixel coordinates of each pixel that constitutes the color image based on the position of the depth camera and the depth image; calculating a virtual plane that is perpendicular to the movement direction of the robot hand and at a predetermined distance from the robot hand using the coordinates of the robot hand and the movement direction of the robot hand; generating a virtual plane image using a predetermined portion on the virtual plane; and generating a composite image by combining the virtual plane image and the color image. Effect of the Invention

[0011] An advantage of the present invention is that it is possible to provide an image processing device or the like that makes it easy to grasp the sense of distance between a robot hand and an object without using a special display. [Brief description of the drawings]

[0012] [Figure 1]FIG. 1 is a block diagram illustrating an example of an image processing device according to the present disclosure. [Diagram 2] 10 is a flowchart illustrating an example of an operation of the image processing device of the present disclosure. [Diagram 3] FIG. 2 is a block diagram showing an example hardware configuration of a display system according to the present disclosure. [Figure 4] FIG. 1 is a block diagram illustrating an example of an image processing system according to the present disclosure. [Diagram 5] FIG. 1 is a block diagram illustrating an example of a robot hand manipulation system according to the present disclosure. [Figure 6] FIG. 2 is a schematic diagram showing a coordinate system of the robot hand manipulation system of the present disclosure. [Figure 7] 1 is a schematic diagram showing a first specific example of a composite image of the present disclosure. [Figure 8] 13 is a schematic diagram showing a second specific example of a composite image of the present disclosure. FIG. [Figure 9] 13 is a schematic diagram showing a third specific example of a composite image of the present disclosure. FIG. [Figure 10] 13 is a schematic diagram showing a fourth specific example of a composite image of the present disclosure. FIG. [Figure 11] FIG. 13 is a schematic diagram showing a fifth specific example of a composite image according to the present disclosure. [Figure 12] 13 is a schematic diagram showing a sixth specific example of a composite image of the present disclosure. FIG. [Figure 13] FIG. 13 is a schematic diagram showing a seventh specific example of a composite image according to the present disclosure. [Figure 14] FIG. 13 is a schematic diagram showing an eighth specific example of a composite image according to the present disclosure. [Figure 15] 13 is a schematic diagram showing another example of the eighth specific example of a composite image of the present disclosure. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments of the present invention will be described in detail with reference to the drawings. However, the embodiments described below are limited in a technically preferable manner for carrying out the present invention, but the scope of the invention is not limited to the following. Note that the same components in each drawing are given the same numbers, and the description may be omitted.

[0014] (First embodiment) 1 is a block diagram showing an example of an image processing device 100 according to the present disclosure. The image processing device 100 includes a robot hand coordinate acquisition means 110, a robot hand movement direction acquisition means 120, a depth image acquisition means 130, a color image acquisition means 140, and a pixel coordinate calculation means 150. The image processing device 100 also includes a virtual plane calculation means 160, a virtual plane image generation means 170, and a composite image generation means 180.

[0015] The robot hand coordinates are acquired by the robot hand coordinate acquisition means 110. The robot hand is capable of moving relative to a reference point.

[0016] The moving direction of the robot hand is acquired by a robot hand moving direction acquisition means 120. In this embodiment, when the robot hand is stationary, the moving direction is defined as a predetermined front of the robot hand.

[0017] A depth image of the robot hand is acquired by a depth camera that captures an image of the robot hand, and the depth image acquisition means 130 acquires the depth image of the robot hand. In this embodiment, the depth camera is capable of acquiring a depth image and a color image simultaneously.

[0018] A color image acquisition means 140 acquires a color image of the robot hand from a depth camera.

[0019] The pixel coordinates of each pixel constituting the color image are calculated by pixel coordinate calculation means 150. The above calculation is performed based on the position of the depth camera and the depth image.

[0020] The virtual plane calculation means 160 calculates a virtual plane. The virtual plane is perpendicular to the moving direction of the robot hand. The virtual plane calculation means 160 calculates a virtual plane that is a predetermined distance from the robot hand. The virtual plane is calculated using the coordinates of the robot hand and the moving direction of the robot hand.

[0021] A virtual plane image is generated by the virtual plane image generating means 170. The virtual plane image is generated using a predetermined portion of the virtual plane.

[0022] The composite image generating means 180 generates a composite image by combining the virtual plane image and the color image. In the composite image, the virtual plane image is displayed between the robot hand and the object. This makes it easier for the operator to grasp the sense of distance.

[0023] Next, the operation of the image processing device 100 will be described. Fig. 2 is a flowchart showing an example of the operation of the image processing device 100 of the present disclosure. First, the robot hand coordinate acquisition means 110 acquires the robot hand coordinates (S101). Although details will be described later, these coordinates are determined, for example, in the coordinate system of the robot to which the robot hand is attached. The robot hand can move relative to a reference point of the robot coordinate system.

[0024] Next, the moving direction of the robot hand is acquired by the robot hand moving direction acquisition means 120 (S102). Note that in this embodiment, when the robot hand is stationary, the moving direction is defined as a predetermined front of the robot hand.

[0025] Next, the depth image acquisition means 130 acquires a depth image of the robot hand from a depth camera that captures an image of the robot hand (S103). The depth image is obtained as a point cloud.

[0026] Next, the color image acquisition means 140 acquires a color image of the robot hand from the depth camera (S104). Here, the color image is a general two-dimensional image. The color image is, for example, a color image with a preset number of pixels. However, the color image may be a monochrome image.

[0027] Next, the pixel coordinates of each pixel constituting the color image are calculated by the pixel coordinate calculation means 150 (S105). Here, the pixel coordinates are the coordinates of a point corresponding to a pixel in the color image. That is, each pixel in the color image is associated with the coordinates of a point in the corresponding depth image. The above association is performed based on the position of the depth camera and the depth image.

[0028] Next, the virtual plane calculation means 160 calculates a virtual plane (S106). The virtual plane is perpendicular to the moving direction of the robot hand. The virtual plane is at a predetermined distance from the robot hand. The virtual plane is calculated using the coordinates of the robot hand and the moving direction of the robot hand.

[0029] Next, the virtual plane image generating means 170 generates a virtual plane image (S107). The virtual plane image is generated using a predetermined portion of the virtual plane. Although details will be described later, for example, a predetermined rectangular or circular portion of the virtual plane becomes the virtual plane image.

[0030] Next, the composite image generating means 180 generates a composite image by combining the virtual plane image and the color image (S108). This composite image is output to the display device, and the composite image is displayed on the display device. In the composite image, the virtual plane image is displayed between the robot hand and the object. This makes it easier for the operator to grasp the sense of distance.

[0031] Next, a hardware configuration of the image processing device 100 will be described. FIG. 3 is a block diagram showing an example of a hardware configuration of the image processing device 100 of the present disclosure. The image processing device 100 is implemented in, for example, a general computer. This computer includes, for example, a processor 101, a memory 102, a display unit 103, an input / output unit 104, a storage unit 105, and a communication interface 106. The processor 101 performs calculations necessary for image processing. The memory 102 stores data used in the calculations. The display unit 103 displays data and images. The input / output unit 104 is a device that inputs and outputs data. The input / output unit 104 includes, for example, a keyboard, a mouse, an audio signal output unit, and a print signal output unit. The storage unit 105 stores data. The communication interface 106 performs interface processing for communication. Image processing and display are possible with the above hardware configuration.

[0032] An image processing system that acquires an image and processes the image can be configured using the image processing device 100 of the present disclosure. Fig. 4 is a block diagram showing an example of the image processing system 1000 of the present disclosure. The image processing system 1000 includes the image processing device 100 and a depth camera 200.

[0033] The depth camera 200 is a camera that can measure the distance (depth) from the camera to an object. Methods for measuring distance include a stereo camera method, a ToF (Time of Flight) method, and a structured lighting method. In this embodiment, real-time performance is important. For this reason, the stereo camera method and the ToF method are suitable. The depth camera 200 in this example is also equipped with a camera that captures normal two-dimensional images. A color image is acquired by this camera.

[0034] In the image processing system 1000, a depth image and a color image obtained by a depth camera 200 are acquired by an image processing device 100. Then, a composite image is generated by the image processing device 100.

[0035] Next, a robot hand operating system using the image processing system 1000 will be described. FIG. 5 is a block diagram showing an example of the robot hand operating system 10000 of the present disclosure. The robot hand operating system 10000 has the image processing system 1000, a robot hand 2000, and a robot hand operating unit 3000. The robot hand operating unit 3000 is provided at a location away from the robot hand 2000. In other words, the robot hand 2000 is remotely controlled by the robot hand operating unit 3000. The robot hand operating unit 3000 is provided with a display device 3100 and a robot hand controller 3200.

[0036] The robot hand controller 3200 is operated by an operator. The robot hand operates in response to an input to the robot hand controller 3200. A composite image in which an image of the robot hand 2000 and a virtual plane image are combined is displayed on the display device 3100. Since the operator operates the robot hand controller while viewing the composite image, it becomes possible to bring the behavior of the robot hand closer to the feeling of operation of the operator. Specifically, the virtual plane image is displayed between the robot hand and the object. This makes it easy to grasp the sense of distance between the robot hand and the object.

[0037] FIG. 6 is a schematic diagram showing a robot hand coordinate system in the robot hand operation system 10000. This is a diagram for explaining the coordinate systems of the stereo camera 4 and the robot 2. As shown in FIG. 6, in this example, a world coordinate system WCS, a camera coordinate system CCS, a robot coordinate system RCS, and a hand coordinate system HCS are defined. The world coordinate system WCS is a coordinate system that is a reference for the entire environment in which the robot hand 2000 is installed. The position and orientation of the depth camera 200 are expressed as coordinate values ​​of the world coordinate system WCS. Similarly, the position and orientation of the robot hand 2000 are expressed as coordinate values ​​of the world coordinate system WCS. However, if the robot coordinate system RCS is immovable relative to the world coordinate system WCS, the robot coordinate system RCS may be used as a reference coordinate system instead of the world coordinate system WCS. For this reason, in the following description, the robot coordinate system RCS is used as a reference coordinate system instead of the world coordinate system WCS. The robot coordinate system RCS is a coordinate system with the reference point of the robot 2100 as the origin.

[0038] The camera coordinate system CCS is a coordinate system based on the depth camera 200. The distance (depth) to an object in the depth image is acquired as the Z coordinate of the camera coordinate system. Here, the position and attitude (orientation) of the depth camera 200 can be expressed in the robot coordinate system RCS. Therefore, the coordinates of each point in the depth image can be expressed by the coordinate value of the robot coordinate system RCS.

[0039] The hand coordinate system HCS is a coordinate system based on the hand. In the example of Fig. 6, a configuration is illustrated in which a robot hand 2000 includes a claw 2001. Here, for example, the direction of the tip of the claw 2001 is set as X in the hand coordinate system. The X direction is set as the moving direction of the robot hand 2000 or the front direction. The opening and closing direction of the claw 2001 is the Y direction of the hand coordinate system HCS. The direction perpendicular to X and Y is the Z direction.

[0040] (Example 1) Next, a method for generating a composite image will be described. First, a virtual plane in the hand coordinate system HCS is calculated by the following equation. ax1+by+cz+d=0, a, b, c, d are parameters (Equation 1) Here, x1 is a constant determined during calculation and corresponds to the distance from the front of the robot hand.

[0041] Next, the portion of the virtual plane to be displayed is set to generate a virtual plane image. For example, if the following four line segments are specified, a rectangular wire frame is specified. y = y1, -z1 ≤ z ≤ z1 (Equation 2) y = -y1, -z1 ≤ z ≤ z1 (Equation 3) z=z1, -y1≦y≦y1 (Formula 4) z=-z1, -y1≦y≦y1 (Formula 5) Then, the coordinates of the wire frame in the hand coordinate system HCS are transformed into coordinates in the camera coordinate system CCS, thereby obtaining a virtual planar image.

[0042] The virtual planar image and the color image obtained above are synthesized by the synthetic image generating means 180. In this way, a synthetic image is generated. Fig. 7 is a schematic diagram showing a first specific example of the synthetic image of the present disclosure thus obtained. Fig. 7 shows an image in which an image of the robot hand 2000 and a virtual planar image p are synthesized.

[0043] In the above description, the virtual planar image p is an example of a wire frame, but the virtual planar image may have a shape other than a rectangle. For example, the inside of the wire frame may be colored. This coloring may be performed semi-transparently. A pattern such as a grid may be added.

[0044] (Example 2) In FIG. 7, an example of one virtual plane image p is shown, but there may be multiple virtual plane images p. FIG. 8 is a schematic diagram showing a second specific example of a composite image of the present disclosure. In FIG. 8, three wire frames are composited. The distances from the origin of the robot hand are x1, x2, and x3, respectively. In this way, by compositing multiple virtual plane images, it becomes easier to grasp the sense of distance to the object. Here, for example, multiple wire frames p1, p2, and p3 are arranged at equal intervals. For example, the distances between the robot hand 2000 and each of the wire frames p1, p2, and p3 are set so that x2=2·x1 and x3=3·x1.

[0045] (Example 3) When the robot hand 2000 approaches an object, the faster the moving speed is, the faster the robot hand 2000 approaches the object. Therefore, by adjusting the interval between the virtual planar images according to the moving speed, more intuitive operation can be facilitated.

[0046] Fig. 8 is a schematic diagram showing a second specific example of a composite image of the present disclosure. The diagram on the left side of Fig. 9 is a composite image when the moving speed is v1. The diagram on the right side is a composite image when the moving speed is v2, which is greater than v1. In this specific example, a composite image is generated such that the interval between virtual planar images becomes wider as the moving speed becomes faster.

[0047] For example, the interval is set according to the following formula: d=d0×v / v0 (formula 6) Here, d0 is a reference interval and v0 is a reference speed, and d0 and v0 are determined in advance.

[0048] In this way, the interval between the virtual planar images increases according to the moving speed of the robot hand 2000. This makes it easier for the operator to grasp the change in the distance between the object and the robot hand 2000.

[0049] (Example 4) 10 is a schematic diagram showing a fourth specific example of a composite image of the present disclosure. In this specific example, the width of the virtual plane image p is set according to the opening width w of the claws 2001 provided on the robot hand 2000. The pair of claws 2001 pinch the object, and the robot hand 2000 grasps the object. That is, the opening width of the claws 2001 varies. Therefore, in the fourth specific example, the widths of the wire frames p1, p2, and p3 vary according to the opening width of the claws 2001. This operation makes it easier to predict interference between the robot hand 2000 and the object or other obstacles.

[0050] (Example 5) FIG. 11 is a schematic diagram showing a fifth specific example of a composite image of the present disclosure. In this example, a plurality of virtual plane images p are displayed in different colors. In FIG. 11, three wire frames p1, p2, and p3 are colored red, yellow, and green. In this example, green, yellow, and red are displayed in the order of furthest from the robot hand 2000. The closer the robot hand 2000 is to the object 4000, the more concentration is required of the operator. In the above configuration, this situation is made known to the operator.

[0051] (Example 6) 12 is a schematic diagram showing a sixth specific example of a composite image of the present disclosure. In this specific example, the display of a portion of the virtual plane image p that is behind the object 4000 is omitted. In the example of FIG. 12, three wire frames p1, p2, and p3 are composited into the image. Then, the display of a portion of the wire frame p3 that is behind the object 4000 is omitted. By using such a display, it becomes easier for the operator to grasp the positional relationship between the object 4000 and the robot hand 2000.

[0052] To realize the above display, first, the coordinates of each point of the wire frame are converted into the robot coordinate system RCS (or the world coordinate system WCS). Furthermore, the coordinates of each point of the wire frame are converted into the coordinates of the camera coordinate system CCS. Then, the Z coordinate of this point in the camera coordinate system is compared with the Z coordinate of each point cloud data of the depth image of the object 4000. As a result, the pixels of the wire frame that have a larger Z coordinate than the point cloud data are excluded from the composite image.

[0053] (Example 7) FIG. 13 is a schematic diagram showing a seventh specific example of a composite image of the present disclosure. In this specific example, the object 4000 is colored according to the positional relationship with the virtual planar image p. In the example of FIG. 13, a wire frame p1 is composited at the tip of the nail 2001. A wire frame p2 is composited at a position a predetermined distance away from the tip of the nail. A wire frame p3 is composited at a position a predetermined distance away from the tip of the nail. The area between the wire frame p1 and the wire frame p2 of the object 4000 is designated as A. The area between the wire frame p2 and the wire frame p3 of the object 4000 is designated as B. The area beyond the wire frame p3 of the object 4000 is designated as C. The area A, the area B, and the area C are colored in different colors. This makes it easier for the operator to grasp the positional relationship between the robot hand 2000 and the object 4000.

[0054] (Example 8) In the specific examples 1 to 7, examples have been mainly described in which the robot hand 2000 has the openable / closable claw 2001. Also, examples have been described in which the virtual planar image p is rectangular. However, the tool attached to the robot hand 2000 is not limited to the claw 2001. Also, the virtual planar image p is not limited to a rectangle.

[0055] Fig. 14 is a schematic diagram showing an eighth specific example of a composite image of the present disclosure. In this example, an adsorption chuck 2002 is attached to a robot hand. The tip of the adsorption chuck 2002 is circular, and the virtual plane image p is also drawn as a circle. In Fig. 14, two circular wire frames p1 and p2 are drawn.

[0056] Fig. 15 is a schematic diagram showing another example of the eighth specific example of the composite image of the present disclosure. As shown in Fig. 15, a circular wire frame is composited into a color image. In Fig. 15, two circular wire frames p1 and p2 are composited. The inside of the wire frame p2 that overlaps with the object 4000 is colored.

[0057] It should be noted that the tools attached to the robot hand 2000 are not limited to the claws 2001 and the suction chuck 2002. For example, they may be drivers, sockets, etc. Furthermore, the shape of the virtual plane image p is not limited to a rectangle or a circle.

[0058] The image processing device 100 and the like of the present disclosure have been described above. The image processing device 100 of the present disclosure includes a robot hand coordinate acquisition means 110, a robot hand movement direction acquisition means 120, a depth image acquisition means 130, a color image acquisition means 140, and a pixel coordinate calculation means 150. The image processing device 100 also includes a virtual plane calculation means 160, a virtual plane image generation means 170, and a composite image generation means 180. The robot hand coordinate acquisition means 110 acquires robot hand coordinates. The robot hand 2000 can move relative to a reference point. The robot hand movement direction acquisition means 120 acquires the movement direction of the robot hand 2000. The depth image acquisition means 130 acquires a depth image of the robot hand 2000 from a depth camera that captures the robot hand 2000. The color image acquisition means 140 acquires a color image of the robot hand 2000 from the depth camera 200. The pixel coordinate calculation means 150 calculates the pixel coordinates of each pixel constituting the color image. This calculation is performed based on the position of the depth camera and the depth image. The virtual plane calculation means 160 calculates a virtual plane. The virtual plane is perpendicular to the moving direction of the robot hand 2000. The virtual plane calculation means 160 calculates a virtual plane at a predetermined distance from the robot hand 2000. The virtual plane is calculated using the coordinates of the robot hand and the moving direction of the robot hand. The virtual plane image generation means 170 generates a virtual plane image p. The virtual plane image p is generated using a predetermined portion of the virtual plane. The composite image generation means 180 generates a composite image by combining the virtual plane image p and the color image.

[0059] In the above configuration, the virtual plane image p is displayed between the robot hand 2000 and the target object 4000. This makes it easier for the operator to grasp the sense of distance.

[0060] According to one embodiment, the virtual plane calculation means 160 calculates a plurality of virtual planes in the image processing device 100. The virtual planes have mutually different distances from the robot hand 2000.

[0061] In the above configuration, a plurality of virtual planes having different distances from the robot hand 2000 are calculated. Then, a plurality of virtual plane images p are displayed. This makes it easier for the operator to grasp the sense of distance.

[0062] According to one embodiment, in the image processing device 100, the virtual plane image generating means 170 generates a plurality of virtual plane images p in different colors. According to the above configuration, the virtual plane images having different distances from the robot hand 2000 are displayed in different colors, which makes it easier for the operator to grasp the sense of distance.

[0063] According to one embodiment, in the image processing device 100, the virtual plane calculation means 160 sets intervals between the multiple virtual planes in accordance with the moving speed of the robot hand 2000. In the above configuration, the moving speed of the robot hand 2000 is easily recognized by the operator.

[0064] Moreover, according to one embodiment, in the image processing device 100, the virtual planar image generating means 170 generates the virtual planar image p as a wire frame.

[0065] By displaying the virtual planar image p as a wire frame, the virtual planar image p in the composite image can be easily recognized. Also, the amount of data of the virtual planar image p is reduced.

[0066] According to one embodiment, in the image processing device 100, the claws 2001 are attached to the robot hand 2000. Then, the virtual plane image generating means 170 generates a virtual plane image so as to correspond to the opening width of the claws. This process is performed when the robot hand 2000 is configured to open and close the claws 2001.

[0067] The above configuration makes it easy to understand the movement range of the robot hand 2000. This makes it easy to prevent interference between the robot hand 2000 and an object.

[0068] According to one embodiment, in the image processing device 100, only a portion of the virtual plane image p that is in front of the object is synthesized with the color image by the synthetic image generating means 180. This process is performed when an object other than the robot hand 2000 exists in the color image.

[0069] With the above configuration, it becomes easy to grasp the positional relationship between the robot hand 2000 and an object in the depth direction.

[0070] The image processing system 1000 of the present disclosure includes an image processing device 100 and a depth camera 200.

[0071] In this configuration, a depth image and a color image of the robot hand 2000 are acquired by the depth camera 200. Then, the image processing device 100 generates a composite image by combining the color image with a virtual planar image p.

[0072] The composite image makes it easier for the operator to grasp the sense of distance.

[0073] Moreover, the robot hand operating system 10000 of the present disclosure includes an image processing device 100, a robot hand 2000, and a robot hand operating unit 3000 provided at a location remote from the robot hand 2000. The robot hand operating unit 3000 includes a display device 3100 and a robot hand controller 3200. The robot hand 2000 is remotely operated by the robot hand controller 3200. A composite image generated by the image processing device 100 is displayed on the display device 3100.

[0074] In the above configuration, the operator can operate the robot hand controller 3200 while viewing a composite image that allows the operator to easily grasp the sense of distance. This enables more accurate remote control of the robot hand 2000.

[0075] In the image processing method of the present disclosure, the computer acquires robot hand coordinates. The robot hand 2000 can move relative to a reference point. The computer acquires the moving direction of the robot hand 2000. The computer acquires a depth image of the robot hand 2000 from the depth camera 200. The computer acquires a color image of the robot hand 2000 from the depth camera 200. The depth camera 200 captures a depth image and a color image of the robot hand 2000. The computer calculates pixel coordinates of each pixel constituting the color image. This calculation is performed based on the position of the depth camera 200 and the depth image. Next, the computer calculates a virtual plane. This calculation is performed using the coordinates of the robot hand 2000 and the moving direction of the robot hand 2000. The virtual plane is perpendicular to the moving direction of the robot hand 2000. The virtual plane is at a predetermined distance from the robot hand 2000. The computer generates a virtual plane image p using a predetermined portion of the virtual plane. Then, the computer generates a composite image by combining the virtual planar image p and the color image.

[0076] By using the image processing method configured as described above, the virtual plane image p is displayed between the robot hand 2000 and the object 4000. This makes it easier for the operator to grasp the sense of distance.

[0077] The image processing program of the present disclosure causes a computer to execute the following processes. One of the processes is a process of acquiring the robot hand coordinates of the robot hand 2000. The robot hand 2000 can move relative to a reference point. Another process is a process of acquiring the moving direction of the robot hand 2000. Another process is a process of acquiring a depth image and a color image from the depth camera 200. The depth camera 200 captures a depth image and a color image of the robot hand 2000. Another process is a process of calculating pixel coordinates of each pixel constituting the color image. This calculation is performed based on the position of the depth camera 200 and the depth image. Another process is a process of calculating a virtual plane. This calculation is performed using the coordinates of the robot hand 2000 and the moving direction of the robot hand. The virtual plane is perpendicular to the moving direction of the robot hand 2000. The virtual plane is at a predetermined distance from the robot hand 2000. Another process is a process of generating a virtual plane image p. A virtual plane image p is generated using a predetermined portion of the virtual plane. This is a process for generating the virtual plane image p. Then, the computer is caused to execute a process for generating a composite image by combining the virtual plane image p and a color image.

[0078] With the above configuration, the virtual plane image p is displayed between the robot hand 2000 and the target object 4000. This makes it easier for the operator to grasp the sense of distance.

[0079] The scope of the present invention also includes a program for causing a computer to execute the processes of the above-described embodiment and a recording medium storing the program. As the recording medium, for example, a magnetic disk, a magnetic tape, an optical disk, a magneto-optical disk, a semiconductor memory, etc. can be used.

[0080] The present invention has been described above by taking the above-mentioned embodiment as an exemplary example. However, the present invention is not limited to the above-mentioned embodiment. That is, the present invention can be applied in various aspects that can be understood by a person skilled in the art within the scope of the present invention.

[0081] Some or all of the above embodiments may be described as follows: However, the scope of the present disclosure is not limited to the following.

[0082] (Appendix 1) a robot hand coordinate acquisition means for acquiring a robot hand coordinate of a robot hand that is movable relative to a reference point; A robot hand movement direction acquisition means for acquiring a movement direction of the robot hand; A depth image acquisition means for acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; A color image acquisition means for acquiring a color image of the robot hand from the depth camera; a pixel coordinate calculation means for calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; a virtual plane calculation means for calculating a virtual plane that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand, using the coordinates of the robot hand and the moving direction of the robot hand; a virtual plane image generating means for generating a virtual plane image using a predetermined portion of the virtual plane; a composite image generating means for generating a composite image by combining the virtual planar image and the color image; 13. An image processing device comprising:

[0083] (Appendix 2) The virtual plane calculation means calculating a plurality of the virtual planes having different distances from the robot hand; 2. The image processing device according to claim 1 .

[0084] (Appendix 3) The virtual plane image generating means generating a plurality of the virtual planar images in different colors; 3. The image processing device according to claim 2,

[0085] (Appendix 4) The virtual plane calculation means setting intervals between the plurality of virtual planes in accordance with a moving speed of the robot hand; 4. The image processing device according to claim 2 or 3.

[0086] (Appendix 5) The virtual plane image generating means generating the virtual planar image as a wireframe; 5. The image processing device according to claim 1,

[0087] (Appendix 6) When the robot hand is configured to open and close the claws, The virtual plane image generating means generating the virtual planar image so as to correspond to the opening width of the nails; 6. The image processing device according to claim 1,

[0088] (Appendix 7) When an object other than the robot hand is present in the color image, The composite image generating means Only a portion of the virtual plane image that is in front of the object is synthesized with the color image. 7. The image processing device according to claim 1,

[0089] (Appendix 8) An image processing device according to any one of claims 1 to 7, The depth camera; 1. An image processing system comprising:

[0090] (Appendix 9) An image processing device according to any one of claims 1 to 8, The robot hand; a robot hand operation unit provided at a location separate from the robot hand and including a display device for displaying the composite image and a robot hand controller for operating the robot hand; A robot hand operation system comprising:

[0091] (Appendix 10) The computer Obtaining robot hand coordinates of a robot hand that can move relative to a reference point; Obtaining a moving direction of the robot hand; Acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; acquiring a color image of the robot hand from the depth camera; Calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; Using the coordinates of the robot hand and the moving direction of the robot hand, a virtual plane is calculated that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand; generating a virtual plane image using a predetermined portion of the virtual plane; generating a composite image by combining the virtual planar image and the color image; 13. An image processing method comprising:

[0092] (Appendix 11) obtaining robot hand coordinates of a robot hand that is movable relative to a reference point; A process of acquiring a moving direction of the robot hand; A process of acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; acquiring a color image of the robot hand from the depth camera; calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; A process of calculating a virtual plane that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand, using the coordinates of the robot hand and the moving direction of the robot hand; generating a virtual plane image using a predetermined portion of the virtual plane; A process of generating a composite image by combining the virtual planar image and the color image; An image processing program characterized by causing a computer to execute the above steps. [Explanation of symbols]

[0093] 100 Image processing device 110 Robot hand coordinate acquisition means 120 Robot hand movement direction acquisition means 130 Depth image acquisition means 140 Color image acquisition means 150 Pixel coordinate calculation means 160 Virtual plane calculation means 170 Virtual Plane Image Generation Means 180 Synthetic image generation means 200 Depth Camera 1000 Image Processing System 2000 Robot Hand 2100 Robot 3000 Robot hand operation unit 3100 display device 3200 Robot Hand Controller 4000 objects

Claims

1. a robot hand coordinate acquisition means for acquiring a robot hand coordinate of a robot hand that is movable relative to a reference point; A robot hand movement direction acquisition means for acquiring a movement direction of the robot hand; A depth image acquisition means for acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; A color image acquisition means for acquiring a color image of the robot hand from the depth camera; a pixel coordinate calculation means for calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; a virtual plane calculation means for calculating a virtual plane that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand, using the coordinates of the robot hand and the moving direction of the robot hand; a virtual plane image generating means for generating a virtual plane image using a predetermined portion of the virtual plane; a composite image generating means for generating a composite image by combining the virtual planar image and the color image; 13. An image processing device comprising:

2. The virtual plane calculation means calculating a plurality of the virtual planes having different distances from the robot hand; 2. The image processing device according to claim 1,

3. The virtual plane image generating means generating a plurality of the virtual planar images in different colors; 3. The image processing device according to claim 2.

4. The virtual plane calculation means setting intervals between the plurality of virtual planes in accordance with a moving speed of the robot hand; 4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.

5. The virtual plane image generating means generating the virtual planar image as a wireframe; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

6. When the robot hand is configured to open and close the claws, The virtual plane image generating means generating the virtual planar image so as to correspond to the opening width of the nails; 4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

7. When an object other than the robot hand is present in the color image, The composite image generating means Only a portion of the virtual plane image that is in front of the object is synthesized with the color image.

4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.

8. An image processing device according to any one of claims 1 to 3, The robot hand; a robot hand operation unit provided at a location separate from the robot hand and including a display device for displaying the composite image and a robot hand controller for operating the robot hand; A robot hand operation system comprising:

9. The computer Obtaining robot hand coordinates of a robot hand that can move relative to a reference point; Obtaining a moving direction of the robot hand; Acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; acquiring a color image of the robot hand from the depth camera; Calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; Using the coordinates of the robot hand and the moving direction of the robot hand, a virtual plane is calculated that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand; generating a virtual plane image using a predetermined portion of the virtual plane; generating a composite image by combining the virtual planar image and the color image; 13. An image processing method comprising:

10. obtaining robot hand coordinates of a robot hand that is movable relative to a reference point; A process of acquiring a moving direction of the robot hand; A process of acquiring a depth image of the robot hand from a depth camera that photographs the robot hand; acquiring a color image of the robot hand from the depth camera; calculating pixel coordinates of each pixel constituting the color image based on the position of the depth camera and the depth image; A process of calculating a virtual plane that is perpendicular to the moving direction of the robot hand and is at a predetermined distance from the robot hand, using the coordinates of the robot hand and the moving direction of the robot hand; generating a virtual plane image using a predetermined portion of the virtual plane; A process of generating a composite image by combining the virtual planar image and the color image; An image processing program characterized by causing a computer to execute the above steps.

Images