Processing system, processing method, and program

Abstract

To provide a processing system capable of supporting a remote operation.SOLUTION: A processing system includes: a first processing section that draws a first line segment that is a line segment passing through a distal end position of a robot arm and parallel to a depth direction during remote operation of the robot arm; a second processing section that draws a plane along a vertical direction from the first line segment; and a third processing section that draws a line segment in the plane from the distal end position of the robot arm in the vertical direction.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] The present disclosure relates to a processing system, a processing method, and a program. [Background technology]

[0002] Robots are used in a variety of fields. Patent Document 1 discloses a related technique relating to a control device for remotely controlling a robot. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 3924495 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technical field related to the remote control of robots as disclosed in Patent Document 1, there is a demand for technology that can support remote control.

[0005] One of the objectives of each aspect of the present disclosure is to provide a processing system, a processing method, and a program that can solve the above-mentioned problems. [Means for solving the problem]

[0006] According to one aspect of the present disclosure, a processing system includes a first processing unit that, when remotely operating a robot arm, draws a first line segment that passes through the tip position of the robot arm and is parallel to the depth direction, a second processing unit that draws a surface along a vertical direction from the first line segment, and a third processing unit that draws a line segment within the surface in a vertical direction from the tip position of the robot arm.

[0007] According to another aspect of the present disclosure, a processing method includes, when remotely operating a robot arm, drawing a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction, drawing a plane along a vertical direction from the first line segment, and drawing a line segment within the plane in a vertical direction from the tip position of the robot arm.

[0008] According to another aspect of the present disclosure, a program causes a computer, when remotely operating a robot arm, to draw a first line segment that passes through the tip position of the robot arm and is parallel to the depth direction, draw a plane extending vertically from the first line segment, and draw a line segment within the plane extending vertically from the tip position of the robot arm. [Effects of the Invention]

[0009] According to each aspect of the present disclosure, remote operation can be supported. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 illustrates an example of a configuration of a processing system according to some embodiments of the present disclosure. [Figure 2] FIG. 1 is a diagram illustrating an example of a processing flow of a processing system according to some embodiments of the present disclosure. [Figure 3] FIG. 10 is a diagram illustrating an example of a line segment drawn by an image processing device according to some embodiments of the present disclosure. [Figure 4] 10A and 10B are diagrams for explaining line segments drawn by an image processing device according to some embodiments of the present disclosure. [Figure 5] 1A and 1B are diagrams for explaining a surface drawn by an image processing device according to some embodiments of the present disclosure. [Figure 6] 1A and 1B are diagrams for explaining lines drawn on the surface of an object by an image processing device according to some embodiments of the present disclosure. [Figure 7] 10A and 10B are diagrams for explaining line segments drawn by an image processing device according to some embodiments of the present disclosure. [Figure 8] FIG. 2 is a first diagram for explaining correction performed by an image processing device according to some embodiments of the present disclosure. [Figure 9] FIG. 2 is a second diagram for explaining correction performed by an image processing device according to some embodiments of the present disclosure. [Figure 10] FIG. 1 illustrates an example of a configuration of a processing system according to some embodiments of the present disclosure. [Figure 11] FIG. 2 is a diagram illustrating an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. [Figure 12] FIG. 1 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, the embodiments will be described in detail with reference to the drawings. <Embodiment> A processing system 1 according to an embodiment of the present disclosure will be described with reference to the drawings. The processing system 1 is a system that assists the operator of the robot arm 10 in grasping the position of the robot arm 10 in space by drawing lines and planes along an operation coordinate system of the robot arm 10 on a monitor 40 (described later).

[0012] (Configuration of the processing system of the present disclosure) 1 is a diagram illustrating an example of a configuration of a processing system 1 according to some embodiments of the present disclosure. As shown in FIG. 1, the processing system 1 according to one embodiment of the present disclosure includes a robot arm 10, an imaging device 20, an image processing device 30, a monitor 40, and an input device 50.

[0013] The robot arm 10 is a device that is remotely controlled by an operator. The imaging device 20 includes a camera 201 and a 3D (Dimensional) sensor 202. The camera 201 and the 3D sensor 202 each capture an image of the operating area of ​​the robot arm 10. The image processing device 30 generates an image to be displayed on the monitor 40. For example, the image processing device 30 is a computer. The monitor 40 displays the image generated by the image processing device 30. The input device 50 is a device that inputs remote commands to the robot arm 10. For example, the input device 50 is a mouse.

[0014] The above-described processing performed by the processing system 1 according to the embodiment of the present disclosure is merely an example, and the processing system 1 is not limited to the above-described processing. For example, the processing system 1 may perform the processing described below.

[0015] (Processing performed by the processing system of the present disclosure) 2 is a diagram illustrating an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Here, a description will be given of the drawing processing performed by the processing system 1 illustrated in FIG.

[0016] The camera 201 captures a 2D image by capturing an image of the operating area of ​​the robot arm 10. The 3D sensor 202 captures an image of the operating area of ​​the robot arm 10 to capture depth data. That is, the imaging device 20 captures a 2D image and depth data (step S1). The depth data is associated with each pixel of the 2D image, and the imaging device 20 captures the position in three-dimensional space of the object surface captured by each pixel.

[0017] The image processing device 30 identifies the position of the robot arm 10 and the position of the object based on the acquired 2D image and depth data (step S2). The image processing device 30 draws a line segment based on the identified position of the robot arm 10 (step S3). FIG. 3 is a diagram illustrating an example of a line segment drawn by the image processing device 30 according to some embodiments of the present disclosure. FIG. 4 is a diagram for explaining a line segment drawn by the image processing device 30 according to some embodiments of the present disclosure. In FIG. 3, the line segment is indicated by line segment 1. In this case, the image processing device 30 draws line segment 1 from the near front surface A to the rear surface B, passing through the tip of the robot arm 10, and parallel to the depth direction from the near front surface A to the rear surface B, as shown in FIG. 4. In this drawing, line segment 1 is intersected by the near front surface A and the rear surface B regardless of the depth position (i.e., the position in the Y-axis direction) of the robot arm 10. Therefore, the image processing device 30 can represent the position of the XZ plane between the near front surface and the rear surface by drawing as shown in FIG. 3.

[0018] The image processing device 30 draws a surface in the Z-axis direction based on the line segment (step S4). FIG. 5 is a diagram for explaining a surface drawn by the image processing device 30 according to some embodiments of the present disclosure. In FIG. 5, the surface is indicated by surface 4. At this time, the image processing device 30 draws surface 4 in the Z-axis direction from line segment 1 as shown in FIG. 5. The image processing device 30 expresses boundary 5 on the near side and boundary 6 on the rear side using surface 4. As a result, the operator can easily recognize the positions of the near side and rear side in the depth direction (i.e., positions in the Y-axis direction).

[0019] The image processing device 30 draws a line on the surface of the object in the depth direction based on the depth position of the robot arm 10 (step S5). FIG. 6 is a diagram for explaining the line drawn on the surface of the object by the image processing device 30 according to some embodiments of the present disclosure. In FIG. 6, the line drawn on the surface of the object is indicated by a line 7. At this time, as shown in FIG. 6, the image processing device 30 draws the overlapping portion of the XZ plane and the surface of the object as the line 7 at the depth position of the object identified in the processing of step S2. This line 7 makes it easier for the operator to recognize the relative positions of the object and the tip of the robot arm 10 in space.

[0020] The image processing device 30 draws a line segment that passes through the tip of the robot arm 10 and is parallel to the Z axis on the plane drawn by the processing of step S4 (step S6). FIG. 7 is a diagram for explaining a line segment drawn by the image processing device 30 according to some embodiments of the present disclosure. In FIG. 7, the line segment is indicated by line segment 8. At this time, the image processing device 30 draws line segment 8 on plane 4, as shown in FIG. 7, as line segment 8 that passes through the tip of the robot arm 10 and is parallel to the Z axis. This line segment 8 makes it easier for the operator to recognize the position of the target object in the depth direction (i.e., the position in the Y axis direction) even if there is no other object around the target object.

[0021] If the above-mentioned surface 4, line segment 1, or line segment 8 is hidden behind an object, the image processing device 30 does not render the surface 4, line segment 1, or line segment 8 hidden behind the object on the monitor 40. The image processing device 30 also determines the foreground / background relationship between the object and each of the above-mentioned surface 4, line segment 1, and line segment 8, using depth data obtained from the imaging device 20.

[0022] Furthermore, the image processing device 30 records the relative positional relationship between the robot arm 10 and the camera 201. Then, when drawing, the image processing device 30 corrects the difference in coordinates caused by the difference (i.e., the difference) between the coordinate system of the imaging device 20 and the coordinate system of the robot arm 10. Since the image processing device 30 makes this correction possible, the imaging device 20 can be installed in any position. Furthermore, it is no longer necessary to consider the relationship between the coordinate system of the robot arm 10 and the optical axis.

[0023] (advantage) The processing system 1 according to an embodiment of the present disclosure has been described above. In the processing system 1, the image processing device 30 (an example of a first processing unit) draws a first line segment that passes through the tip position of the robot arm and is parallel to the depth direction when remotely operating the robot arm. The image processing device 30 (an example of a second processing unit) draws a plane extending vertically from the first line segment. The image processing device 30 (an example of a third processing unit) draws a line segment within the plane extending vertically from the tip position of the robot arm. This processing system 1 makes it easier for a user to grasp the depth position of the robot arm. As a result, the user's operation of the robot arm can be brought closer to the desired operation.

[0024] In another embodiment of the present disclosure, the image processing device 30 may perform correction to associate a coordinate system of the operation direction by the input device 50 with a coordinate system displayed on the monitor 40. FIG. 8 is a first diagram for explaining correction performed by the image processing device 30 according to some embodiments of the present disclosure. FIG. 9 is a second diagram for explaining correction performed by the image processing device 30 according to some embodiments of the present disclosure. For example, when the input device 50 is a mouse, the image processing device 30 performs correction to correspond left and right movement of the mouse to the X-axis direction and up and down movement to the Z-axis direction, as shown in FIG. 8. Furthermore, the image processing device 30 performs correction to correspond wheel operation of the mouse to the Y-axis direction, as shown in FIG. 9. This correction by the image processing device 30 improves operability for the operator.

[0025] In another embodiment of the present disclosure, the position of the robot arm 10 may be obtained from the robot's encoder value. In one embodiment of the present disclosure, if the imaging device 20 can acquire "the position in three-dimensional space of the object surface captured for each pixel," this is sufficient for the subsequent rendering process (steps S3 to S6), and there is no need to know the object's position (for example, the position of the object's center or an estimate of a bounding box). Therefore, in another embodiment of the present disclosure, it is not necessarily necessary to perform the process of step S2.

[0026] 10 is a diagram illustrating an example of the configuration of a processing system 1 according to some embodiments of the present disclosure. As illustrated in FIG. 10, the processing system 1 includes a first processing unit 301, a second processing unit 302, and a third processing unit 303.

[0027] When a robot arm is remotely controlled, a first processing unit 301 draws a first line segment that passes through the tip of the robot arm and is parallel to the depth direction. A second processing unit 302 draws a plane extending vertically from the first line segment. A third processing unit 303 draws a line segment within the plane extending vertically from the tip of the robot arm.

[0028] The first processing unit 301 can be realized, for example, by using the functions of the image processing device 30 illustrated in Fig. 1. The second processing unit 302 can be realized, for example, by using the functions of the image processing device 30 illustrated in Fig. 1. The third processing unit 303 can be realized, for example, by using the functions of the image processing device 30 illustrated in Fig. 1.

[0029] Next, processing performed by the processing system 1 according to some embodiments of the present disclosure will be described. Fig. 11 is a diagram showing an example of a processing flow of the processing system 1 according to some embodiments of the present disclosure. Here, the processing of the processing system 1 will be described with reference to Fig. 11.

[0030] When remotely controlling a robot arm, the first processing unit 301 draws a first line segment that passes through the tip of the robot arm and is parallel to the depth direction (step S101). The second processing unit 302 draws a plane extending vertically from the first line segment (step S102). The third processing unit 303 draws a line segment within the plane extending vertically from the tip of the robot arm (step S103).

[0031] The processing system 1 according to some embodiments of the present disclosure has been described above. This processing system 1 can support remote operation.

[0032] The order of the processes in each embodiment of the present disclosure may be changed as long as the processes are performed appropriately.

[0033] Each embodiment of the present disclosure has been described, but the processing system 1, robot arm 10, imaging device 20, image processing device 30, monitor 40, input device 50, and other control devices may have a computer system built therein. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the computer reads and executes this program to perform the above processing. Specific examples of computers are shown below.

[0034] 12 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 12, the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9.

[0035] For example, the above-described processing system 1, robot arm 10, imaging device 20, image processing device 30, monitor 40, input device 50, and other control devices are each implemented in a computer 5. The operations of each of the above-described processing units are stored in the form of a program in a storage 8. A CPU 6 reads the program from the storage 8, loads it into a main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the above-described storage units in accordance with the program.

[0036] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. In addition, when this program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-mentioned processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0037] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already recorded in the computer system, a so-called differential file (differential program).

[0038] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0039] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0040] (Appendix 1) a first processing unit that, when remotely operating a robot arm, draws a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; a second processing unit that draws a surface along a vertical direction from the first line segment; a third processing unit that draws a line segment in the plane in a vertical direction from the tip position of the robot arm; A processing system comprising:

[0041] (Appendix 2) a fourth processing unit that draws a line on a surface of an object that exists at the same depth as the depth of the tip position of the robot arm; 2. The processing system of claim 1, comprising:

[0042] (Appendix 3) a fifth processing unit that corrects a coordinate system in a real space where the robot arm exists so that the coordinate system on the screen to be drawn coincides with the coordinate system; 3. The processing system of claim 1 or 2, comprising:

[0043] (Appendix 4) a sixth processing unit that corrects a coordinate system in an operation for remotely controlling the robot arm so that the coordinate system on the screen to be drawn coincides with the coordinate system; 4. The processing system of any one of claims 1 to 3, comprising:

[0044] (Appendix 5) When remotely operating the robot arm, drawing a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; Drawing a surface along a vertical direction from the first line segment; Drawing a line segment in the plane in a vertical direction from the tip position of the robot arm; A processing method comprising:

[0045] (Appendix 6) Drawing a line on the surface of an object that exists at the same depth as the depth of the tip position of the robot arm; Attachment 5, a processing method comprising:

[0046] (Appendix 7) Correcting the coordinate system in the real space where the robot arm exists so that it coincides with the coordinate system on the screen where it is drawn; 6. The method of claim 5, further comprising:

[0047] (Appendix 8) Correcting the coordinate system used to remotely control the robot arm so that it coincides with the coordinate system on the screen to be drawn; 8. The processing method according to any one of appendices 5 to 7,

[0048] (Appendix 9) On the computer, When remotely operating the robot arm, drawing a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; Drawing a surface along a vertical direction from the first line segment; Drawing a line segment in the plane in a vertical direction from the tip position of the robot arm; A program that executes the following.

[0049] (Appendix 10) Drawing a line on the surface of an object that exists at the same depth as the depth of the tip position of the robot arm; 10. The program according to claim 9, which causes the computer to execute the above steps.

[0050] (Appendix 11) Correcting the coordinate system in the real space where the robot arm exists so that it coincides with the coordinate system on the screen where it is drawn; 11. The program according to claim 9 or 10, which causes the computer to execute the above.

[0051] (Appendix 12) Correcting the coordinate system used to remotely control the robot arm so that it coincides with the coordinate system on the screen to be drawn; 12. The program according to any one of claims 9 to 11, which causes the computer to execute the above. [Explanation of symbols]

[0052] 1. Processing System 5. Computer 6 CPU 7. Main memory 8. Storage 9. Interface 10. Robot arm 20. Imaging device 30. Image processing device 40···Monitor 50 Input device 301 First processing section 302 Second processing section 303 Third processing section

Claims

1. a first processing unit that, when remotely operating a robot arm, draws a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; a second processing unit that draws a surface along a vertical direction from the first line segment; a third processing unit that draws a line segment in the plane in a vertical direction from a tip position of the robot arm; A processing system comprising:

2. a fourth processing unit that draws a line on a surface of an object that exists at the same depth as the depth of the tip position of the robot arm; The processing system of claim 1 , comprising:

3. a fifth processing unit that corrects a coordinate system in a real space where the robot arm exists so that the coordinate system on the screen to be drawn coincides with the coordinate system; The processing system according to claim 1 or 2, comprising:

4. a sixth processing unit that corrects a coordinate system used in remotely controlling the robot arm so that the coordinate system on the screen to be drawn coincides with the coordinate system; The processing system of claim 1 , comprising:

5. When remotely operating the robot arm, drawing a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; drawing a surface along a vertical direction from the first line segment; Drawing a line segment in the plane in a vertical direction from the tip position of the robot arm; A processing method comprising:

6. On the computer, When remotely operating the robot arm, drawing a first line segment that passes through a tip position of the robot arm and is parallel to a depth direction; drawing a surface along a vertical direction from the first line segment; Drawing a line segment in the plane in a vertical direction from the tip position of the robot arm; A program that executes the following.

Images