Processing device, processing system, and processing method

Abstract

A processing device (20) comprising a measurement point cloud display unit (21) that three-dimensionally displays a point cloud including an object on a screen, a region model display unit (22) that three-dimensionally displays a region model (M) on the screen, a positioning unit (23) that positions the region model by changing the position and the posture of the region model and thereby sets the point cloud of the object so as to be at least partially surrounded by the region model, and a setting unit (24) that sets a region which is within the screen and is surrounded by the positioned region model as the model of the object.

Description

Processing device, processing system, and processing method

[0001] The present invention relates to a processing device, a processing system, and a processing method.

[0002] It is common to use a 3D camera to detect a workpiece as an object and determine its position and orientation. To detect a workpiece, it is necessary to teach the processing device that performs the detection process the workpiece model and the reference workpiece position and orientation in advance.

[0003] Regarding model teaching, there is a method of setting a model based on a point cloud of a workpiece obtained from an image captured by a camera (for example, Patent Publication No. 2021-062416).

[0004] Patent Publication No. 2021-062416

[0005] However, creating a model based on the point cloud of the workpiece requires complex calculations, which is cumbersome.

[0006] Therefore, there is a demand for a technique that can easily create a model of a workpiece based on the point cloud of the workpiece.

[0007] According to a first aspect of the present disclosure, there is provided a processing device comprising: a screen; a measurement point cloud display unit that displays a point cloud of an object measured by a three-dimensional camera in 3D on the screen; an area model display unit that displays an area model in 3D on the screen; a positioning unit that changes the position and orientation of the area model displayed by the area model display unit to position the area model so that the point cloud of the object is at least partially surrounded by the area model; and a setting unit that sets the area within the screen surrounded by the area model positioned by the positioning unit as a model of the object.

[0008] Furthermore, there is provided a processing system comprising the processing device of the first aspect and the three-dimensional camera.

[0009] Furthermore, a processing method is provided in which a point cloud of an object measured by a three-dimensional camera is displayed in 3D on a screen, an area model is displayed in 3D on the screen, the position and orientation of the area model is changed to position the area model, thereby causing the point cloud of the object to be at least partially surrounded by the area model, and the area within the screen surrounded by the positioned area model is set as a model of the object.

[0010] The objects, features and advantages of the present invention will become more apparent from the following description of the embodiments taken in conjunction with the accompanying drawings.

[0011] 5A is a schematic diagram of a system including a processing device in a first embodiment of the present invention. FIG. 5B is a block diagram of a system including a processing device in a first embodiment of the present invention. FIG. 5C is a flowchart showing the operation of a processing device in a first embodiment of the present invention. FIG. 5D is a diagram showing a point cloud of a workpiece and an area model in a first embodiment. FIG. 5E is a schematic diagram of an area model. FIG. 5F is a diagram showing an overall enlargement of the area model shown in FIG. 5A. FIG. 5G is a diagram showing the area model shown in FIG. 5A with a changed posture. FIG. 5H is a diagram showing a partial enlargement of the area model shown in FIG. 5A. FIG. 5I is a diagram showing another area model. FIG. 5J is a diagram showing yet another area model. FIG. 5J is a diagram showing a menu screen for selecting an area model. FIG. 5I is a diagram showing another workpiece. FIG. 5J is a diagram showing yet another workpiece. FIG. 5J is a diagram showing an additional area model etc. in a second embodiment. FIG. 5J is a diagram showing an area model etc. in a third embodiment. FIG. 9A is a diagram showing a workpiece etc. shown in FIG. 9A is a flowchart showing the operation of a processing device in a third embodiment of the present invention.

[0012] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Corresponding components throughout the drawings are designated by common reference numerals. FIG. 1 is a schematic diagram of a system including a processing device according to a first embodiment of the present invention. As shown in FIG. 1, the system 1 mainly includes a robot 5, such as an articulated robot, equipped with an end effector 6, a three-dimensional camera 9, and a teaching pendant 30 equipped with an input unit 32. A workpiece W is placed on a table T disposed near the robot 5. The workpiece W has a three-dimensional shape. For example, the workpiece W shown in FIG. 1 has a shape consisting of a central disk portion and a shaft portion penetrating the disk portion. The disk portion and the shaft portion may be integrally formed. The shape of the workpiece W is not limited to this.

[0013] 1 shows a configuration in which an end effector 6 of a robot 5 grips or machines a workpiece W. Note that a configuration having a three-dimensional scanner may be used instead of the three-dimensional camera 9.

[0014] 2 is a block diagram of a system including a processing device according to a first embodiment of the present invention. The control device 10 of the system 1 is a computer, and includes a storage unit 11, such as a ROM and RAM, a CPU (Central Processing Unit) 12, and a bus connecting the storage unit 11 and the CPU 12. The storage unit 11 stores images captured by the three-dimensional camera 9, as well as an operation program for the robot 5, an area model M (described later), and various parameters.

[0015] 1, the input unit 32 is a part of the teaching pendant 30, but the input unit 32 may be an external device such as a keyboard, a mouse, or a touch panel. The screen 31 may be a part of the teaching pendant 30, or may be an independent screen 31 such as a CRT or LCD monitor. A configuration including the three-dimensional camera 9, the storage unit 11, the processing device 20, the screen 31, and the input unit 32 is referred to as a processing system 7.

[0016] The CPU 12 serves as a robot control unit 29 that controls the processing device 20 and the robot 5, which will be described in detail later. The processing device 20 includes a measurement point cloud display unit 21 that 3D displays a point cloud including an object (workpiece) W in a 3D image captured by the 3D camera 9 on a screen 31, and an area model display unit 22 that 3D displays (three-dimensionally displays) an area model M on the screen 31. The processing device 20 further includes a positioning unit 23 that changes the position and orientation of the area model M displayed by the area model display unit 22 to position the area model M so that the point cloud of the object W is at least partially surrounded by the area model M, and a setting unit 24 that sets the area within the screen 31 surrounded by the area model M positioned by the positioning unit 23 as a model of the object W. The processing device 20 further includes a color change unit 25 that changes the color of at least one of the point cloud of the object W and the area model M in response to an input operation from the input unit 32.

[0017] The measurement point cloud display unit 21, area model display unit 22, positioning unit 23, setting unit 24, and color change unit 25 are functional modules realized by, for example, a computer program executed on the processor 12. The computer program for executing the processes of the measurement point cloud display unit 21 to the color change unit 25 possessed by the processor 12 may be provided in a form recorded on a computer-readable recording medium such as a semiconductor memory, a magnetic recording medium, or an optical recording medium.

[0018] In addition, the processing device 20 or the robot control unit 29 may have an imaging control function for the three-dimensional camera 9, a setting function for the model M, MW and detection parameters (score threshold and reference position of the workpiece), etc., a screen display function for teaching the model M, MW to the robot 5, and a workpiece detection function using the taught model M, MW.

[0019] 3 is a flowchart showing the operation of the processing device in the first embodiment of the present invention. The contents shown in FIG. 3 are assumed to be stored in advance in the storage unit 11 of the processing device's operation program.

[0020] First, in step S1, the three-dimensional camera 9 captures an image of the workpiece W placed on the table T. The three-dimensional image of the workpiece W is stored in the memory unit 11. Next, in step S2, the measurement point cloud display unit 21 displays a three-dimensional image including the point cloud of the workpiece W on the screen 31. Fig. 4A is a diagram showing the point cloud of the workpiece and the like in the first embodiment. In Fig. 4A, the point cloud of the workpiece W is placed on the point cloud of the table T.

[0021] Next, in step S3, the area model display unit 22 displays the area model M in 3D on the screen 31 in response to an operation by the operator. Fig. 4B is a diagram showing a point cloud of a workpiece and an area model in the first embodiment. The area model M shown in Fig. 4B is a rectangular parallelepiped frame displayed in three dimensions. Note that, as will be described later, the area model M may have other shapes, such as a sphere or a capsule shape.

[0022] The initial values ​​for the position and orientation of the area model M and the initial values ​​for the size may be any values ​​determined according to the installation location and size of the workpiece W. Alternatively, the area model display unit 22 may calculate the initial values ​​for the position and orientation of the area model M and the initial values ​​for the size of the area model M so that all point clouds measured by the three-dimensional camera are included in the area model M, and then display the area model M on the screen 31. In the following, a case where the area model M is a rectangular parallelepiped will be described.

[0023] Figure 5A is a schematic diagram of a region model. In Figure 5A, each vertex A1 to A8 of the region model M and the center C0 of the region model M are highlighted with white circles. These white circles are called operation points. Note that for the purpose of simplicity, the operation points may be omitted from the illustration.

[0024] Next, in step S4, the operator operates the input unit 32 to position the area model M so that the point cloud of the workpiece W is at least partially surrounded by the area model M. This operation is performed through the positioning unit 23.

[0025] 5B to 5D are diagrams showing the area model shown in FIG. 5A. For example, in FIG. 5B, the operator operates the input unit 32 to perform a drag operation in the diagonal direction of one surface of the area model M while specifying the operation point A2. This changes the position of the vertex A2, thereby enlarging the area model M as a whole. It is also possible to shrink the area model M by a similar operation.

[0026] 5C, the operator operates the input unit 32 to translate or rotate the area model M by dragging while keeping the operation point C0 specified. This allows the area model to translate or the area model M to rotate around the center C0 in accordance with the dragging operation.

[0027] With regard to the operation point C0, when the operation point C0 is specified, a menu regarding whether to translate or rotate the area model M may be displayed on the screen 31. Similarly, when the vertex A2 is specified, a similar menu screen regarding whether to scale or rotate the area model M may be displayed on the screen 31. Alternatively, when one of the arrows indicating the X, Y, and Z directions shown in Fig. 5C is specified, the area model M may be rotated around the specified direction. In this way, the operation regarding the area model M can be changed depending on the specified position of the area model M.

[0028] 5D, an operation point C1 is placed at the center of one surface of the area model M. The operator operates the input unit 32 to drag the operation point C1 in the depth direction of the area model M (the normal direction of the surface on which the operation point C1 is placed) while keeping the operation point C1 specified. This allows the area model M to be partially enlarged / reduced.

[0029] If the area model M has another shape, for example, a sphere, as shown in FIG. 6A, the area model M can be enlarged or reduced in size by manipulating the operation point C0′ located at the center of the sphere to change its position, or by manipulating the operation point C1′ located at the edge of the sphere, as described above.

[0030] Alternatively, as shown in Figure 6B, if the area model M is capsule-shaped (a combination of a cylinder and two hemispheres connected to the end faces of the cylinder), the position and orientation can be changed by operating the operation point C0'' located at the center of the cylinder, or the area model M can be enlarged or reduced in the same manner as described above by operating the operation point C2' located at the center of the end face of the cylinder or the operation point C3' located on the edge of the hemisphere.

[0031] Furthermore, Fig. 6C is a diagram showing a menu screen for selecting an area model. The menu screen 39 shown in Fig. 6C may be displayed on the screen 31 between steps S2 and S3 in Fig. 3. The operator then uses the input unit 32 to select an area model M of a desired shape. This allows the operator to select an area model M that is optimal for the shape of the workpiece W, and it will be understood that the operations described below can be performed more accurately.

[0032] Also, numerical values ​​relating to the position and orientation of the area model M and numerical values ​​relating to the size may be separately displayed on the screen 31. Furthermore, the operator may directly input numerical values ​​relating to the position and orientation of the area model M and numerical values ​​relating to the size of the area model M using the input unit 32, thereby changing the position and orientation and size of the area model M.

[0033] 5B to 5D, the operator positions the area model M so that the point cloud of the workpiece W is at least partially surrounded by the area model M. As a result, as shown in Fig. 4B, the point cloud of the workpiece W is at least partially surrounded by the area model M. It is preferable that the point cloud of the workpiece W is completely surrounded by the area model M. In addition, it is preferable that the area model M is of the minimum size that can at least partially surround the workpiece W.

[0034] Next, in step S5, the area within the screen 31 surrounded by the positioned area model M is set as a model MW of the workpiece W. Therefore, in the present invention, a simple model MW can be easily created based on the point cloud of the workpiece W.

[0035] Next, the position and orientation of the workpiece W is taught to the robot 5 using a known method based on the model MW. Then, the robot 5 is operated in accordance with the operation program of the robot 5 based on the taught position and orientation of the workpiece W. Alternatively, the model MW may be used to detect the workpiece W in another image captured by the three-dimensional camera 9.

[0036] 7A and 7B are diagrams showing other workpieces. Workpiece W1 shown in Fig. 7A has a shape in which a large rectangular parallelepiped is placed between a cylinder and a small rectangular parallelepiped. In contrast, workpiece W2 shown in Fig. 7B has a shape in which a large rectangular parallelepiped is placed between a truncated cone and a small rectangular parallelepiped. In other words, workpieces W1 and W2 have a common part consisting of a small rectangular parallelepiped and a large rectangular parallelepiped.

[0037] In such a case, as shown in Figures 7A and 7B, only the common portion of the workpieces W1 and W2 is surrounded by the area model M0. In other words, the workpieces W1 and W2 are only partially surrounded by the area model M0. In this case, a model MW0 of the workpiece W can be set using a common area model M0 for the different workpieces W1 and W2. In other words, in another image in which multiple workpieces W1 and W2 are captured, a common model MW0 can be used to set the model MW0 of the workpieces W1 and W2. Alternatively, the common model MW0 can be used to detect different workpieces W1 and W2 in yet another image captured by the three-dimensional camera 9.

[0038] In addition, the color change unit 25 not only changes the color of at least one of the area model M and the point cloud of the object W, but may also change the type of line that makes up the area model M, such as a solid line, a dashed line, or a dotted line, in accordance with the operation of the input unit 32.

[0039] In the second embodiment of the present invention, the same processes S1 to S5 as those in the first embodiment are performed. In the second embodiment, in step S6 of FIG. 3, an additional area model M' is displayed in 3D on the screen. In step S7, the position and orientation of the additional area model M' are changed to position the additional area model M'. FIG. 8 is a diagram showing the additional area model and the like in the second embodiment. In FIG. 8, the additional area model M' is positioned so as to at least partially surround the point cloud of the table T. For the sake of simplicity, the area model M for the workpiece W is not shown in FIG.

[0040] 8 is a point cloud that is not necessary to at least partially surround the point cloud of the workpiece W with the area model M. In the second embodiment, such a point cloud of the table T is surrounded by an additional area model M′, and then the area surrounded by the additional area model M′ is masked.

[0041] In this way, by masking point clouds unrelated to the workpiece W, such as the table T and background, with an additional area model M', it becomes easier to set up a model of the workpiece W. Note that steps S6 and S7 may be performed first to surround unnecessary point clouds with the area model M', and then steps S1 to S5 may be performed to create a model WM of the workpiece W. It will be apparent that this makes it easier and more accurate to create the model WM of the workpiece W. Note that multiple area models M' may be used to perform masking for multiple locations.

[0042] Furthermore, Fig. 9A is a diagram showing an area model etc. in the third embodiment, and Fig. 9B is a diagram showing the workpiece etc. shown in Fig. 9A. Furthermore, Fig. 10 is a flowchart showing the operation of the processing device in the third embodiment of the present invention. Steps S1 to S3 in Fig. 10 are the same as those described above, so repeated explanation will be omitted.

[0043] 9A is a roughly X-shaped workpiece W3 consisting of two elongated portions. When setting such a workpiece W3 as a model, first, in step S4′, the position and orientation of the area model M1 is changed to position the area model M1 so that a part of the workpiece W3, for example, one of the elongated portions, is at least partially surrounded.

[0044] Next, in step S5', the additional area model M2 is displayed on the screen 31. Then, in step S6', the position and orientation of the area model M2 is changed to position the area model M2 so that another portion of the workpiece W3, for example, the other elongated portion, is at least partially surrounded.

[0045] Finally, in step S7', the area in the screen 31 surrounded by the positioned area model M1 and the additional area model M2 is set as a model MW3 of the workpiece W3. Naturally, two or more area models may be used to surround the workpiece W. This makes it possible to easily set the model MW3 even for a workpiece W3 with a complex shape.

[0046] 9B, the image captured by the three-dimensional camera 9 may include foreign objects other than the workpiece. Thus, even if foreign objects are present around the workpiece W3, the process described with reference to FIG. 10 can be used to surround the workpiece W3 with the area models M1 and M2 so as not to include the foreign objects, thereby enabling the model MW3 to be set accurately. Therefore, the third embodiment is advantageous for setting a model of a workpiece having a complex shape.

[0047] Although the embodiments of the present disclosure have been described in detail, the present disclosure is not limited to the individual embodiments described above. Various additions, substitutions, modifications, partial deletions, etc. are possible to these embodiments without departing from the gist of the invention or the concept and spirit of the present invention derived from the content of the claims and their equivalents. For example, in the above-described embodiments, the order of each operation and the order of each process are shown as examples and are not limited to these. The same applies when numerical values ​​or mathematical formulas are used in the description of the above-described embodiments. Furthermore, appropriate combinations of several of the above-described embodiments are within the scope of the present disclosure.

[0048] REFERENCE SIGNS LIST 1 System 5 Robot 7 Processing system 10 Control device 11 Memory unit 20 Processing device 21 Measurement point cloud display unit 22 Area model display unit 23 Positioning unit 24 Setting unit 25 Color change unit 30 Teaching operation panel 31 Screen 32 Input unit 39 Menu screen M, M', M0, M1, M2 Area model MW, MW0, MW3 Work model W, W1, W2, W3 Work

Claims

1. Screen and a measurement point cloud display unit that displays a point cloud including an object measured by a three-dimensional camera on the screen in 3D; an area model display unit for displaying an area model in 3D on the screen; a positioning unit that changes a position and orientation of the area model displayed by the area model display unit to position the area model so that the point cloud of the object is at least partially surrounded by the area model; a setting unit that sets an area within the screen surrounded by the area model positioned by the positioning unit as a model of the object.

2. The processing device according to claim 1 , wherein the area model is a rectangular parallelepiped, a sphere, or a capsule.

3. The processing device according to claim 2 , wherein the area model has at least one of an operation point for resizing the area model and an operation point for changing a position and orientation of the area model.

4. The processing device according to claim 1 , further comprising an input unit for inputting a position, orientation and size of the area model.

5. The processing device according to claim 1 , further comprising a color change unit that changes a color of at least one of the point cloud of the object and the area model.

6. The area model display unit displays the additional area model in 3D on the screen, The processing device according to claim 1 , wherein the positioning unit positions the additional area model by changing a position and orientation of the additional area model displayed by the area model display unit, so that a point cloud unnecessary for at least partially surrounding the point cloud of the object by the area model is masked by the additional area model.

7. The area model display unit displays the additional area model in 3D on the screen, The processing device according to claim 1 , wherein the positioning unit positions the additional area model by changing a position and orientation of the additional area model displayed by the area model display unit, thereby at least partially surrounding the point cloud of the object by the area model and the additional area model.

8. A processing device according to claim 1 or 2, and said three-dimensional camera.

9. The point cloud including the object measured by the 3D camera is displayed in 3D on the screen. Displaying the area model in 3D on the screen; repositioning the area model to position the area model such that the object point cloud is at least partially enclosed by the area model; A processing method in which an area within the screen surrounded by the positioned area model is set as a model of the object.

10. Furthermore, displaying an additional area model in 3D on the screen; The processing method of claim 9, further comprising changing a position and orientation of the additional area model to position the additional area model, such that points not required for at least partially surrounding the point cloud of the object by the area model are masked by the additional area model.

11. Furthermore, displaying an additional area model in 3D on the screen; The method of claim 9 , further comprising: positioning the additional area model by changing a position and orientation of the additional area model, such that the point cloud of the object is at least partially enclosed by the area model and the additional area model.