Robot simulation device

Abstract

A robot simulation device includes: a model configuration unit (132) for configuring a robot model, a vision sensor model, and a workpiece model in a virtual space; a workpiece model position calculation unit (134) for calculating the position and posture of the workpiece model based on the robot model or the vision sensor model in the virtual space by aligning the three-dimensional position information of the workpiece model obtained by the vision sensor in the work space with the shape features of the workpiece model; and a simulation execution unit (135) for performing the following simulation actions: measuring the workpiece model using the vision sensor model and performing operations on the workpiece model using the robot model. The model configuration unit configures the workpiece model in the virtual space based on the position and posture calculated by the workpiece model position calculation unit based on the robot model or the vision sensor model.

Description

[Technical Field]

[0001] This invention relates to a robot simulation device. [Previous Technology]

[0002] The following technology is known: In a robot system having a robot, a vision sensor and a workpiece in the work space, in a virtual space that represents the work space in a three-dimensional manner, a robot model of the robot, a vision sensor model of the vision sensor and a workpiece model are configured, and the following simulation is performed: the workpiece model is measured by the vision sensor model, and the robot model performs operations on the workpiece model (e.g., Patent Document 1).

[0003] Patent Document 2 describes "an information processing apparatus comprising: a first selection unit, which selects one of a plurality of coordinate systems contained in a virtual space according to a first instruction input, wherein the virtual space is configured with a first model based on CAD data containing position information in the virtual space; a first acquisition unit, which acquires first information indicating a second model that does not contain position information in the virtual space; a second acquisition unit, which acquires second information indicating the position in the coordinate system selected by the first selection unit; and a setting unit, which sets the position of the second model in the virtual space to the aforementioned position based on the first and second information" (abstract). Previous Art Documents Patent Documents

[0004] Patent Document 1: Japanese Patent Application Publication No. 2015-171745; Patent Document 2: Japanese Patent Application Publication No. 2020-97061 [Summary of the Invention]

[0005] The problem the invention aims to solve

[0006] The simulation device described in Patent Document 1 uses, for example, random numbers to generate the bulk state of a workpiece model in a virtual space. Simulation technology for efficiently creating motion programs for robots capable of achieving higher precision workpiece removal actions remains highly anticipated. Means for solving this problem...

[0007] One aspect of this disclosure is a robot simulation device for simulating the operation performed by the robot on the workpiece in a robot system comprising a robot, a vision sensor, and a workpiece disposed in a workspace. The robot simulation device includes: a model placement unit that places a robot model of the robot, a vision sensor model of the vision sensor, and a workpiece model in a virtual space that represents the workspace in three dimensions; a workpiece model position calculation unit that calculates the position and posture of the workpiece model based on the robot model or the vision sensor model in the virtual space by aligning the three-dimensional position information of the workpiece obtained by the vision sensor in the workspace with the shape features of the workpiece model; and a simulation execution unit that performs the following simulation actions: measuring the workpiece model using the vision sensor model and performing operations on the workpiece model using the robot model. In the aforementioned virtual space, the model configuration unit configures the aforementioned workpiece model according to the aforementioned position and posture calculated by the aforementioned workpiece model position calculation unit, based on the aforementioned robot model or the aforementioned vision sensor model. Effects of the Invention

[0008] Since the robot model performs the simulated action of reproducing the bulk state of the workpiece in the work space in the virtual space, it is possible to efficiently create motion programs that can perform high-precision extraction actions.

[0009] These objects, features, and advantages of the invention, as well as other objects, features, and advantages, will become more apparent from the detailed description of the typical embodiments of the invention shown in the accompanying drawings.

Implementation Method

[0011] Form used to implement the invention

[0012] Next, embodiments of the present disclosure will be described with reference to the drawings. In the referenced drawings, the same reference numerals are used for the same constituent parts or functional parts. For ease of understanding, the scale of these drawings has been appropriately changed. Furthermore, the forms shown in the drawings are examples for implementing the present invention, and the present invention is not limited to the forms shown in the drawings.

[0013] Figure 1 is a diagram showing the configuration of a robot simulation device 30 connected to a robot system 100 according to an embodiment. The robot system 100 includes: a robot 10, a robot control device 20 for controlling the movement of the robot 10, a vision sensor 70, and a workpiece W placed in a bulk state in a container 81. The robot 10 has a hand 11 mounted on its wrist flange. The various objects constituting the robot system 100 are arranged in the workspace. The robot simulation device 30 is a device for executing a simulation of the motion program for creating the robot 10. The robot simulation device 30 is connected to the robot control device 20 in a wired or wireless manner. Furthermore, the robot simulation device 30 can also be connected to the robot control device 20 remotely.

[0014] The robot simulation device 30 of this embodiment arranges models of various objects, including the robot 10, the vision sensor 70, and the workpiece W loosely packed in the container 81, in a virtual space, and simulates the movement of these models to simulate the action of the vision sensor 70 detecting the workpiece W and the robot 10 (hand 11) removing the workpiece W. In this case, the robot simulation device 30 obtains the actual three-dimensional position information of the workpiece W already loosely packed in the container 81, and reproduces the actual loose state of the workpiece W in the virtual space to perform the simulation, thereby efficiently creating a motion program that can perform workpiece removal actions with higher precision.

[0015] The vision sensor 70 may be a two-dimensional camera that acquires two-dimensional images, or a three-dimensional position detector that acquires the three-dimensional position of an object. In this embodiment, the vision sensor 70 is configured as a range sensor capable of acquiring the three-dimensional position of an object. The vision sensor 70 includes a projector 73 and two cameras 71 and 72 disposed facing each other and sandwiching the projector 73. The projector 73 is configured to project desired patterned light, such as point light or slit light, onto the surface of the object. The projector has a light source such as a laser diode or a light-emitting diode. The cameras 71 and 72 are digital cameras equipped with imaging elements such as CCD or CMOS sensors.

[0016] Furthermore, Figure 1 also illustrates a robot coordinate system C1 set on the robot 10 and a sensor coordinate system C2 set on the vision sensor 70. As an example, the robot coordinate system C1 is set at the base of the robot 10, and the sensor coordinate system C2 is set at the position of the lens of the vision sensor 70. The position and posture of these coordinate systems can be controlled in the robot control device 20. In Figure 1, although a configuration in which the vision sensor 70 is mounted on the front end of the arm of the robot 10 is shown as an example, there is also a configuration in which the vision sensor 70 is fixed at a known position in the workspace.

[0017] Figure 2 is a diagram showing an example of the hardware configuration of the robot control device 20 and the robot simulation device 30. The robot control device 20 may also have the following configuration as a general computer: a memory 22 (ROM, RAM, non-volatile memory, etc.) connected to the processor 21 via a bus, an input / output interface 23, an operation unit 24 including various operation switches, etc. The robot simulation device 30 may also have the following configuration as a general computer: a memory 32 (ROM, RAM, non-volatile memory, etc.) connected to the processor 31 via a bus, a display unit 33, an operation unit 34 composed of an input device such as a keyboard (or soft keyboard), an input / output interface 35, etc. Various information processing devices such as personal computers, laptop PCs, and tablet terminals can be used as the robot simulation device 30.

[0018] Figure 3 is a functional block diagram showing the functional configuration of the robot simulation device 30. The robot simulation device 30 includes a virtual space creation unit 131, a model configuration unit 132, a vision sensor model position setting unit 133, a workpiece model position calculation unit 134, and a simulation execution unit 135.

[0019] The Virtual Space Production Department 131 will create a virtual space that represents the work space in a three-dimensional way.

[0020] The model configuration unit 132 configures the models of each object constituting the robot system 100 in a virtual space. The state of each object model configured in the virtual space by the model configuration unit 132 can also be displayed on the display unit 33.

[0021] The vision sensor model position setting unit 133 obtains information indicating the position of the vision sensor 70 within the workspace from the robot control device 20. For example, the vision sensor model position setting unit 133 can obtain information (calibration data) indicating the relative position of the robot coordinate system C1 and the sensor coordinate system C2, which has been stored in the robot control device 20, as a file. Specifically, the information indicating the relative position is the position and orientation of the vision sensor 70 (sensor coordinate system C2) relative to the robot 10 (robot coordinate system C1) within the workspace. The information indicating the relative position of the robot coordinate system C1 and the sensor coordinate system C2 has been obtained by performing calibration of the vision sensor 70 in advance in the robot system 100 and is stored in the robot control device 20.

[0022] Here, the correction can be achieved, for example, by using the vision sensor 70 to measure the visual marker that has been attached to a predetermined reference position on the robot, thereby obtaining the position and orientation of the vision sensor 70 relative to the visual marker. By obtaining the position and orientation of the vision sensor 70 relative to the visual marker that has been positioned at a known location, the position and orientation of the vision sensor 70 relative to the robot 10 can be obtained.

[0023] The model configuration unit 132 configures the vision sensor model in the virtual space such that the relative position between the robot model coordinate system set in the robot model and the sensor model coordinate system set in the vision sensor model in the virtual space is the same as the relative position between the robot coordinate system and the sensor coordinate system in the work space.

[0024] The workpiece model position calculation unit 134 calculates the position and orientation of the workpiece model based on the robot 10 or the vision sensor 70, obtained by the vision sensor 70 within the workspace, by aligning it with the shape features of the workpiece model. The workpiece model is then positioned in the virtual space based on the robot model or the vision sensor model. The model placement unit 132 uses the calculated position and orientation to place the workpiece model in the virtual space.

[0025] The simulation execution unit 135 performs the following simulation: it arranges a workpiece model in a bulk state according to the calculated position and posture, measures it using a vision sensor model, and then removes it using a robot model. Furthermore, in this specification, when referring to simulation or simulated action, it includes not only the case of performing numerical simulation of the actions of a robot or the like, but also the case of simulating the actions of various object models such as the robot model on a display screen.

[0026] Figure 4 is a flowchart showing the simulated actions performed under the control of the processor 31 of the robot simulation device 30.

[0027] First, the virtual space creation unit 131 creates a virtual space that represents the work space in a three-dimensional manner (step S1). Then, the model configuration unit 132 configures the robot model 10M in the virtual space (step S2). Figure 5 shows the state where the robot model 10M has been configured in the virtual space. Furthermore, the simulation execution unit 135 sets the robot model coordinate system M1 for the robot model 10M at the position in the virtual space corresponding to the robot coordinate system C1 already defined in the work space.

[0028] Next, the vision sensor model position setting unit 133 sets the position and posture of the vision sensor model 70M based on the position and posture of the vision sensor 70 based on the robot 10 in the workspace (step S3). The position and posture of the vision sensor based on the robot 10 in the workspace are calculated by performing correction on the vision sensor 70 in the robot system 100, and are memorized, for example, as the relative position of the robot coordinate system C1 and the sensor coordinate system C2 in the robot control device 20. In step S3, the vision sensor model position setting unit 133 obtains information on the relative position of the robot coordinate system C1 and the sensor coordinate system C2 from the robot control device 20.

[0029] Next, in step S4, the model configuration unit 132 configures the visual sensor model 70M in the virtual space such that the relative positions of the robot model coordinate system M1 and the sensor model coordinate system M2 are the same as the relative positions of the robot coordinate system C1 and the sensor coordinate system C2 in the work space.

[0030] Figures 6 and 7 show the state in which the model configuration unit 132 configures the vision sensor model 70M in the virtual space according to information indicating the relative position of the vision sensor 70 relative to the robot 10. Furthermore, Figure 6 shows an example in which the vision sensor 70 is used as a fixed camera fixed at a predetermined position in the work space, and Figure 7 shows an example in which the vision sensor 70 is mounted on the forearm of the robot 10. As shown in Figures 6 and 7, the vision sensor model 70M includes a projector model 73M and two camera models 71M and 72M arranged facing each other while sandwiching the projector model 73M. As shown in Figures 6 and 7, in the virtual space, a sensor model coordinate system M2 is set at a position corresponding to the sensor coordinate system C2.

[0031] Next, in step S5, the workpiece model position calculation unit 134 makes the three-dimensional information of the workpiece W, which is obtained by the vision sensor 70 in the work space and is based on the robot 10 or the vision sensor 70, coincide with the shape features of the workpiece model WM, thereby calculating the position and posture of the workpiece model WM based on the robot model 10M or the vision sensor model 70M in the virtual space.

[0032] The three-dimensional position information of the workpiece W is measured by the vision sensor 70, and is stored in the robot control device 20 as a set of three-dimensional coordinates based on, for example, the robot coordinate system C1 or the sensor coordinate system C2. The workpiece model position calculation unit 134 obtains the three-dimensional position information of the workpiece W from the robot control device 20, and calculates the position and orientation of the workpiece model WM by superimposing the features of the workpiece model WM's shape.

[0033] Here, a method for obtaining three-dimensional position information of a workpiece W in a bulk state, as measured by the vision sensor 70, will be described with reference to FIGS. 8 to 10. In this embodiment, the vision sensor 70 is a distance sensor capable of acquiring the distance to an object. The distance sensor acquires the three-dimensional information of the workpiece in the form of, for example, a distance image or a three-dimensional map. The distance image is an image that represents the distance from the distance sensor to the workpiece within the measurement distance by using the brightness or color of each pixel. The three-dimensional map is a map that represents the three-dimensional position of the workpiece within the measurement area by using the set of three-dimensional coordinate values ​​of points on the surface of the workpiece.

[0034] The two cameras 71 and 72 of the vision sensor 70 are oriented in different directions such that their fields of view at least partially overlap. The projection range of the projector 73 is configured to at least partially overlap with the fields of view of each camera 71 and 72. Figure 8 shows the measurement of the condition of the workpiece W by the vision sensor 70 when the vision sensor 70 is a fixed camera fixed at a predetermined position in the workspace. Figure 9 shows the measurement of the condition of the workpiece W by the vision sensor 70 when the vision sensor 70 is mounted on the forearm of the robot 10 arm.

[0035] The intersection lines of the first plane group and the second plane group are subjected to multiple operations to calculate the three-dimensional coordinates of the intersection points of the intersection lines and the workpiece surface, which are used as the three-dimensional position information of the workpiece W. The first plane group is a plane group that is obtained by dividing the field of view of the area where the workpiece W is arranged by the two cameras 71 and 72 into equal intervals. The second plane group is equivalent to the interface between the brightness and darkness of the pattern light 160 when the projector 73 projects a strip pattern light 160 into the area where the workpiece W is arranged into the area that is the object of measurement.

[0036] In Figure 10, the field of view (the area of ​​the measurement object) captured by the two cameras 71 and 72 is represented as the field of view FV, and an imaginary line dividing the field of view at equal intervals is represented by a single-point chain line. In Figure 10, a strip-shaped patterned light 160 projected onto the area where the workpiece W is disposed is illustrated, a plane of the first plane group (hereinafter referred to as the first plane 151), and a plane of the second plane group (hereinafter referred to as the second plane 152). Furthermore, in Figure 10, the strip-shaped patterned light 160 extends from the inside towards the near front side and is represented as a light and dark pattern (the representation formed by the presence or absence of shadows). Also, in Figure 10, the intersection line L1 of the first plane 151 and the second plane 152 is illustrated, as well as the intersection point P of the intersection line L1 and the surface of the workpiece W.

[0037] Thus, the first plane group and the second plane group are calculated, and the intersection line of the first plane group and the second plane group is calculated. Then, the three-dimensional information of the multiple intersection points P of the calculated multiple intersection lines and the multiple intersection points P of the surface of the bulk workpiece W is calculated.

[0038] The robot control device 20 obtains three-dimensional coordinates for all workpieces W by performing multiple workpiece removal steps.

[0039] The three-dimensional coordinates of all workpieces W obtained in the robot system 100 through the above procedure have been saved in the robot control device 20.

[0040] The workpiece model position calculation unit 134 obtains the three-dimensional coordinates (based on the robot coordinate system C1 or the sensor coordinate system C2) of the plurality of intersection points P on the workpiece surface obtained as described above from the robot control device 20, and uses them as the three-dimensional information of the workpiece W. Furthermore, the workpiece model position calculation unit 134 compares the three-dimensional information of the workpiece W with the shape features of the workpiece model (face data, edge data, vertex data, etc.) to search for possible positions and orientations of the workpiece model, and calculates the position and orientation of the workpiece model with the highest degree of consistency between the set of three-dimensional coordinates and the shape information of the workpiece model. In this way, the workpiece model position calculation unit 134 obtains the position and orientation of the workpiece model WM in virtual space corresponding to the position and orientation of the workpiece W in the workspace.

[0041] Figure 11 shows the state in which the workpiece model WM is configured to coincide with the three-dimensional position information (multiple intersection points P) of the workpiece W through such a procedure. Furthermore, Figure 11 illustrates the range Q in which the three-dimensional position of the workpiece W is obtained. Also, Figure 11 illustrates the workpiece model coordinate system M3 set on each workpiece model WM. The workpiece model coordinate system M3 can also be set at the center of gravity of each workpiece model WM, for example, when each workpiece model WM is cuboid in shape.

[0042] Next, in step S6, the model configuration unit 132 configures the workpiece model WM in the virtual space with the position and posture of the workpiece model W based on the robot model 10M or the vision sensor model 70M. Figure 12 shows the state in which the workpiece model WM is configured in the virtual space based on the position and posture of the workpiece model WM calculated in step S5 when the vision sensor model 70M is a fixed sensor with its position fixed. Figure 13 shows the state in which the workpiece model WM is configured in the virtual space based on the position and posture of the workpiece model WM calculated in step S5 when the vision sensor model M is mounted on the robot model 10M. As shown in Figures 12 and 13, the position and posture of the workpiece model WM can also be obtained as the position and posture of the workpiece model coordinate system M3 relative to the robot model coordinate system M1 or the vision sensor model coordinate system M2. In this way, the actual configuration of the workpiece W, which is already bulked in the workspace, can be reproduced in the virtual space.

[0043] Next, in step S7, the simulation execution unit 135 will perform the following simulation operation when the workpiece model WM has been arranged in the virtual space as shown in FIG12 or FIG13: measuring the workpiece model WM by means of the vision sensor model 70M, and taking out the workpiece model WM one by one by means of the hand model 11M that has been mounted on the robot model 10M.

[0044] The simulation execution unit 135 measures the position and pose of the workpiece model WM in a simulated manner in virtual space, similar to the measurement action using the vision sensor 70, through the following procedure: (a1) Calculate the first plane group based on the position and measurement area of ​​the two camera models 71M and 72M configured in the vision sensor model 70M in the virtual space. (a2) Next, calculate the second plane group based on the position and measurement area of ​​the projector model 73M. (a3) ​​Calculate the multiple intersection lines of the first plane group and the second plane group. (a4) Calculate the three-dimensional coordinates of the intersection points of the intersection lines and the workpiece model WM. (a5) Calculate the position and pose of the workpiece model WM based on the three-dimensional coordinates of the workpiece model WM. (a6) Simulate the following action: based on the calculated position and pose of the workpiece model WM, move the robot model 10M to a position where it can hold the object workpiece model, and use the hand model 11M to remove the object workpiece model.

[0045] Figure 14 shows the state in which the simulated action of retrieving the workpiece model WM by the robot model 10M is being performed by the simulation execution unit 135. Such actions can also be displayed on the display unit 33 of the robot simulation device 30.

[0046] Thus, according to this embodiment, since the simulated actions of the robot model can be performed in a state where the bulk state of the workpiece in the work space has been reproduced in the virtual space, it is possible to efficiently produce an action program that can perform a high-precision extraction action.

[0047] Although the present invention has been described above using typical embodiments, it should be understood that anyone skilled in the art can make changes, omissions, or additions to the above embodiments without departing from the scope of the present invention.

[0048] The functional blocks of the robot simulation device 30 shown in Figure 3 can also be implemented by the processor 31 of the robot simulation device 30 executing software that has been stored in the memory device; or, it can also be implemented by using hardware such as ASIC (Application Specific Integrated Circuit) as the main body.

[0049] The program that executes the simulation action of FIG4 in the above embodiment can be recorded on various computer-readable recording media (such as semiconductor memory such as ROM, EEPROM, flash memory, magnetic recording media, CD-ROM, DVD-ROM, etc.). [Simplified Explanation of the Diagram]

[0010] Figure 1 is a diagram showing the configuration of a robot simulation device connected to a robot system according to an embodiment. Figure 2 is a diagram showing an example of the hardware configuration of the robot control device and the robot simulation device. Figure 3 is a functional block diagram showing the functional configuration of the robot simulation device. Figure 4 is a flowchart showing the simulated actions performed by the robot simulation device. Figure 5 is a diagram showing the state in which the robot model has been configured in the virtual space. Figure 6 is a diagram showing the state in which the robot model and the vision sensor model are configured in the virtual space when the vision sensor model is a fixed sensor fixed in the virtual space. Figure 7 is a diagram showing the state in which the robot model and the vision sensor model are configured in the virtual space when the vision sensor model is mounted on the robot model. Figure 8 is a diagram showing the state of the workpiece measured by the vision sensor when the vision sensor is a fixed sensor fixed in the workspace. Figure 9 is a diagram showing the state of the workpiece measured by the vision sensor when the vision sensor is mounted on the robot. Figure 10 is a diagram showing the state in which the workpiece is measured by projecting patterned light onto the workpiece by the vision sensor. Figure 11 shows the situation where multiple intersection points are measured on the surface of the workpiece. Figure 12 shows the state where the workpiece model is positioned in the virtual space based on the calculated position and orientation of the workpiece model when the vision sensor model is a fixed sensor fixed in the virtual space. Figure 13 shows the state where the workpiece model WM is positioned in the virtual space based on the calculated position and orientation of the workpiece model when the vision sensor model is mounted on the robot model. Figure 14 shows the state where the simulated action of retrieving the workpiece model by the robot model is being executed by the simulation actuator.

Claims

1. A robot simulation apparatus for simulating operations performed by a robot on a workpiece in a robot system comprising a robot, a vision sensor, and a workpiece disposed in a workspace, the robot simulation apparatus comprising: a model placement unit for placing a robot model of the robot, a vision sensor model of the vision sensor, and a workpiece model in a virtual space representing the workspace in three dimensions; a workpiece model position calculation unit for comparing three-dimensional position information of the workpiece obtained by the vision sensor within the workspace, based on the robot or the vision sensor, with the shape features of the workpiece model to search for possible positions and poses of the workpiece model, and calculating the degree of consistency between the three-dimensional position information of the workpiece and the shape information of the workpiece model as the maximum position and pose of the workpiece model, thereby calculating the position and pose of the workpiece model based on the robot model or the vision sensor model in the virtual space; and a simulation execution unit for performing the following simulation actions: measuring the workpiece model using the vision sensor model, and performing operations on the workpiece model using the robot model. In the aforementioned virtual space, the aforementioned model configuration unit configures the aforementioned workpiece model based on the aforementioned position and posture calculated by the aforementioned workpiece model position calculation unit, using the aforementioned robot model or the aforementioned vision sensor model as a reference.

2. The robot simulation apparatus of claim 1, wherein the aforementioned three-dimensional position information of the aforementioned workpiece obtained by the aforementioned vision sensor in the aforementioned work space includes the three-dimensional position information of all the aforementioned workpieces in bulk in the aforementioned work space measured by the aforementioned vision sensor.

3. The robot simulation apparatus of claim 2, wherein the three-dimensional position information of the aforementioned workpiece is a set of three-dimensional points of the aforementioned workpiece measured by the aforementioned vision sensor.

4. The robot simulation device according to any one of claims 1 to 3 further comprises: a vision sensor model position setting unit, which sets the position and posture of the vision sensor model based on the position and posture of the vision sensor based on the robot in the aforementioned work space, and configures the vision sensor model in the aforementioned virtual space to the position and posture of the vision sensor model that has been set.

5. The robot simulation device of claim 4, wherein the position and posture of the aforementioned visual sensor based on the aforementioned robot in the aforementioned work space are included in the calibration data, which is obtained by calibrating the aforementioned visual sensor in the aforementioned work space.

Images