Robot simulation device

Abstract

The present invention is a robot simulation device comprising: a model arrangement unit (132) that arranges a robot model, a visual sensor model, and a workpiece model in a virtual space; a workpiece model position calculation unit (134) that, by superimposing three-dimensional position information for a workpiece, acquired by a visual sensor in a workspace and based on a robot or the visual sensor, and shape characteristics of a workpiece model, calculates a position and orientation of the workpiece model based on a robot model or a visual sensor model in the virtual space; and a simulation execution unit (135) that measures the workpiece model using the visual sensor model and executes a simulation operation in which work is performed on the workpiece model by the robot model. The model arrangement unit arranges, in the virtual space, the workpiece model in the position and the orientation that were calculated by the workpiece model position calculation unit and are based on the robot model or the visual sensor model.

Description

Technical Field

[0001] The present invention relates to a robot simulation device.

Background Art

[0002] In a robot system having a robot, a vision sensor, and a workpiece in a work space, a robot model of the robot, a vision sensor model of the vision sensor, and a workpiece model of the workpiece are arranged in a virtual space that three-dimensionally represents the work space, and the workpiece model is measured by the vision sensor model, and a simulation is performed in which the robot model performs work on the workpiece model is known (for example, Patent Document 1).

[0003] Patent Document 2 describes "a first selection unit that selects one coordinate system based on a first instruction input among a plurality of coordinate systems included in a virtual space in which a first model based on CAD data including position information in the virtual space is arranged, a first acquisition unit that acquires first information indicating a second model that does not include position information in the virtual space, a second acquisition unit that acquires second information indicating a position in the coordinate system selected by the first selection unit, and a setting unit that sets the position of the second model in the virtual space to the position based on the first and second information" (abstract).

Prior Art Documents

Patent Documents

[0006] One aspect of the present disclosure is a robot simulation device for simulating operations performed by a robot on a workpiece in a robot system including a robot, a vision sensor, and a workpiece arranged in a workspace, comprising: a model arrangement unit that arranges a robot model of the robot, a vision sensor model of the vision sensor, and a workpiece model of the workpiece in a virtual space that three-dimensionally represents the workspace; a workpiece model position calculation unit that calculates the position and orientation of the workpiece in the virtual space with respect to the robot model or the vision sensor model by superimposing three-dimensional position information of the workpiece, acquired by the vision sensor in the workspace, with respect to the robot or the vision sensor, and the shape characteristics of the workpiece model; and a simulation execution unit that measures the workpiece model using the vision sensor model and executes a simulation operation in which the robot model performs operations on the workpiece model. The three-dimensional position information of the workpiece acquired by the visual sensor within the workspace includes the three-dimensional position information measured using the visual sensor for all of the workpieces that are randomly stacked within the workspace. The aforementioned model arrangement section is In such a way that the arrangement of all the workpieces that are scattered within the aforementioned workspace is reproduced in the virtual space, This robot simulation device arranges the work model in the virtual space at a position and orientation calculated by the work model position calculation unit, based on the robot model or the vision sensor model. [Effects of the Invention]

[0007] Since the robot model's work simulation is executed in a virtual space that replicates the random stacking of workpieces in the actual workspace, it is possible to efficiently create motion programs that can perform highly accurate picking operations.

[0008] These and other objects, features, and advantages of the present invention will become even clearer from the detailed description of typical embodiments of the present invention shown in the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows a robot simulation device according to one embodiment, connected to a robot system. [Figure 2] This diagram shows an example of the hardware configuration of a robot control device and a robot simulation device. [Figure 3] This is a functional block diagram showing the functional configuration of a robot simulation device. [Figure 4] This is a flowchart illustrating the simulated operation of a robot simulation device. [Figure 5] This diagram shows a robot model placed within a virtual space. [Figure 6] This diagram shows the state in which the robot model and the vision sensor model are placed in a virtual space, where the vision sensor model is a fixed sensor fixed within the virtual space. [Figure 7] This diagram shows the state in which the robot model and the vision sensor model are placed in a virtual space, when the vision sensor model is mounted on the robot model. [Figure 8] This diagram illustrates how a visual sensor measures a workpiece when the visual sensor is a fixed sensor that is fixed within the workspace. [Figure 9] This diagram illustrates how a workpiece is measured using a vision sensor when it is mounted on a robot. [Figure 10] This diagram illustrates a situation where a visual sensor projects patterned light onto a workpiece to measure its dimensions. [Figure 11] This diagram illustrates a situation where multiple intersection points have been measured on the surface of a workpiece. [Figure 12]This shows the state in which the workpiece is positioned in the virtual space based on the calculated position and orientation of the workpiece, in the case where the visual sensor model is a fixed sensor fixed within the virtual space. [Figure 13] This shows the state in which the workpiece model (WM) is placed in a virtual space based on the calculated position and orientation of the workpiece model, when the vision sensor model is mounted on the robot model. [Figure 14] This diagram shows the state in which the simulation execution unit is performing a simulation operation to extract a work model using a robot model. [Modes for carrying out the invention]

[0010] Next, embodiments of the present disclosure will be described with reference to the drawings. In the drawings, similar components or functional parts are given the same reference numerals. For ease of understanding, the scale of these drawings has been appropriately changed. Furthermore, the embodiments shown in the drawings are just one example of how to carry out the present invention, and the present invention is not limited to the illustrated embodiments.

[0011] Figure 1 shows a configuration in which a robot simulation device 30 according to one embodiment is connected to a robot system 100. The robot system 100 includes a robot 10, a robot control device 20 that controls the movement of the robot 10, a vision sensor 70, and workpieces W placed in a bulk state in a container 81. The robot 10 has a hand 11 mounted on its wrist plunger. Each object constituting the robot system 100 is arranged in the workspace. The robot simulation device 30 is a device for performing simulations to create an operation program for the robot 10. The robot simulation device 30 is connected to the robot control device 20 by wire or wireless. The robot simulation device 30 may also be connected remotely to the robot control device 20.

[0012] The robot simulation device 30 according to this embodiment arranges models of each object including the robot 10, the vision sensor 70, and the work W stacked in the container 81 in a virtual space, detects the work W with the vision sensor 70, and simulates the operation of taking out the work W with the robot 10 (hand 11) by operating these models in a simulated manner. In this case, the robot simulation device 30 acquires the actual three-dimensional position information of the work W stacked in the container 81, reproduces the actual stacked state of the work W in the virtual space, and executes the simulation, thereby enabling the efficient creation of an operation program capable of executing a more accurate work taking-out operation.

[0013] The vision sensor 70 may be a two-dimensional camera that acquires a two-dimensional image, or may be a three-dimensional position detector that acquires the three-dimensional position of the object. In this embodiment, it is assumed that the vision sensor 70 is a range sensor capable of acquiring the three-dimensional position of the object. The vision sensor 70 includes a projector 73 and two cameras 71 and 72 arranged at positions facing each other with the projector 73 interposed therebetween. The projector 73 is configured to be able to project a desired pattern light such as spot light or slit light onto the surface of the object. The projector includes, for example, a light source such as a laser diode or a light emitting diode. The cameras 71 and 72 are digital cameras equipped with imaging elements such as CCDs or CMOS sensors.

[0014] Note that FIG. 1 also shows the robot coordinate system C1 set for the robot 10 and the sensor coordinate system C2 set for the vision sensor 70. As an example, the robot coordinate system C1 is set at the base of the robot 10, and the sensor coordinate system C2 is set at the position of the lens of the vision sensor 70. The positions and orientations of these coordinate systems are grasped by the robot control device 20. In FIG. 1, as an example, a configuration in which the vision sensor 70 is attached to the tip of the arm of the robot 10 is shown, but there is also a configuration example in which the vision sensor 70 is fixed at a known position in the work space.

[0015] Figure 2 shows an example of the hardware configuration of the robot control device 20 and the robot simulation device 30. The robot control device 20 may have a configuration similar to a general computer, with a processor 21 connected to memory 22 (ROM, RAM, non-volatile memory, etc.), input / output interface 23, and an operation unit 24 including various operation switches via a bus. The robot simulation device 30 may have a configuration similar to a general computer, with a processor 31 connected to memory 32 (ROM, RAM, non-volatile memory, etc.), a display unit 33, an operation unit 34 consisting of input devices such as a keyboard (or software keys), and an input / output interface 35 via a bus. Various information processing devices such as personal computers, notebook PCs, and tablet terminals can be used as the robot simulation device 30.

[0016] Figure 3 is a functional block diagram showing the functional configuration of the robot simulation device 30. The robot simulation device 30 comprises a virtual space creation unit 131, a model placement unit 132, a vision sensor model position setting unit 133, a work model position calculation unit 134, and a simulation execution unit 135.

[0017] The virtual space creation unit 131 creates a virtual space that represents the workspace in three dimensions.

[0018] The model placement unit 132 places models of each object that constitute the robot system 100 within the virtual space. The state in which each object model has been placed in the virtual space by the model placement unit 132 may be displayed on the display unit 33.

[0019] The visual sensor model position setting unit 133 acquires information representing the position of the visual sensor 70 within the workspace from the robot control device 20. For example, the visual sensor model position setting unit 133 acquires information (calibration data) indicating the relative position between the robot coordinate system C1 and the sensor coordinate system C2, which is stored in the robot control device 20, as a file from the robot control device 20. Specifically, this relative position information represents the position and orientation of the visual sensor 70 (sensor coordinate system C2) relative to the robot 10 (robot coordinate system C1) within the workspace. The information representing the relative position between the robot coordinate system C1 and the sensor coordinate system C2 is acquired by performing a calibration of the visual sensor 70 in advance in the robot system 100 and is stored in the robot control device 20.

[0020] Here, calibration is achieved, for example, by measuring a visual marker attached to a predetermined reference position on the robot with the visual sensor 70, thereby obtaining the position and orientation of the visual sensor 70 relative to the visual marker. By obtaining the position and orientation of the visual sensor 70 relative to a visual marker placed at a known position, the position and orientation of the visual sensor 70 relative to the robot 10 is obtained.

[0021] The model placement unit 132 places the vision sensor model in the virtual space such that the relative positions of the robot model coordinate system set for the robot model and the sensor model coordinate system set for the vision sensor model in the virtual space are the same as the relative positions of the robot coordinate system and the synth coordinate system in the workspace.

[0022] The work model position calculation unit 134 calculates the position and orientation of the work model relative to the robot model or the visual sensor model in the virtual space by superimposing the three-dimensional position information of the robot 10 or the work relative to the visual sensor 70, which is acquired in the workspace by the visual sensor 70, with the shape characteristics of the work model. The model placement unit 132 places the work model in the virtual space at the calculated position and orientation.

[0023] The simulation execution unit 135 measures the workpiece models, which are arranged in a bulk stack at the calculated positions and orientations, using a visual sensor model, and performs a simulation of the robot model picking them up. In this specification, the terms "simulation" or "simulation operation" include not only numerical simulations of robot movements, but also the operation of various object models, such as robot models, on a display screen.

[0024] Figure 4 is a flowchart showing the simulation operation performed under the control of the processor 31 of the robot simulation device 30.

[0025] First, the virtual space creation unit 131 creates a virtual space that represents the workspace in three dimensions (step S1). Then, the model placement unit 132 places the robot model 10M within the virtual space (step S2). Figure 5 shows the state after the robot model 10M has been placed within the virtual space. Furthermore, the simulation execution unit 135 sets the robot model coordinate system M1 for the robot model 10M in the virtual space at a position corresponding to the robot coordinate system C1 defined in the workspace.

[0026] Next, the vision sensor model position setting unit 133 sets the position and orientation of the vision sensor model 70M relative to the robot model 10M in the virtual space, based on the position and orientation of the vision sensor 70 relative to the robot 10 in the workspace (step S3). The position and orientation of the vision sensor relative to the robot 10 in the workspace are stored in the robot control device 20, for example, as the relative position between the robot coordinate system C1 and the sensor coordinate system C2, by performing a calibration of the vision sensor 70 in the robot system 100. In step S3, the vision sensor model position setting unit 133 acquires information from the robot control device 20 as the relative position between the robot coordinate system C1 and the sensor coordinate system C2.

[0027] Next, in step S4, the model placement unit 132 places the vision sensor model 70M in the virtual space such that the relative positions of the robot model coordinate system M1 and the sensor model coordinate system M2 are equivalent to the relative positions of the robot coordinate system C1 and the sensor coordinate system C2 in the workspace.

[0028] Figures 6 and 7 show the state in which the model placement unit 132 places the vision sensor model 70M in the virtual space according to information representing the relative position of the vision sensor 70 with respect to the robot 10. Figure 6 shows an example in which the vision sensor 70 is used as a fixed camera fixed at a predetermined position in the workspace, and Figure 7 shows an example in which the vision sensor 70 is attached to the tip of the arm of the robot 10. As shown in Figures 6 and 7, the vision sensor model 70M comprises a projector model 73M and two camera models 71M and 72M positioned opposite each other with the projector model 73M in between. As shown in Figures 6 and 7, in the virtual space, the sensor model coordinate system M2 is set at the position corresponding to the sensor coordinate system C2.

[0029] Next, in step S5, the work model position calculation unit 134 calculates the position and orientation of the work model WM relative to the robot model 10M or the vision sensor model 70M in the virtual space by superimposing the three-dimensional information of the robot 10 or the work W relative to the vision sensor 70, which was acquired by the vision sensor 70 in the workspace, with the shape characteristics of the work model WM.

[0030] The three-dimensional position information of the workpiece W is stored in the robot control device 20 as a set of three-dimensional coordinates based on, for example, the robot coordinate system C1 or the sensor coordinate system C2, by measuring the workpiece W with the vision sensor 70. The workpiece model position calculation unit 134 acquires the three-dimensional position information of the workpiece W from the robot control device 20 and calculates the position and orientation of the workpiece model WM by superimposing it with the shape characteristics of the workpiece model WM.

[0031] Here, the method for acquiring three-dimensional position information of a workpiece W in a randomly stacked state using the visual sensor 70 will be explained with reference to Figures 8 to 10. In this embodiment, the visual sensor 70 is a range sensor that can acquire the distance to an object. The range sensor acquires three-dimensional information of the workpiece in the form of, for example, a distance image or a three-dimensional map. A distance image is an image that represents the distance from the range sensor to the workpiece within the measurement distance by the brightness or color of each pixel. A three-dimensional map represents the three-dimensional position of the workpiece within the measurement area as a set of three-dimensional coordinate values ​​of points on the surface of the workpiece.

[0032] The two cameras 71 and 72 of the vision sensor 70 are oriented in different directions from each other so that their fields of view overlap at least partially. The projection range of the projector 73 is positioned so that it at least partially overlaps with the fields of view of each camera 71 and 72. Figure 8 shows the situation in which the vision sensor 70 measures the workpiece W when the vision sensor 70 is a fixed camera fixed in a predetermined position in the workspace. Figure 9 shows the situation in which the vision sensor 70 measures the workpiece W when the vision sensor 70 is mounted on the tip of the arm of the robot 10.

[0033] The three-dimensional position information of the workpiece W is calculated by calculating multiple intersection lines between a first group of planes that divides the field of view captured by the two cameras 71 and 72 at equal intervals, passing through the focal points of the two cameras 71 and 72, which cover the area to be measured where the workpiece W is placed, and a second group of planes that corresponds to the boundary surface between light and dark when a striped pattern light 160 is projected by the projector 73 onto the area to be measured in the area where the workpiece W is placed. The three-dimensional coordinates of the intersection points between these intersection lines and the workpiece surface are then calculated (see Figure 10).

[0034] Figure 10 shows the field of view (the area to be measured) captured by the two cameras 71 and 72 as the field of view FV, and the dashed lines that divide the field of view into equal intervals are shown as dashed lines. Figure 10 illustrates the striped pattern light 160 projected onto the area where the workpiece W is placed, one of the first plane groups (hereinafter referred to as the first plane 151), and one of the second plane groups (hereinafter referred to as the second plane 152). In Figure 10, the striped pattern light 160 is shown as a light and dark pattern (represented by the presence or absence of shading) extending from the back to the front of the figure. Figure 10 also illustrates the intersection line L1 of the first plane 151 and the second plane 152, and the intersection point P of the intersection line L1 and the surface of the workpiece W.

[0035] In this way, the first and second plane groups are calculated, and the intersection lines between the first and second plane groups are calculated. Then, three-dimensional information of multiple intersection points P between the calculated intersection lines and the surface of the randomly stacked workpieces W is calculated.

[0036] The robot control device 20 performs the workpiece removal process multiple times to acquire 3D coordinates for all workpieces W.

[0037] The three-dimensional coordinates of all workpieces W, acquired by the robot system 100 using the procedure described above, are stored in the robot control device 20.

[0038] The work model position calculation unit 134 acquires the three-dimensional coordinates (coordinates based on the robot coordinate system C1 or sensor coordinate system C2) of the multiple intersection points P on the work surface determined as described above from the robot control device 20 as three-dimensional information of the work W. The work model position calculation unit 134 then compares the three-dimensional information of the work W with the shape characteristics of the work model (face data, edge data, vertex data, etc. of the work model) to search for possible positions and orientations of the work model, and calculates the position and orientation of the work model that maximizes the degree of agreement between the set of three-dimensional coordinates and the shape information of the work model. As a result, the work model position calculation unit 134 obtains the position and orientation of the work model WM in the virtual space that corresponds to the position and orientation of the work W in the working space.

[0039] Figure 11 shows the state in which the work model WM is superimposed and positioned relative to the 3D position information (multiple intersection points P) of the workpiece W using the procedure described above. Figure 11 also illustrates the range Q in which the 3D position of the workpiece W is obtained. Furthermore, Figure 11 shows the work model coordinate system M3 set for each work model WM. The work model coordinate system M3 may be set at the centroid position of each work model WM, for example, if the work model WM is a rectangular parallelepiped.

[0040] Next, in step S6, the model placement unit 132 places the work model WM in the virtual space at a position and orientation of the work model W relative to the robot model 10M or the vision sensor model 70M. Figure 12 shows the state in which the work model WM is placed in the virtual space based on the position and orientation of the work model WM calculated in step S5, when the vision sensor model 70M is a fixed sensor with a fixed position. Figure 13 shows the state in which the work model WM is placed in the virtual space based on the position and orientation of the work model WM calculated in step S5, when the vision sensor model M is mounted on the robot model 10M. As shown in Figures 12 and 13, the position and orientation of the work model WM may also be determined as the position and orientation of the work model coordinate system M3 relative to the Romot model coordinate system M1 or the vision sensor model coordinate system M2. This reproduces in the virtual space the actual arrangement of the workpieces W that are piled up in the workspace.

[0041] Next, in step S7, the simulation execution unit 135 performs a simulation of the workpiece model WM being measured by the vision sensor model 70M and the workpiece model WM being picked up one by one by the hand model 11M mounted on the robot model 10M, with the workpiece model WM positioned in the virtual space as shown in Figure 12 or Figure 13.

[0042] The simulation execution unit 135, in the same manner as the measurement operation using the visual sensor 70, simulates the position and orientation of the work model WM in the virtual space by following the procedure below. (a1) A first set of planes is calculated based on the positions and measurement areas of two camera models 71M and 72M in the visual sensor model 70M placed in the virtual space. (a2) Next, a second set of planes is calculated based on the position of the projector model 73M and the measurement area. (a3) Calculate multiple intersection lines between the first group of planes and the second group of planes. (a4) Calculate the three-dimensional coordinates of the intersection point between the intersection line and the work model WM. (a5) The position and orientation of the work model WM are calculated based on the three-dimensional coordinates of the work model WM. (a6) Based on the calculated position and orientation of the work model WM, the robot model 10M is moved to a position where it can grasp the target work model, and the operation of picking up the target work model with the hand model 11M is simulated.

[0043] Figure 14 shows the state in which the simulation execution unit 135 is performing a simulation operation to extract the work model WM from the robot model 10M. Such an operation may also be displayed on the display unit 33 of the robot simulation device 30.

[0044] Thus, according to this embodiment, since the robot model's work simulation is performed in a virtual space that reproduces the bulk stacking state of workpieces in the workspace, it is possible to efficiently create an operation program that can perform highly accurate picking operations.

[0045] Although the present invention has been described above using typical embodiments, those skilled in the art will understand that modifications to the above embodiments and various other modifications, omissions, and additions can be made without departing from the scope of the present invention.

[0046] The functional blocks of the robot simulation device 30 shown in Figure 3 may be realized by the processor 31 of the robot simulation device 30 executing software stored in a memory device, or they may be realized by a hardware-based configuration such as an ASIC (Application Specific Integrated Circuit).

[0047] The program that performs the simulation operation shown in Figure 4 in the above-described embodiment can be recorded on various computer-readable recording media (for example, semiconductor memories such as ROM, EEPROM, and flash memory, magnetic recording media, and optical discs such as CD-ROM and DVD-ROM). [Explanation of symbols]

[0048] 10 Robots 10M Robot Model 11 Hand 11M Hand Model 20 Robot control devices 21 processors 22 memory 23 Input / Output Interfaces 24 Control section 30 Robot Simulation Device 31 processors 32 memory 33 Display section 34 Control section 35 Input / Output Interfaces 70. Vision Sensors 70M Vision Sensor Model 71, 72 Camera 71M, 72M camera models 73 Projectors 73M Projector Model 81 Container 81M container model 100 Robot Systems 131 Virtual Space Creation Department 132 Model Placement Section 133 Visual sensor model position setting unit 134 Work Model Position Calculation Unit 135 Simulation Execution Unit

Claims

1. A robot simulation device for simulating the work performed by a robot on a workpiece in a robot system including a robot, a vision sensor, and a workpiece arranged in a workspace, A model placement unit that places the robot model of the robot, the vision sensor model of the vision sensor, and the workpiece model of the workpiece in a virtual space that represents the aforementioned workspace in three dimensions, A work model position calculation unit calculates the position and orientation of the work model relative to the robot model or the vision sensor model in the virtual space by superimposing the three-dimensional position information of the workpiece, which is acquired by the vision sensor within the aforementioned workspace, with the shape characteristics of the work model. The system includes a simulation execution unit that measures the workpiece model using the visual sensor model and performs a simulation operation in which the robot model performs work on the workpiece model, The three-dimensional position information of the workpiece acquired by the visual sensor within the aforementioned workspace includes the three-dimensional position information measured using the visual sensor for all of the workpieces that are randomly stacked within the aforementioned workspace. The model placement unit is a robot simulation device that places the workpiece model in the virtual space at the position and orientation calculated by the workpiece model position calculation unit, based on the robot model or the vision sensor model, so that the arrangement of all the workpieces that are randomly stacked in the workspace is reproduced in the virtual space.

2. The robot simulation apparatus according to claim 1, wherein the three-dimensional position information of the workpiece is a set of three-dimensional points of the workpiece measured using the vision sensor.

3. The system further includes a vision sensor model position setting unit that sets the position and orientation of the vision sensor model relative to the robot model in the virtual space based on the position and orientation of the vision sensor relative to the robot in the aforementioned workspace, The robot simulation apparatus according to claim 1 or 2, wherein the model placement unit places the visual sensor model in the virtual space at the set position and orientation of the visual sensor model.

4. The robot simulation apparatus according to claim 3, wherein the position and orientation of the vision sensor relative to the robot within the workspace are data included in calibration data obtained by performing calibration of the vision sensor in the workspace.

Images