Simulation device, control system, and modeling method

Abstract

A simulation device 100 is provided with: an actual shape model generation unit 112 for generating, on the basis of measurement data, an actual shape model indicating a three-dimensional actual shape of a machine system 2 that includes robots 4A, 4B; and a model correction unit 113 that corrects a simulation model 310 of the machine system 2 on the basis of a comparison between the simulation model 310 and an actual shape model 210.

Description

Technical Field

[0001] The present disclosure relates to a simulation device, a control system, and a modeling method.

Background Art

[0002] Patent Document 1 discloses a robot simulator including a model storage unit that stores model information regarding a robot and an obstacle, and an information processing unit that generates a path capable of moving the tip of the robot from a start position to an end position while avoiding a collision between the robot and the obstacle based on the model information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present disclosure provides a simulation device effective for improving the reliability of simulation.

Means for Solving the Problems

[0005] A simulation device according to one aspect of the present disclosure includes an actual shape model generation unit that generates an actual shape model representing a three-dimensional actual shape of a machine system including a robot based on measurement data, and a model correction unit that corrects a simulation model based on a comparison between the simulation model of the machine system and the actual shape model.

[0006] A control system according to another aspect of the present disclosure further includes the above-described simulation device including a simulator that simulates the operation of a machine system based on a simulation model, and a control device that controls the machine system based on a simulation result by the simulator.

[0007] Another modeling method according to another aspect of the present disclosure includes generating an actual shape model representing the three-dimensional actual shape of a machine system including a robot based on measurement data, and correcting the simulation model based on a comparison between the simulation model of the machine system and the actual shape model.

Advantages of the Invention

[0008] According to the present disclosure, it is possible to provide a simulation device effective for improving the reliability of simulation.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Figure 14

Figure 15

Figure 16

Figure 17

Figure 18

Figure 19

Figure 20

Figure 21

Figure 22

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments will be described in detail with reference to the drawings. In the description, the same reference numerals are given to the same elements or elements having the same function, and redundant descriptions are omitted.

[0011] 〔Automation System〕 The automation system 1 shown in FIG. 1 is a system for operating at least a robot in a machine system including at least a robot. As a specific example of the automation system 1, a production system for operating at least a robot so as to produce a product in a machine system can be mentioned, but the use of the machine system is not necessarily limited to the production of products.

[0012] The automation system 1 includes a machine system 2 and a control system 50. The machine system 2 includes a plurality of objects 3. Each of the plurality of objects 3 is a substantial object that occupies a part of the three-dimensional real space. The plurality of objects 3 includes at least one controlled object 4 to be controlled and at least one peripheral object 5.

[0013] At least one controlled object 4 includes at least one robot. In FIG. 1, two robots 4A and 4B are shown as at least one controlled object 4, and a main stage 5A, sub-stages 5B and 5C, and a frame 5D are shown as at least one peripheral object 5.

[0014] FIG. 2 is a schematic diagram illustrating the schematic configuration of the robots 4A and 4B. For example, the robots 4A and 4B are 6-axis vertical articulated robots and have a base 11, a turning part 12, a first arm 13, a second arm 14, a third arm 17, a tip part 18, and actuators 41, 42, 43, 44, 45, and 46. The base 11 is installed around the conveyor 5 A. The turning part 12 is provided on the base 11 so as to turn around a vertical axis 21. The first arm 13 is connected to the turning part 12 so as to swing around an axis 22 that intersects (for example, is orthogonal to) the axis 21. The intersection includes a case where the relationship is a twisted relationship like a so-called spatial intersection. The second arm 14 is connected to the tip part of the first arm 13 so as to swing around an axis 23 that is substantially parallel to the axis 22. The second arm 14 includes an arm base 15 and an arm end 16. The arm base 15 is connected to the tip part of the first arm 13 and extends along an axis 24 that intersects (for example, is orthogonal to) the axis 23. The arm end 16 is connected to the tip part of the arm base 15 so as to turn around the axis 24. The third arm 17 is connected to the tip part of the arm end 16 so as to swing around an axis 25 that intersects (for example, is orthogonal to) the axis 24. The tip part 18 is connected to the tip part of the third arm 17 so as to turn around an axis 26 that intersects (for example, is orthogonal to) the axis 25.

[0015] In this way, the robots 4A and 4B include a joint 31 that connects the base 11 and the swivel unit 12, a joint 32 that connects the swivel unit 12 and the first arm 13, a joint 33 that connects the first arm 13 and the second arm 14, a joint 34 that connects the arm base 15 and the arm end 16 in the second arm 14, a joint 35 that connects the arm end 16 and the third arm 17, and a joint 36 that connects the third arm 17 and the tip 18.

[0016] The actuators 41, 42, 43, 44, 45, and 46 include, for example, an electric motor and a speed reducer, and drive the joints 31, 32, 33, 34, 35, and 36, respectively. For example, the actuator 41 rotates the swivel unit 12 around the axis 21, the actuator 42 swings the first arm 13 around the axis 22, the actuator 43 swings the second arm 14 around the axis 23, the actuator 44 rotates the arm end 16 around the axis 24, the actuator 45 swings the third arm 17 around the axis 25, and the actuator 46 rotates the tip 18 around the axis 26.

[0017] Note that the specific configuration of the robots 4A and 4B can be changed as appropriate. For example, the robots 4A and 4B may be 7-axis redundant robots in which one more axis joint is added to the above-mentioned 6-axis vertical articulated robot, or so-called scalar type articulated robots.

[0018] The main stage 5A supports the robots 4A and 4B, the sub-stages 5B and 5C, and the frame 5D. The sub-stage 5B supports the workpiece by the robot 4A. The sub-stage 5C supports the workpiece by the robot 4B. The frame 5D holds various objects (not shown) in the upper space of the main stage 5A. Specific examples of the objects held by the frame 5D include environmental sensors such as laser sensors, or tools used by the robots 4A and 4B.

[0019] Note that the configuration of the machine system 2 shown in FIG. 1 is an example. As long as it includes at least one robot, the configuration of the machine system 2 can be changed as appropriate. For example, the machine system 2 may include three or more robots.

[0020] The control system 50 controls at least one controlled object 4 included in the machine system 2 based on an operation program prepared in advance. The control system 50 may include a plurality of controllers that respectively control a plurality of controlled objects 4, and a host controller that outputs control commands to the plurality of controllers so as to cooperate the plurality of controlled objects 4. FIG. 1 shows controllers 51 and 52 that respectively control robots 4A and 4B, and a host controller 53. The host controller 53 outputs control commands to the controllers 51 and 52 so as to cooperate the robots 4A and 4B.

[0021] The control system 50 further includes a simulation device 100. The simulation device 100 simulates the state of the machine system 2. Simulating the state of the machine system 2 includes simulating the static arrangement relationship of a plurality of objects 3. Simulating the state of the machine system 2 may further include simulating the dynamic arrangement relationship of a plurality of objects 3 that changes due to the operations of controlled objects 4 such as robots 4A and 4B.

[0022] The simulation is useful for evaluating the appropriateness of the operations of the robots 4A and 4B based on the operation program before actually operating the robots 4A and 4B. However, if the reliability of the simulation is low, even an operation evaluated as appropriate according to the simulation result may cause problems such as collisions between the objects 3 when the operation is actually executed by the robots 4A and 4B.

[0023] The operations of robots 4A and 4B are simulated by performing a kinematic calculation that reflects the operation results of robots 4A and 4B on a simulation model that includes the arrangement information of a plurality of objects 3 including robots 4A and 4B and the structure / dimension information of each of the plurality of objects 3.

[0024] In order to improve the reliability of the simulation, it is important to improve the accuracy of the simulation model. In contrast, the simulation device 100 is configured to generate a real shape model representing the three-dimensional real shape of the machine system 2 based on measurement data, and to correct the simulation model based on a comparison between the simulation model of the machine system 2 and the real shape model. Thereby, the accuracy of the simulation model can be easily improved.

[0025] For example, as shown in FIG. 3, the simulation device 100 has, as a functional configuration, a simulation model storage unit 111, a real shape model generation unit 112, and a model correction unit 113.

[0026] The simulation model storage unit 111 stores a simulation model of the machine system 2. The simulation model includes at least the arrangement information of a plurality of objects 3 and the structure / dimension information of each of the plurality of objects 3. The simulation model is prepared in advance based on design data of the machine system 2 such as three-dimensional CAD data. The simulation model may include a plurality of object models corresponding to the plurality of objects 3 respectively. Each of the plurality of object models includes the arrangement information and the structure / dimension information of the corresponding object 3. The arrangement information of the object 3 includes the position / attitude of the object 3 in a predetermined simulation coordinate system.

[0027] The actual shape model generation unit 112 generates an actual shape model representing the three-dimensional actual shape of the machine system 2 based on the measurement data. The measurement data is data obtained by measuring the machine system 2 in the real space. As a specific example of the measurement data, a three-dimensional actual image of the machine system 2 captured by a three-dimensional camera can be mentioned. As a specific example of the three-dimensional camera, a stereo camera, a TOF (Time of Flight) camera, etc. can be mentioned. The three-dimensional camera may be a three-dimensional laser displacement meter.

[0028] As an example, the control system 50 has at least one three-dimensional camera 54, and the actual shape model generation unit 112 generates an actual shape model based on the three-dimensional actual image of the machine system 2 captured by the three-dimensional camera 54. The actual shape model generation unit 112 may generate an actual shape model representing the three-dimensional shape of the surface of the machine system 2 as a point cloud. The actual shape model generation unit 112 may generate an actual shape model representing the three-dimensional shape of the surface of the machine system 2 as a fine polygon group.

[0029] The control system 50 may have a plurality of three-dimensional cameras 54, and the actual shape model generation unit 112 may acquire a plurality of three-dimensional actual images from the plurality of three-dimensional cameras 54 and generate an actual shape model by combining the plurality of three-dimensional actual images. The actual shape model generation unit 112 may acquire a plurality of three-dimensional actual images including images of a common synthesis object from the plurality of three-dimensional cameras 54, and generate an actual shape model by combining the plurality of three-dimensional actual images so that the portions corresponding to the synthesis object in each of the plurality of three-dimensional actual images are matched to the known shape of the synthesis object.

[0030] FIG. 4 is a schematic diagram illustrating an object to be photographed by two three-dimensional cameras 54. To simplify the description, in FIG. 4, the machine system 2 is represented by objects 6A and 6B with simplified shapes. As shown in FIG. 5, according to the three-dimensional camera 54A in the upper left of FIG. 4, a three-dimensional image 221 is acquired, and according to the three-dimensional camera 54B in the lower right of FIG. 4, a three-dimensional image 222 is acquired. The three-dimensional image 221 includes the three-dimensional shape of at least the part of the machine system 2 facing the three-dimensional camera 54A. The three-dimensional image 222 includes the three-dimensional shape of at least the part of the machine system 2 facing the three-dimensional camera 54B.

[0031] For example, the actual shape model generation unit 112 generates an actual shape model 220 by combining the three-dimensional image 221 and the three-dimensional image 222 with the object 6B as the above-described object for synthesis. For example, the actual shape model generation unit 112 matches the three-dimensional shape of the object 6B included in the three-dimensional images 221 and 222 to the known three-dimensional shape of the object 6B. The matching here means moving each of the three-dimensional images 221 and 222 so that the three-dimensional shape of the object 6B included in the three-dimensional images 221 and 222 fits the known three-dimensional shape of the object 6B. By moving each of the three-dimensional images 221 and 222 so that the three-dimensional shape of the object 6B included in the three-dimensional images 221 and 222 fits the known three-dimensional shape of the object 6B, as shown in FIG. 6, the three-dimensional images 221 and 222 are combined, and an actual shape model 220 of the objects 6A and 6B is generated. Note that the actual shape model generation unit 112 may synthesize the three-dimensional images of the plurality of three-dimensional cameras 54 with any one of the robots 4A and 4B, the main stage 5A, the sub-stages 5B and 5C, and the frame 5D as the object for synthesis.

[0032] The model correction unit 113 corrects the simulation model based on the comparison between the simulation model stored in the simulation model storage unit 111 and the actual shape model generated by the actual shape model generation unit 112. The model correction unit 113 may correct the simulation model by individually matching a plurality of object models to the actual shape model. The matching here means correcting the position and orientation of each of the plurality of object models so as to fit the actual shape model. The model correction unit 113 may correct the simulation model by repeating a matching process including selecting one matching target model from the plurality of object models and matching the matching target model to the actual shape model.

[0033] In the matching process, the model correction unit 113 may exclude from the actual shape model the portions that have already been matched to other object models and match the matching target model to the actual shape model. In the matching process, the model correction unit 113 may select, as the matching target model, the largest object model among one or more object models that have not been selected as the matching target model.

[0034] By repeating the matching process, the arrangement of the plurality of object models is individually corrected. However, there may be differences between the simulation model and the actual shape model that cannot be eliminated only by correcting the arrangement of the plurality of object models. For example, the actual shape model may include portions that do not correspond to any of the plurality of object models. Also, any of the plurality of object models may include portions that do not correspond to the actual shape model.

[0035] In contrast, the simulation device 100 may further include an object addition unit 114 and an object deletion unit 115. After the matching process is completed for all of the plurality of object models, the object addition unit 114 extracts, from the actual shape model, portions that do not match any of the object models, and adds a new object model to the simulation model based on the extracted portions. After the matching process is completed for all of the plurality of object models, the object deletion unit 115 extracts, from the simulation model, portions that do not match the actual shape model, and deletes the extracted portions from the simulation model.

[0036] Hereinafter, with reference to the drawings, the correction of the simulation model by the model correction unit 113, the addition of the object model by the object addition unit 114, and the deletion of unnecessary portions of the simulation model by the object deletion unit 115 will be specifically illustrated.

[0037] FIG. 7 is a diagram illustrating an actual shape model of the machine system 2, and FIG. 8 is a diagram illustrating a simulation model of the machine system 2. The actual shape model 210 shown in FIG. 7 includes a portion 211 corresponding to the robot 4A, a portion 212 corresponding to the robot 4B, a portion 213 corresponding to the main stage 5A, a portion 214 corresponding to the sub-stage 5B, a portion 215 corresponding to the sub-stage 5C, and a portion 216 corresponding to the frame 5D.

[0038] The simulation model 310 shown in FIG. 8 includes a robot model 312A corresponding to the robot 4A, a robot model 312B corresponding to the robot 4B, a main stage model 313A corresponding to the main stage 5A, a sub-stage model 313B corresponding to the sub-stage 5B, and a frame model 313D corresponding to the frame 5D. The simulation model 310 does not include a sub-stage model 313C (see FIG. 15) corresponding to the sub-stage 5C.

[0039] First, the model correction unit 113 selects the largest main stage model 313A among the robot models 312A and 312B, the main stage model 313A, the sub-stage models 313B, and the frame model 313D. Here, "large" means that the occupied area in the three-dimensional space is large.

[0040] As shown in FIGS. 9 and 10, the model correction unit 113 matches the main stage model 313A to the actual shape model 210. As shown by the hatched portion in FIG. 10, the main stage model 313A matches the portion 213 corresponding to the main stage 5A in the actual shape model 210.

[0041] As shown in FIG. 11, the model correction unit 113 excludes the portion 213 that has already been matched to the main stage model 313A from the actual shape model 210. Although the portion 213 is erased in FIG. 11, excluding the portion 213 from the actual shape model 210 does not mean deleting the portion 213 from the actual shape model 210. The portion 213 may be left in the actual shape model 210 without deleting the portion 213, and the portion 213 may be excluded from the matching target in the subsequent matching processes. The same applies to the exclusion of other portions of the actual shape model 210.

[0042] Next, the model correction unit 113 selects the largest sub-stage model 313B among the robot models 312A and 312B, the sub-stage model 313B, and the frame model 313D, and as shown in FIG. 12, matches the sub-stage model 313B to the actual shape model 210. As shown by the hatched portion in FIG. 12, the sub-stage model 313B matches the portion 214 corresponding to the sub-stage 5B in the actual shape model 210. As shown by the dotted pattern portion in FIG. 12, the sub-stage model 313B includes a portion 313b that does not match the portion 214.

[0043] As shown in FIG. 13, the model correction unit 113 excludes the portion 214 that has already been matched to the sub-stage model 313B from the actual shape model 210. Next, the model correction unit 113 selects the largest robot model 312B among the robot model 312A, the robot model 312B, and the frame model 313D, and matches the robot model 312B to the actual shape model 210. As shown by the hatched portion in FIG. 13, the robot model 312B matches the portion 212 corresponding to the robot 4B in the actual shape model 210.

[0044] As shown in FIG. 14, the model correction unit 113 excludes the portion 212 that has already been matched to the robot model 312B from the actual shape model 210. Next, the model correction unit 113 selects the largest robot model 312A among the robot model 312A and the frame model 313D, and matches the robot model 312A to the actual shape model 210. As shown by the hatched portion in FIG. 14, the robot model 312A matches the portion 211 corresponding to the robot 4A in the actual shape model 210.

[0045] As shown in FIG. 15, the model correction unit 113 excludes the portion 211 that has already been matched to the robot model 312A from the actual shape model 210. Next, the model correction unit 113 selects the frame model 313D and matches the frame model 313D to the actual shape model 210. As shown by the hatched portion in FIG. 15, the frame model 313D matches the portion 216 corresponding to the frame 5D in the actual shape model 210.

[0046] Thus, all the matching processes for the robots 4A, 4B, the main stage 5A, the sub-stage 5B, and the frame 5D are completed. However, since the simulation model 310 does not include an object model corresponding to the sub-stage 5C, the portion 215 of the actual shape model 210 remains without matching to any of the object models included in the simulation model 310.

[0047] In contrast, the object addition unit 114 extracts the portion 215 and adds a sub-stage model 313C corresponding to the sub-stage 5C to the simulation model 310 based on the portion 215, as shown in FIG. 16.

[0048] Also, the portion 313b that did not match the actual shape model 210 will remain without matching any portion of the actual shape model 210. In contrast, the object addition unit 114 extracts the portion 313b and deletes the portion 313b from the simulation model 310. Thus, the correction of the simulation model by the model correction unit 113, the addition of the object model by the object addition unit 114, and the deletion of unnecessary portions by the object deletion unit 115 are completed.

[0049] Here, when generating an actual shape model based on the three-dimensional actual image of the machine system 2 captured by the three-dimensional camera 54, the actual shape model may include hidden portions that are not visible in the three-dimensional camera 54. Even when generating an actual shape model based on a plurality of three-dimensional actual images of the machine system 2 respectively captured by a plurality of three-dimensional cameras 54, the actual shape model may include overlapping hidden portions that are not visible in any of the plurality of three-dimensional cameras 54.

[0050] FIG. 17 is a schematic diagram illustrating the imaging targets by two three-dimensional cameras 54. For simplicity of explanation, in FIG. 17, the machine system 2 is represented by objects 7A, 7B, 7C, 7D with simplified shapes.

[0051] FIG. 18 shows an actual shape model 230 generated based on the three-dimensional image captured by the left three-dimensional camera 54A in FIG. 17 and the three-dimensional image captured by the right three-dimensional camera 54B in FIG. 17.

[0052] The actual shape model 230 includes a hidden portion 230a that is not visible in the three-dimensional camera 54A, a hidden portion 230b that is not visible in the three-dimensional camera 54B, and a duplicate hidden portion 230c that is not visible in either of the three-dimensional cameras 54A and 54B. The duplicate hidden portion 230c is a portion where the hidden portion 230a and the hidden portion 230b overlap.

[0053] Even though the actual shape model includes hidden portions, if the simulation model does not include hidden portions, the accuracy of matching the object model to the actual shape model may decrease. In contrast, the simulation apparatus 100 may generate a preprocessed model in which virtual hidden portions corresponding to the hidden portions that are not visible in the three-dimensional camera 54 are excluded from the simulation model, and correct the simulation model based on a comparison between the preprocessed model and the actual shape model.

[0054] When generating an actual shape model based on three-dimensional real images of the machine system 2 captured by a plurality of three-dimensional cameras 54, the simulation apparatus 100 may generate a preprocessed model in which virtual duplicate hidden portions corresponding to the duplicate hidden portions that are not visible in any of the plurality of three-dimensional cameras 54 are excluded from the simulation model, and correct the simulation model based on a comparison between the preprocessed model and the actual shape model.

[0055] For example, the simulation apparatus 100 may further include a camera position calculation unit 121, a preprocessing unit 122, a re-division unit 123, and a preprocessed model storage unit 124.

[0056] The camera position calculation unit 121 calculates the position of the three-dimensional virtual camera so as to match a three-dimensional virtual image obtained by photographing the simulation model with a three-dimensional virtual camera corresponding to the three-dimensional camera 54 to the three-dimensional real image. The camera position calculation unit 121 may calculate the position of the three-dimensional virtual camera so as to match a portion corresponding to a predetermined calibration object in the three-dimensional virtual image to a portion corresponding to the calibration object in the three-dimensional real image.

[0057] The camera position calculation unit 121 may use any one of the plurality of objects 3 except as a calibration object, or two or more of the plurality of objects 3 may be used as calibration objects. For example, the camera position calculation unit 121 may use the robot 4A or the robot 4B as a calibration object.

[0058] For example, after the camera position calculation unit 121 calculates a three-dimensional virtual image on the condition that a three-dimensional virtual camera is arranged at a predetermined initial position, the camera position calculation unit 121 evaluates the difference between the calibration object in the three-dimensional virtual image and the calibration object in the three-dimensional real image, and changes the position of the three-dimensional virtual camera, and repeats these operations until the evaluation result of the difference falls below a predetermined level, thereby calculating the position of the three-dimensional virtual camera. Note that the position of the three-dimensional virtual camera includes the posture of the three-dimensional virtual camera.

[0059] The camera position calculation unit 121 may calculate the positions of the plurality of three-dimensional virtual cameras so as to match the plurality of three-dimensional virtual images obtained by photographing the simulation model with the plurality of three-dimensional virtual cameras respectively corresponding to the plurality of three-dimensional cameras 54 to the plurality of three-dimensional real images.

[0060] The preprocessing unit 122 calculates a virtual hidden part based on the position of the three-dimensional virtual camera and the simulation model, generates a preprocessed model obtained by excluding the virtual hidden part from the simulation model, and stores the preprocessed model in the preprocessed model storage unit 124. For example, the preprocessing unit 122 extracts a visible surface facing the three-dimensional virtual camera from the simulation model, and calculates a part located behind the visible surface as a virtual hidden part.

[0061] The preprocessing unit 122 may calculate a virtual overlapping hidden part based on the positions of the plurality of three-dimensional virtual cameras and the simulation model, generate a preprocessed model obtained by excluding the virtual overlapping hidden part from the simulation model, and store the preprocessed model in the preprocessed model storage unit 124.

[0062] FIG. 19 is a diagram illustrating a preprocessed model 410 generated for the machine system 2 of FIG. 17. The preprocessing unit 122 calculates a virtual hidden portion 410a corresponding to the hidden portion 230a based on the position of the three-dimensional virtual camera 321A corresponding to the three-dimensional camera 54A of FIG. 17 and the simulation model. Further, the preprocessing unit 122 calculates a virtual hidden portion 410b corresponding to the hidden portion 230b based on the position of the three-dimensional virtual camera 321B corresponding to the three-dimensional camera 54B of FIG. 17 and the simulation model. Furthermore, the preprocessing unit 122 calculates a virtual overlapping hidden portion 410c that is not captured by either of the three-dimensional virtual cameras 321A and 321B. The virtual overlapping hidden portion 410c is a portion where the virtual hidden portion 410a and the virtual hidden portion 410b overlap.

[0063] The preprocessing unit 122 may generate a preprocessed model in the same data format as the actual shape model. For example, when the actual shape model generation unit 112 generates an actual shape model representing the three-dimensional shape of the surface of the machine system 2 as a point cloud, the preprocessing unit 122 may generate a preprocessed model representing the three-dimensional shape of the surface of the machine system 2 as a point cloud. When the actual shape model generation unit 112 generates an actual shape model representing the three-dimensional shape of the surface of the machine system 2 as a fine polygon group, the preprocessing unit 122 may generate a preprocessed model representing the three-dimensional shape of the surface of the machine system 2 as a fine polygon group.

[0064] By matching the data formats of the preprocessed model and the actual shape model, it becomes easier to compare the preprocessed model and the actual shape model. Even if the data formats of the preprocessed model and the actual shape model are different, it is possible to compare the preprocessed model and the actual shape model, so it is not essential to match the data format of the preprocessed model to the data format of the actual shape model.

[0065] The re-division unit 123 divides the pre-processed model into a plurality of pre-processed object models respectively corresponding to the plurality of objects 3. For example, the re-division unit 123 divides the pre-processed model into a plurality of pre-processed object models based on the comparison between each of the plurality of object models stored in the simulation model storage unit 111 and the pre-processed object model.

[0066] For example, the re-division unit 123 sets the portion corresponding to the object model of object 7A in the pre-processed model 410 as the pre-processed object model 411 of object 7A, sets the portion corresponding to the object model of object 7B in the pre-processed model 410 as the pre-processed object model 412 of object 7B, sets the portion corresponding to the object model of object 7C in the pre-processed model 410 as the pre-processed object model 413 of object 7C, and sets the portion corresponding to the object model of object 7D in the pre-processed model 410 as the pre-processed object model 414 of object 7D.

[0067] When the simulation device 100 includes a camera position calculation unit 121, a pre-processing unit 122, a re-division unit 123, and a pre-processed model storage unit 124, the model correction unit 113 corrects the simulation model based on the comparison between the pre-processed model stored in the pre-processed model storage unit 124 and the actual shape model generated by the actual shape model generation unit 112. For example, the model correction unit 113 matches each of the plurality of object models to the actual shape model based on the comparison between the corresponding pre-processed object model and the actual shape model.

[0068] In addition, when the actual shape model does not include a hidden portion, or when the influence of the hidden portion on the matching accuracy of the object model with respect to the actual shape model can be ignored, the virtual hidden portion is removed from the simulation model excludingIt is not essential to generate a removed preprocessed model. Even in such a case, preprocessing may be performed to match the data format of the simulation model with the data format of the actual shape model.

[0069] The simulation device 100 may further include a simulator 125. The simulator 125 simulates the operation of the machine system 2 based on the simulation model corrected by the model correction unit 113. For example, the simulator 125 simulates the operation of the machine system 2 by performing a kinematic operation (e.g., forward kinematic operation) that reflects the operation results of the controlled objects 4 such as the robots 4A and 4B on the simulation model.

[0070] The simulation device 100 may further include a program generation unit 126. The program generation unit 126 (planning support device) supports the operation planning of the machine system 2 based on the simulation results by the simulator 125. For example, the program generation unit 126 repeatedly evaluates an operation program for controlling the controlled objects 4 such as the robots 4A and 4B based on the simulation results by the simulator 125, and corrects the operation program based on the evaluation results to generate an operation program.

[0071] The program generation unit 126 may transmit the generated operation program to the upper controller 53 so as to control the controlled object 4 based on the operation program. Thereby, the upper controller 53 (control device) controls the machine system based on the simulation results by the simulator 125.

[0072] FIG. 20 is a block diagram illustrating the hardware configuration of the simulation apparatus 100. As shown in FIG. 20, the simulation apparatus 100 has a circuit 190. The circuit 190 includes one or more processors 191, a memory 192, a storage 193, an input / output port 194, and a communication port 195. The storage 193 has a computer-readable storage medium such as, for example, a non-volatile semiconductor memory. The storage 193 stores a program for causing the simulation apparatus 100 to at least generate an actual shape model representing the three-dimensional actual shape of the machine system 2 based on measurement data, and correct the simulation model based on a comparison between the simulation model of the machine system 2 and the actual shape model. For example, the storage 193 stores a program for causing the simulation apparatus 100 to construct the above-described functional configuration.

[0073] The memory 192 temporarily stores a program loaded from the storage medium of the storage 193 and calculation results by the processor 191. The processor 191 cooperates with the memory 192 to execute the above program, thereby constituting each functional block of the simulation apparatus 100. The input / output port 194 inputs and outputs information to and from the three-dimensional camera 54 in accordance with a command from the processor 191. The communication port 195 communicates with the host controller 53 in accordance with a command from the processor 191.

[0074] Note that the circuit 190 is not necessarily limited to one that configures each function by a program. For example, at least a part of the functions of the circuit 190 may be configured by a dedicated logic circuit or an ASIC (Application Specific Integrated Circuit) integrating the same.

[0075] 〔Modeling Procedure〕 Next, as an example of the modeling method, a correction procedure for the simulation model executed by the simulation apparatus 100 will be exemplified. This procedure includes generating a real shape model representing the three-dimensional real shape of the machine system 2 based on the measurement data, and correcting the simulation model based on the comparison between the simulation model of the machine system 2 and the real shape model.

[0076] As shown in FIG. 21, the simulation apparatus 100 first executes steps S01, S02, S03, S04, S05, S06, S07, and S08 in order. In step S01, the real shape model generation unit 112 acquires a plurality of three-dimensional real images of the machine system 2 respectively captured by the plurality of three-dimensional cameras 54. In step S02, the real shape model generation unit 112 recognizes, in each of the plurality of three-dimensional real images acquired in step S01, a portion corresponding to the above-described synthesis object. In step S03, the real shape model generation unit 112 generates a real shape model by combining the plurality of three-dimensional real images so that the portions corresponding to the synthesis object in each of the plurality of three-dimensional real images are matched to the known shape of the synthesis object.

[0077] In step S04, the camera position calculation unit 121 recognizes, in each of the plurality of three-dimensional real images, a portion corresponding to the calibration object. In step S05, the camera position calculation unit 121 calculates the position of the three-dimensional virtual camera so that the portion corresponding to the calibration object in the three-dimensional virtual image is matched to the portion corresponding to the calibration object in the three-dimensional real image, for each of the plurality of three-dimensional virtual cameras. In step S06, the preprocessing unit 122 calculates, for each of the plurality of three-dimensional virtual cameras, a virtual hidden portion in the simulation model that is not visible to the three-dimensional virtual camera based on the position of the three-dimensional virtual camera and the simulation model.

[0078] In step S07, based on the calculation result of the virtual hidden part in step S06, the preprocessing unit 122 generates a preprocessed model by excluding the virtual overlapping hidden part that does not appear in any of the plurality of three-dimensional virtual cameras from the simulation model, and stores it in the preprocessed model storage unit 124. In step S08, the re-division unit 123 divides the preprocessed model stored in the preprocessed model storage unit 124 into a plurality of preprocessed object models corresponding to the plurality of objects 3 respectively.

[0079] Next, as shown in FIG. 22, the simulation device 100 executes steps S11, S12, S13, and S14. In step S11, the model correction unit 113 selects the largest object model among one or more object models that are not selected as the matching target model in the plurality of object models as the matching target model. In step S12, the model correction unit 113 matches the matching target model to the actual shape model based on the comparison between the preprocessed object model corresponding to the matching target model and the actual shape model.

[0080] In step S13, the model correction unit 113 excludes the part of the actual shape model that matches the matching target model from the target of the matching process in subsequent times. In step S14, the model correction unit 113 checks whether the matching process for all object models is completed.

[0081] In step S14, if it is determined that there are remaining object models for which the matching process is not completed, the simulation device 100 returns the process to step S11. Thereafter, until the matching for all object models is completed, the selection of the matching target model and the matching of the matching target model to the actual shape model are repeated.

[0082] When it is determined in step S14 that the matching process for all object models has been completed, the simulation apparatus 100 executes step S15. In step S15, the object addition unit 114 extracts from the actual shape model the portions that do not match any of the object models, and adds new object models to the simulation model based on the extracted portions. Further, the object deletion unit 115 extracts from the simulation model the portions that do not match the actual shape model, and deletes the extracted portions from the simulation model. Thus, the correction procedure for the simulation model is completed.

[0083] 〔Effects of the Present Embodiment〕 As described above, the simulation apparatus 100 includes an actual shape model generation unit 112 that generates an actual shape model 210 representing the three-dimensional actual shape of the machine system 2 including the robots 4A and 4B based on measurement data, and a model correction unit 113 that corrects the simulation model 310 based on a comparison between the simulation model 310 of the machine system 2 and the actual shape model 210.

[0084] According to this simulation apparatus 100, the accuracy of the simulation model 310 can be easily improved. Therefore, this simulation apparatus 100 is effective in improving the reliability of the simulation.

[0085] The machine system 2 includes a plurality of objects 3 including the robots 4A and 4B, the simulation model 310 includes a plurality of object models respectively corresponding to the plurality of objects 3, and the model correction unit 113 may correct the simulation model 310 by individually matching the plurality of object models with the actual shape model 210. In this case, by performing the matching with the actual shape model 210 for each of the plurality of object models, the simulation model 310 can be corrected with higher accuracy.

[0086] The model correction unit 113 may correct the simulation model 310 by repeating the matching process including selecting one matching target model from a plurality of object models and matching the matching target model to the actual shape model 210. In this case, the matching for each of the plurality of objects can be performed easily and reliably.

[0087] In the matching process, the model correction unit 113 may exclude the portions already matched to other object models from the actual shape model 210 and match the matching target model to the actual shape model 210. In this case, a new matching target model can be matched to the actual shape model 210 without being affected by the portions already matched to other object models. Therefore, the simulation model 310 can be corrected with higher accuracy.

[0088] In the matching process, the model correction unit 113 may select the largest object model among one or more object models that have not yet been selected as the matching target model as the matching target model. In this case, by performing the matching in order from the larger object models and excluding the portions matched to the object models from the actual shape model 210, it becomes easier to narrow down the portions to be matched for the matching target model in each matching process. Therefore, the simulation model 310 can be corrected with higher accuracy.

[0089] The simulation apparatus 100 may further include an object addition unit 114 that extracts, from the actual shape model 210, the portions that are not matched to any of the object models after the matching process is completed for all of the plurality of object models, and adds a new object model to the simulation model 310 based on the extracted portions. In this case, the simulation model 310 can be corrected with higher accuracy.

[0090] After the matching process is completed for all of the plurality of object models, the simulation apparatus 100 may further include an object deletion unit 115 that extracts, from the simulation model 310, portions that do not match the actual shape model 210 and deletes the extracted portions from the simulation model 310. In this case, the simulation model 310 can be corrected with higher accuracy.

[0091] The actual shape model generation unit 112 generates an actual shape model 230 based on the three-dimensional actual image of the machine system 2 captured by the three-dimensional camera 54. The simulation apparatus 100 further includes a preprocessing unit 122 that generates a preprocessed model 410 in which virtual hidden portions 410a and 410b corresponding to hidden portions 230a and 230b that are not captured by the three-dimensional camera 54 are excluded from the simulation model 310. The model correction unit 113 may correct the simulation model 310 based on a comparison between the preprocessed model 410 and the actual shape model 210. In this case, by using, as a comparison target with the actual shape model 230, the preprocessed model 410 in which portions that cannot be represented by the actual shape model 210 because they are not captured by the three-dimensional camera 54 are excluded from among the plurality of objects 3, the simulation model 310 can be corrected with higher accuracy.

[0092] The actual shape model generation unit 112 generates an actual shape model 230 based on the three-dimensional actual image of the machine system 2 captured by the three-dimensional camera 54. The simulation device 100 includes a preprocessing unit 122 that generates a preprocessed model 410 by excluding virtual hidden parts 410a and 410b corresponding to hidden parts 230a and 230b that are not captured by the three-dimensional camera 54 from the simulation model 310 of the machine system 2, and a re-division unit 123 that divides the preprocessed model 410 into a plurality of preprocessed object models corresponding to the plurality of objects 3 respectively. The model correction unit 113 may match each of the plurality of object models to the actual shape model 210 based on the comparison between the corresponding preprocessed object model and the actual shape model. In this case, by improving the matching accuracy for each of the plurality of object models, the simulation model 310 can be corrected with higher accuracy.

[0093] The simulation device 100 further includes a camera position calculation unit 121 that calculates the positions of the three-dimensional virtual cameras 321A and 321B corresponding to the three-dimensional camera 54 so as to match the three-dimensional virtual image obtained by photographing the simulation model 310 with the three-dimensional virtual cameras 321A and 321B to the three-dimensional actual image. The preprocessing unit 122 may calculate the virtual hidden parts 410a and 410b based on the positions of the three-dimensional virtual cameras 321A and 321B and the simulation model 310. In this case, by making the virtual hidden parts 410a and 410b correspond to the hidden parts 230a and 230b with higher accuracy, the simulation model 310 can be corrected with higher accuracy.

[0094] The camera position calculation unit 121 may calculate the positions of the three-dimensional virtual cameras 321A and 321B so that a portion of the three-dimensional virtual image corresponding to a predetermined calibration object is matched with a portion of the three-dimensional real image corresponding to the calibration object. In this case, by performing the matching between the three-dimensional virtual image and the three-dimensional real image only for the portion corresponding to the calibration object, the positions of the three-dimensional virtual cameras 321A and 321B can be corrected more easily.

[0095] The actual shape model generation unit 112 acquires a plurality of three-dimensional real images from a plurality of three-dimensional cameras 54 including the three-dimensional cameras 54A and 54B, combines the plurality of three-dimensional real images to generate an actual shape model 210, and the preprocessing unit 122 may generate a preprocessed model 410 in which a virtual overlapping hidden portion 410c corresponding to an overlapping hidden portion 230c that does not appear in any of the plurality of three-dimensional cameras 54A and 54B is excluded from the simulation model 310. In this case, the simulation model 310 can be corrected with higher accuracy by reducing the virtual overlapping hidden portion 410c.

[0096] The actual shape model generation unit 112 may acquire a plurality of three-dimensional real images including images of a common synthesis object from a plurality of three-dimensional cameras 54 including the three-dimensional cameras 54A and 54B, and combine the plurality of three-dimensional real images to generate an actual shape model 210 such that a portion corresponding to the synthesis object in each of the plurality of three-dimensional real images is matched with the known shape of the synthesis object. In this case, the plurality of three-dimensional real images can be easily combined to generate an actual shape model 210 with fewer hidden portions.

[0097] The simulation device 100 further includes a camera position calculation unit 121 that calculates the positions of a plurality of three-dimensional virtual cameras 321A and 321B so as to match a plurality of three-dimensional virtual images obtained by photographing a simulation model 310 with the plurality of three-dimensional virtual cameras 321A and 321B respectively corresponding to the plurality of three-dimensional cameras 54A and 54B. The preprocessing unit 122 may calculate a virtual overlapping hidden part 410c based on the positions of the plurality of three-dimensional virtual cameras 321A and 321B and the simulation model 310. In this case, the simulation model 310 can be corrected with higher accuracy by making the virtual overlapping hidden part 410c correspond to the overlapping hidden part 230c with higher accuracy.

[0098] The actual shape model generation unit 112 may generate an actual shape model 210 that represents the three-dimensional actual shape of the machine system 2 as a point cloud, and the preprocessing unit 122 may generate a preprocessed model 410 that represents the three-dimensional virtual shape of the simulation model 310 as a virtual point cloud. In this case, the difference between the actual shape model 210 and the preprocessed model 410 can be easily evaluated.

[0099] The actual shape model generation unit 112 generates an actual shape model 210 that represents the three-dimensional actual shape of the machine system 2 as a point cloud, and the simulation device 100 further includes a preprocessing unit that generates a preprocessed model 410 that represents the three-dimensional virtual shape of the simulation model 310 as a virtual point cloud. The model correction unit 113 may correct the simulation model 310 based on a comparison between the preprocessed model 410 and the actual shape model 210. In this case, the difference between the actual shape model 210 and the preprocessed model 410 can be easily evaluated.

[0100] Although the embodiments have been described above, the present disclosure is not necessarily limited to the above-described embodiments, and various modifications are possible without departing from the gist thereof.

Explanation of Reference Numerals

[0101] 2... Machine system, 3... Object, 4A, 4B... Robot, 50... Control system, 53... Host controller (control device), 54, 54A, 54B... Three-dimensional camera, 100... Simulation device, 112... Actual shape model generation unit, 113... Model correction unit, 114... Object addition unit, 115... Object deletion unit, 121... Camera position calculation unit, 122... Preprocessing unit, 123... Re-division unit, 125... Simulator, 126... Program generation unit (planning support device), 210, 220, 230... Actual shape model, 230a, 230b... Hidden part, 230c... Overlapping hidden part, 310... Simulation model, 321A, 321B... Three-dimensional virtual camera, 410... Preprocessed model, 410a, 410b... Virtual hidden part, 410c... Virtual overlapping hidden part, 411, 412, 413, 414... Preprocessed object model.

Claims

1. An actual shape model generation unit that generates an actual shape model representing the three-dimensional actual shape of the surface of a machine system including a robot based on measurement data; A model correction unit that corrects the simulation model based on a comparison between the simulation model of the machine system and the actual shape model, and The machine system includes a plurality of objects including the robot, The simulation model includes a plurality of object models respectively corresponding to the plurality of objects, The model correction unit corrects the simulation model by individually matching the plurality of object models to the actual shape model representing the three-dimensional actual shape of the surface of the machine system including the plurality of objects. A simulation device.

2. The simulation device according to claim 1, wherein the model correction unit corrects the simulation model by repeating a matching process including selecting one matching target model from the plurality of object models and matching the matching target model to the actual shape model.

3. The simulation device according to claim 2, wherein in the matching process, the model correction unit excludes a portion that has already been matched to another object model from the actual shape model and matches the matching target model to the actual shape model.

4. The simulation device according to claim 3, wherein in the matching process, the model correction unit selects the largest object model among one or more object models that have not yet been selected as the matching target model as the matching target model.

5. After the matching process is completed for all of the plurality of object models, an object addition unit that extracts a portion that is not matched to any object model from the actual shape model and adds a new object model to the simulation model based on the extracted portion. The simulation device according to any one of claims 2 to 4.

6. After the matching process is completed for all of the plurality of object models, an object deletion unit is further provided that extracts from the simulation model portions that do not match the actual shape model and deletes the extracted portions from the simulation model. The simulation apparatus according to any one of claims 2 to 5.

7. The actual shape model generation unit generates the actual shape model based on a three-dimensional actual image of the machine system captured by a three-dimensional camera. The simulation apparatus further includes a preprocessing unit that generates a preprocessed model from which virtual hidden portions corresponding to hidden portions not shown in the three-dimensional camera are excluded from the simulation model. The model correction unit corrects the simulation model based on a comparison between the preprocessed model and the actual shape model. The simulation apparatus according to any one of claims 1 to 6.

8. The actual shape model generation unit generates the actual shape model based on a three-dimensional actual image of the machine system captured by a three-dimensional camera. The simulation apparatus a preprocessing unit that generates a preprocessed model from which virtual hidden portions corresponding to hidden portions not shown in the three-dimensional camera are excluded from the simulation model of the machine system; a re-division unit that divides the preprocessed model into a plurality of preprocessed object models respectively corresponding to the plurality of objects; The model correction unit matches each of the plurality of object models to the actual shape model based on a comparison between the corresponding preprocessed object model and the actual shape model. The simulation apparatus according to any one of claims 1 to 6.

9. The simulation apparatus further includes a camera position calculation unit that calculates the position of the three-dimensional virtual camera so as to match a three-dimensional virtual image obtained by photographing the simulation model with the three-dimensional virtual camera corresponding to the three-dimensional camera to the three-dimensional actual image. The preprocessing unit calculates the virtual hidden portion based on the position of the three-dimensional virtual camera and the simulation model. The simulation apparatus according to claim 7 or 8.

10. The camera position calculation unit calculates the position of the three-dimensional virtual camera so that a portion of the three-dimensional virtual image corresponding to a predetermined calibration object is matched with a portion of the three-dimensional real image corresponding to the calibration object. The simulation apparatus according to claim 9.

11. The actual shape model generation unit acquires a plurality of three-dimensional real images from a plurality of three-dimensional cameras including the three-dimensional camera, and generates the actual shape model by combining the plurality of three-dimensional real images. The preprocessing unit generates a preprocessed model in which a virtual overlapping hidden portion corresponding to an overlapping hidden portion that does not appear in any of the plurality of three-dimensional cameras is excluded from the simulation model. The simulation apparatus according to claim 7 or 8.

12. The actual shape model generation unit acquires a plurality of three-dimensional real images including images of a common composite object from a plurality of three-dimensional cameras including the three-dimensional camera, and in each of the plurality of three-dimensional real images, a portion corresponding to the composite object is matched with a known shape of the composite object. The simulation apparatus according to claim 11, which generates the actual shape model by combining the plurality of three-dimensional real images.

13. The simulation apparatus further includes a camera position calculation unit that calculates the positions of the plurality of three-dimensional virtual cameras so that a plurality of three-dimensional virtual images obtained by photographing the simulation model with the plurality of three-dimensional virtual cameras respectively corresponding to the plurality of three-dimensional cameras are matched with the plurality of three-dimensional real images. The preprocessing unit calculates the virtual overlapping hidden portion based on the positions of the plurality of three-dimensional virtual cameras and the simulation model. The simulation apparatus according to claim 11 or 12.

14. The actual shape model generation unit generates the actual shape model that represents the three-dimensional actual shape of the machine system as a point cloud. The preprocessing unit generates the preprocessed model that represents the three-dimensional virtual shape of the simulation model as a virtual point cloud. The simulation apparatus according to any one of claims 7 to 13.

15. The actual shape model generation unit generates the actual shape model that represents the three-dimensional actual shape of the machine system as a point cloud. The simulation apparatus further includes a preprocessing unit that generates a preprocessed model that represents the three-dimensional virtual shape of the simulation model as a virtual point cloud. The simulation apparatus according to any one of claims 1 to 6, wherein the model correction unit corrects the simulation model based on a comparison between the pre-processed model and the actual shape model.

16. The simulation apparatus according to any one of claims 1 to 15, further comprising a simulator that simulates the operation of the machine system based on the simulation model.

17. A control system comprising the simulation apparatus according to claim 16, and a control device that controls the machine system based on a simulation result by the simulator.

18. Generating an actual shape model representing the three-dimensional actual shape of the surface of a machine system including a robot based on measurement data, and correcting the simulation model based on a comparison between the simulation model of the machine system and the actual shape model, wherein the machine system includes a plurality of objects including the robot, the simulation model includes a plurality of object models respectively corresponding to the plurality of objects, and a modeling method of correcting the simulation model by individually matching the plurality of object models to the actual shape model representing the three-dimensional actual shape of the surface of the machine system including the plurality of objects.

19. An actual shape model generation unit that generates an actual shape model representing the three-dimensional actual shape of a machine system including a robot based on a three-dimensional actual image of the machine system captured by a three-dimensional camera, a pre-processing unit that generates a pre-processed model excluding a virtual hidden part corresponding to a hidden part not shown in the three-dimensional camera from the simulation model of the machine system, and a model correction unit that corrects the simulation model based on a comparison between the pre-processed model and the actual shape model.

Images