Three-dimensional model generation system, generation method, and program

Abstract

Provided is a generation system, generation method, and program for a three-dimensional model. The three-dimensional model generation system according to the present embodiment includes: an end effector having a flexible gripping portion that grips an object; a deformation sensor that detects a deformed shape of the gripping portion in a state in which the object is gripped; a calculation portion that calculates a normal direction of a surface of the object based on surface shape data of the object; a determination portion that determines a gripping position of the object gripped by the end effector based on the normal direction; and a coordinate system setting portion that sets a reference coordinate system based on the deformed shape of the gripping portion.

Description

Technical Field

[0001] This disclosure relates to systems, methods, and programs for generating three-dimensional models. Background Technology

[0002] Patent Document 1 (Japanese Patent Application Publication No. 2021-85781) discloses an information processing apparatus for calculating the position and orientation of an object. The information processing apparatus of Patent Document 1 obtains a three-dimensional shape model (edge ​​information, point cloud information) from the object's distance point cloud information and determines the rotational symmetry about a first axis. Furthermore, the information processing apparatus determines the inverse symmetry about a second axis different from the first axis. Summary of the Invention

[0003] The technical problem that the invention aims to solve

[0004] Patent Document 1 establishes rotational symmetry about the first axis and inversion symmetry about the second axis. Therefore, manual setting of the coordinate axes is required, making automatic setting difficult. Consequently, it is difficult to efficiently generate 3D models of various objects. This invention enables the creation of 3D models using a reference coordinate system that takes into account the symmetry of objects.

[0005] In the generation of 3D shape models, appropriate information can be annotated by setting coordinate axes that take into account the symmetry of the object. Therefore, learning data with added position and pose information related to the object's position and pose can be generated. An inferrer for the object can be generated through teacher-guided learning using the object's position and pose information as correct answer labels (teacher data).

[0006] This disclosure was made to solve such a problem, and provides a system, method, and program for generating three-dimensional models that can efficiently and well generate three-dimensional models in appropriate coordinate systems.

[0007] Technical solutions for solving the problem

[0008] The three-dimensional model generation system in this embodiment includes: an end effector having a flexible gripping part for holding an object; a deformation sensor for detecting the deformation shape of the gripping part in the state of holding the object; a calculation unit for calculating the normal direction of the surface of the object based on the surface shape data of the object; a determination unit for determining the gripping position of the object held by the end effector based on the normal direction; and a coordinate system setting unit for setting a reference coordinate system based on the deformation shape of the gripping part.

[0009] The aforementioned three-dimensional model generation system may also include: a three-dimensional shape measuring device that measures the three-dimensional shape of the object by scanning it; and a drive control unit that drives the end effector so that the end effector places the object within the scanning range of the three-dimensional shape measuring device.

[0010] In the aforementioned three-dimensional model generation system, the three-dimensional shape measuring device may also include a depth sensor that measures the distance to the object, allowing the object's pose relative to the depth sensor to change around the axis of the reference coordinate system.

[0011] In the aforementioned three-dimensional model generation system, the three-dimensional shape measuring device may also include a table for rotating the object, and the end effector may place the object in a manner in which the rotation axis of the table is aligned with the axis of the reference coordinate system.

[0012] In the above-mentioned three-dimensional model generation system, a surface shape sensor may also be included. The surface shape sensor measures the surface shape data of the object before the end effector grasps the object. In the surface shape data, the part with the cross-section that is most approximately circular is selected as the grasping position of the end effector.

[0013] In the aforementioned 3D model generation system, it is also possible to determine the symmetry of the object relative to each axis of the reference coordinate system, and correct the axes of the reference coordinate system based on the symmetry determination result.

[0014] The method for generating a three-dimensional model in this embodiment includes: calculating the normal direction of the object's surface based on the object's surface shape data; setting the object's gripping position based on the normal direction; using an end effector with a flexible gripping part to grip the object at the gripping position; detecting the deformation shape of the gripping part when gripping the object; and setting a reference coordinate system based on the deformation shape of the gripping part.

[0015] In the above-described method for generating a 3D model, the end effector can also be driven so that it places the object within the scanning range of a 3D shape meter, and the 3D shape meter scans the object to measure its 3D shape.

[0016] In the above-described method for generating a three-dimensional model, the three-dimensional shape measuring device may also include a depth sensor that measures the distance to the object, causing the pose of the object relative to the depth sensor to change around the axis of the reference coordinate system.

[0017] In the above-described method for generating a three-dimensional model, the three-dimensional shape measuring instrument may also include a stage for rotating the object on which it is placed, and the end effector may place the object in such a manner that the rotation axis of the stage is aligned with the axis of the reference coordinate system.

[0018] In the above-described method for generating a three-dimensional model, a surface shape sensor may measure the surface shape data of the object before the end effector grasps the object, and the part whose cross-section is most approximately circular in the surface shape data may be selected as the grasping position of the end effector.

[0019] In the above-described method for generating a 3D model, it is also possible to determine the symmetry of the object relative to each axis of the reference coordinate system, and then correct the axes based on the determination result of the symmetry.

[0020] The program in this embodiment causes a computer to perform: a step of calculating the normal direction of the object's surface based on the object's surface shape data; a step of setting the object's gripping position based on the normal direction; a step of controlling an end effector having a flexible gripping part so that the end effector grips the object at the gripping position; a step of obtaining deformation shape data obtained by detecting the deformation shape of the gripping part when gripping the object; and a step of setting a reference coordinate system based on the deformation shape of the gripping part.

[0021] Invention Effects

[0022] According to this disclosure, a system, method, and program for generating three-dimensional models that can efficiently generate three-dimensional models in appropriate coordinate systems are provided.

[0023] The foregoing and other objects, features and advantages of this disclosure will become more fully understood from the detailed description given below and the accompanying drawings, which are given by way of example only, and therefore should not be considered as limiting this disclosure. Attached Figure Description

[0024] Figure 1 This is a schematic diagram showing the system's structure.

[0025] Figure 2 This is a top view schematically showing the structure of an end-effecter.

[0026] Figure 3 It is a block diagram showing the configuration of the processing device.

[0027] Figure 4 This is a flowchart illustrating the method for generating a three-dimensional model according to this embodiment.

[0028] Figure 5This is a flowchart illustrating the method for generating a three-dimensional model according to this embodiment.

[0029] Figure 6 This is a flowchart representing the determination process for rotational symmetry.

[0030] Figure 7 This is a diagram used to illustrate the cross-sectional shapes Sc and r(θ) of the convex hull of an object.

[0031] Figure 8 This is a coordinate graph representing r'(θ) and its Fourier transform.

[0032] Figure 9 It represents r(θ) and r(θ+ω) max The coordinate graph of the difference of ).

[0033] Figure 10 This is a flowchart representing the process of determining symmetry.

[0034] Figure 11 This is a flowchart representing the process of determining symmetry.

[0035] Figure 12 It is a diagram showing the cross-sectional shape in the process of determining surface symmetry.

[0036] Figure 13 This diagram illustrates the process of correcting the axis in the determination of face symmetry.

[0037] Figure 14 This is a block diagram illustrating an example of the hardware configuration of the processing device 100. Detailed Implementation

[0038] The present invention will now be described through embodiments thereof, but this does not limit the invention to the embodiments described in the claims. Furthermore, all the configurations described in the embodiments are not necessarily required as means to solve the problem.

[0039] In recent years, machine learning models have been used as estimators to determine the position and orientation of objects. To generate such estimators using machine learning, a large amount of training data needs to be collected. This training data includes not only images captured by cameras but also sometimes synthetic images generated through simulation.

[0040] Renderers and simulators generate composite images from 3D models (also known as 3D shape models or 3D models) that represent the 3D shape of an object. By using simulators and the like, a large number of composite images with altered positions and poses of objects can be easily generated. Furthermore, simulators and the like can annotate the composite images with labels such as position and pose.

[0041] For creating 3D models, sensors such as stereo cameras and depth sensors can be used. For example, a depth sensor measures the point cloud data of an object. Point cloud data includes information such as the position coordinates of each point in an XYZ three-dimensional orthogonal coordinate system. In addition, by transforming the point cloud data into edge data, mesh data, or surface data, a 3D model can be formed.

[0042] Referring to the accompanying drawings, the system and method for generating the three-dimensional model involved in this embodiment will be described. Figure 1 This is a block diagram representing the structure of System 1. System 1 includes a processing unit 100, a three-dimensional shape measuring device 200, an end effector 300, an arm mechanism 400, and a ranging sensor 500. System 1 generates a three-dimensional model representing the three-dimensional shape of object O. In addition, the processing unit 100 can also perform processing for determining the symmetry of object O.

[0043] The end effector 300 is a gripping mechanism (robot hand) for holding objects. The end effector 300 is capable of releasing the held object. The end effector 300 has a flexible gripping part. Specifically, in the end effector 300, the part in contact with the object O is formed of a flexible elastomer or the like. The configuration of the end effector 300 will be described later. The end effector 300 is mounted at the front end of the arm mechanism 400.

[0044] The arm mechanism 400 is a drive mechanism that drives the end effector 300. The arm mechanism 400 is a robotic arm with multiple joints. The arm mechanism 400 can change the position and orientation of the object O held by the end effector 300. For example, the arm mechanism 400 changes the position and orientation of the object O by changing the position and orientation of the end effector 300. The arm mechanism 400 can move the end effector 300 to a holding position of the object O. Furthermore, when the end effector 300 holds the object O, the arm mechanism 400 can move the object O to a desired position. The positional relationship between the arm mechanism 400 and the three-dimensional shape measuring device 200 (described later) is known.

[0045] For example, the arm mechanism 400 and the end effector 300 can place an object O on the stage 210. After the end effector 300 holds the object O, the arm mechanism 400 moves the end effector 300. As a result, the object O moves onto the stage 210. With the object O in contact with the mounting surface of the stage 210, the end effector 300 releases its grip. Thus, the arm mechanism 400 and the end effector 300 can move the object O onto the stage 210.

[0046] The end effector 300 has a flexible gripping part. Furthermore, the gripping part deforms when the end effector 300 grips the object O. The end effector 300 has a built-in sensor for measuring the deformation shape of the gripping part. For example, the end effector described in Non-Patent Document 1 can be used as the end effector 300.

[0047] Non-patent literature 1: Andreas ten Pas published an article entitled Grasp PoseDetection in Point Clouds on arXiv on June 29, 2017.

[0048] Figure 2 This is a schematic diagram showing an example of an end effector 300. The end effector 300 includes a main body 311, a base 313, a gripping part 314, a claw part 315, a deformation sensor 316, and an actuator 317.

[0049] A pair of claws 315 are rotatably provided on the main body 311. The claws 315 are gripping mechanisms driven to grasp the object O. Here, the end effector 300 is a two-finger type with two claws 315. Of course, the end effector 300 may also be provided with three or more fingers. The end effector 300 is not limited to a gripping mechanism with fingers.

[0050] A gripping part 314 is mounted on the front end of each of the two claw parts 315. The gripping part 314 is fixed to the claw part 315 via a base 313. The pair of gripping parts 314 are configured to clamp an object O from both sides. The object O is positioned between the two gripping parts 314. For example, in Figure 2 In this arrangement, one gripping part 314 is positioned on the right side of the object O, and the other gripping part 314 is positioned on the left side of the object O. The gripping parts 314 are formed from a resin film or sheet. Therefore, the gripping parts 314 deform according to the shape of the object O at the gripping position. That is, the gripping parts 314 deform according to the shape of the object O at the point of contact with it.

[0051] Main body 311 installed in Figure 1The arm mechanism 400 shown in the figure has an actuator 317 for driving two claws 315 in the main body 311. The actuator 317 drives the claws 315 in the direction of the arrow, causing the front ends of the two claws 315 to move closer and further apart. The actuator 317 grips the object O while the two claws 315 are holding it. When the end effector 300 performs a gripping action on the object O, the actuator 317 drives the claws 315, causing the pair of claws 315 to move closer together. As a result, the gripping part 314 comes into contact with the object O and can clamp the object O. When a release action is performed to release the grip on the object O, the actuator 317 drives the claws 315, causing the pair of claws 315 to separate. As a result, the gripping part 314 leaves the object O, and thus the object O can be released.

[0052] Additionally, a deformation sensor 316 is mounted on the base 313. The deformation sensor 316 detects the deformation shape of the gripping part 314. The deformation sensor 316 is built into the gripping part 314 and the base 313. For example, the gripping part 314 is formed of a flexible membrane. The gripping part 314 is spatially mounted on the base 313. The space between the base 313 and the gripping part 314 is hollow, and the deformation sensor 316 is disposed in this hollow space.

[0053] The deformation sensor 316 is an optical sensor such as a camera. A pattern that can be detected by the deformation sensor 316 is provided on the surface of the base 313 side of the gripping part 314. For example, a point cloud pattern arranged in a two-dimensional manner is formed on the gripping part 314.

[0054] The deformation sensor 316 captures an image of the point cloud pattern disposed on the gripping portion 314. Due to the deformation of the gripping portion 314, the positional relationships of the points in the captured image change. That is, the position of each point in the captured image changes according to the amount of deformation of the gripping portion 314. Therefore, by capturing the point cloud pattern of the gripping portion 314, the deformed shape of the gripping portion 314 can be detected. The shape of the contact area where the gripping portion 314 contacts the object O is detected. The deformation sensor 316 can determine the shape of only the contact area of ​​the object O, that is, the local shape.

[0055] return Figure 1 Description: The 3D shape measuring device 200 is a measuring device used to measure the 3D shape of an object O. The 3D shape measuring device 200 is, for example, a 3D scanner. The 3D shape measuring device 200 includes a table 210 and a depth sensor 220. The table 210 has a mounting surface parallel to a horizontal plane. A user or robot places the object O, which is the object to be measured, on the table 210.

[0056] Depth sensor 220 measures the distance (depth) up to object O on stage 210. For example, depth sensor 220 may be a LiDAR, stereo camera, or depth camera, capturing depth images. Stage 210 rotates object O about an axis parallel to the vertical direction. During this rotation, depth sensor 220 measures the distance up to the surface of object O.

[0057] Therefore, the three-dimensional shape measuring device 200 can measure measurement data representing the three-dimensional shape of the entire circumference of the object O. The measurement data includes multiple distance measurements, which represent the distance from the depth sensor 220 to the object O. Since the object O can be measured during the rotation of the stage 210, the distance to the surface of the object O from any direction can be measured. Furthermore, in order to measure the shape of the bottom and top surfaces of the object O, the orientation of the object O on the stage 210 can be changed, and the three-dimensional shape measuring device 200 can perform the measurements.

[0058] The three-dimensional shape data of the object is obtained through the detection results of the depth sensor 220. That is, the depth sensor 220 obtains the three-dimensional shape data by detecting the distance up to the surface of the object. The three-dimensional shape data is a depth image representing the surface shape of object O, that is, the boundary between object O and space.

[0059] The measurement data obtained by the 3D shape measuring device 200 is, for example, point cloud data with multiple points. Each point includes information about its 3D coordinates in the sensor coordinate system of the 3D shape measuring device 200. Alternatively, the measurement data can also be mesh data. Mesh data consists of a tri-mesh connecting three adjacent points and a quad-mesh connecting four adjacent points. The 3D shape measuring device 200 outputs the measurement data to the processing device 100.

[0060] The range sensor 500 measures the distance to object O in order to determine the surface shape of object O. The range sensor 500 can be a lidar, laser rangefinder, depth camera, ultrasonic sensor, stereo camera, etc. The data measured by the range sensor 500 is used as surface shape data. The range sensor 500 measures surface shape data representing at least a portion of the surface shape of object O. The surface shape data is not limited to data representing the entire surface shape of object O; data representing a portion of the surface shape is sufficient. It can determine the surface shape of object O even without CAD data, etc.

[0061] The processing device 100 is an information processing device for a personal computer. For example, the processing device 100 includes a memory, a processor, various interfaces, input devices, output devices, a monitor, etc. The processor of the processing device 100 executes a program stored in the memory to perform the processing described later. Furthermore, the processing device 100 is an information processing device capable of wireless or wired communication.

[0062] The processing unit 100 acquires surface shape data from the ranging sensor 500. The processing unit 100 also acquires three-dimensional shape data from the three-dimensional shape measuring device 200. Furthermore, the processing unit 100 establishes a coordinate system for the three-dimensional shape data and creates a three-dimensional model. Further, the processing unit 100 controls the arm mechanism 400 and the end effector 300.

[0063] use Figure 3 The processing device 100 will be described in detail. Figure 3 This is a control block diagram showing the configuration of the processing device 100. The processing device 100 includes a surface shape data acquisition unit 101, a calculation unit 102, a gripping position setting unit 103, and a drive control unit 105. The processing device 100 also includes a measurement data acquisition unit 110, a coordinate system setting unit 120, a first determination unit 130, a second determination unit 140, a third determination unit 150, and a correction unit 160.

[0064] (Acquisition of 3D shape data)

[0065] First, the processing for acquiring the three-dimensional shape data of object O will be described. The surface shape data acquisition unit 101 acquires the surface shape data measured by the ranging sensor 500. The surface shape data is data representing the local shape of object O. The surface shape data includes data representing distances from the ranging sensor 500 to multiple points on object O. The surface shape data may be, for example, point cloud data or mesh data.

[0066] The calculation unit 102 calculates the normal direction of the surface of object O based on the surface shape data. Furthermore, the calculation unit 102 determines the distribution of the normal directions on the surface of object O. That is, the calculation unit 102 calculates the normal at each point on object O. The surface shape data can be, for example, point cloud data or mesh data. The calculation unit 102 can determine the normal direction from the point cloud data or mesh data. A normal is a direction orthogonal to the surface of object O. The normal direction varies depending on the position of object O.

[0067] The gripping position setting unit 103 sets the gripping position of the object O held by the end effector 300 based on the normal direction. The gripping position setting unit 103 searches for the optimal gripping position based on the distribution of the normal direction.

[0068] The drive control unit 105 controls the end effector 300 and the arm mechanism 400 so that the end effector 300 grips the object O in a gripping position. The drive control unit 105 outputs command values ​​to each motor of the end effector 300 and the arm mechanism 400. In this way, the end effector 300 can grip the object O in the gripping position set by the gripping position setting unit 103. In the gripping position of the object O, the drive claw 315 drives the gripping part 314 to clamp the object O.

[0069] For example, the gripping position setting unit 103 determines the cross section (also called the gripping section) that becomes the gripping position of object O based on the distribution of the normal direction. The gripping position of object O can be set to the position where the shape (outline) of the cross section of object O is closest to a circle. That is, the gripping position setting unit 103 sets the cross section of object O that is closest to a circle as the gripping position (gripping section).

[0070] Specifically, as shown in Non-Patent Document 1, the gripping position setting unit 103 searches for the antipodal point of the object O and outputs candidate gripping postures. An antipodal point is a point located on the opposite side of the object O, passing through its center. The gripping position setting unit 103 takes any point of the object O and its antipodal point as a pair and calculates their normal directions. For each candidate, it calculates the point cloud normals on the plane and selects the part that best fits a circle as the gripping position posture. When the cross-section is circular, the normal directions of the paired points become parallel. Therefore, by calculating the deviation of the normal directions of the paired points, a cross-section close to a circle can be searched. The gripping position setting unit 103 sets the gripping position of the object O based on surface shape data.

[0071] The drive control unit 115 drives the arm mechanism 400 to move the end effector 300 to a gripping position. For example, the drive control unit 115 outputs control command values ​​to the motors of each joint of the arm mechanism 400. As a result, the end effector 300 moves in a plane passing through the gripping section. In the gripping section, gripping portions 314 are arranged on both sides of the object O. Furthermore, the drive control unit 105 drives the actuator 317 of the end effector. Thus, as... Figure 2 As shown, the gripping part 314 contacts the object O, and the object O is gripped. The gripping part 314 contacts the object O in the gripping section.

[0072] In this way, the section closest to a circle is used as the gripping position. Figure 2 The gripping part 314 shown clamps the object O. The end effector 300 can stably grip the object O. The end effector 300 can grip the object O with a stable position and attitude.

[0073] The coordinate system setting unit 120 sets the initial reference coordinate system for the three-dimensional model of object O. The coordinate system setting unit 120 sets the XYZ three-dimensional orthogonal coordinate system as the initial reference coordinate system. The coordinate system setting unit 120 sets the origin and the directions of the X-axis, Y-axis, and Z-axis with respect to the three-dimensional model. The X-axis, Y-axis, and Z-axis are mutually orthogonal directions.

[0074] The coordinate system setting unit 120 sets an initial reference coordinate system based on the deformation shape data from the deformation sensor 316. One axis of the coordinate system is orthogonal to the gripping section. In the gripping section, it passes through the center of the two gripping parts 314.

[0075] The shape of the object O at the gripping section is determined based on the deformed shape of the gripping part 314. In the gripping section, the object O becomes approximately circular. The coordinate system setting part 120 sets an initial reference coordinate system such that the axis of the initial reference coordinate system passes through the center or center of gravity of the object O. That is, the center (origin) of the initial reference coordinate system is aligned with the center (center of gravity) of the gripping section. For example, if the object O is rotationally symmetric, the axis of symmetry of the object O can be aligned with the axis of the initial reference coordinate system.

[0076] Then, the drive control unit 105 controls the arm mechanism 400 and the end effector 300 to place the object O within the scanning range of the three-dimensional shape measuring instrument 200. Through control based on the drive control unit 105, the arm mechanism 400 and the end effector 300 place the object O onto the stage 210. Specifically, the arm mechanism 400 moves the object O so that the axes of the initial reference coordinate system are aligned with the center of the scanning range. For example, as... Figure 1 As shown, the axis of the initial reference coordinate system becomes the central axis passing through the center or center of gravity of object O. Therefore, the drive control unit 105 controls the arm mechanism 400 and the end effector 300 to align the central axis with the rotation axis of the stage. Then, the three-dimensional shape measuring device 200 rotates the stage 210 to measure the three-dimensional shape of object O.

[0077] The center of object O can be aligned with the center of the scanning range in an orientation as close as possible to the holding posture. This reduces the number of times object O needs to be repositioned for 3D scanning using the 3D shape meter 200. For example, it prevents a portion of object O from exceeding the scanning range at any scanning angle (rotation angle) on the stage 210. Furthermore, the 3D shape meter 200 performs 3D measurements stably and efficiently. It can generate a 3D model in an appropriate coordinate system.

[0078] The measurement data acquisition unit 110 acquires the measurement data measured by the three-dimensional shape measuring device 200. As described above, the measurement data becomes point cloud data and mesh data representing the three-dimensional shape of the object. That is, the processing device 100 acquires data (three-dimensional data) representing the three-dimensional shape of the three-dimensional model. Furthermore, the measurement data is acquired in a coordinate system set by the coordinate system setting unit 120. That is, the measurement data is represented by the position coordinates (XYZ coordinates) on the XYZ axes of the initial reference coordinate system set by the coordinate system setting unit 120.

[0079] The following explanation assumes that the measurement data acquisition unit 110 acquires a mesh model representing a three-dimensional shape. Alternatively, the measurement data acquisition unit 110 may acquire point cloud data from the three-dimensional shape measuring device 200, and the processing device 100 may generate mesh data from the point cloud data. The relative positional relationship between the three-dimensional shape measuring device 200 and the placement points is known. Therefore, it is possible to associate the three-dimensional model of object O with the axes derived from the end effector 300.

[0080] (Symmetry determination)

[0081] Furthermore, the processing device 100 may optionally determine the symmetry of the three-dimensional model. Specifically, the processing device 100 determines the symmetry with respect to each axis of the reference coordinate system. And, based on the symmetry determination result, the processing device 100 corrects the axes of the reference coordinate system.

[0082] Here, in a three-dimensional model representing the three-dimensional shape of an object, it is desirable to appropriately set the coordinate axes of the coordinate system. For example, in the case where the object is rotationally symmetric, by setting coordinate axes consistent with the rotation axis, the position and orientation of the object can be appropriately changed. It is desirable to appropriately determine the symmetry of the object. In the method according to this embodiment, the symmetry of the object can be appropriately determined. In this way, a coordinate system useful for estimating the 6-axis position and orientation in the estimator can be set. Hereinafter, the configuration related to the determination of symmetry will be described.

[0083] The first determination unit 130 determines the rotational symmetry of object O. The first determination unit 130 determines whether it is rotationally symmetric about each of the three axes. For example, if object O is rotationally symmetric about the X-axis, the X-axis becomes the axis of rotational symmetry. Therefore, when the object is rotated about the X-axis by a certain angle, the shape of the object remains the same.

[0084] The second determination unit 140 determines whether the object O is rotationally symmetric with an angular period. In the case of an axis of rotational symmetry, the second determination unit 140 determines whether the rotational symmetry about that axis has an angular period. For example, if the X-axis is the axis of rotational symmetry, the second determination unit 140 determines whether the shape is rotationally symmetric with an angular period about the X-axis.

[0085] An example of a shape with rotational symmetry having an angular period in a cross-section orthogonal to the axis is a regular polygon. Specifically, when the cross-section is a regular quadrilateral, the object becomes a rotationally symmetric shape with fourth-order symmetry. For example, an example of a shape without rotational symmetry having an angular period in a cross-section orthogonal to the axis is a circle.

[0086] The third determination unit 150 determines the symmetry. If the shape of the object is symmetrical, it is a shape that is reverse symmetrical with respect to a plane. If it is reverse symmetrical with respect to a plane containing the axis, the third determination unit 150 determines that the object is symmetrical.

[0087] The correction unit 160 corrects the reference coordinate system. If the second determination unit 140 determines that the system is rotationally symmetric with an angular period, the correction unit 160 corrects the reference coordinate system. This establishes a reference coordinate system different from the initial reference coordinate system. Specifically, the correction unit 160 corrects the orientation of axes that are not axes of rotational symmetry. For example, if the X-axis is an axis of rotational symmetry, the correction unit 160 changes the orientation of the Y-axis and Z-axis. The correction unit 160 rotates the Y-axis and Z-axis around the X-axis.

[0088] Furthermore, if the third determination unit 150 determines that object O has surface symmetry, the correction unit 160 corrects the reference coordinate system. That is, it changes the direction of the axes of the reference coordinate system so that the axes are contained within the plane of symmetry. In this way, a reference coordinate system can be set with two axes parallel to the plane of symmetry as coordinate axes.

[0089] use Figure 4 The method for generating the three-dimensional model involved in this embodiment will be described. Figure 4 This is a flowchart illustrating the method for generating a three-dimensional model according to this embodiment.

[0090] First, the surface shape data acquisition unit 101 acquires the surface shape data of object O (S11). This acquires the surface shape data measured by the ranging sensor 500. Here, the surface shape data is point cloud data or mesh data representing the surface of object O. We will assume it to be mesh data for explanation.

[0091] Next, the calculation unit 102 calculates the normal direction of the surface of object O based on the surface shape data (S12). Thus, the distribution of the normal direction of the surface of object O is determined. The gripping position setting unit 103 sets the gripping position based on the normal direction (S13). Here, the calculation unit 102 sets a near-circular cross-section as the gripping position of object O, as described above.

[0092] The end effector 300 holds the object O in the holding position (S14). That is, after the arm mechanism 400 moves the end effector 300 to the holding position, the end effector 300 holds the object O.

[0093] Here, the flexible gripping part 314 grips the object O. Therefore, the gripping part 314 deforms according to the shape of the object O. Then, the deformation sensor 316 detects the deformed shape of the gripping part 314 (S15).

[0094] The coordinate system setting unit 120 sets an initial reference coordinate system based on the deformed shape of the gripping part 314 (S16). As described above, the gripping part 314 is made of a soft material, and therefore deforms along the object O being gripped. Therefore, the coordinate system setting unit 120 can set the axis direction of the initial reference coordinate system based on the deformed shape of the gripping part 314.

[0095] The end effector 300 and the arm mechanism 400 place the object O on the stage 210 (S17). Here, the arm mechanism 400 moves the object O so that it is placed within the scanning range of the three-dimensional shape measuring device 200. More specifically, the object O is positioned on the three-dimensional shape measuring device 200 such that the axis direction of the initial reference coordinate system is aligned with the scanning axis.

[0096] The three-dimensional shape measuring device 200 scans the object O (S18). The three-dimensional shape measuring device 200 scans the entire circumference of the object O to perform measurement. The measurement data acquisition unit 110 acquires the measurement data in the initial reference coordinate system (S19). That is, a three-dimensional shape model of the object O in the XYZ three-dimensional orthogonal coordinate system, which is set in accordance with the deformation shape of the holding unit 314, is obtained.

[0097] use Figure 5 The process for determining the symmetry of a 3D model is explained. Figure 5 This is a flowchart illustrating an example of the process for determining symmetry. In the following description, the symmetry of each axis relative to the initial reference coordinate system derived in step S15 is determined. Thus, the symmetry of the three-dimensional model corresponding to the measurement data measured in step S18 is determined.

[0098] The first determination unit 130 selects the axes (x, y, z) in the initial reference coordinate system (S103). That is, the first determination unit 130 selects one axis from the x-axis, y-axis, and z-axis. The first determination unit 130 determines whether the object O is rotationally symmetric about the selected axis (hereinafter also referred to as the selected axis) (S104). Rotational symmetry is a shape whose shape is the same as the original shape when rotated about the selected axis.

[0099] If object O is rotationally symmetric (S104: Yes), the second determination unit 140 determines whether the rotational symmetry has an angular period (S105). That is, the third determination unit 150 determines whether object O has a rotationally symmetric shape with an angular period. A rotationally symmetric shape with an angular period is a shape whose shape is consistent with the original shape when rotated about the selection axis by a predetermined angle (an angle greater than 0° and less than 360°). In other words, the shape is inconsistent with the original shape when rotated about the axis by an angle other than the angle represented by the angular period.

[0100] If the rotational symmetry has an angular period (S105: Yes), the correction unit 160 corrects the reference coordinate system (S106). The correction unit 160 changes the orientation of the two axes other than the selected axis in the initial reference coordinate system. This determines the orientation of the two axes in a plane orthogonal to the selected axis. Then, the process proceeds to step S109. If the rotational symmetry does not have an angular period (S105: No), the process proceeds to step S109.

[0101] If object O is not rotationally symmetric (S104: No), the third determination unit 150 determines whether object O is plane-symmetric (S107). Plane symmetry is the shape of object O when it is reversed in a plane containing the selection axis, and the shape is the same as the original shape. If object O is plane-symmetric (S107: Yes), the correction unit 160 corrects the reference coordinate system (S108). The correction unit 160 changes the orientation of the two axes other than the selection axis in the initial reference coordinate system. Specifically, it makes one of the axes other than the selection axis parallel to the plane of symmetry (reversed plane). Then, the process moves to S109.

[0102] If object O is not symmetrical (S107: No), the process proceeds to S109. Then, the first determination unit 130 determines whether the processing has ended for all axes (S109). If the processing has not ended for all axes (S109: No), the process returns to step S103, and the first determination unit 130 selects an axis. That is, for axes for which symmetry has not been determined, the processing device 100 performs the above-described processing. For example, the processing of S104 to S108 is performed in the order of z-axis, y-axis, and x-axis. If the processing has ended for all axes (S109: Yes), the processing ends.

[0103] Figure 6 This is a flowchart illustrating an example of the determination process in the first determination unit 130 and the second determination unit 140. Here, the axis selected in step S103 of the diagram will be defined as 'a'. That is, axis a is any one of the x-axis, y-axis, and z-axis in the initial reference coordinate system. In the following explanation, axis a will be defined as the z-axis, and the plane P orthogonal to axis a will be defined as the xy-plane.

[0104] The first determination unit 130 creates the object model and the cross-sectional shapes So and Sc of its convex hull in a plane P orthogonal to axis a (S301). The cross-sectional shapes So and Sc are determined based on the intersection of plane P and the object model. The cross-sectional shape So of the object becomes the shape representing the outline of the object in plane P. The convex hull in plane P becomes the smallest polygon containing the object (i.e., the cross-sectional shape So). The first determination unit 130 changes its position on axis a to generate the cross-sectional shapes So and Sc. Therefore, for an object O, the first determination unit 130 generates multiple cross-sectional shapes So and multiple cross-sectional shapes Sc.

[0105] The first determination unit 130 calculates the distance r (S302) between the intersection point of the line l and the outermost point of the cross-sectional shape Sc (or So) in plane P and the origin Po. Figure 7 The cross-sectional shape Sc and r are described. Figure 7 Plane P is shown. Here, as... Figure 7 As shown, line l is a line parallel to plane P and passes through the origin Po of plane P. Furthermore, line l takes the origin P0 as its starting point. In plane P, the angle between line l and the x-axis is θ. On line l, distance r represents the distance between the origin Po and the outer shape of the cross-section Sc. Moreover, when there are two or more intersections between line l and the cross-section Sc, the distance between the outermost intersection point and the origin Po is taken as r.

[0106] As described above, the first determination unit 130 can also use the cross-sectional shape So of the object model instead of the cross-sectional shape Sc. Therefore, in the following process, the cross-sectional shape Sc can also be replaced with the cross-sectional shape So. When using the cross-sectional shape So, the distance from the origin Po to the outline of the cross-sectional shape So on the straight line l is taken as the distance r. In the following explanation, an example of processing the cross-sectional shape Sc will be described. Therefore, as... Figure 7 As shown, the distance from the origin Po to the intersection of the line l and the cross-sectional shape Sc will be used as the distance r for explanation.

[0107] Furthermore, the first determination unit 130 changes θ to calculate the distance r. That is, the first determination unit 130 calculates r by covering an entire circle centered at the origin Po in plane P. For example, the first determination unit 130 sets θ to δ θThe intervals are varied to obtain r. Thus, the first determination unit 130 can obtain a discrete function r(θ). For example, θ takes discrete values ​​from 0° to 360°. r(θ) becomes the polar coordinate representation of the cross-sectional shape Sc. That is, r(θ) corresponds to the function obtained by performing a polar coordinate transformation on the cross-sectional shape Sc. Furthermore, the first determination unit 130 calculates r(θ) for each of the multiple cross-sectional shapes Sc generated by changing the position on axis a (in this case, the z position).

[0108] The first determination unit 130 standardizes r(θ) (S303). For example, the first determination unit 130 multiplies r(θ) by a certain coefficient so that the maximum value of r(θ) becomes 1. Thus, r(θ) falls within the range of 0 to 1. The first determination unit 130 determines whether the value of r(θ) is constant (S304). That is, the first determination unit 130 determines whether the cross-sectional shape Sc is circular. More specifically, the first determination unit 130 determines whether the shape of the cross-sectional shape Sc is circular.

[0109] When r(θ) is constant (S304: Yes), the first determination unit 130 determines that the object is circular (rotationally symmetric) (S305). When it is determined to be circular, the processing of the first determination unit 130 and the second determination unit 140 ends. That is, in Figure 5 In step S105, the determination is "No". Examples of three-dimensional objects with a circular cross-sectional shape include spheres, cylinders, cones, and rugby balls. Additionally, there are objects such as bins and bottles whose diameter changes depending on the position of axis a. These shapes are rotationally symmetric about their central axis.

[0110] If r(θ) is not constant (S304: No), the first determination unit 130 determines that the object is not circular (S306). Next, the second determination unit 140 determines whether it possesses rotational symmetry with an angular period. That is, the implementation... Figure 5 The decision processing in step S105.

[0111] The second decision unit 140 performs a discrete Fourier transform on r'(θ) (S307). Here, r'(θ) is determined based on r(θ) and the average value of r. ave It can be calculated using r'(θ) = r(θ) - r ave Let's calculate the average value r. ave This corresponds to the DC component of r. Furthermore, the Fourier series (amplitude spectrum) R(ω) obtained through the discrete Fourier transform is expressed as... Figure 8 . Figure 8 The left-hand coordinate graph represents r'(θ), and the right-hand coordinate graph represents R(ω). Furthermore, in situations such as... Figure 7When the cross-sectional shape Sc is a regular hexagon as shown, in Figure 8 In the R(w) shown, there is a peak every 60°. Additionally, in... Figure 8 In the above, a Fourier transform was performed on the real function, thus making it positive and negative symmetric.

[0112] The second determination unit 140 calculates the amplitude R based on R(ω) to obtain the maximum angular period ω. max (S308). Here, ω max Set to above 0° and below 180°. Figure 8 The figure shows that the peak is largest at R (60°), therefore, ω max It becomes 60° (=π / 3[rad]).

[0113] The second decision section 140 calculates the difference (r(θ) - r(θ + ω)). max The area E (S309) of the absolute value of )). Here, the difference represents the function r(θ) and the function r(θ) shifted by the angular period ω. max The function r(θ + ω) after max The difference value of ) is given by the area E of the absolute value. The area E can be calculated by the following equation (1).

[0114]

[0115] The area E is obtained by integrating the absolute value over the range of 0° to 360° with respect to θ. Furthermore, in equation (1), θ takes discrete values; therefore, the sum of the absolute values ​​of θ over the range of 0° to 360° constitutes the area E. The second determination unit 140 determines whether the area E is less than a threshold (S310). That is, the third determination unit 150 compares the area E with a preset threshold.

[0116] If the value is less than the threshold (S310: Yes), the second determination unit 140 determines that it has an angular period ω. max The shape is rotationally symmetric (linearly symmetric) (S311). That is, it has been displaced by an angular period ω. max If the difference in the function is small, the second determination unit 140 determines that it is a rotationally symmetric (linearly symmetric) shape with an angular period. If the difference is not less than the threshold (S310: No), the second determination unit 140 determines that it is not a rotationally symmetric (linearly symmetric) shape with an angular period (S312).

[0117] exist Figure 9 The diagram shows r(θ) and r(θ+ω). max ).exist Figure 9 In the diagram, the left side shows the coordinate graph for the non-rotationally symmetric shape, and the right side shows the coordinate graph for the rotationally symmetric case. In the non-rotationally symmetric case, r(θ) and r(θ+ω)max The difference between θ and r(θ + ω) is not zero, therefore, the area E obtained by equation (1) increases. In the case of a rotationally symmetric shape with an angular period, it becomes r(θ) and r(θ + ω). max The function is roughly the same as r(θ). Therefore, at any θ, r(θ) is approximately the same as r(θ + ω). max The difference between the two values ​​is approximately 0, and the area E is approximately 0. Therefore, the second determination unit 140 can determine whether it is a rotationally symmetric structure with an angular period by comparing the area E with a preset threshold.

[0118] Furthermore, multiple cross-sectional shapes Sc are generated by changing the position on axis a. Rotational symmetry is determined for each cross-sectional shape Sc individually. Therefore, if the value of r(θ) is constant for all cross-sectional shapes Sc, it is determined to be rotationally symmetric without angular periodicity. If the area E of more than one cross-sectional shape Sc is not less than a threshold, it is determined not to be rotationally symmetric. If the area E of some cross-sectional shapes Sc is less than the threshold, and r(θ) is constant for all other cross-sectional shapes, it is determined to be rotationally symmetric with angular periodicity.

[0119] Furthermore, even when the area E is less than the threshold for the entire cross-sectional shape Sc, when the angular period ω max At the same time, it is sometimes not rotationally symmetric with an angular period. Specifically, the angular period ω in all cross-sectional shapes Sc max When the elements are offset but not overlapped, they do not constitute rotational symmetry with an angular period. For example, in a certain cross-sectional shape, the angular period ω... max The angle period is 60°, and in the remaining cross-sectional shapes, the angle period is ω. max At a value of 72°, it does not constitute rotational symmetry with an angular period. Conversely, in a certain cross-sectional shape, the angular period ω... max The angle period is 60°, and in the remaining cross-sectional shapes, the angle period is ω. max When the angle is 30°, it becomes a rotationally symmetric system with an angular period. That is, it becomes a 6th-order rotationally symmetric system.

[0120] Furthermore, when the object possesses rotational symmetry with an angular period (S311), such as Figure 5 As in step S106, the correction unit 160 corrects the initial reference coordinate system. The correction unit 160 may also correct the axial direction of the initial reference coordinate system based on the peak surface. For example, the correction unit 160 sets the reference coordinate system such that r(θ) becomes the maximum angle θ. max The direction becomes the axial direction. The correction unit 160 determines that r becomes the maximum θ. max Here, ω max Set to 0° or higher and less than 360°.

[0121] Become r(θ) max The position of the cross-sectional shape Sc or cross-sectional shape So is called the peak position. The correction unit 160 will tilt the x-axis by an angle θ. max The straight line after that is taken as axis b. Figure 7 In, θ max The angle is 0°. Axis b becomes a straight line passing through the origin and the peak position. The plane containing the selection axis (axis a) and axis b becomes the peak surface. Furthermore, in the case where multiple r(θ) become the largest angle θ, axis b can be set based on one of its peak positions.

[0122] Following step S312, as in step S107, the third determination unit 150 performs a symmetry determination. Using Figures 10-12 An example of determining symmetry will be used to illustrate this. Figure 10 , Figure 11 This is a flowchart illustrating an example of a decision-making process. Figure 12 This is a diagram representing a symmetrical cross-sectional shape. Furthermore, in the following explanation, the cross-sectional shape Sc of the convex hull is processed, but the cross-sectional shape So of the object can also be processed. Therefore, in the following process, it is also possible to replace the cross-sectional shape Sc with the cross-sectional shape So.

[0123] The third determination section 150 is spaced at a certain angle δ. φ Draw the origin P through plane P. O The straight line L(φ) (S401). Therefore, multiple straight lines L(φ) are defined in plane P. Here, as... Figure 12 As shown, line L(φ) is a line on plane P that passes through the origin Po and forms an angle φ with respect to the x-axis. That is, angle φ in plane P is the angle between the x-axis and line L(φ). Furthermore, φ is set to a range of 0° or greater and less than 180°. Line L(φ) is defined as traversing the cross-sectional shape Sc.

[0124] The third determination unit 150 takes sampling points Qm (S402) at regular intervals δm along the straight line L (φ). This allows sampling of multiple points along the straight line (φ). The third determination unit 150 records the xy coordinates of each sampling point Qm. Figure 12 The diagram shows a portion of the sampling points Qm on the straight line L(φ). Here, only three sampling points placed on a single straight line L(φ) are shown. Furthermore, the third determination unit 150 draws multiple straight lines L(φ) by changing the angle φ. The third determination unit 150 determines multiple sampling points Qm on each straight line L(φ). In the following processing, unless otherwise specifically mentioned, the same processing is performed for each cross-sectional shape.

[0125] The third determination unit 150, for all sampling points Qm, draws an orthogonal line K(Qm) passing through Qm and orthogonal to the line L(φ) (S403). The third determination unit 150 calculates the intersection points Qmu and Qml of K(Qm) with the cross-sectional shape Sc (S404). Here, as... Figure 12 As shown, K(Qm) is a straight line that crosses the cross-sectional shape Sc of the convex hull. Therefore, K(Qm) intersects the cross-sectional shape Sc at two locations. Thus, as... Figure 11 As shown, the intersection point of K(Qm) with the positive side of the cross-sectional shape Sc is designated as Qmu, and the intersection point on the negative side is designated as Qml. In addition, when K(Qm) intersects with the cross-sectional shape Sc at 3 or more locations, the intersection point on the positive side with the outermost point is designated as Qmu, and the intersection point on the negative side with the outermost point is designated as Qml.

[0126] The third determination unit 150 calculates the distances du and dl between the intersection points Qmu and Qml from the sampling point Qm (S405). For example... Figure 12 As shown, the third determination unit 150 sets the distance between the sampling point Qm and the intersection point Qmu as du, and sets the distance between the sampling point Qm and the intersection point Qml as dl.

[0127] Next, the third determination unit 150 calculates the signed distance difference d = du - dl (S406). The third determination unit 150 calculates the average value d. ave (S407). The third determination unit 150 calculates the sum of d for multiple sampling points Qm on a straight line L(φ), and divides the sum by the number of sampling points to obtain the average value d. ave The third determination unit 150 calculates d and d ave The sum of the absolute values ​​of the differences d sum (S408). Total d sum It can be obtained using the following formula (2).

[0128]

[0129] The third decision section 150 calculates the total d. sum To become the smallest angle φ min (S409). As mentioned above, the signed distance difference d, the average d ave Total d sum These are calculated for each angle φ. That is, the third determination unit 150 performs the above processing on the line L(φ) to calculate the signed distance difference d and the average value d. ave Total d sum The third decision section addresses multiple sums d. sum Compare the results and find the smallest sum d. sumFind the angle φ of the line L(φ) at that time. The angle φ at which the cross-sectional shape Sc is closest to line symmetry (reverse symmetry) with respect to the line L(φ) is set as angle φ. min .

[0130] The third determination unit 150 calculates the angle φ. min The average value of the signed distance difference d at time d ave＿min Sum of all d sum＿min (S410). Determine the total sum d. sum＿min Is it less than the threshold (S411)? The third determination unit 150 will sum d sum＿min Compare with a pre-set threshold.

[0131] In total d sum＿min If the value is not less than the threshold (S411: No), the third determination unit 150 determines that it is not a symmetrical shape (S412). That is, with respect to axis a, the cross-sectional shape Sc (or cross-sectional shape So) is neither rotationally symmetric nor surface symmetric. Therefore, the object has no symmetry about axis a.

[0132] In total d sum＿min If the value is less than the threshold (S411: Yes), the correction unit 160 is based on φ. min and d ave＿min Let axis b be defined (S413). Specifically, axis b is inclined at an angle φ relative to the x-axis and moves away from the origin Po from d. ave＿min The straight line after. In Figure 13 The diagram shows axis b in plane P, orthogonal to the z-axis. Axis b is equivalent to shifting line L (φmin) parallel to the line by a distance d in a direction orthogonal to the line. ave＿min The straight line after that. In addition, the position of the origin Po is also moved parallel to it. The direction orthogonal to axis b and orthogonal to axis a becomes the remaining axis c. The third determination unit 150 determines that it is symmetrical with respect to the plane (reversal surface) containing axis a and axis b (S414). Thus, when the object O has surface symmetry, the correction unit 160 corrects the axis direction of the initial reference coordinate system so that it has an axis along the reversal surface.

[0133] In addition, for each cross-sectional shape Sc, Figure 10 , Figure 11 The symmetry determination process is shown. Furthermore, regarding all cross-sectional shapes Sc and the sum d sum＿min If the values ​​are less than the threshold and axis b is common, it is determined to be face symmetric. Furthermore, Figures 3 to 11 The process shown is an example of a determination process, and this embodiment is not limited to the above process.

[0134] System 1 performs the aforementioned determination process on each of the three axes of the initial reference coordinate system. That is, it determines the symmetry about each of the three axes. If the shape is determined to be rotationally symmetric about two or more axes without angular periodicity, it is determined to be a sphere. If it is rotationally symmetric about one axis and planar symmetric about another, it becomes an object with two symmetries, such as a cylinder or a rugby ball. If it is rotationally symmetric about one axis, it can be determined whether it has angular periodicity (discrete symmetry) or simply angular periodicity. Discrete symmetric cross-sectional shapes include regular polygons, star shapes, etc. Through the above algorithm, various types of symmetry can be determined.

[0135] In summary, System 1 includes an end effector 300, a deformation sensor 316, a calculation unit 102, a gripping position setting unit 103, and a coordinate system setting unit 120. The end effector 300 has a flexible gripping part 314 that grips an object O. The deformation sensor 316 detects the deformation shape of the gripping part while gripping the object. The calculation unit 102 calculates the normal direction of the object's surface based on the object's surface shape data. The gripping position setting unit 103 determines the gripping position of the object O held by the end effector 300 based on the normal direction. The coordinate system setting unit 120 sets a reference coordinate system based on the deformation shape of the gripping part 314.

[0136] In addition, the method for generating 3D models involves the following processing.

[0137] (A) The calculation unit 102 calculates the normal direction of the surface of the object based on the surface shape data of the object.

[0138] (B) The gripping position setting unit 103 sets the gripping position of object O based on the normal direction.

[0139] (C) An end effector 300 with a soft gripping part 314 grips an object in a gripping position.

[0140] (D) Deformation sensor 316 detects the deformation shape of an object when it is held.

[0141] (E) Set a reference coordinate system according to the deformed shape of the gripping part 314.

[0142] In this way, it is easy to generate a 3D model in a suitable coordinate system.

[0143] In addition, the processing device 100 can also perform the following processes (1) to (6).

[0144] (1) Obtain measurement data representing the three-dimensional shape of the object in the reference coordinate system.

[0145] (2) Select the axes of the initial reference coordinate system.

[0146] (3) Determine whether there is rotational symmetry relative to the axis.

[0147] (4) Determine whether it has rotational symmetry with an angular period relative to the axis.

[0148] (5) In the absence of the stated rotational symmetry, determine the planar symmetry with respect to the stated axis.

[0149] (6) By changing the axis selected in (2), the processing of (3) to (5) is performed on the three axes of the initial reference coordinate system respectively.

[0150] In this way, an appropriate coordinate system can be set. And, the three-dimensional model of the object can be represented using the three-dimensional coordinate system set as described above. That is, the three-dimensional model representing the three-dimensional shape of the object is represented by an appropriate coordinate system. For example, the position coordinates of the point cloud and mesh representing the three-dimensional model are represented by an appropriate coordinate system. Information related to various symmetries can be annotated. A three-dimensional shape model can be created using coordinate axes that take symmetry into account. Of course, the processing in (1) to (6) can also be omitted.

[0151] The processing device 100 derives initial reference coordinates from the 3D model. Furthermore, the processing device 100 can perform calculations efficiently by sequentially searching for rotational symmetry and surface symmetry along each axis. It can annotate information related to the coordinate system setting and symmetry of the 3D shape model of the object. It can efficiently generate a 3D model in an appropriate coordinate system.

[0152] In the case of symmetrical objects, the axes of the coordinate system define the axes of symmetry and the planes of symmetry (reversal planes). In the case of rotational symmetry with angular periodicity or surface symmetry, the correction unit 160 corrects the initial reference coordinate system. By using the axes of the corrected coordinate system, the position and orientation of the object can be appropriately changed. Learning data can be easily and appropriately generated. Furthermore, position and orientation information, etc., representing the position and orientation, can be appropriately and easily annotated onto the learning data. A coordinate system useful for estimating the 6-axis position and orientation in the estimator can be set.

[0153] Alternatively, the coordinate system setting unit 120 may derive the initial reference coordinates based on the direction of the normal to each grid of the mesh model representing the three-dimensional shape. This allows for the easy determination of an appropriate initial reference coordinate system.

[0154] The coordinate system setting unit 120 uses the cuboid containing object O as a three-dimensional bounding box. Furthermore, the coordinate system setting unit 120 derives an initial reference coordinate system with the center of the cuboid as the origin and the directions of each side of the cuboid as the x, y, and z axes.

[0155] When the object has surface symmetry, the correction unit 160 corrects the axis direction of the initial reference coordinate system so that it has an axis along the reverse surface.

[0156] When an object has rotational symmetry with an angular period, the correction unit 160 corrects the axial direction of the initial reference coordinate system based on the peak surface.

[0157] The above processing method can be implemented by computer programs and hardware. That is, the processing device 100 executes a predetermined program, functioning as a generation device or generation system. Figure 14 The diagram shows an example of the hardware configuration of the processing device 100. The processing device 100 includes a processor 10, a memory 20, and an interface 30. The memory 20 stores programs, various parameters, measurement data, etc. The processor 10 executes the program stored in the memory 20. The interface 30 transmits data to the 3D shape sensor 200, the end effector 300, the arm mechanism 400, the distance sensor 500, etc. Additionally, the interface 30 receives data from the 3D shape sensor 200, the end effector 300, the arm mechanism 400, the distance sensor 500, etc.

[0158] By executing a program through the processor 10 of the processing device 100, the method described in this embodiment can be performed. The processing device 100 may have at least one processor 10. Furthermore, the above-described processing is performed by executing a program stored in memory through one or more processors 10. The processing device 100 is not physically limited to a single device and may be distributed among multiple devices. That is, the above-described method may also be performed by distributing the processing among multiple devices.

[0159] Some or all of the above-described processes can also be executed by a computer program. That is, the control of the processing device 100 is performed by a control computer that constitutes the processing device 100 executing a program. The program includes a set of commands (or software code) for causing the computer to perform one or more functions described in the embodiments when read by the computer. The program can also be stored on a non-transitory computer-readable medium or a physical storage medium. As non-limiting examples, computer-readable media or physical storage media include random-access memory (RAM), read-only memory (ROM), flash memory, solid-state drive (SSD) or other memory technologies, CD-ROM, digital versatile disc (DVD), Blu-ray disc or other optical disc storage, magnetic tape cassette, magnetic tape, disk storage or other magnetic storage devices. The program can also be transmitted on a transient computer-readable medium or communication medium. As a non-limiting example, transient computer-readable media or communication media include electrical, optical, acoustic, or other forms of propagated signals.

[0160] The invention described above is based on the embodiments, but the invention is not limited to the above embodiments. It is self-evident that various modifications can be made without departing from its spirit.

[0161] Based on the disclosure described herein, embodiments of this disclosure can obviously be varied in many ways. These variations should not be considered as departing from the spirit and scope of this disclosure, and all such changes intended to be obvious to those skilled in the art are included within the scope of the appended claims.

Claims

1. A three-dimensional model generation system, comprising: An end effector having a flexible gripping part that holds an object; A deformation sensor that detects the deformation shape of the holding part in the state of holding the object; The calculation unit calculates the normal direction of the object's surface based on the object's surface shape data; The decision unit determines the holding position of the object held by the end effector based on the normal direction; as well as The coordinate system setting unit calculates the shape of the gripping section at the gripping position of the object based on the deformed shape of the gripping part, and sets the reference coordinate system in such a way that one axis of the reference coordinate system is orthogonal to the gripping section and the origin of the reference coordinate system is consistent with the center of the gripping section.

2. The three-dimensional model generation system according to claim 1 further comprises: A three-dimensional shape measuring device that measures the three-dimensional shape of the object by scanning it; and A drive control unit drives the end effector so that the end effector places the object within the scanning range of the three-dimensional shape measuring instrument.

3. The three-dimensional model generation system according to claim 2, The three-dimensional shape measuring device includes a depth sensor for determining the distance to the object. The orientation of the object relative to the depth sensor is changed about the axis of the reference coordinate system.

4. The three-dimensional model generation system according to claim 3, The three-dimensional shape measuring device also includes a stage for rotating the object it is mounted on. The end effector places the object in such a manner that the rotation axis of the stage is aligned with the axis of the reference coordinate system.

5. The system for generating a three-dimensional model according to any one of claims 1 to 4, It also includes a surface shape sensor that measures the surface shape data of the object before the end effector grasps it. In the surface shape data, the part with the cross-section that is most approximately circular is selected as the holding position of the end effector.

6. The system for generating a three-dimensional model according to any one of claims 1 to 4, Determine the symmetry of the object with respect to each axis of the reference coordinate system. Based on the symmetry determination result, the axes of the reference coordinate system are corrected.

7. A method for generating a three-dimensional model, comprising: Calculate the normal direction of the object's surface based on the object's surface shape data; The holding position of the object is set based on the normal direction; An end effector with a flexible gripping part grips an object at the gripping position; Detect the deformation shape of the holding part when holding the object; as well as The shape of the gripping section at the gripping position of the object is determined based on the deformed shape of the gripping part. A reference coordinate system is set in such a way that one axis of the reference coordinate system is orthogonal to the gripping section and the origin of the reference coordinate system is consistent with the center of the gripping section.

8. The method for generating a three-dimensional model according to claim 7, The end effector is driven so that it places the object within the scanning range of the three-dimensional shape meter. The three-dimensional shape of the object is measured by scanning the object with the three-dimensional shape measuring device.

9. The method for generating a three-dimensional model according to claim 8, The three-dimensional shape measuring device includes a depth sensor for determining the distance to the object. The orientation of the object relative to the depth sensor is changed about the axis of the reference coordinate system.

10. The method for generating a three-dimensional model according to claim 9, The three-dimensional shape measuring device also includes a stage for rotating the object it is mounted on. The end effector places the object in such a manner that the rotation axis of the stage is aligned with the axis of the reference coordinate system.

11. The method for generating a three-dimensional model according to any one of claims 7 to 10, Before the end effector grasps the object, a surface shape sensor measures the surface shape data of the object. In the surface shape data, the part with the cross-section that is most approximately circular is selected as the holding position of the end effector.

12. The method for generating a three-dimensional model according to any one of claims 7 to 10, Determine the symmetry of the object with respect to each axis of the reference coordinate system. Based on the symmetry determination result, the axis is corrected.

13. A computer-readable medium storing a program that causes at least one computer to perform the following steps, said steps including: The direction of the normal to the surface of the object is calculated based on the object's surface shape data. The holding position of the object is set based on the normal direction; An end effector having a flexible gripping part is controlled such that the end effector grips the object at the gripping position; Deformation shape data is obtained by detecting the deformation shape of the holding part when holding the object; as well as The shape of the gripping section at the gripping position of the object is determined based on the deformed shape of the gripping part. A reference coordinate system is set in such a way that one axis of the reference coordinate system is orthogonal to the gripping section and the origin of the reference coordinate system is consistent with the center of the gripping section.

Images