Model generation device and picking robot

Abstract

The invention comprises: a sensor (101) for acquiring scene information including information of an object workpiece for generating a model; a workpiece information acquiring unit (102) for acquiring information of the object workpiece, i.e., workpiece information, from the scene information; a reference model determining unit (103) for determining a reference model, i.e., a reference model, based on the scene information and the workpiece information; a deformation constraint condition determining unit (104) for determining conditions for deforming the reference model, i.e., deformation constraint conditions, based on a result of analyzing the workpiece shape using the scene information and the workpiece information and the reference model; and an amorphous model generating unit (106) for determining deformation parameters of the reference model within the range of the deformation constraint conditions based on the deformation constraint conditions, and deforming the reference model according to the deformation parameters.

Description

Technical Field

[0001] The present invention relates to a model generating device and a picking robot for generating a model of a target workpiece. Background Art

[0002] In the past, there existed a device that could enter a turnover box and other devices and grip a workpiece that was not aligned in a certain direction. Patent document 1 discloses the technology of a robot simulation device, that is, a model of a target workpiece is input, the scene of the target workpiece is simulated, image data is generated, the gripping position is estimated for the image data, and the removal process of the target workpiece is simulated based on the estimation result. In the case where it is difficult to remove the target workpiece, the robot simulation device described in patent document 1 adjusts the simulation parameters related to the robot action, thereby improving the ratio of the number of successful gripping of the target workpiece to the number of trial gripping actions, that is, the picking success rate.

[0003] Patent Document 1: Japanese Patent Application Publication No. 2018-144158 Summary of the invention

[0004] When a robot picks bulk workpieces, it is necessary to first identify the workpiece to be picked and know the gripping position. In order to achieve a high picking success rate in bulk picking, it is necessary to adjust the recognition parameters of the object recognition method.

[0005] However, in the robot simulation device described in Patent Document 1, only fixed-shaped workpieces for which CAD (Computer Aided Design) models of industrial parts exist can be simulated. Therefore, the robot simulation device described in Patent Document 1 has the problem that it cannot be simulated for workpieces such as food, for which the shapes of the individual products are not aligned and for which CAD models do not exist.

[0006] The present invention has been made in view of the above-mentioned situation, and an object of the present invention is to obtain a model generation device capable of generating a model of a target workpiece having a non-aligned shape and applicable to a simulation.

[0007] In order to solve the above-mentioned problems and achieve the purpose, the model generating device of the present invention is characterized in that it comprises: a sensor, which obtains scene information including information of an object workpiece used to generate a model; a workpiece information acquiring unit, which obtains information of the object workpiece, namely, workpiece information, from the scene information; a baseline model determining unit, which determines a model that serves as a baseline for expressing misaligned shapes, namely, a baseline model, based on the scene information and the workpiece information; a deformation constraint condition determining unit, which determines conditions for deforming the baseline model, namely, deformation constraint conditions, based on the results and the baseline model obtained by analyzing the workpiece shape using the scene information and the workpiece information; and an amorphous model generating unit, which determines deformation parameters of the baseline model within the range of the deformation constraint conditions based on the deformation constraint conditions, and deforms the baseline model according to the deformation parameters.

[0008] Effects of the Invention

[0009] According to the present invention, the model generation device has an effect of being able to generate a model of a target workpiece having a shape that is not aligned and can be applied to a simulation. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a block diagram showing a configuration example of a model generation device according to the first embodiment.

[0011] Figure 2 This is a first flowchart showing the operation of the model generation device according to the first embodiment.

[0012] Figure 3 This is a second flowchart showing the operation of the model generation device according to the first embodiment.

[0013] Figure 4 This is a diagram showing an example in which the processing circuit included in the model generation device according to the first embodiment is configured by a processor and a memory.

[0014] Figure 5 This is a diagram showing an example in which the processing circuit included in the model generation device according to the first embodiment is configured by dedicated hardware.

[0015] Figure 6 This is a block diagram showing a structural example of a reference model determination unit involved in Implementation Example 2.

[0016] Figure 7 This is a flowchart showing the operation of the reference model determination unit according to the second embodiment.

[0017] Figure 8 This is a block diagram showing a configuration example of a picking robot according to the third embodiment.

[0018] Fig. 9This is a block diagram showing a configuration example of a simulation condition setting unit according to the third embodiment.

[0019] Fig.10 This is a block diagram showing a configuration example of a data set generating unit according to the third embodiment.

[0020] Fig.11 This is a block diagram showing a configuration example of a parameter adjustment unit according to the third embodiment.

[0021] Fig.12 This is a block diagram showing a configuration example of a gripping parameter adjustment unit according to the third embodiment.

[0022] Fig.13 This is a flowchart showing the operation of the gripping parameter adjustment unit according to the third embodiment.

[0023] Fig.14 This is a block diagram showing a configuration example of an identification parameter adjustment unit according to the third embodiment.

[0024] Fig.15 This is a flowchart showing the operation of the recognition parameter adjustment unit according to the third embodiment. DETAILED DESCRIPTION

[0025] Hereinafter, a model generation device and a picking robot according to an embodiment of the present invention will be described in detail based on the drawings. In addition, the present invention is not limited to this embodiment.

[0026] Implementation method 1.

[0027] Figure 1 1 is a block diagram showing a structural example of a model generation device 10 involved in Embodiment 1 of the present invention. The model generation device 10 generates a model of an object workpiece, which is a shape that is not aligned and can be applied to simulation. The model of the object workpiece generated by the model generation device 10 can be a model of a 3-dimensional object workpiece or a model of a 2-dimensional object workpiece. In the following, a method for generating a 3-dimensional model of an object workpiece by the model generation device 10 is specifically described. In addition, the model of the object workpiece is sometimes simply referred to as a model. When the model generation device 10 generates a 2-dimensional model, it is possible to realize a 2-dimensional model by a dimensional compression method such as projecting the 3-dimensional model onto a plane. In addition, the object workpiece for which the model generation device 10 can generate a model is not limited to an object workpiece whose shapes are not aligned, that is, an amorphous workpiece, but also includes an object workpiece whose shapes are aligned, that is, a fixed-shaped workpiece.

[0028] like Figure 1As shown, the model generation device 10 includes a sensor 101, a workpiece information acquisition unit 102, a reference model determination unit 103, a deformation restriction condition determination unit 104, a storage unit 105, and an indefinite model generation unit 106. The details of each structure will be described together with the operation of the model generation device 10. Figure 2 This is a first flowchart showing the operation of the model generation device 10 according to the first embodiment. Figure 2 The flowchart shows the operations from the sensor 101 to the deformation restriction condition determination unit 104 . Figure 3 This is a second flowchart showing the operation of the model generation device 10 according to the first embodiment. Figure 3 The flowchart shows the operation of the indefinite shape model generating unit 106.

[0029] In this embodiment, the sensor 101 measures the target workpiece that is the object of the generated model (step S101) and obtains the scene information of the target workpiece. The scene information is information including the information of the target workpiece that is the object of the generated model, that is, the workpiece information and the background of the target workpiece. The workpiece information is information indicating the shape, color, etc. related to the target workpiece. The sensor 101 has, for example, a 2D (Dimensions) camera, a 3D sensor, etc. The sensor 101 can use a 2D camera to obtain 2D data related to a grayscale image or a color image, and can also use a 3D sensor to obtain 3D data related to a 3D point group or a distance image. The data about the scene information can be measurement data or image data. In the following description, 2D is sometimes referred to as 2D, and 3D is sometimes referred to as 3D.

[0030] The workpiece information acquisition unit 102 acquires information of the target workpiece to be generated as a model, that is, workpiece information, from the scene information acquired by the sensor 101 (step S102). The workpiece information acquisition unit 102 can acquire information such as a size range and a color represented by RGB (Red Green Blue) from the data acquired by the sensor 101. Regarding the method of acquiring the workpiece information, for example, one target workpiece is arranged on a rotating table rotating on one axis, and the target workpiece is rotated by the rotating table while being measured by the sensor 101. The data does not need to be data corresponding to the entire circumference of the target workpiece, and the measurement by the sensor 101 may be performed only once. The workpiece information acquisition unit 102 deletes the background and the like that are not necessary for model generation from the acquired data, and acquires only the data of the target workpiece. The workpiece information acquisition unit 102 calculates the size range of the target workpiece, the average color information, and the like based on the acquired data of the target workpiece. The size range of the target workpiece is, for example, a range determined by each coordinate of the X coordinate, Y coordinate, and Z coordinate of the three-dimensional point group.

[0031] The reference model determination unit 103 determines a model that serves as a reference for representing misaligned shapes, i.e., a reference model, based on the scene information obtained by the sensor 101 and the workpiece information obtained by the workpiece information acquisition unit 102 (step S103). The reference model determined by the reference model determination unit 103 can be a primitive shape such as a rectangular parallelepiped, a sphere, a triangular pyramid or other geometric shape, or a model created by a user, or a combination of geometric shapes or models created by a user. The representation method of the model is a set of points with information on three-dimensional coordinates, that is, it can be a three-dimensional point group, a mesh model, a numerical value that can represent a geometric shape, or a combination thereof. The representation method of the reference model is not limited to the contents described in this embodiment, and other representation methods can be used. Here, regarding the determination method of the reference model, two methods are considered: automatic determination and manual determination.

[0032] In the method for determining a reference model by automatic determination, the reference model determination unit 103 selects one scene information from a plurality of scene information obtained by the sensor 101, for example, and calculates the center of gravity position of the target workpiece represented by the workpiece information contained in the scene information. The reference model determination unit 103 uses the calculated center of gravity position as the center point of the reference model and generates several different reference models based on the center point. The reference model determination unit 103 determines the size of the generated reference model based on the workpiece information obtained by the workpiece information acquisition unit 102. The reference model determination unit 103 can calculate the distribution of normal lines of the measurement data and the reference model, and automatically determine the reference model that is closest to the normal line distribution of the measurement data as the reference model for representing the misaligned shape of the target workpiece. The method for determining a reference model by automatic determination as described above can be expressed by the following formula (1). That is, the reference model determination unit 103 executes the following formula (1) when determining the reference model by automatic determination. In addition, in equation (1), M is the measurement data, P is the reference model, g(M) is the normal distribution of the measurement data, and g(P) is the normal distribution of the reference model.

[0033] [Formula 1]

[0034]

[0035] In the method of determining the reference model by manual determination, the user can determine the geometric shape that is most similar to the target workpiece.

[0036] The deformation constraint determination unit 104 determines the conditions for deforming the reference model, i.e., the deformation constraint, based on the result obtained by analyzing the workpiece shape using the scene information and the workpiece information and the reference model (step S104). The deformation constraint determination unit 104 stores the determined deformation constraint in the storage unit 105. Here, in the present embodiment, the deformation constraint determination unit 104 defines what kind of deformation the reference model is to undergo as parameterization. In order to make the shapes of the various models not aligned, the deformation constraint determination unit 104 sets the deformation parameters to different values ​​for each model. In order to deform the reference model and reproduce the features of a shape close to the target workpiece, it is necessary to set constraints according to the degree of deformation by appropriately determining the range of values ​​of each deformation parameter. Therefore, the deformation constraint determination unit 104 determines the range of values ​​that can be obtained for the deformation parameters.

[0037] As deformation parameters representing the shape of the target workpiece, for example, the size of the target workpiece, the deformation amount of the reference model when the reference model is deformed, the curvature, the frequency of occurrence of convexoconcavity, the range of occurrence of convexoconcavity, etc. are considered. The deformation amount is, for example, the amount of movement of mesh vertices in the case of a mesh model. That is, the deformation restriction condition includes at least one of the frequency of occurrence of convexoconcavity of the target workpiece, the range of occurrence of convexoconcavity of the target workpiece, and the curvature as a numerical condition. The deformation restriction condition determination unit 104 can improve the reproducibility of the model by using numerical conditions such as the frequency of occurrence of convexoconcavity.

[0038] The deformation constraint determination unit 104 determines the deformation constraint based on the scene information obtained by the sensor 101, the workpiece information obtained by the workpiece information acquisition unit 102, and the reference model determined by the reference model determination unit 103. For example, the deformation constraint determination unit 104 assigns constraints such as assuming that the vectors formed by the mesh vertices of the reference model and the center point of the reference model are the movement range of the mesh vertices of the reference model with respect to the movement amount of the mesh vertices of the reference model. In this way, the deformation constraint determination unit 104 can prevent model generation failures such as intersections of other meshes or inversion of the front and back faces of meshes when the mesh vertices are moved. The deformation constraint determination unit 104 can determine the size range of the target workpiece based on the workpiece information obtained by the workpiece information acquisition unit 102. The deformation restriction condition determination unit 104 can perform normal estimation on the scene information obtained by the sensor 101 regarding the occurrence frequency of convexity and concavity. If the distribution of the normal estimation is within a predetermined threshold, the occurrence frequency of convexity and concavity is set as one of the deformation parameters to determine the range of the occurrence frequency of convexity and concavity.

[0039] The deformation constraint determination unit 104 can extract and utilize the contour of the workpiece for the analysis of the workpiece shape. That is, the workpiece shape of the target workpiece analyzed by the deformation constraint determination unit 104 includes the contour of the target workpiece. As a method for extracting the contour, the following method is considered, that is, the deformation constraint determination unit 104 sets an arbitrary section for the measurement data of the workpiece, for example, and searches for the measurement data contained within a certain distance from the set section. With respect to the measurement data of the workpiece, the contour is not limited to one. For example, if it is the aforementioned method, the contour is obtained for each position and posture set by the section. As a method for analyzing the workpiece contour, it is considered that the deformation constraint determination unit 104 performs curve fitting on the workpiece contour, and determines the range of the frequency of occurrence of convex and concave based on the result of the curve fitting. The deformation constraint determination unit 104 analyzes the information of the workpiece contour of the target workpiece, thereby easily determining the deformation constraint corresponding to the misalignment of the shape of the target workpiece. As a result, the model generation device 10 can improve the reproducibility of the model.

[0040] The deformation constraint determination unit 104 can automatically obtain shape features by a learning method using a neural network instead of manually designing shape features such as curves obtained from curvature, polynomial approximation, etc. The deformation constraint determination unit 104 can consider, for example, taking data obtained by imaging the normal distribution obtained from the measurement data of the target workpiece as input information, learning a neural network that classifies the target workpiece and workpieces other than the target workpiece, and using the feature map of the intermediate layer of the learned neural network as the shape feature. As described above, the deformation constraint determination unit 104 can use the feature map of the intermediate layer of the neural network obtained by learning the shape of the target workpiece as a deformation parameter. The model generation device 10 can obtain deformation parameters suitable for the target workpiece through machine learning, so it can reproduce complex workpiece shapes with high accuracy. Therefore, in the deformation constraint determination unit 104, it is expected that a general and highly expressive deformation constraint can be obtained compared with manually designing shape features. The structure of the neural network is not limited, and the learning object is not limited to the class classification problem. However, it is preferable that the network structure and learning object are such that the feature map of the intermediate layer can well express the characteristics of the workpiece shape.

[0041] The storage unit 105 stores the deformation restriction conditions which are the ranges of deformation parameters determined by the deformation restriction condition determination unit 104 .

[0042] The indefinite model generation unit 106 reads the deformation constraint condition from the storage unit 105, determines the deformation parameter of the reference model within the range of the deformation constraint condition based on the read deformation constraint condition (step S105), and deforms the reference model according to the deformation parameter (step S106). As described above, in the model generation device 10, it is assumed that a plurality of indefinite models are generated. Therefore, the indefinite model generation unit 106 receives the designation of the number of models to be generated from the user. When the indefinite model generation unit 106 fails to generate the number of models designated by the user (step S107: No), it returns to step S105 and repeats the above-mentioned operation.

[0043] When the indefinite shape model generating unit 106 generates the number of models designated by the user (step S107: Yes), the operation ends.

[0044] Specifically, the indefinite model generation unit 106 determines the deformation parameters based on the deformation constraint determined by the deformation constraint determination unit 104. For example, when the output model is a mesh model, the indefinite model generation unit 106 can search for the closest vector in each measurement data for the movement amount of the mesh vertex of the reference model, and determine the intersection between the normal line descending from the measurement data to the vector and the vector as the movement target of the mesh vertex of the reference model. Since the indefinite model generation unit 106 expresses the number of meshes of the reference model used in the curvature, the number of meshes of the reference model can be determined by increasing the number of meshes for the part with high density of point groups in the measurement data. The indefinite model generation unit 106 can perform normal line estimation for the measurement data in the occurrence frequency of convexity and concavity, and determine the occurrence frequency of convexity and concavity based on the distribution of normal lines. The reason is that the wider the range of normal line distribution, the higher the occurrence frequency of convexity and concavity is considered. The indefinite model generation unit 106 may estimate the convexoconcave portion in the convexoconcave appearance range based on the normal distribution estimated for the measurement data, and calculate the convexoconcave range at the estimated portion.

[0045] When calculating the value of each deformation parameter, the indefinite model generation unit 106 randomly changes the value so that each model has a different shape. However, it is necessary to make the changed parameter value fall within the range of the deformation constraint condition. After the deformation parameter is determined, the indefinite model generation unit 106 deforms the reference model based on the determined deformation parameter.

[0046] The amorphous model generation unit 106 can use the contour information of the target workpiece extracted from the scene information when moving the mesh vertices. The amorphous model generation unit 106, for example, replaces all or part of the contour of the reference model with the contour information of the target workpiece, and sets the moving target of the mesh vertices to a 3D point on the contour with the smallest distance between the mesh vertices. The contour information of the target workpiece is, for example, a group of 3D points constituting the contour. As the replaced contour information, for example, it is considered that instead of directly using the workpiece contour selected by randomly selecting the workpiece contour extracted from different workpieces, contour information is used, which is the contour information generated by synthesizing different workpiece contours used to generate other contour information by scaling the contour in the range of the deformation restriction condition relative to the convexity or the longitudinal direction of the contour.

[0047] That is, the amorphous model generation unit 106 can use, as information on the workpiece contour, a workpiece contour generated by synthesizing two or more different workpiece contours. Thus, the amorphous model generation unit 106 can increase the variation of the shape of the generated model by synthesizing the contour information of the target workpiece, i.e., the actual workpiece, to generate a new contour. In addition, the amorphous model generation unit 106 can use, as information on the workpiece contour, a workpiece contour generated by enlarging or reducing the scale of the workpiece contour. Thus, the amorphous model generation unit 106 can increase the variation of the shape of the generated model by further increasing the variation of the contour used. In addition, when determining the deformation amount of the reference model, the amorphous model generation unit 106 can expect to generate a more natural workpiece model by effectively using the information on the workpiece contour actually obtained from the workpiece. Thus, the amorphous model generation unit 106 can generate a model that is more like the target workpiece by using the information on the workpiece contour of the target workpiece, i.e., the actual workpiece.

[0048] In addition, in the model generation device 10, the amorphous model generation unit 106 can be replaced by a model generation method implemented by a neural network. The neural network is configured to take at least one of the scene information, workpiece information, deformation restriction conditions, and deformation parameters as input, and output the model of the target workpiece in the specified expression method. For example, if the neural network is a 2D model such as an image, a random initial value is taken as input and a contour image is output. The input value can be one or a vector with multiple values. In the above case, the correct data of the contour model of the workpiece associated with each value group as the input is prepared in advance as a learning data set for the neural network, and the trained neural network is obtained by learning their transformation.

[0049] Regarding neural networks, for example, when it is desired to output a 3D model, a method of constructing a network in such a way that the output portion of the network corresponds to the voxel space is considered. As a framework for realizing the above-mentioned functions, GAN (Generative Adversarial Networks) and the like have been proposed. In addition, the aforementioned network structure, the content of the learning data set, etc. are an example and are not limited to these, and other network structures, the content of the learning data set, etc. may also be used. The model generation device 10 generates a model by utilizing the high expressive power involved in the neural network, so that versatility can be expected compared to the model generation algorithm constructed by hand, and the application range of the workpiece can be expanded.

[0050] Next, the hardware structure of the model generation device 10 is described. In the model generation device 10, the sensor 101 is a measuring device such as a camera or a laser. The storage unit 105 is a memory. The workpiece information acquisition unit 102, the reference model determination unit 103, the deformation restriction condition determination unit 104, and the amorphous model generation unit 106 are implemented by a processing circuit. The processing circuit can be a processor and a memory that executes a program stored in the memory, or it can be dedicated hardware.

[0051] Figure 4 This is a diagram showing an example in which the processing circuit of the model generation device 10 involved in the first embodiment is composed of a processor and a memory. In the case where the processing circuit is composed of a processor 91 and a memory 92, each function of the processing circuit of the model generation device 10 is implemented by software, firmware, or a combination of software and firmware. The software or firmware is described as a program and stored in the memory 92. In the processing circuit, each function is implemented by the processor 91 reading and executing the program stored in the memory 92. That is, the processing circuit has a memory 92, and the memory 92 is used to store the program that finally executes the processing of the model generation device 10. In addition, these programs can be said to cause the computer to execute the procedures and methods of the model generation device 10.

[0052] Here, the processor 91 may be a CPU (Central Processing Unit), a processing device, an arithmetic device, a microprocessor, a microcomputer, or a DSP (Digital Signal Processor), etc. In addition, the memory 92 may be a non-volatile or volatile semiconductor memory such as a RAM (Random Access Memory), a ROM (Read Only Memory), a flash memory, an EPROM (Erasable Programmable ROM), an EEPROM (registered trademark) (Electrically EPROM), a magnetic disk, a floppy disk, an optical disk, a compact disk, a mini disk, or a DVD (Digital Versatile Disc), etc.

[0053] Figure 5 1 is a diagram showing an example in which the processing circuit of the model generation device 10 according to the first embodiment is configured by dedicated hardware. Figure 5 The processing circuit 93 shown is, for example, a single circuit, a composite circuit, a programmed processor, a parallel programmed processor, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or a combination thereof. Each function of the model generation device 10 may be implemented by the processing circuit 93 individually, or each function may be integrated and implemented by the processing circuit 93.

[0054] In addition, each function of the model generation device 10 may be partially implemented by dedicated hardware and partially implemented by software or firmware. As described above, the processing circuit can implement each function described above by dedicated hardware, software, firmware, or a combination thereof.

[0055] As described above, according to the present embodiment, the model generation device 10 measures the target workpiece through the sensor 101, and determines the reference model based on the information obtained by the measurement through the sensor 101. The model generation device 10 determines the deformation restriction conditions for the reference model, determines the deformation parameters of the reference model within the range of the deformation restriction conditions, and deforms the reference model according to the deformation parameters. Thus, the model generation device 10 can generate a model even if the target workpiece is an irregular workpiece. Therefore, the model generation device 10 can be effectively used in food factories that frequently handle irregular workpieces, etc. By effectively using the model generated by the model generation device 10 through simulation, it is possible to reduce the time and cost required for on-site introduction of industrial robots for the purpose of saving manpower and verification until operation.

[0056] According to the present embodiment, the model generation device 10 obtains some measurement data of the target workpiece, and deforms the reference model based on the deformation restriction conditions of the target workpiece determined according to the analysis results of the measurement data, so that even if the target workpiece is an irregular workpiece, a model can be generated. The model generation device 10 can improve the reproducibility of the model by analyzing the numerical conditions such as the frequency of occurrence of convex and concave and the contour information of the target workpiece. In addition, the model generation device 10 can generate a model that is more like the target workpiece by using the contour information of the target workpiece, that is, the actual workpiece. The model generation device 10 can increase the variation of the shape of the generated model by synthesizing the contour information of the target workpiece, that is, the actual workpiece to generate a new contour. The model generation device 10 can increase the variation of the shape of the generated model by further increasing the variation of the contour used. In addition, the model generation device 10 can obtain deformation parameters suitable for the target workpiece through machine learning, so that a complex workpiece shape can be reproduced with high accuracy. In the model generation by the neural network, the model generation device 10 generates a model with high expression ability, so it can be expected to be more versatile than the model generation algorithm constructed by hand, and the application range of the workpiece can be expanded.

[0057] Implementation method 2.

[0058] In the second embodiment, the specific structure and operation of the reference model determination unit 103 included in the model generation device 10 will be described.

[0059] Figure 6 1 is a block diagram showing a configuration example of the reference model determination unit 103 according to the second embodiment. Figure 6 As shown, the reference model determination unit 103 includes an individual model determination unit 201, a registration unit 202, an average shape generation unit 203, a shape element segmentation unit 204, an individual difference category determination unit 205, a workpiece main axis determination unit 206, a primitive comparison unit 207, and a fitting model determination unit 208. The details of each structure are described together with the operation of the reference model determination unit 103. Figure 7 This is a flowchart showing the operation of the reference model determination unit 103 according to the second embodiment. Figure 7 The flowchart of FIG. 1 shows the operation of the reference model determination unit 103, that is, Figure 2 The details of the operation of step S103 in the flowchart.

[0060] The individual model determination unit 201 models each workpiece information acquired by the workpiece information acquisition unit 102, and determines an individual model obtained by modeling each workpiece information (step S201). An individual model is a model obtained by modeling each of the plurality of workpiece information acquired by the workpiece information acquisition unit 102. Specifically, the individual model determination unit 201 deletes the background that is not necessary for model generation for the data acquired by the sensor 101, and extracts only the data of the target workpiece. The individual model determination unit 201 may generate a model by deforming the geometric shape relative to the acquired data of the target workpiece, or may directly determine the acquired data of the target workpiece as an individual model. When generating a model, the individual model determination unit 201 may automatically determine the geometric shape that is most similar to the measurement data of the target workpiece based on the normal distribution, similarly to the determination of the reference model described in the first embodiment, or may allow the user to manually determine the geometric shape that is most similar to the target workpiece.

[0061] The registration unit 202 automatically integrates the plurality of individual models determined by the individual model determination unit 201 (step S202). The integration is, for example, alignment of the plurality of individual models. The alignment method used may be ICP (Iterative Closest Points) or an alignment method based on feature points.

[0062] The average shape generation unit 203 generates an average shape model of the target workpiece based on the integrated result obtained by the registration unit 202 (step S203). The average shape model is a model obtained by averaging the features of multiple individual models integrated by the registration unit 202. For example, when the output model is a mesh model, the average shape generation unit 203 can calculate the average value of the mesh vertices and the average value of the normals based on the aligned individual models, and generate a model composed of them as an average shape model. For example, when the output model is a 3D point group, the average shape generation unit 203 can calculate the average value of the 3D point group of each individual model after alignment, and generate a model composed of them as an average shape model. The reference model determination unit 103 can easily determine the reference model by using the average shape model of the target workpiece, thereby improving the reproducibility of the model.

[0063] The shape element segmentation unit 204 segments the average shape model generated by the average shape generation unit 203 or the individual model determined by the individual model determination unit 201 into at least one or more shape elements constituting each model (step S204). The shape element segmentation unit 204 performs clustering processing on the average shape model or the individual model, thereby segmenting each shape element constituting the average shape model or the individual model. In the shape element segmentation unit 204, if the input data is a 3D point group, it is considered to cluster the 3D point group for each shape element by using the k-means method.

[0064] The individual difference category determination unit 205 determines an individual difference category that indicates the misalignment degree of the workpiece shape by level for the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204 (step S205). Specifically, the individual difference category determination unit 205 determines the individual difference category for the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204.

[0065] Even in an object with shape fluctuations for each individual, the degree of fluctuation, the tendency of fluctuation, etc. are different for each object. For example, although the general tendency of the shape of unprocessed fruits is consistent, the size, shape, etc. are different according to the individual. Therefore, it is difficult to apply the object detection method involved in template matching on 2D images used in the past. In addition, paste-like snack products such as cylindrical fish cakes and processed foods are heat-treated after being formed by a mold. Therefore, the color of the surface is slightly different due to the fluctuation of the degree of grilling, or there are differences in the fine convexities and concavities on the surface, but the size itself can be regarded as roughly the same. In addition, fried foods such as fried foods wrapped in batter are fried foods in which the shape of the batter is different for each individual, or if it is the ingredients inside, specifically if it is fried foods, the shape of the chicken is not aligned. Therefore, there are also ingredients with irregular shapes. In order to generate a model that is more like the object workpiece, the fluctuation of the shape of the workpiece is classified into several categories according to the nature of the workpiece, that is, classified into individual difference categories, and it is effective to apply the model generation method existing in each classification.

[0066] The individual difference category determination unit 205, for example, determines the individual difference category by calculating the normal distribution of the average shape model and comparing the normal distribution of the average shape model with the normal distribution of each workpiece information extracted from the scene information. The individual difference category determination unit 205 classifies workpieces with large differences into the category of irregular shapes and classifies workpieces with small differences into the category of regular shapes.

[0067] The workpiece main axis determination unit 206 determines the main axis of inertia, i.e., the workpiece main axis, based on the result obtained by the individual difference category determination unit 205 for the average shape model generated by the average shape generation unit 203 or the individual model determined by the individual model determination unit 201 (step S206). Specifically, the workpiece main axis determination unit 206 determines the workpiece main axis by targeting the average shape model generated by the average shape generation unit 203 and divided into one or more shape elements by the shape element division unit 204 or the individual model determined by the individual model determination unit 201 and divided into one or more shape elements by the shape element division unit 204. In the case of a category in which the shape has no regularity, the workpiece main axis determination unit 206 can determine that there is no workpiece main axis because the workpiece main axis cannot be obtained.

[0068] The primitive comparison unit 207 performs primitive comparison of the average shape model generated by the average shape generation unit 203 with primitive shapes such as a rectangular parallelepiped, a sphere, a triangular pyramid, and other geometric shapes (step S207). Specifically, the primitive comparison unit 207 sets the average shape model generated by the average shape generation unit 203 and divided into more than or equal to one shape element by the shape element division unit 204 as an object and performs primitive comparison. The primitive comparison unit 207 can calculate the similarity based on the normal distribution of the average shape model and the normal distribution of the geometric shape. As a method for calculating the similarity, the primitive comparison unit 207 can, for example, calculate the difference between the normal distribution of the average shape model and the normal distribution of the geometric shape, prepare in advance a lookup table associated with the similarity corresponding to the difference in the normal distribution, and calculate the similarity based on the calculated difference in the normal distribution and the lookup table.

[0069] The fitting model determination unit 208 determines a reference model, i.e., a fitting model, based on the comparison result of the primitive comparison obtained by the primitive comparison unit 207 (step S208). The fitting model determination unit 208 may regard that a fitting model cannot be determined and determine the average shape model as the fitting model when the calculated similarity is less than or equal to a predetermined threshold. The fitting model determination unit 208 may determine the geometric shape with the greatest similarity as the fitting model when the similarity is greater than or equal to the threshold. The reference model determination unit 103 can determine a reference model that is close to the target workpiece, thereby improving the reproducibility of the generated model. The fitting model determination unit 208 can determine a reference model, i.e., a fitting model, for each shape element segmented by the shape element segmentation unit 204. The model generation device 10 can generate a model of a target workpiece such as skewered food connected by multiple objects by determining a fitting model for each element constituting the segmented target workpiece.

[0070] In the model generation device 10, the deformation constraint determination unit 104 determines the deformation constraint of the contour based on the numerical conditions and the shape analysis result of the reference model (step S104). The model generation device 10 can generate a model based on the acquired contour information even when the contour shape of the target workpiece is complex. The deformation constraint determination unit 104 can include the contour information of the average shape model in the acquisition of the contour information. In addition, the deformation constraint determination unit 104 can set the contour information of the average shape model as a reference when determining the deformation constraint. For example, there are methods such as setting the height of the convex and concave to the contour of the average shape model instead of the model center, or standardizing the height of the convex and concave by the height of the contour of the average shape model. When the shape of the workpiece is complex, it is difficult for the deformation constraint determination unit 104 to fit the shape of the workpiece by a simple curve, but it is standardized by the average shape model, so that it is expected to be approximated by a simple sine wave, polynomial, etc.

[0071] In the model generation device 10 , the operation of other components is the same as that of the first embodiment.

[0072] As described above, according to the present embodiment, the model generation device 10 determines the reference model based on the results of the individual difference category and the average shape model, thereby being able to improve the reproducibility of the model in accordance with the individual difference of the target workpiece. Thus, by using the model generated by the model generation device 10, the reproducibility can also be improved in the simulation process before the introduction of the industrial robot, which has the effect of easily and accurately performing the robot configuration or action confirmation for the picking operation. In addition, the model generation device 10 can generate a model of the target workpiece connected by multiple objects, thereby, for example, being able to generate models of meatballs, skewered ingredients, etc. that are processed at the food site, and the application range of the workpiece that becomes the object of model generation is expanded.

[0073] According to the present embodiment, the model generation device 10 uses the average shape model generated by the average shape generation unit 203 as a reference model. Thus, the model generation device 10 can easily determine the reference model by the average shape model, and even in the case of a workpiece that is difficult to express by a geometric shape such as a cuboid, a sphere, or a triangular pyramid, by using the average shape model as a reference model, a model of the target workpiece can be generated. In addition, the model generation device 10 can generate a model based on the acquired contour information even when the contour shape of the target workpiece is complex. In addition, the model generation device 10 determines the reference model based on the workpiece main axis and the average shape model, so that the individual difference category of the target workpiece can be easily and accurately determined based on the workpiece main axis. Thus, the model generation device 10 can determine a reference model close to the target workpiece according to the individual difference of each target workpiece, thereby improving the reproducibility of the model. In addition, the model generation device 10 determines a fitting model for each element constituting the segmented target workpiece, so that a model of a target workpiece connected by multiple objects such as skewered food can also be generated.

[0074] Implementation method 3.

[0075] In the third embodiment, a picking robot including the model generation device 10 described in the first or second embodiment will be described.

[0076] Figure 8 This is a block diagram showing a structural example of a picking robot 20 involved in Embodiment 3. The picking robot 20 has a model generation device 10, a simulation condition setting unit 107, a scene generation unit 108, a data set generation unit 109, a parameter adjustment unit 110, a recognition processing unit 111, and a picking unit 112. The model generation device 10 may be any structure of Embodiment 1 or Embodiment 2.

[0077] The simulation condition setting unit 107 reads the deformation restriction condition from the storage unit 105, generates a model of the target workpiece based on the deformation restriction condition by the indefinite model generating unit 106, and sets the simulation condition including at least one of the sensor information and the workpiece information in the simulation using the generated model. The deformation restriction condition stored in the storage unit 105 is determined by the model generating device 10 as described above. The indefinite model generating unit 106 can also generate the model of the target workpiece by controlling other structures. Fig. 9 1 is a block diagram showing a configuration example of the simulation condition setting unit 107 according to the third embodiment. Fig. 9 As shown, the simulation condition setting unit 107 includes a sensor information setting unit 301 , a workpiece information setting unit 302 , and an environment information setting unit 303 .

[0078] The sensor information setting unit 301 sets at least one of the specifications and setting information of the sensor 101 in the simulation. The sensor information setting unit 301 can set parameters related to the specifications of the sensor 101 or the sensor setting information in the simulation, such as the field of view angle, working distance, setting angle of the sensor 101, and resolution of the sensor 101. The sensor information setting unit 301 can generate at least one model of the target workpiece by the amorphous model generating unit 106 based on the deformation restriction conditions stored in the storage unit 105, and use the generated model to confirm in advance whether the target workpiece is within the field of view of the sensor 101 in the simulation.

[0079] The workpiece information setting unit 302 sets the workpiece information of the target workpiece. The workpiece information setting unit 302 may generate at least one model of the target workpiece by the indefinite model generating unit 106 based on the deformation restriction condition stored in the storage unit 105, and use the generated model to set the color information and size of the target workpiece, the number of models of the target workpiece used in the scene generating unit 108, and the like.

[0080] The environment information setting unit 303 sets the environment information in the simulation. The environment information setting unit 303 can set the light source environment such as diffuse reflection or ambient light, the box size for setting to a bulk state, or the supply method of the workpiece. As the supply method of the workpiece, a bulk state in which the position and posture of the target workpiece are completely irregular, a line state in which the configuration position and posture of the target workpiece are determined, etc. can be considered. The environment information setting unit 303 can generate a model of the target workpiece by the amorphous model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and use the generated model to confirm the supply state in the simulation.

[0081] The simulation condition setting unit 107 sets the simulation conditions of the sensor information, workpiece information and environment information using the sensor information setting unit 301, the workpiece information setting unit 302 and the environment information setting unit 303. The picking robot 20 simulates the actual environment of the user on the simulator, and it is easy to adjust the parameters corresponding to the actual environment of the user, thereby improving the picking success rate.

[0082] The scene generation unit 108 generates a model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction conditions stored in the storage unit 105, and simulates the scene of the target workpiece using the generated model and the simulation conditions set by the simulation condition setting unit 107. The scene information obtained by the measurement of the sensor 101 includes information on the background of the target workpiece, whereas the scene of the target workpiece simulated by the scene generation unit 108 only uses the target workpiece as the object. When the scene generation unit 108 simulates the scene, the generated models do not need to be different, and the same generated model may be used. For example, when the scene generation unit 108 generates a scene using 50 models, the variation of the generated model of the target workpiece is set to 10, and each type is copied in units of 5, and the total number of models used for scene generation is set to 50. In the case where the calculation cost of model generation is large, it is considered to be effective in reducing the calculation cost.

[0083] The data set generation unit 109 generates a data set for adjusting parameters for controlling the picking operation of the target workpiece based on the scenario generated by the scenario generation unit 108. Specifically, the data set is used to adjust parameters for controlling the operation of the picking unit 112 for picking the target workpiece. Fig.10 1 is a block diagram showing a configuration example of the data set generating unit 109 according to the third embodiment. Fig.10 As shown, the data set generating unit 109 includes a 2D data generating unit 401 , a 3D data generating unit 402 , and an annotation data generating unit 403 .

[0084] The 2D data generating unit 401 is a two-dimensional data generating unit that generates 2D data for the scene generated by the scene generating unit 108. The 2D data generating unit 401 may generate a grayscale image or an RGB image.

[0085] The 3D data generating unit 402 is a 3D data generating unit that generates 3D data for the scene generated by the scene generating unit 108. The 3D data generating unit 402 may generate a 3D point group or a range image.

[0086] The annotation data generator 403 generates annotation data for the scene generated by the scene generator 108. The annotation data generator 403 may generate color-coded label data for each type of object or data including position and posture information of each workpiece in the data or a type name.

[0087] The data generated by the data set generation unit 109 is simulation data, and therefore it is considered that the data deviates from the actual image due to the degree of noise addition, etc. Therefore, the data set generation unit 109 may have a function of reproducing the measurement noise. In addition, the data generated by the data set generation unit 109, that is, the data generated by the 2D data generation unit 401, the 3D data generation unit 402, and the annotation data generation unit 403 are collectively referred to as a data set. The picking robot 20 can automatically generate the data required for parameter adjustment in simulation through the data set generation unit 109, thereby reducing the load of the user's data collection work.

[0088] The parameter adjustment unit 110 generates a model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction conditions stored in the storage unit 105 , and automatically adjusts parameters related to picking using the generated model and the data set generated by the data set generation unit 109 . Fig.11 1 is a block diagram showing a configuration example of the parameter adjustment unit 110 according to Embodiment 3. Fig.11 As shown, the parameter adjustment unit 110 includes a gripping parameter adjustment unit 501 and a recognition parameter adjustment unit 502 .

[0089] The gripping parameter adjustment unit 501 generates a model of the target workpiece by the amorphous model generation unit 106 based on the deformation restriction conditions stored in the storage unit 105, and uses the generated model to adjust the parameters related to the robot hand possessed by the picking unit 112. The gripping parameter adjustment unit 501 sets a robot hand such as a tweezer hand, a parallel hand, a suction cup as an object, and adjusts the opening width or claw width of the robot hand that is easy to grip the target workpiece. The type of robot hand is determined by the user. The gripping parameter adjustment unit 501 can adjust all types or multiple gripping parameters at the same time during the adjustment of the gripping parameters, or can adjust the gripping parameters individually in a predetermined order. Hereinafter, the case of continuously adjusting the gripping parameters individually will be mainly described.

[0090] The detailed structure and operation of the gripping parameter adjustment unit 501 will be described. Fig.12 is a block diagram showing a configuration example of the gripping parameter adjustment unit 501 according to Embodiment 3. Fig.12 As shown, the gripping parameter adjustment unit 501 includes a gripping parameter adjustment range determination unit 601, a gripping parameter change unit 602, a model rotation unit 603, a gripping evaluation unit 604, a gripping parameter value determination unit 605, and a gripping parameter adjustment completion determination unit 606. The details of each structure will be described together with the operation of the gripping parameter adjustment unit 501. Fig.13 This is a flowchart showing the operation of the gripping parameter adjustment unit 501 according to the third embodiment.

[0091] The gripping parameter adjustment range determination unit 601 generates a model of the target workpiece by the amorphous model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and determines the gripping parameter adjustment range based on the generated model (step S301). The gripping parameter adjustment range determination unit 601 may generate at least one model of the target workpiece by the amorphous model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and determine the gripping parameter adjustment range, the gripping parameter initial value, etc. based on the size of the generated model, etc.

[0092] The gripping parameter changing unit 602 changes the type of gripping parameter to be adjusted (step S302 ).

[0093] The model rotation unit 603 generates a model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and rotates the generated model (step S303). The model rotation unit 603 may generate at least one model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and disable the generated model or rotate it in a predetermined order.

[0094] The gripping evaluation unit 604 rotates the model by the model rotating unit 603 to perform gripping evaluation (step S304). The gripping evaluation unit 604 can fit the model of the robot hand when the model of the target workpiece is rotated by the model rotating unit 603, and evaluate the frequency F of deviation between the main axis of the target workpiece and the gripping direction. Ang and the interference frequency F of the target workpiece Col The evaluation function of the gripping parameter composed of these two evaluation values ​​is calculated.

[0095] When the rotation of the model by the model rotating unit 603 is not completed (step S305: No), the gripping parameter adjusting unit 501 returns to step S303 and continues the rotation of the model by the model rotating unit 603 and the gripping evaluation by the gripping evaluation unit 604. When the rotation of the model by the model rotating unit 603 is completed (step S305: Yes), the gripping parameter adjusting unit 501 proceeds to the operation of step S306.

[0096] The gripping parameter value determination unit 605 determines the value of the gripping parameter (step S306) based on the evaluation result of the gripping evaluation unit 604. Specifically, the gripping parameter value determination unit 605 searches for the gripping parameter value that minimizes the evaluation function calculated by the gripping evaluation unit 604, and determines the gripping parameter value found by the search as the adjustment parameter value.

[0097] The gripping parameter adjustment completion determination unit 606 determines whether the adjustment of the adjustment object, i.e., all gripping parameters, has been completed (step S307). If the adjustment of the adjustment object, i.e., all gripping parameters, has not been completed (step S307: No), the gripping parameter adjustment completion determination unit 606 returns to the action of step S302 and instructs each unit to perform the same action as above. If the adjustment of the adjustment object, i.e., all gripping parameters, has been completed (step S307: Yes), the gripping parameter adjustment completion determination unit 606 ends the action of the gripping parameter adjustment unit 501.

[0098] As described above, the gripping parameter adjustment unit 501 automatically adjusts the gripping parameters in the picking robot 20. The picking robot 20 can automatically adjust the parameters related to the robot hand that easily grips the target workpiece using the generated model, thereby reducing the user's parameter adjustment work load.

[0099] return Fig.11 The recognition parameter adjustment unit 502 generates a model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction conditions stored in the storage unit 105, and uses the generated model, the data set generated by the data set generation unit 109, and the gripping parameters adjusted by the gripping parameter adjustment unit 501 to adjust the recognition parameters of the object recognition method. The recognition parameter adjustment unit 502 can adjust the recognition parameters of the method for detecting the gripping position that becomes a candidate from 2D data or 3D data, or can adjust the parameters of the learning model for gripping position detection involved in deep learning, or can adjust the parameters of the learning model for deep learning used as segmentation for auxiliary processing to improve the recognition rate. Hereinafter, the recognition parameter adjustment of the method for detecting the gripping position that becomes a candidate from 2D data will be mainly described. In addition, in the adjustment of the recognition parameters, the recognition parameter adjustment unit 502 can adjust all types or multiple recognition parameters at the same time, or can adjust the recognition parameters individually in a predetermined order. Hereinafter, the case where the recognition parameters are adjusted individually and continuously will be mainly described.

[0100] The detailed structure and operation of the recognition parameter adjustment unit 502 will be described. Fig.14 is a block diagram showing a configuration example of the recognition parameter adjustment unit 502 according to the third embodiment. Fig.14 As shown, the recognition parameter adjustment unit 502 includes a recognition parameter adjustment range determination unit 701, a recognition parameter change unit 702, a recognition trial unit 703, a recognition evaluation unit 704, a recognition parameter value determination unit 705, and a recognition parameter adjustment completion determination unit 706. The details of each structure will be described together with the operation of the recognition parameter adjustment unit 502. Fig.15 This is a flowchart showing the operation of the recognition parameter adjustment unit 502 according to the third embodiment.

[0101] The recognition parameter adjustment range determination unit 701 generates a model of the target workpiece by the indefinite model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and determines the adjustment range of the recognition parameter using the generated model (step S401). The recognition parameter adjustment range determination unit 701 may generate a model of at least one target workpiece by the indefinite model generation unit 106 based on the deformation restriction condition stored in the storage unit 105, and determine the adjustment range of the recognition parameter, the initial value of the recognition parameter, etc. based on the size of the generated model, etc.

[0102] The recognition parameter changing unit 702 changes the type of recognition parameter to be adjusted (step S402 ).

[0103] The recognition trial unit 703 performs recognition processing on the data generated by the data set generation unit 109 using the gripping parameters adjusted by the gripping parameter adjustment unit 501 (step S403). If the recognition processing is not completed (step S404: No), the recognition trial unit 703 returns to step S403 to continue the recognition processing. If the recognition processing is completed (step S404: Yes), the recognition result is output to the recognition evaluation unit 704.

[0104] The recognition evaluation unit 704 evaluates the recognition result obtained by the recognition trial unit 703 based on the recognition result obtained from the recognition trial unit 703 (step S405). The recognition evaluation unit 704 can evaluate the deviation E between the gripping position during recognition by the recognition trial unit 703 and the optimal gripping position. Pos , the deviation E between the main axis of the target workpiece and the gripping direction Ang , the evaluation value E of the number of recognition Num The evaluation function of the identification parameters composed of these three components is calculated. The optimal gripping position may be the center of gravity position of the target workpiece or a position predetermined by the user.

[0105] The recognition parameter value determination unit 705 determines the value of the recognition parameter based on the evaluation result of the recognition evaluation unit 704 (step S406). Specifically, the recognition parameter value determination unit 705 searches for the value of the recognition parameter that minimizes the evaluation function calculated by the recognition evaluation unit 704, and determines the value of the recognition parameter found by the search as the value of the adjustment parameter.

[0106] The recognition parameter adjustment completion determination unit 706 determines whether the adjustment of the adjustment object, i.e., all the recognition parameters, has been completed (step S407). If the adjustment of the adjustment object, i.e., all the recognition parameters, has not been completed (step S407: No), the recognition parameter adjustment completion determination unit 706 returns to the operation of step S402 and instructs each unit to perform the same operation as above. If the adjustment of the adjustment object, i.e., all the recognition parameters, has been completed (step S407: Yes), the recognition parameter adjustment completion determination unit 706 ends the operation of the recognition parameter adjustment unit 502.

[0107] As described above, in the picking robot 20, the recognition parameter adjustment unit 502 automatically adjusts the recognition parameters. By using the generated model and data set, the picking robot 20 can automatically adjust the recognition parameters that can easily improve the recognition rate, which can reduce the user's parameter adjustment work load. The parameter adjustment unit 110 automatically adjusts the gripping parameters and the recognition parameters. The picking robot 20 automatically adjusts the parameters, thereby eliminating the need for the user to adjust the parameters, which can reduce the user's parameter adjustment work load.

[0108] return Figure 8 The storage unit 105 may store the parameters related to picking adjusted by the parameter adjustment unit 110 .

[0109] The recognition processing unit 111 performs object recognition processing on the data acquired by the sensor 101 based on the parameters adjusted by the parameter adjustment unit 110. The recognition processing unit 111 can perform object recognition processing on the data acquired by the sensor 101 based on the adjusted parameters stored in the storage unit 105 to detect the grasping position that is a candidate. In addition, the recognition processing unit 111 can directly obtain the parameters adjusted by the parameter adjustment unit 110 from the parameter adjustment unit 110.

[0110] The picking unit 112 has a robot hand. The picking unit 112 performs a picking operation of the target workpiece by the robot hand based on the recognition processing result obtained by the recognition processing unit 111. The picking unit 112 can make the robot hand cut into the gripping position with the highest gripping likelihood based on the recognition processing result to pick up the target workpiece, and repeat such an operation of moving to a predetermined position.

[0111] In addition, the picking robot 20 may be equipped with sensors other than the sensor 101, such as a force sensor for controlling force, a proximity sensor, and a tactile sensor. In addition, the model generation device 10 may also set a plurality of picking robots 20 as an object, or may set a picking robot 20 having a plurality of picking parts 112 as an object. The picking robot 20 simulates the actual environment of the user on the simulator, and uses the generated data set and the generated model of the target workpiece to adjust parameters, thereby improving the success rate of picking the target workpiece.

[0112] The hardware structure of the picking robot 20 is described. In the picking robot 20, the picking unit 112 is implemented by a robot hand and a robot hand control unit. The simulation condition setting unit 107, the scene generation unit 108, the data set generation unit 109, the parameter adjustment unit 110, the recognition processing unit 111, and the robot hand control unit of the picking unit 112 are implemented by a processing circuit. The processing circuit may be a processor and a memory that executes a program stored in a memory, or may be dedicated hardware, as in the case of the model generation device 10.

[0113] As described above, according to this embodiment, the picking robot 20 uses the data set generated on the simulator and the model of the generated object workpiece to automatically adjust the parameters related to picking. As a result, the picking robot 20 can improve the picking success rate and reduce the trial and error period and cost involved in the introduction of industrial robots and verification until operation.

[0114] According to the present embodiment, the picking robot 20 simulates the actual environment of the user on the simulator, and can automatically generate the data required for parameter adjustment using the model of the generated workpiece instead of the actual machine, thereby reducing the load of the user's data collection operation. In addition, the picking robot 20 can automatically adjust the parameters that improve the recognition rate and make it easier to grasp the target workpiece, thereby reducing the load of the user's parameter adjustment operation and improving the success rate of picking the target workpiece.

[0115] The configuration described in the above embodiment is an example of the content of the present invention, and can be combined with other known technologies. Part of the configuration can be omitted or changed without departing from the scope of the present invention.

[0116] Description of the label

[0117] 10 model generation device, 20 picking robot, 101 sensor, 102 workpiece information acquisition unit, 103 reference model determination unit, 104 deformation restriction condition determination unit, 105 storage unit, 106 amorphous model generation unit, 107 simulation condition setting unit, 108 scene generation unit, 109 data set generation unit, 110 parameter adjustment unit, 111 recognition processing unit, 112 picking unit, 201 individual model determination unit, 202 registration unit, 203 average shape generation unit, 204 shape element segmentation unit, 205 individual difference category determination unit, 206 workpiece main axis determination unit, 207 primitive comparison unit, 208 fitting model determination unit, 301 sensor information setting unit, 302 workpiece information setting unit, 303 environment information setting unit, 401 2D data generation unit, 402 3D data generation unit, 403 annotation data generation unit, 501 gripping parameter adjustment unit, 502 recognition parameter adjustment unit, 601 gripping parameter adjustment range determination unit, 602 gripping parameter change unit, 603 model rotation unit, 604 gripping evaluation unit, 605 gripping parameter value determination unit, 606 gripping parameter adjustment end judgment unit, 701 recognition parameter adjustment range determination unit, 702 recognition parameter change unit, 703 recognition trial unit, 704 recognition evaluation unit, 705 recognition parameter value determination unit, 706 recognition parameter adjustment end judgment unit.

Claims

1. A model generation device, characterized in that: have: A sensor that acquires scene information including information of a target workpiece for generating a model; a workpiece information acquisition unit that acquires information about the target workpiece, that is, workpiece information, from the scene information; a reference model determination unit that determines a reference model that is a model used to represent the misaligned shape based on the scene information and the workpiece information; a deformation restriction condition determination unit that determines a deformation restriction condition as a condition for deforming the reference model based on a result obtained by analyzing a workpiece shape using the scene information and the workpiece information and the reference model; as well as The indefinite model generating unit determines a deformation parameter of the reference model within a range of the deformation restriction condition based on the deformation restriction condition, and deforms the reference model according to the deformation parameter.

2. The model generation device according to claim 1, characterized in that: The deformation restricting condition includes, as a numerical condition, at least one of the occurrence frequency of the convexoconcavity of the target workpiece, the occurrence range of the convexoconcavity of the target workpiece, and the curvature.

3. The model generation device according to claim 1 or 2, characterized in that: The workpiece shape analyzed by the deformation restriction condition determination unit includes a workpiece contour of the target workpiece.

4. The model generation device according to claim 3, characterized in that: The indeterminate shape model generating unit uses the information of the workpiece contour when determining the deformation amount of the reference model.

5. The model generation device according to claim 4, characterized in that: The indefinite-shape model generating unit uses, as the information of the workpiece contour, a workpiece contour generated by synthesizing two or more different workpiece contours.

6. The model generation device according to claim 4 or 5, characterized in that: The indefinite shape model generating unit uses, as the information of the workpiece contour, a workpiece contour generated by enlarging or reducing the scale of the workpiece contour.

7. The model generation device according to any one of claims 1 to 6, characterized in that: The reference model determination unit includes: an individual model determination unit that models each workpiece information acquired by the workpiece information acquisition unit and determines an individual model obtained by modeling each workpiece information; a registration unit, which integrates the individual models; as well as The average shape generating unit generates an average shape model of the target workpiece based on the integrated result obtained by the registration unit.

8. The model generation device according to claim 7, characterized in that: The average shape model generated by the average shape generating unit is used as the reference model.

9. The model generation device according to claim 7 or 8, characterized in that: The deformation restriction condition determination unit determines the deformation restriction condition of the contour based on the quantified condition and the shape analysis result of the reference model.

10. The model generation device according to any one of claims 7 to 9, characterized in that: The reference model determination unit includes an individual difference category determination unit that determines an individual difference category that indicates the degree of misalignment of the workpiece shape in a graded manner with respect to the average shape model. The reference model determination unit determines the reference model based on the result of the individual difference category and the average shape model.

11. The model generation device according to any one of claims 7 to 10, characterized in that: The reference model determination unit includes a workpiece main axis determination unit that determines a workpiece main axis as a main axis of inertia for the average shape model or the individual model. The reference model determination unit determines the reference model based on the determined workpiece main axis and average shape model.

12. The model generation device according to any one of claims 7 to 11, characterized in that: The reference model determination unit includes: A primitive comparison unit that compares the average shape model with a geometric shape; and a fitting model determination unit that determines a reference model based on the comparison result of the primitive comparison unit, The reference model determination unit determines a fitting model to be the reference model.

13. The model generation device according to any one of claims 7 to 12, characterized in that: The reference model determination unit includes a shape element division unit that divides the average shape model or the individual model into at least one or more shape elements constituting each model. The reference model determination unit determines the reference model for each of the segmented shape elements.

14. The model generation device according to any one of claims 1 to 13, characterized in that: The deformation restriction condition determination unit uses, as the deformation parameter, a feature map of an intermediate layer of a neural network obtained by learning the shape of the target workpiece.

15. The model generation device according to any one of claims 1 to 14, characterized in that: The indefinite model generation unit is a neural network that outputs a model of the target workpiece, and an input of the neural network includes at least one of the scene information, the workpiece information, the deformation restriction condition, and the deformation parameter.

16. A picking robot, characterized in that: have: The model generating device according to any one of claims 1 to 15; a simulation condition setting unit that generates a model of the target workpiece based on the deformation restriction condition determined by the model generation device, and sets simulation conditions including at least one of sensor information and workpiece information in the simulation using the generated model; a scene generation unit that generates a model of the target workpiece according to the deformation restriction condition, and simulates the scene of the target workpiece using the generated model and the simulation condition; a data set generating unit for generating a data set for adjusting parameters for controlling a picking operation of a target workpiece based on the scenario; a parameter adjustment unit that generates a model of the target workpiece according to the deformation restriction condition, and automatically adjusts parameters related to picking using the generated model and the data set; a recognition processing unit that performs object recognition processing on the data acquired by the sensor based on the parameters adjusted by the parameter adjustment unit; as well as The picking unit performs a picking operation of a target workpiece by a robot hand based on the recognition processing result obtained by the recognition processing unit.

17. The picking robot according to claim 16, characterized in that: The simulation condition setting unit comprises: a sensor information setting unit configured to set at least one of the specification and setting information of the sensor in the simulation; a workpiece information setting unit configured to set workpiece information of the target workpiece; as well as An environment information setting unit is used to set environment information in the simulation. The simulation condition setting unit sets simulation conditions for sensor information, workpiece information, and environmental information.

18. The picking robot according to claim 16 or 17, characterized in that: The data set generating unit comprises: a two-dimensional data generating unit that generates two-dimensional data for the scene generated by the scene generating unit; A 3D data generating unit that generates 3D data; as well as an annotation data generating unit that generates annotation data, The data set generation unit generates a data set based on the scenario.

19. The picking robot according to any one of claims 16 to 18, characterized in that: The parameter adjustment unit has: a gripping parameter adjustment unit that generates a model of the target workpiece according to the deformation restriction condition and adjusts gripping parameters related to the robot hand using the generated model; as well as a recognition parameter adjustment unit that generates a model of the target workpiece according to the deformation restriction condition, and adjusts the recognition parameters of the object recognition method using the generated model, the data set, and the gripping parameters adjusted by the gripping parameter adjustment unit, The parameter adjustment unit automatically adjusts the grasping parameters and the recognition parameters.

20. The picking robot according to claim 19, characterized in that: The gripping parameter adjustment unit comprises: a gripping parameter adjustment range determination unit that generates a model of the target workpiece according to the deformation restriction condition and determines the gripping parameter adjustment range based on the generated model; a gripping parameter changing section that changes the type of gripping parameter to be adjusted; a model rotating unit that generates a model of the target workpiece according to the deformation restriction condition and rotates the generated model; a grip evaluation unit that performs grip evaluation by rotating the model through the model rotating unit; a gripping parameter value determination unit that determines a gripping parameter value based on an evaluation result of the gripping evaluation unit; as well as The gripping parameter adjustment completion determination unit determines whether the adjustment of the adjustment object, that is, all gripping parameters, has been completed. The gripping parameter adjustment unit automatically adjusts the gripping parameters.

21. The picking robot according to claim 19 or 20, characterized in that: The identification parameter adjustment unit comprises: a recognition parameter adjustment range determination unit that generates a model of the target workpiece according to the deformation restriction condition and determines the adjustment range of the recognition parameter using the generated model; an identification parameter changing unit that changes the type of identification parameter to be adjusted; a recognition trial unit that performs recognition processing on the data generated by the data set generation unit using the gripping parameters adjusted by the gripping parameter adjustment unit; an identification evaluation unit that evaluates the identification result obtained by the identification trial unit; an identification parameter value determination unit that determines a value of an identification parameter based on a result of the identification evaluation unit; as well as The recognition parameter adjustment completion determination unit determines whether the adjustment of all recognition parameters to be adjusted has been completed. The recognition parameter adjustment unit automatically adjusts the recognition parameter.

Images