DETERMINATION DEVICE, PROCESSING SYSTEM, TRANSMISSION SYSTEM, DETERMINATION METHOD AND STORAGE MEDIUM

Abstract

According to one embodiment, a determination device is configured to acquire an image of an object to be grasped and input the image to a determination part. The determination device is configured to receive a grasp point acquisition method from the determination part. According to the input of the image, the determination part outputs one of a first method, a second method, or a third method as the acquisition method. The first method is a method of calculating external shape data of the object based on the image and calculating the grasp point using the external shape data. The second method is a method of inputting the image to a second model and acquiring the grasp point output from the second model, thereby training the second model. The third method is a method of acquiring the grasp point by designating the grasp point.

Description

AREA

[0001] Embodiments of the invention generally relate to a determination device, a processing system, a transmission system, a determination method and a storage medium. BACKGROUND

[0002] There is a device for gripping and transferring objects. A technology that can increase the processing efficiency of the transfer device is desirable. BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a schematic view showing a configuration of a processing system according to an embodiment; Fig. 2 is a schematic view showing a specific configuration of the processing system; Fig. 3 is a schematic view showing an example of a determining part; Fig. 4 is a schematic view showing another example of a determining part; Fig. 5A and Fig. 5B are schematic views showing an example of the training method of the determination part; Fig. 6 is a perspective view schematically showing the method of calculating the external shape data of an object; Fig. 7 is a plan view schematically showing the method of calculating the external shape data of the object; Fig. 8 is a schematic view for describing the distance data of the distance between the center of a contact surface and the center of gravity of the object; Fig. 9 is a schematic view showing the state when a suction mechanism grips the object; Fig. 10 is a schematic view showing the state when a clamping mechanism grips the object; Fig. 11 is a schematic view of a second model; Fig. 12 is a schematic view showing an example of a graphical user interface; Fig. 13A and Fig. 13B are schematic views showing examples of the graphical user interface; Fig. 14 is a schematic view showing an example of a graphical user interface; Fig. 15A to 15C are schematic views showing examples of the graphical user interface; Fig. 16 is a flowchart illustrating a processing method according to the embodiment; Fig. 17 is a flowchart showing a processing method according to a first modification of the embodiment; Fig. Fig. 18 is a schematic view showing a configuration of a processing system according to a modification of the embodiment; and Fig. 19 is a schematic view showing a hardware configuration. DETAILED DESCRIPTION

[0003] According to one embodiment, a determination device is configured to acquire an image of an object to be grasped. The determination device is configured to input the image to a determination part, wherein the determination part contains a first model, and the first model is trained. The determination device is configured to receive from the determination part a detection method for a gripping point at which a gripper is to grasp the object. Depending on the input of the image, the determination part outputs one of a first method, a second method, or a third method as the detection method. The first method is a method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data.The second method is a method for inputting the image into a second model and detecting the grasp point output by the second model, thereby training the second model. The third method is a method for detecting the grasp point by labeling the grasp point.

[0004] Embodiments of the invention will now be described with reference to the drawings. The drawings are schematic or conceptual; and the relationships between the thicknesses and widths of parts, the size ratios between parts, etc., do not necessarily correspond to actual values. The dimensions and / or proportions may be shown differently in the drawings, even if the same part is depicted. In the drawings and the description of the application, components similar to those described therein are designated by the same reference numerals, and a detailed description is omitted where appropriate.

[0005] Fig. 1 is a schematic view showing a configuration of a processing system according to an embodiment.

[0006] The processing system 1 according to the embodiment includes a determination device 10, a storage device 15, a computing device 20, a sensor 30, a control device 40, a transfer device 50, and a terminal device 60.

[0007] The transfer device 50 can handle and transfer an object. For example, the transfer device 50 grasps an item stored in a container. The transfer device 50 lifts the grasped object and transfers it to another container. The transfer device 50 releases the grasped object into the other container. The object is placed in the container and stored. A series of operations including grasping, transferring, and depositing is referred to herein as "picking." The transfer device 50 is, for example, a picking robot.

[0008] The transfer device 50 includes a gripper 55. The gripper 55 can grip an object by pinching, suction, clamping, etc. The gripper 55 can include both a clamping mechanism and a suction mechanism, and both mechanisms can be used selectively.

[0009] The control device 40 controls the transfer device 50. The control device 40 is a so-called robot controller. The control device 40 moves the gripper 55 to a gripping point when the transfer device 50 grasps the object. The "gripping point" is represented by the three-dimensional position (X, Y, Z) and the three-dimensional angle (ϕ, θ, ψ). The gripper 55 grasps the object at the gripping point.

[0010] Sensor 30 detects the object to be grasped. Sensor 30 is, for example, an image sensor. A camera containing sensor 30 is used. The camera captures an image by recording the object. The camera can capture a video image. In such a case, a still image is cropped from the video image. Sensor 30 captures an RGB image or a depth image. Preferably, both an RGB image and a depth image are captured. One sensor 30 can be used to capture an RGB image and another sensor 30 can be used to capture a depth image.

[0011] Sensor 30 can also be a distance sensor. For example, a laser rangefinder (LRF) containing sensor 30 is used. The LRF can capture a distance image by measuring the distances to objects in the surrounding area. Either a camera or an LRF can be used as sensor 30.

[0012] Computing device 20 determines the gripping point at which gripper 55 should grip the object. Determining device 10 determines a detection method for the gripping point. Computing device 20 executes the detection method determined by determining device 10. Storage device 15 stores data from the processing of determining device 10 and computing device 20, data obtained by the processing, etc.

[0013] The determination device 10 uses a determination part 11 to determine the detection method. The determination part 11 includes a first model that is trained. The determination part 11 outputs the detection method of the grasping point according to the input image. The determination device 10 inputs the image captured by the sensor 30 to the determination part 11 and receives the detection method output from the determination part 11. The determination part 11 outputs a first method, a second method, or a third method as the detection method. The computing device 20 executes the first, second, or third method.

[0014] In the first method, the computing device 20 calculates the outer shape data of the object's outer shape (the contour) based on the image. The computing device 20 generates a combination (a pattern) of the outer shape portion to be grasped and the grasping point of the gripper 55. The computing device 20 generates multiple patterns by changing the portion to be grasped, the position of the gripper 55, the posture of the gripper 55, and so on. The computing device 20 calculates the score for each pattern. The grasping point of the pattern for which the highest score is obtained is used as the actual grasping point to be used.

[0015] In the second method, the computing device 20 inputs the image into a second model, which is then trained. The second model is previously machine-learned to calculate the grasping point based on the image input. The second model includes a neural network. The computing device 20 receives the grasping point output from the second model.

[0016] In the third method, the grasping point is designated by a person; and the computing device 20 detects the grasping point by determining the grasping point. First, the computing device 20 requests the terminal device 60 to specify the grasping point. Then, the user of the terminal device 60 specifies the grasping point according to the request of the computing device 20. The terminal device 60 accepts the grasping point designated by the user and transmits the grasping point to the computing device 20. The computing device 20 adopts the grasping point transmitted from the terminal device 60.

[0017] When grasping an object, computing device 20 determines the grasp point by performing only one of the first, second, or third methods. When grasping one object, computing device 20 does not perform the remaining two of the first method, the second method, or the third method.

[0018] The determination device 10, the storage device 15, the computing device 20, and the sensor 30 are connected to each other via a network, wireless communication, or wired communication. A device can perform the functions of both the determination device 10 and the computing device 20. The control device 40 and the terminal device 60 are each connected to the computing device 20 via a network, wireless communication, or wired communication.

[0019] The determination device 10 sends the selected acquisition method to the computing device 20. The sensor 30 transmits the acquired image to the computing device 20. The computing device 20 acquires the grasping point using the received acquisition method. The computing device 20 transmits the acquired grasping point to the control device 40. The control device 40 moves the gripper 55 to the received grasping point and causes the gripper 55 to grasp the object at the grasping point. The transfer device 50 transfers the grasped object. The processing system 1 functions as a transfer system that transfers (handles) the object.

[0020] The inventions according to the embodiments will now be described in more detail. Transfer device

[0021] Fig. Figure 2 is a schematic view showing a specific configuration of a processing system.

[0022] In the Fig. In the example shown in Figure 2, the transfer device 50 is a vertically articulated robot. The transfer device 50 comprises a manipulator 51 having a plurality of links 51a and a plurality of rotation axes 51b. The links 51a are connected to one another by the rotation axes 51b.

[0023] The position and angle of the distal end of the manipulator 51 are changed by actuating the rotation axes 51b. It is advantageous if the distal end of the manipulator 51 has six degrees of freedom. The gripper 55 is attached to the distal end of the manipulator 51. In the example shown, the gripper 55 comprises a suction mechanism 56 and a clamping mechanism 57.

[0024] The suction mechanism 56 holds the object by suction. The suction mechanism 56 includes one or more suction pads 56a. The interior of the suction pad 56a is decompressed by a pressure relief device (not shown) when the suction pad 56a is in contact with the object. As a result, the object is sucked by the suction pad 56a. The number of suction pads 56a may be larger or smaller than in the example shown.

[0025] The clamping mechanism 57 grips the object by clamping. The clamping mechanism 57 comprises a plurality of rod-shaped support parts 57a. The object is clamped and held by the plurality of holding parts 57a. The clamping mechanism 57 may comprise more support parts 57a than in the example shown. The support part 57a may have a finger-like configuration with one or more joints.

[0026] The gripper 55 also includes a switching mechanism 58. The suction mechanism 56 and the clamping mechanism 57 are coupled to the switching mechanism 58. The switching mechanism 58 rotates the suction mechanism 56 and the clamping mechanism 57. The mechanism used to grip the object can be switched by rotating the suction mechanism 56 and the clamping mechanism 57.

[0027] The gripper 55 is not limited to the example shown and can also contain only one of the two mechanisms, the suction mechanism 56 or the clamping mechanism 57. In this case, the switching mechanism 58 is superfluous.

[0028] Two containers C1 and C2 are located near the transfer device 50. The transfer device 50 grasps an object O stored in the container C1 and transfers it to the container C2.

[0029] Sensor 30 is used to detect the condition inside container C1. The object O stored in container C1 is detected from above by sensor 30. In addition to sensor 30 for detecting the condition in container C1, another sensor 30 can be provided for detecting the condition in container C2. Sensor 30 can be attached to transfer device 50.

[0030] Unlike the illustrated example, the transfer device 50 can also be a parallel-joint robot that includes the gripper 55. The transfer device 50 can be an aircraft (a drone or similar) that includes the gripper 55. Regardless of which transfer device 50 is used, the gripper 55 of the transfer device 50 grips the object at the gripping point detected by the computing device 20. First model

[0031] Fig. 3 is a schematic view showing an example of a determination part.

[0032] It is advantageous if the first model of the determining part 11 contains a neural network. For example, the first model contains a convolutional neural network. Fig. Figure 3 is an example of a first model that includes a convolutional neural network. Fig. The example shown in Figure 3 uses a convolutional neural network based on EfficientNetV2.

[0033] Specifically, the Fig. 3, the first model 100 illustrates blocks 101 to 110. The numerical values under blocks 101 to 110 indicate the number of layers of each block. Blocks 101 to 108 convolve the data. Blocks 101 to 108 repeat the extraction of feature points from the image and the compression of the extracted feature points. Block 109 uses pooling to further reduce the data compressed by blocks 101 to 108. Block 110 fully concatenates the data output by block 109.

[0034] For example, block 110 outputs the probabilities of the first, second, and third methods. The probabilities each represent the possibility that the object can be grasped when the methods are executed. The method with the highest probability is the optimal capture method. The determining device 10 determines the capture method as the method with the highest probability.

[0035] The first model is previously machine-learned. Supervised learning can be used in machine learning. Supervised learning uses multiple datasets. Each dataset contains training images and labels. The object to be grasped is visible in the training image. The label indicates the optimal method for grasping the object visible in the training image. One of the first, second, or third methods is specified as the label. The first model is repeatedly trained to output the method specified by the label according to the training image input.

[0036] Fig. 4 is a schematic view showing another example of a determining part.

[0037] As in Fig. As shown in Figure 4, the first model of the determination part 11 may include a variational autoencoder (VAE) 120 and a clustering model 130. The VAE 120 includes an encoder 121 and a decoder 122. When an image is input to the VAE 120, the encoder 121 dimensionally compresses the image into one or more latent variables. The latent variables are derived from the input data and are features of the input data. The decoder 122 reconstructs the original image from the one or more latent variables. The determination device 10 inputs the image to the VAE 120, acquires the one or more latent variables output from the encoder 121, and inputs the one or more latent variables to the clustering model 130. The determination device 10 may extract a part of the latent variables and input these latent variables into the cluster model 130.The cluster model 130 is previously machine-learned to cluster the latent variables. The cluster model 130 clusters the input latent variables into a first cluster, a second cluster, and a third cluster. The first cluster corresponds to the first method. The second cluster corresponds to the second method. The third cluster corresponds to the third method.

[0038] The encoder 121 can output a probability distribution from the input image. The determination device 10 can input the probability distribution into the cluster model 130 to classify the input image into one of the first to third clusters.

[0039] As a specific example, the output of encoder 121 may contain a two-dimensional map of the latent variables for three methods. One of the three methods may be determined using a method that combines the k-means method with semi-supervised learning. For example, classification may be performed by specifying the centroid of the latent variable as labeled data and then determining the clusters of each point. When training a model, the VAE is first learned, and the result is then used for semi-supervised learning with k-means.

[0040] Fig. 5A and Fig. 5B are schematic views showing an example of the training method of the determination part.

[0041] To train the VAE 120, several training images are used. The object to be grasped is visible in each training image. The VAE 120 is trained to output the same image as the input training image. After the VAE 120 has been trained, several training images are input to the VAE 120 one after the other. Each time the training image is input to the VAE 120, the latent variable output by the encoder 121 is recorded. As shown in Fig. 5A, a plurality of latent variables 125 are obtained. The cluster model 130 clusters the plurality of latent variables 125. Unsupervised learning such as the k-means algorithm or the like is used for clustering. The number of clusters is set to the three clusters of the first method cluster, the second method cluster, and the third method cluster. Through clustering, as shown in Fig. As shown in Figure 5B, the multiple latent variables 125 are clustered into the three clusters of a cluster CL1 of the first method, a cluster CL2 of the second method, and a cluster CL3 of the third method.

[0042] Or, the determination part 11 may include a classification model that outputs the probabilities of the first, second, and third methods according to the input of the latent variable 125. In such a case, the classification model is previously machine-learned through supervised learning. When supervised learning is performed, multiple datasets are used, similar to the training of the first model 100 described above. Each dataset contains the latent variables based on the input of the image and labels. One of the first, second, or third methods is designated as the label. The classification model is repeatedly trained to output the method designated by the label according to the input of the latent variable. First method

[0043] In the first method, the safety factor is estimated at each grasp point by thoroughly searching several grasp points. The safety factor indicates the probability that object O can be handed over without falling and corresponds to the score described above. From the multiple grasp points, the one with the highest safety factor is selected.

[0044] The following describes an example of a method for calculating the safety factor when using the suction mechanism 56. The computing device 20 acquires the data acquired by the sensor 30 and detects the status of various components related to the control of the transfer device 50. For example, the computing device 20 performs prescribed image processing on the image to calculate "object external shape data," "object center of gravity data," etc., to indicate the status of the various components.

[0045] Fig. 6 is a perspective view schematically showing the method of calculating the external shape data of the object. Fig. 7 is a plan view schematically showing the method of calculating the external shape data of the object.

[0046] The “object outer shape data” is calculated using the image of the object O. The object outer shape data indicates the outer shape of the object O stored in the container C1 from which the object O is to be taken. As in Fig. 6, the object outer shape data includes, for example, data relating to a first surface F1 and a second surface F2 of a rectangular parallelepiped that circumscribes the object O. The second surface F2 is adjacent to the first surface F1. When an object surface is not flat (when the object surface contains an unevenness), the computing device 20 recognizes the rectangular parallelepiped shape that circumscribes the object surface as the outer shape data of the object, as shown in Fig. 7. The computing device 20 recognizes the external shape, when the object is viewed along a specific direction, as a graspable area Fc of the object. The graspable area Fc is a flat part of the object's surface that can be grasped with a suction cup.

[0047] The computing device 20 calculates the safety factor based on the contact area data and the distance data. The contact area data indicates the area where the gripper 55 and the graspable area of the object O touch. The distance data indicates the distance L between a center point K of the contact area and a center of gravity G of the object O. The contact area is the area where the gripper 55 and the object O touch.

[0048] The pressure with which the contact surface can be suctioned is referred to as the "suction pressure." The stress, i.e., the distance L divided by a second area moment I with the contact surface as the cross-section, is referred to as the "divided stress value." The computing device 20 calculates a safety factor R based on the numerical value of the suction force pressure divided by the divided stress value. The safety factor R is the value of the suction pad pressure at the gripping point divided by the sum of the generated bending stress and other tensile stress. The bending stress is calculated using the following formula (1). σ(x)=Mxl

[0049] Fig. 8 is a schematic view for describing the distance data of the distance between the center of the contact surface and the center of gravity of the object. Fig. 9 is a schematic view showing the state when the suction mechanism grips the object.

[0050] In formula (1), σ (x) is the bending stress, M is the moment, I is the second moment of area and x is the distance from the neutral axis. Fig. Figure 8 shows a model in which a bending stress acts on a structural body. As in Fig. As shown in Figure 8, tensile and compressive stresses are generated in a structural body when a bending stress acts on the structural body.

[0051] When a bending stress is applied to a structural body, the structural body will fracture if the maximum bending stress exceeds the tensile stress the structural body can withstand. In the case of suction gripping, it can be assumed that the suction cup 56a will detach from the object O to be gripped if the maximum bending stress exceeds the negative pressure for each suction cup 56a. The safety factor R is calculated using the following formula (2). R=Pσ+Ts

[0052] In formula (2), P is the vacuum pressure (the suction gripping pressure) of any gripping type, σ is the bending stress, and Ts is the generated other tensile stress. According to the embodiment, the generated other tensile stress is assumed to be Ts. The safety factor R is the value of the suction gripping pressure at the gripping point divided by the bending stress. By omitting the tensile stress Ts, the safety factor R is calculated according to the following formula (3). R=PM×rI

[0053] In formula (3), M is the moment determined by the distance L between the center K and the object's center of gravity G. r is the shortest distance between the contour of the gripping surface and the object's center of gravity G. I is the second area moment determined by any holding method. The example in Fig. 9 shows a case where the object's center of gravity G is located outside the gripping surface. In such a case, the shortest distance r is the distance between the object's center of gravity G and the contour of an effective suction pad 56a1 closest to the object's center of gravity G. The effective suction pads 56a1 refer to the suction pads 56a among the plurality of suction pads 56a used to hold the object O. The contour of the effective suction pad 56a1 closest to the object's center of gravity G is the position where the gripping surface is most easily released.

[0054] As described above, to calculate the safety factor R, it is necessary to calculate the second area moment I. When the gripper 55 includes a plurality of suction pads 56a, the number of combinations of the effective suction pads 56a1 used for gripping is determined according to the number of suction pads 56a for which the internal pressure is independently controllable. A number Q of combinations of the effective suction pads 56a1 used for gripping is calculated by the following formula (4), where the number of suction pads 56a for which the internal pressure is independently controllable is N. For example, the number Q is 31 when the internal pressure is independently controllable for each of five suction pads 56a. Q=∑i=1i=NNCi

[0055] The arrangement directions of the suction pads 56a are assumed to be a first arrangement direction and a second arrangement direction. The first arrangement direction and the second arrangement direction intersect. The computing device 20 calculates the second area torque I about the first and second arrangement directions while rotating the group of effective suction pads 56a1 by 180 degrees along a plane parallel to the first and second arrangement directions, by 1 degree each. The computing device 20 performs the calculation for all combinations.

[0056] The computing device 20 calculates the safety factor R described above for each determined second moment I. For example, if the number Q is 31, there are 31 × 181 = 5611 patterns of combinations of the effective suction pads 56a1 and the angle of the suction mechanism 56. The computing device 20 calculates the second moment I and the safety factor R for each pattern. The computing device 20 selects the pattern for which the highest safety factor R is obtained as the gripping point actually used during gripping.

[0057] Specific methods for calculating the safety factor when using the suction mechanism 56 are discussed, for example, in paragraphs 0061 to 0096 of JP-A 2021-037608 (Kokai), etc.

[0058] An example of a method for calculating the safety factor when using the clamping mechanism 57 will now be described. Here, a method for calculating the safety factor when two holding parts 57a hold the object O will be described. When the clamping mechanism 57 includes three or more holding parts 57a, the computing device 20 can calculate the safety factor by approximating the holding state of the object O as a state in which two holding parts 57a hold the object O.

[0059] Fig. 10 is a schematic view showing the state when the clamping mechanism grips the object.

[0060] The computing device 20 calculates the safety factor using several parameters. The several parameters include at least a diameter D, a distance d, a length L, and a gravitational force mg. The diameter D is a parameter related to the size of the area where the support part 57a and the object O contact each other. Hereinafter, this area is referred to as the contact area CR1. The diameter D is the diameter of a circle Ci1 that describes the contour of the contact area CR1. In this example, the shape of the contact area CR1 is approximated by the circle Ci1. In such a case, the area of the contact area CR1 is estimated to be smaller, so the computing device 20 can calculate the safety factor with greater consideration of the probability of the object O falling.

[0061] The distance d is a parameter that refers to a position P1 where the maximum bending (torsional) stress is generated in the contact area CR1. The position P1 is the point on the contour of the contact area CR1 that is farthest from the object's center of gravity G. The distance d is the distance between the position P1 and a center position P2 of the contact area CR1.

[0062] The length L is a parameter related to the bending moment (torsional moment) that generates the bending stress in the contact area CR1. The length L is the length of the arm where the bending moment is generated. The length L is the distance between the position P1 and a straight line SL1 passing through the object's center of gravity G in the vertical direction. The gravitational force mg is a parameter related to the weight of the object O. The gravitational force mg is the product of the mass m of the object O and the magnitude g of the acceleration due to gravity. A torque T, which is the bending moment generated at the position P1, is represented by the following formula (5). T=Lmg

[0063] A second area polar moment Ip of the circle Ci1, ie the approximate shape of the contact area CR1, is represented by the following formula (6). Ip=πD432

[0064] The bending stress generated by the torque T is represented by the following formula (7). τ=Td2Ip

[0065] The friction pressure Fp generated in the contact area CR1 is represented by the following formula (8). Fp=2μfA

[0066] In formula (8), a parameter f is the holding force of the support part 57a, a parameter µ is the friction coefficient corresponding to the object O, and a parameter A is the area of the contact area CR1. The friction pressure Fp is generated in all directions within the contact area CR1. The safety factor R is represented by the following formula (9). R=Fpτ

[0067] Taking into account the vertically downward stress, the safety factor R is represented by the following formula (10). R=Fpτ+mg

[0068] The computing device 20 calculates the diameter D, the distance d, and the length L based on the data on the object's external shape and the data on the object's center of gravity. The computing device 20 receives the mass data of the mass m of the object O from the storage device 15. The computing device 20 calculates the safety factor R according to the above formula.

[0069] The computing device 20 repeatedly calculates the safety factor while changing the value of at least one of the multiple parameters. In doing so, multiple safety factors are calculated, each corresponding to multiple states. The computing device 20 selects the state in which the highest safety factor is obtained as the gripping point.

[0070] Specific methods for calculating the safety factor when using the clamping mechanism 57 are discussed, for example, in paragraphs 0052 to 0107 of JP-A 2021-146434 (Kokai), etc. Second method

[0071] Fig. Figure 11 is a schematic representation of the second model.

[0072] The second model used in the second method was previously machine-learned. The second model includes a neural network. To obtain better grasp points, it is advantageous if the neural network is a convolutional neural network (CNN) containing convolutional layers.

[0073] The Fig. The second model 200 shown in Figure 11 comprises an input layer 210, intermediate layers 220, and an output layer 230. The image of the object is input to the input layer 210. The intermediate layers 220 comprise convolutional layers. In the example shown, the intermediate layers 220 comprise a ResNet 221 and a fully convolutional network (FCN) 222. The output layer 230 outputs the position (X, Y, Z) and the angle (ϕ, θ, ψ).

[0074] In the Fig. In the example shown in Figure 11, the output layer 230 also outputs the gripping method. Specifically, when the object is gripped by the suction mechanism 56, the output layer 230 outputs the combination (the suction cup pattern) of the suction cups 56a used to grip the object. When the object is gripped by the clamping mechanism 57, the output layer 230 outputs the distance (the width) between the support parts 57a.

[0075] The second model 200 is machine-learned using multiple training data sets. Each training data set contains a combination of the input image and training data. The training data specifies the gripping point at which the object visible in the input image can be reliably grasped, the combination of the suction cups 56a (or the distance between the holding parts 57a), etc. The intermediate layers 220 are trained to output the training data for the input image.

[0076] Or multiple models can be used as the second model. For example, a first submodel and a second submodel are used as the second model. The first submodel outputs the position of the gripper 55 when the object is gripped according to the image input. The second submodel outputs the angle of the gripper 55 when the object is gripped according to the image input and the position output of the first submodel. Third method

[0077] Fig. 12, 13A, 13B, 14 and 15A to 15C are schematic views showing examples of graphical user interfaces.

[0078] When performing the third method, the computing device 20 requests the terminal device 60 to determine the grasping point. The terminal device 60 displays a graphical user interface (GUI) for the user to determine the grasping point. For example, the terminal device 60 displays a graphical user interface for determining the position of the object and a graphical user interface for determining the position and angle of the gripper 55.

[0079] The Fig. The graphical user interface 300 shown in Figure 12 is displayed to determine the position of the object. The GUI 300 consists of an image 301, icons 302 to 305, and a message window 306. The image captured by the sensor 30 is displayed in the image 301. The icon 302 is displayed to select the image displayed in the image 301. For example, the sensor 30 repeatedly and continuously captures images. By clicking the icon 302, the user can select the image to be displayed in the image 301 from the multiple images captured by the sensor 30.

[0080] Icon 303 is displayed to select the labeling method for the object area on the image displayed in image 301. Icon 304 is clicked to determine the outer perimeter of the object for the image displayed in image 301. Icon 305 is clicked to save the position of the taught object. Message window 306 contains instructions for the user. The user enters operations into GUI 310 according to the instructions displayed in message window 306.

[0081] The following labeling methods are available for the icon 303: Manual labeling, labeling by image processing, and labeling by a combination of manual and image processing. The user can select which method to use by clicking on icon 303. With manual labeling, the user sequentially determines the vertices of the object by operating a pointer 307, as shown in Fig. 13A. The user clicks on icon 304 after selecting all vertices. As shown in Fig. As shown in Figure 13B, a line segment connecting the designated vertices is used as the outer boundary of the object, and the region within the outer boundary is enclosed. The region enclosed by the outer boundary is referred to as the position (range) of the object.

[0082] When the label is selected by image processing, the terminal device 60 detects objects based on the image and displays the detected area of each object. For example, edge detection is used to detect the object. For edge detection, the Canny method, the keypoint matching method, or the like can be used. When the label is selected by a combination of manual and image processing, the terminal device 60 first uses image processing to detect the object. The terminal device 60 displays the vertices of the detected object. The user clicks the icon 304 after using the pointer 307 to adjust the positions of the vertices. As a result, the line segment connecting the designated vertices is used as the outer perimeter of the object.

[0083] The Fig. The GUI 310 shown in Figure 14 is displayed to determine the position and angle of the gripper 55. The GUI 310 consists of an image 311, icons 312 to 316, and a message window 317. The image captured by the sensor 30 is displayed in the image 311. In addition, the position of the object being taught in the GUI 300 is represented by a dashed line in the image 311. The icon 312 is displayed to select the image displayed in the image 311. The icon 313 is clicked when the gripping technique of the object is selected. Fig. In the transfer device 50 shown in Figure 2, the gripper 55 includes the two gripping techniques: suction and pinching. By clicking on icon 313, the user can select which of the two gripping techniques, suction and pinching, should be used. The display of icon 313 can be omitted if the gripper 55 includes only one gripping technique.

[0084] Icon 314 is displayed to select the number of suction cups used for grasping when suction grasping is used as the grasping technique. When icon 314 is clicked, the Fig. 15A, the window 320 is displayed. Fig. 15A shows an example in which the gripper 55 contains four suction pads. The gripper 55 includes suction pads 0 through 3. The user can specify which suction pads should be used for gripping by selecting the checkboxes 321 through 324 of window 320 accordingly.

[0085] If the user has marked one or more of the check boxes 321 to 324, the suction cups 311a to 311d and a center point 311e of the suction cups 311a to 311d are displayed on the image 311 of the GUI 310, as shown in Fig. 15B. By clicking on the icon 315, the user can adjust the position and angle of the displayed suction cups 311a to 311d. For example, the user can move the suction cups 311a to 311d by using a pointer 318 to drag and drop the center point 311e. Furthermore, the user can rotate the suction cups 311a to 311d around the center point 311e by dragging and dropping one of the suction cups 311a to 311d.

[0086] Once the number, position, and angle of the suction cups have been finally determined, the user clicks on icon 316. The number, position, and angle of the suction cups specified by the user are then saved.

[0087] When clamping is selected as the gripping technique, the user determines the positions of the support parts of the clamping mechanism. Fig. Figure 15C shows an example in which the clamping mechanism includes two support parts. The user sets the position and angle of each support part in image 311. For example, the user can position the support parts 311f and 311g as shown in Fig. 15C by clicking on the image 311 with the pointer 318.

[0088] The user clicks icon 316 when the support parts 311f and 311g are placed. The support part 311f and the support part 311g are saved as the positions of the support parts. A center point 311h between the support part 311f and the support part 311g is saved as the pinch point. The angle of a line segment from the support part 311f to the support part 311g with respect to the horizontal axes of the image coordinate system is saved as the rotation angle of the clamping mechanism.

[0089] The message window 317 displays an instruction to the user. The user enters operations on the GUI 310 according to the instruction displayed in the message window 317.

[0090] The gripping point is derived when the GUI 310 was used to determine the positions of the suction cups during suction or the positions of the support parts during clamping. The angle of the gripping point is represented by a normal vector of the surface contacted by the suction cups or the surface contacted by the midpoint between the support parts. The inclination of the surface contacted by the suction cups or the surface contacted by the midpoint between the support parts is calculated based on the image. To increase the accuracy of the inclination, it is advantageous if the sensor 30 can capture a depth image.

[0091] Fig. 16 is a flowchart illustrating a processing method according to the embodiment.

[0092] In a Fig. In the processing method M1 shown in Figure 16, the processing system 1 first receives an instruction from the higher-level system of the object to be grasped. When the container in which the object is stored is transferred according to the instruction, the sensor 30 captures an image of the object to be grasped (step S1). The determination device 10 transmits the captured image to the determination part 11 (step S2). The determination part 11 outputs the detection method of the grasping point according to the image input. The determination device 10 receives the output of the detection method from the determination part (step S3). The computing device 20 determines the detection method output by the determination device 10 as one of the first to third methods (step S4).

[0093] If the detection method is the first method, the computing device 20 calculates external shape data representing the external shape of the object based on the image (step S11). The computing device 20 generates multiple combinations (patterns) of the part to be gripped and the gripping point of the gripper 55 (step S12). The computing device 20 calculates the score for each pattern (step S13). The computing device 20 uses the gripping point of the pattern for which the best score was obtained as the actual gripping point to be used (step S14).

[0094] If the acquisition method is the second method, the computing device 20 inputs the image to the second model (step S21). The computing device 20 acquires the grasp point output from the second model (step S22).

[0095] If the detection method is the third method, the computing device 20 requests the terminal device 60 to designate the grasping point (step S31). The terminal device 60 accepts the user's designation of the grasping point (step S32). The computing device 20 receives the grasping point from the terminal device 60 (step S33).

[0096] The control device 40 moves the gripper 55 to the gripping point determined using one of the methods (step S5). The control device 40 causes the gripper 55 to grasp the object at the gripping point (step S6). The control device 40 causes the gripper 55 to transfer the object into another container and deposit it (step S7).

[0097] The advantages of the embodiment will now be described.

[0098] The grasping point is determined when a transfer device grasps an object. The grasping point is represented by a three-dimensional position and a three-dimensional angle. The first method and the second method described above can be regarded as methods for determining the grasping point. In the first method, the external shape data of the object is calculated, and various parameters such as the center of gravity, etc. are calculated based on the external shape data. The grasping point is calculated based on the parameters. In the second method, the second model outputs the grasping point. As a result, there is no need to calculate the external shape data, calculate various parameters based on the external shape data, etc. According to the first method, a grasping point at which the object can be grasped more stably can be calculated compared to the second method.According to the second method, the gripping point can be determined more quickly than the first method.

[0099] Using only the first method to detect the grasp point allows more objects to be grasped more reliably. As a result, errors such as misgrips, dropping objects during handover, etc., can be suppressed.

[0100] If only the second method for detecting the grasping point is used, the grasping point is reached more quickly. As a result, the standby time of the transfer device 50 before the grasping operation can be reduced. For example, the standby time of the transfer device 50 due to detecting the grasping point can be avoided. This allows the number of objects that can be grasped and transferred per unit time to be increased. The processing efficiency of the transfer can be further increased.

[0101] By combining the first method and the second method, the advantages of each method can be utilized. With the first method, a suitable grasping point can be easily obtained even if the object visible in the image has not yet been trained, if the object in the image is difficult to recognize, etc. As a result, the occurrence of grasping errors, dropping the object due to unstable grasping, etc. can be suppressed. With the second method, external shape data and the calculation of parameters and the like based on the external shape data are not required, and the time required for calculation can be reduced. In other words, if the suitable grasping point can be obtained by the second method, the grasping point can be obtained more quickly by performing the second method. For example, if it is difficult for the second method to obtain the grasping point, errors such asGrasping errors can be suppressed by implementing the first method. As a result, the object can be grasped more stably, while processing efficiency is increased.

[0102] Transfer efficiency depends on the time required to transfer the object. The time required to transfer the object includes the calculation time for the control device 40 to operate the transfer device 50, the operation time of the transfer device 50, the retry time when the grasping fails, and so on. If the calculation time by the control device 40 is long, the transfer device 50 must remain on standby until the calculation is completed. In this case, the calculation time can be reduced by determining the grasping point using the second method. Furthermore, grasping errors can be suppressed by using the first method to appropriately detect the grasping point.

[0103] The inventor obtained the following findings by further investigating the combination of the first and second methods. First, by combining the first and second methods, the transfer efficiency can be effectively improved by reducing the average calculation time and reducing the probability of grasping errors. It was found that when combining the first and second methods, long repetition times occurred for the transfer of some objects; and such repetition times reduced the transfer efficiency. For example, grasping errors easily occur for objects that deform easily during grasping, transparent objects, objects that easily reflect light, etc. Even if the first method is used, the success rate of grasping such objects is not high enough for the grasping points.It was found that if the grasp of such an object failed, the grasp failed again during a retry attempt, and the retry time was greatly increased. It was found that a long retry time also occurred when unexpected problems occurred during the transfer process; and such a retry time also reduced the efficiency of the transfer.

[0104] Considering the problems described above, the inventor developed the third method in addition to the first and second methods. In the third method, the grasping point is determined by a person. When a person determines the grasping point, a suitable grasping point can be determined even for objects that are easily deformed, transparent objects, and objects that easily reflect light. Even if unexpected problems occur, a person can take more appropriate countermeasures. This can reduce the probability of an error in grasping. When the grasping point is determined by a person, in addition to the calculation time of the control device 40, the human operation time is also required. However, it has been found that even if the human operation time is added, the transfer efficiency can be increased because the occurrence of a retry, the repetition of retry, etc.can be suppressed.

[0105] According to the embodiment, the method to be used from the first, second, and third methods is output by the determination part 11. Accordingly, the time required for determining the detection method of the grasping point can be reduced, and the transmission efficiency can be further increased. First modification

[0106] The determination device 10 can determine the condition of the object based on the image along with the determination of the grip point detection method. The condition of the object is classified as normal without damage or abnormal with damage. Damage includes a scratch, chip, crack, sharp bend, tear, etc. The determination device 10 inputs the image to a determination part to determine the condition of the object. The determination device 10 detects the condition output by the determination part.

[0107] For example, a determination part for determining the object state can be provided separately from the determination part 11. Advantageously, the determination part 11 assumes the function of the determination part for determining the object state. By determining both the detection method of the grasping point and the state of the object, the time required for the determination part 11 can be shorter than if the detection method of the grasping point and the state of the object were determined by separate determination parts.

[0108] Fig. 17 is a flowchart showing a processing method according to a first modification of the embodiment.

[0109] Compared to the Fig. The processing method shown in Figure 16 contains the Fig. 17, the processing method M2 includes steps S3a and S3b instead of step S3. Here, an example will be described in which the determination part 11 outputs both the detection method and the state of the object. In step S3a, the determination device 10 receives the detection method and the state of the object output from the determination part 11. In step S3b, the determination device 10 determines whether the state of the object is normal or not. If the state of the object is abnormal, the object is not grasped, and the processing ends. If the state of the object is normal, step S4 is performed similarly to the processing method M1. If the object is determined to be abnormal due to damage, another object of the same type is prepared. The processing method M2 is performed again on the other object.

[0110] For example, the object transferred by the transfer device 50 is then shipped. The object to be shipped must be normal and undamaged. To ship normal objects, a method may be considered in which the object is grasped by the transfer device 50, transferred, and then inspected to determine whether the transferred object is normal or not. However, in such a method, the object is first transferred and then inspected, and if the object is abnormal, a normal object must be transferred. In other words, two or more transfer operations are required for one object.

[0111] According to processing method M2, the state of the object is determined before grasping. If the object is determined to be abnormal, grasping is not performed. Accordingly, the probability of grasping and transporting an abnormal object can be reduced. The number of repetitions of the transfer operation due to an abnormal object can be suppressed, and the processing efficiency of the transfer device 50 can be increased.

[0112] To determine the state of the object, it is advantageous to use the image used to detect the grasp point. The image captured by sensor 30 may contain noise. For example, noise such as distortion, blur, "whiteout," "blackout," etc., of the image may be caused by the object reflecting light when the object's lighting conditions are unsuitable, by effects of the object's surface texture, etc. There is a possibility that the grasping success rate will decrease if a noisy image is used to detect the grasp point. To solve this problem, it is useful to use the image used to detect the grasp point to determine the object's state. With this method, the object visible in the image can be classified as abnormal if the noise contained in the image affects the grasping success rate.In other words, it is possible to determine whether the object visible in the image is abnormal or not, in addition to determining whether the object is actually abnormal or not. By classifying the object visible in the image as abnormal when the noise contained in the image affects the grasping success rate, the occurrence of grasping errors can be suppressed. As a result, the processing efficiency of the transfer device 50 can be further increased. The image used to detect the grasping point and the image used to determine the state of the object may not be exactly the same.For example, the images can be considered substantially equal if the imaging conditions of the image used to capture the grasp point and the imaging conditions of the image used to determine the state of the object are substantially equal and / or if the difference between the imaging times is sufficiently small. Imaging conditions include the way the light hits the object, the settings of sensor 30, etc. Training

[0113] Fig. 18 is a schematic view showing a configuration of a processing system according to a modification of the embodiment.

[0114] Compared to processing system 1, the one in Fig. The processing system 1a shown in Figure 18 also includes a training device 70. The training device 70 pre-trains the first and second models. The training device 70 stores the trained first and second models in the storage device 15.

[0115] The training device 70 also retrains the second model based on the results obtained by the processing of the processing system 1a. When the third method is performed, the person in the image indicates the position of the object, the grasping position, and the grasping angle. The image used, the designated position of the object, the designated grasping position, and the designated grasping angle are stored in the storage device 15. The training device 70 trains the second model using the image used in the third method as input data and using the determined position of the object, the determined grasping position, and the determined grasping angle as learning data. This can improve the grasping success rate of the second model.Furthermore, the determination part 11 can be retrained so that the object used to retrain the second model has a higher determination percentage of the second method. This can reduce the determination frequency of the third method for the object used to retrain the second model and further increase the transmission efficiency.

[0116] The training device 70 may be provided separately from the determination device 10 and the calculation device 20, or the determination device 10 or the calculation device 20 may include the function of the training device 70.

[0117] Fig. 19 is a schematic view illustrating a hardware configuration.

[0118] For example, a Fig.19 is used as the determination device 10, the calculation device 20, or the control device 40. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0119] ROM 92 stores programs that control the operation of computer 90. ROM 92 stores programs necessary to cause computer 90 to perform the processing described above. RAM 93 functions as a storage area into which the programs stored in ROM 92 are loaded.

[0120] The CPU 91 contains a processing circuit. The CPU 91 uses the RAM 93 as working memory to execute the programs stored in at least one of the ROM 92 and the storage device 94. When executing the programs, the CPU 91 performs various processing by controlling configurations via a system bus 98.

[0121] The storage device 94 stores the data required for the execution of the programs and / or the data obtained by the execution of the programs.

[0122] The input interface (I / F) 95 can connect the computer 90 and an input device 95a. The input interface 95 is, for example, a serial bus interface such as USB, etc. The CPU 91 can read various data from the input device 95a via the input interface 95.

[0123] The output interface (I / F) 96 can connect the computer 90 and an output device 96a. The output interface 96 is, for example, an image output interface such as Digital Visual Interface (DVI), High-Definition Multimedia Interface (HPMI (registered trademark)), etc. The CPU 91 can transmit data to the output device 96a via the output interface 96 and cause the output device 96a to display an image.

[0124] The communication interface (I / F) 97 can connect the computer 90 and a server 97a external to the computer 90. The communication interface 97 is, for example, a network card such as a LAN card, etc. The CPU 91 can read various data from the server 97a via the communication interface 97.

[0125] The storage device 94 includes at least one selected from a hard disk drive (HDD) and a solid-state drive (SSD). The input device 95a includes at least one selected from a mouse, a keyboard, a microphone (audio input), and a touchpad. The output device 96a includes at least one of the following devices: a monitor, a projector, a printer, and a speaker. A device such as a touch panel can also be used to function as both the input device 95a and the output device 96a.

[0126] The processing performed by the determination device 10, the computing device 20, or the control device 40 may be implemented by one computer 90 or by the cooperation of multiple computers 90. One computer 90 may function as two or more computers selected from the determination device 10, the computing device 20, and the control device 40.

[0127] The processing of the various data described above may be recorded as a program that can be executed by a computer on a magnetic disk (a flexible disk, a hard disk, etc.), an optical disk (CD-ROM, CD-R, CD-RW, DVD-ROM, DVD± R, DVD± RW, etc.), a semiconductor memory or other non-transitory computer-readable storage medium.

[0128] The information recorded on the recording medium can be read by a computer (or an embedded system), for example. The recording format (the storage format) of the recording medium is arbitrary. For example, the computer reads a program from the recording medium and causes a CPU to execute the instructions specified in the program based on the program. In the computer, the acquisition (or reading) of the program can occur over a network.

[0129] The embodiments of the invention include the following features. Feature 1

[0130] A determination device that configures: to capture an image of an object to be grasped; inputting the image into a determination part, the determination part containing a first model that has been trained; and to obtain from the determination part a method for detecting a gripping point at which a gripper should grip the object, according to the input of the image, whereby the determination part outputs as a detection method one of a first method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, thereby training the second model, or a third method for detecting the grasping point by designating the grasping point. Feature 2

[0131] The determination device according to feature 1, in which the determination part outputs the detection method and classifies a state of the object as normal without damage or abnormal with damage. Feature 3

[0132] The determination device according to feature 2, in which the first model contains a neural network and the first model outputs the detection method and a classification result of the object state. Feature 4

[0133] A processing system including: the determining device according to one of features 1 to 3; and a computing device configured to detect the grasping point using the detection method determined by the determining device. Feature 5

[0134] The processing system according to feature 4, further comprising: a transfer device containing the gripper; and a control device that moves the gripper to the gripping point determined by the computing device. Feature 6

[0135] The processing system according to feature 5, further comprising: a terminal device connected to the computing device via a network, when the determining device determines the detection method as the third method, the computing device requests the terminal device to designate the grasping point and receives the grasping point designated by the terminal device. Feature 7

[0136] The processing system according to feature 6, in which the terminal device displays the image and accepts a designation of the grasping point on the image. Feature 8

[0137] The processing system according to any one of features 4 to 7, further comprising: a training device configured to train the second model, the training device trains the second model using the image and the grasping point designated by the third method. Feature 9

[0138] A transfer system including: a transfer device having a gripper, the transfer device being configured to handle an object with the gripper; and a determining device configured to capture an image of an object, inputting the image into a determination part with a first model, wherein the first model is trained, and to obtain from the determination part a method for detecting a gripping point at which the gripper should grip the object, according to the input of the image, whereby the determination part outputs as a detection method one of a first method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, wherein the second model is trained, or a third method for detecting the grasping point by designating the grasping point. Feature 10

[0139] A determination method including: to cause a computer to to capture an image of an object to be grasped, inputting the image into a determination part with a first model, wherein the first model is trained, and a detection method of a gripping point at which a gripper is to grasp the object from the determination part, according to the input of the image, wherein the determination part outputs as a detection method one of a first method for calculating outer shape data of the object based on the image and for calculating the grasping point using the outer shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, wherein the second model is trained, or a third method for detecting the grasping point by designating the grasping point. Feature 11

[0140] A program which, when executed by the computer according to feature 10, causes the computer to perform the determination method according to feature 10. Feature 12

[0141] A storage medium configured to store the program in accordance with feature 11.

[0142] According to the above embodiments, a determination apparatus, a processing system, a transmission system, a determination method, a program, and a storage medium are provided, with which the efficiency of transmission processing can be increased.

[0143] While specific embodiments have been described, these embodiments have been presented only by way of example and are not intended to limit the scope of the inventions. Indeed, the novel embodiments described herein may be embodied in a variety of other forms; furthermore, various omissions, substitutions, and changes may be made in the form of the embodiments described herein without departing from the spirit of the inventions. The appended claims and their equivalents are intended to cover such forms or modifications that would fall within the scope and spirit of the invention. Furthermore, the above embodiments may be combined and practiced with one another. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] JP-A 2021-037608

[0057] JP-A 2021-146434

[0070]

Claims

[1] Determination device configured to: to capture an image of an object to be grasped; inputting the image into a determination part, the determination part containing a first model that has been trained; and to obtain from the determination part a detection method of a gripping point at which a gripper should grip the object, according to the input of the image, whereby the determination part outputs as a detection method one of a first method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, wherein the second model is trained, or a third method for detecting the grasping point by designating the grasping point. [2] The determination device according to claim 1, wherein the determination part outputs the detection method and classifies a state of the object as normal without damage or abnormal with damage. [3] The determination device according to claim 2, wherein the first model includes a neural network, and the first model outputs the detection method and a classification result of the object state. [4] Determining device according to claim 1, wherein the first model includes an encoder configured to output a variable or probability distribution according to the input image, and a clustering model configured to cluster the encoder's output. [5] The determination device according to claim 1, wherein the cluster model is configured to cluster the output of the encoder into a first cluster corresponding to the first method, a second cluster corresponding to the second method, and a third cluster corresponding to the third method. [6] Processing system comprising: the determining device according to one of claims 1 to 5; and a computing device configured to detect the grasping point using the detection method determined by the determining device. [7] Processing system according to claim 6, further comprising: a transfer device containing the gripper; and a control device that moves the gripper to the gripping point determined by the computing device. [8] Processing system according to claim 7, further comprising: a terminal device connected to the computing unit via a network, when the determining device determines the detection method as the third method, the computing device requests the terminal device to determine the grasping point and receives the grasping point determined by the terminal device. [9] The processing system according to claim 8, wherein the terminal device displays the image and accepts a designation of the grasp point on the image. [10] Processing system according to one of claims 6 to 9, further comprising: a training device configured to train the second model, wherein the training device trains the second model using the image and the grasping point designated by the third method. [11] Transfer system comprising: a transfer device having a gripper, the transfer device being configured to handle an object with the gripper; and a determining device configured take a picture of an object, inputting the image into a determination part with a first model, wherein the first model is trained, and to obtain from the determination part a detection method of a gripping point at which the gripper should grip the object, according to the input of the image, whereby the determination part outputs as a detection method one of a first method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, wherein the second model is trained, or a third method for detecting the grasping point by designating the grasping point. [12] Determination method comprising: to cause a computer to to capture an image of an object to be grasped, inputting the image into a determination part with a first model, wherein the first model is trained, and from the determination part a method for detecting a gripping point at which a gripper should grip the object, according to the input of the image, whereby the determination part outputs as a detection method one of a first method for calculating external shape data of the object based on the image and for calculating the gripping point using the external shape data, a second method for inputting the image to a second model and capturing the grasp point output from the second model, wherein the second model is trained, or a third method for detecting the grasping point by designating the grasping point. [13] Determination method according to claim 12, wherein the first model includes an encoder configured to output a variable or probability distribution according to the input image, and a clustering model configured to cluster the encoder's output. [14] The determination method according to claim 13, wherein the cluster model is configured to cluster the output of the encoder into a first cluster corresponding to the first method, a second cluster corresponding to the second method, and a third cluster corresponding to the third method. [15] Storage medium configured to: to save a program, the program, when executed by the computer according to any one of claims 12 to 14, causes the computer to perform the determination method according to any one of claims 12 to 14.

Images