Determination device, processing system, conveyance system, determination method, program, and memory medium

Abstract

To provide a determination device, a processing system, a conveyance system, a determination method, a program, and a memory medium, which can improve the processing efficiency.SOLUTION: A determination device obtains an image in which an object to be gripped is imaged. The determination device inputs the image to a determination tool including a learnt first model. The determination device obtains, from the determination tool, a method for obtaining a gripping point for a gripping part to grip an object. The determination tool outputs any of a first method, a second method, and a third method, in response to input of the image. In the first method, data on an outer shape of the object is calculated from the image and the gripping point is calculated using the data on the outer shape. In the second method, the image is inputted to a learnt second model and the gripping point outputted from the second model is obtained. In the third method, a designated gripping point is obtained.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] FIELD Embodiments of the present invention relate to a determination apparatus, a processing system, a transport system, a determination method, a program, and a storage medium. [Background technology]

[0002] There are devices that grip and transport objects, and there is a demand for technology that can improve the processing efficiency of these transport devices. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] US Patent Application Publication No. 2023 / 205184 Summary of the Invention [Problem to be solved by the invention]

[0004] The problem to be solved by the present invention is to provide a determination device, a processing system, a transport system, a determination method, a program, and a storage medium that can improve processing efficiency. [Means for solving the problem]

[0005] A determination device according to an embodiment acquires an image of an object to be grasped. The determination device inputs the image to a determiner including a trained first model. The determination device obtains from the determiner a method for acquiring a grip point at which a gripper grasps the object. The determiner outputs one of a first method, a second method, and a third method in response to the input of the image. In the first method, outline data of the object is calculated from the image, and the grip point is calculated using the outline data. In the second method, the image is input to a trained second model, and the grip point output from the second model is acquired. In the third method, the specified grip point is acquired. [Brief explanation of the drawings]

[0006] [Figure 1] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. [Figure 2] FIG. 2 is a schematic diagram illustrating a specific configuration of the processing system. [Figure 3] FIG. 3 is a schematic diagram illustrating an example of a determiner. [Figure 4] FIG. 4 is a schematic diagram showing another example of the determiner. [Figure 5] 5(a) and 5(b) are schematic diagrams showing an example of a method for learning a classifier. [Figure 6] FIG. 6 is a perspective view illustrating a method for calculating the outer shape data of an object. [Figure 7] FIG. 7 is a plan view illustrating a method for calculating the outer shape data of an object. [Figure 8] FIG. 8 is a schematic diagram for explaining distance data indicating the distance between the center of the contact surface and the center of gravity of the object. [Figure 9] FIG. 9 is a schematic diagram showing the state when the suction mechanism grips an object. [Figure 10] FIG. 10 is a schematic diagram showing how the clamping mechanism grips an object. [Figure 11] FIG. 11 is a schematic diagram illustrating the second model. [Figure 12] FIG. 12 is a schematic diagram showing an example of a graphical user interface. [Figure 13] 13(a) and 13(b) are schematic diagrams showing an example of a graphical user interface. [Figure 14] FIG. 14 is a schematic diagram showing an example of a graphical user interface. [Figure 15] 15(a) to 15(c) are schematic diagrams showing an example of a graphical user interface. [Figure 16] FIG. 16 is a flowchart illustrating a processing method according to an embodiment. [Figure 17]FIG. 17 is a flowchart showing a processing method according to a first modified example of the embodiment. [Figure 18] FIG. 18 is a schematic diagram showing the configuration of a processing system according to a modified example of the embodiment. [Figure 19] FIG. 19 is a schematic diagram showing the hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0007] Hereinafter, each embodiment of the present invention will be described with reference to the drawings. The drawings are schematic or conceptual, and the relationship between the thickness and width of each part, the size ratio between parts, etc. are not necessarily the same as those in reality. Furthermore, even when the same part is shown, the dimensions and ratios may be different depending on the drawing. In this specification and each drawing, elements similar to those already described are designated by the same reference numerals, and detailed description will be omitted as appropriate.

[0008] FIG. 1 is a schematic diagram showing the configuration of a processing system according to an embodiment. The processing system 1 according to the embodiment includes a determination device 10, a storage device 15, a calculation device 20, a sensor 30, a control device 40, a transport device 50, and a terminal device 60.

[0009] The transport device 50 is capable of transporting an object. For example, the transport device 50 grasps an object stored in a container. The transport device 50 lifts the grasped object and transports it to another container. The transport device 50 releases the grasped object in the other container. The object is placed in the container and stored therein. Here, a series of operations including grasping, transporting, and placing is referred to as "picking." For example, the transport device 50 is a picking robot.

[0010] The transport device 50 includes a gripping unit 55. The gripping unit 55 can grip an object by clamping, suction, jamming, or the like. The gripping unit 55 includes both a clamping mechanism and a suction mechanism, and either mechanism may be selectively used.

[0011] The control device 40 controls the transport device 50. The control device 40 is a so-called robot controller. When the transport device 50 grasps an object, the control device 40 moves the gripping unit 55 to a gripping point. The "grasping point" is represented by a three-dimensional position (X, Y, Z) and a three-dimensional angle (φ, θ, ψ). The gripping unit 55 grasps the object at the gripping point.

[0012] The sensor 30 detects the object being grasped. For example, the sensor 30 is an image sensor. A camera including the sensor 30 is used. The camera captures an image of the object and acquires an image. The camera may also acquire a video. In this case, a still image is extracted from the video. The sensor 30 acquires an RGB image or a depth image. Preferably, both an RGB image and a depth image are acquired. A sensor 30 for acquiring an RGB image and another sensor 30 for acquiring a depth image may be used.

[0013] Alternatively, the sensor 30 may be a distance measurement sensor. For example, a laser range finder (LRF) including the sensor 30 is used. The LRF can measure the distance to surrounding objects and acquire a distance image. Both a camera and an LRF may be used as the sensor 30.

[0014] The arithmetic device 20 acquires a gripping point for the gripping unit 55 to grip an object. The determination device 10 determines a method for acquiring the gripping point. The arithmetic device 20 executes the acquisition method determined by the determination device 10. The storage device 15 stores data for processing by the determination device 10 and the arithmetic device 20, data obtained by these processes, etc.

[0015] The determination device 10 uses a determiner 11 to determine the acquisition method. The determiner 11 includes a trained first model. The determiner 11 outputs a method for acquiring a grip point in response to an input of an image. The determination device 10 inputs an image acquired by the sensor 30 to the determiner 11, and obtains an acquisition method output from the determiner 11. The determiner 11 outputs any one of the first method, the second method, or the third method as the acquisition method. The calculation device 20 executes any one of the first method, the second method, or the third method.

[0016] In the first method, the arithmetic device 20 calculates contour data indicating the contour (outline) of the object from an image. The arithmetic device 20 generates combinations (patterns) of the parts to be grasped in the contour and the grasping points of the grasping unit 55. The arithmetic device 20 generates a large number of patterns while changing the parts to be grasped, the position of the grasping unit 55, the posture of the grasping unit 55, etc. The arithmetic device 20 calculates a score for each pattern. The grasping points of the pattern with the best score are adopted as the grasping points to be actually used.

[0017] In the second method, the computing device 20 inputs an image into a trained second model. The second model is trained in advance by machine learning to calculate grip points in response to the input image. The second model includes a neural network. The computing device 20 acquires the grip points output from the second model.

[0018] In the third method, a gripping point is designated by a person, and the arithmetic device 20 acquires the designated gripping point. First, the arithmetic device 20 requests the terminal device 60 to designate a gripping point. Next, the user of the terminal device 60 designates a gripping point in response to the request from the arithmetic device 20. The terminal device 60 accepts the gripping point designated by the user and transmits it to the arithmetic device 20. The arithmetic device 20 acquires the gripping point transmitted from the terminal device 60.

[0019] When an object is grasped, the computing device 20 executes any one of the first method, the second method, and the third method to obtain a grasp point, and when the object is grasped, the computing device 20 does not execute the remaining two of the first method, the second method, and the third method.

[0020] The determination device 10, the storage device 15, the arithmetic device 20, and the sensor 30 are connected to one another via a network, wireless communication, or wired communication. One device may have the functions of both the determination device 10 and the arithmetic device 20. The control device 40 and the terminal device 60 are each connected to the arithmetic device 20 via a network, wireless communication, or wired communication.

[0021] The determination device 10 transmits the selected acquisition method to the calculation device 20. The sensor 30 transmits the acquired image to the calculation device 20. The calculation device 20 acquires the gripping point using the received acquisition method. The calculation device 20 transmits the acquired gripping point to the control device 40. The control device 40 moves the gripping unit 55 to the received gripping point and causes the gripping unit 55 to grip the object at that gripping point. The transport device 50 transports the gripped object. The processing system 1 functions as a transport system that transports (handles) objects.

[0022] The invention according to the embodiment will be described in more detail below.

[0023] (Transportation device) FIG. 2 is a schematic diagram illustrating a specific configuration of the processing system. 2, the transfer device 50 is a vertical articulated robot. The transfer device 50 includes a manipulator 51 including a plurality of links 51a and a plurality of rotation shafts 51b. The links 51a are connected to each other by the rotation shafts 51b.

[0024] Operation of each rotation axis 51b changes the position and angle of the tip of the manipulator 51. The tip of the manipulator 51 preferably has six degrees of freedom. A gripping unit 55 is attached to the tip of the manipulator 51. In the illustrated example, the gripping unit 55 includes a suction mechanism 56 and a clamping mechanism 57.

[0025] The suction mechanism 56 grips an object by suction. The suction mechanism 56 includes one or more suction pads 56a. With the suction pads 56a in contact with the object, the pressure inside the suction pads 56a is reduced by a pressure reducing device (not shown). This causes the object to be adsorbed to the suction pads 56a. The number of suction pads 56a may be greater or less than that shown in the example.

[0026] The clamping mechanism 57 grips an object by pinching. The clamping mechanism 57 includes a plurality of rod-shaped support portions 57a. The object is pinched and gripped by the plurality of support portions 57a. The clamping mechanism 57 may include more support portions 57a than the example shown in the figure. The support portions 57a may be configured in a finger shape including one or more joints.

[0027] The gripping unit 55 further includes a switching mechanism 58. The suction mechanism 56 and the clamping mechanism 57 are connected to the switching mechanism 58. The switching mechanism 58 rotates the suction mechanism 56 and the clamping mechanism 57. By rotating the suction mechanism 56 and the clamping mechanism 57, it is possible to switch the mechanism used to grip the object.

[0028] Not limited to the illustrated example, the gripping unit 55 may include only one of the suction mechanism 56 and the clamping mechanism 57. In that case, the switching mechanism 58 is not necessary.

[0029] Two containers C1 and C2 are placed near the transport device 50. The transport device 50 grasps an object O contained in the container C1 and transports it to the container C2.

[0030] The sensor 30 is provided to detect the internal condition of the container C1. The sensor 30 detects an object O contained in the container C1 from above. In addition to the sensor 30 for detecting the internal condition of the container C1, another sensor 30 may be provided to detect the internal condition of the container C2. The sensor 30 may be attached to the conveying device 50.

[0031] In addition to the examples shown in the drawings, the conveying device 50 may be a parallel link robot including a gripping unit 55. The conveying device 50 may be an air vehicle (such as a drone) including a gripping unit 55. Whichever conveying device 50 is used, the gripping unit 55 of the conveying device 50 grips the object at the gripping point acquired by the computing device 20.

[0032] (1st model) FIG. 3 is a schematic diagram illustrating an example of a determiner. The first model of the determiner 11 preferably includes a neural network. For example, the first model includes a convolutional neural network. FIG. 3 shows an example of the first model including a convolutional neural network. In the example shown in FIG. 3, a convolutional neural network based on EfficientNetV2 is used.

[0033] Specifically, the first model 100 shown in FIG. 3 includes blocks 101 to 110. The numbers written below the blocks 101 to 110 indicate the number of layers in each block. The blocks 101 to 108 perform convolution of data. The blocks 101 to 108 repeatedly extract feature points from an image and compress the extracted feature points. The block 109 further reduces the data compressed by the blocks 101 to 108 by pooling. The block 110 performs full coupling on the data output from the block 109.

[0034] For example, the block 110 outputs a probability for each of the first, second, and third methods. The probability indicates the possibility that the object can be grasped when each method is executed. The method with the highest probability indicates that it is the most suitable acquisition method. The determination device 10 determines the method with the highest probability as the acquisition method.

[0035] The first model is machine-learned in advance. Supervised learning can be used for machine learning. In supervised learning, multiple data sets are used. Each data set includes training images and labels. The training images depict objects to be grasped. The labels indicate the optimal method for grasping the object depicted in the training images. Either the first method, the second method, or the third method is set as the label. The first model is repeatedly trained so as to output the method set in the label in response to the input of training images.

[0036] FIG. 4 is a schematic diagram showing another example of the determiner. As shown in FIG. 4, the first model of the classifier 11 may include a variable autoencoder (VAE) 120 and a clustering model 130. The VAE 120 includes an encoder 121 and a decoder 122. When an image is input to the VAE 120, the encoder 121 reduces the dimension of the image to latent variables. The latent variables are variables derived from the input data and indicate characteristics of the data. The decoder 122 restores the latent variables to the original image. The determination device 10 inputs the image to the VAE 120 and obtains the latent variables output from the encoder 121. The determination device 10 inputs the latent variables to the clustering model 130. The clustering model 130 is machine-trained in advance to cluster the latent variables. The clustering model 130 clusters the input latent variables into any of the first method clusters, the second method clusters, and the third method clusters.

[0037] 5(a) and 5(b) are schematic diagrams showing an example of a method for learning a classifier. A plurality of training images are used for training the VAE 120. Each training image depicts an object to be grasped. The VAE 120 is trained to output an image identical to the input training image. After the VAE 120 has trained, a plurality of training images are sequentially input to the VAE 120. Each time a training image is input to the VAE 120, a latent variable output from the encoder 121 is acquired. As a result, a plurality of latent variables 125 are obtained, as shown in FIG. 5(a). The clustering model 130 clusters the plurality of latent variables 125. Unsupervised learning such as the k-means method is used for the clustering. The number of clusters is set to three: a cluster of the first method, a cluster of the second method, and a cluster of the third method. As a result of the clustering, the plurality of latent variables 125 are clustered into three clusters: a cluster CL1 of the first method, a cluster CL2 of the second method, and a cluster CL3 of the third method, as shown in FIG. 5(b).

[0038] Alternatively, the determinator 11 may include a classification model that outputs the probability of each of the first method, the second method, and the third method in response to the input of the latent variable 125. In this case, the classification model is machine-learned in advance by supervised learning. When supervised learning is performed, multiple data sets are used, similar to the learning of the first model 100 described above. Each data set includes a latent variable based on the input of an image and a label. Either the first method, the second method, or the third method is set as the label. The classification model is repeatedly trained so as to output the method set as the label in response to the input of the latent variable.

[0039] (1st method) In the first method, multiple gripping points are comprehensively searched and a safety factor is estimated for each gripping point. The safety factor indicates the degree of possibility that the object O can be transported without dropping it, and corresponds to the score described above. The gripping point with the highest safety factor is selected from the multiple gripping points.

[0040] An example of a method for calculating the safety factor when the suction mechanism 56 is used will be described. The calculation device 20 acquires data detected by the sensor 30 and recognizes the status of various elements related to the control of the transport device 50. For example, the calculation device 20 performs predetermined image processing on the image to calculate "object outer shape data," "object center of gravity data," and the like, which indicate the status of various elements.

[0041] Fig. 6 is a perspective view showing a typical example of a method for calculating the outer shape data of an object, and Fig. 7 is a plan view showing a typical example of a method for calculating the outer shape data of an object. The "object outer shape data" is calculated using an image of the object O. The object outer shape data indicates the outer shape of the object O contained in the container C1 from which the object O was taken. For example, as shown in FIG. 6, the object outer shape data includes data on a first face F1 and a second face F2 of a rectangular parallelepiped shape circumscribing the object O. The second face F2 is adjacent to the first face F1. If the object surface is not flat (if the object surface has irregularities), as shown in FIG. 7, the calculation device 20 recognizes the rectangular parallelepiped shape circumscribing the object surface as the object outer shape data. The calculation device 20 recognizes the outer shape of the object when viewed from a specific direction as the object's holdable area Fc. The holdable area Fc is a flat portion of the object surface that can be adsorbed.

[0042] The calculation device 20 calculates a safety factor based on the contact area data and the distance data. The contact area data indicates the area of contact between the gripping portion 55 and the holdable region of the object O. The distance data indicates the distance L between the center K of the contact surface and the center of gravity G of the object O. The contact surface is the surface where the gripping portion 55 and the object O come into contact.

[0043] The pressure that can adhere to the contact surface is defined as the "adhesion pressure." The stress obtained by dividing the distance L by the moment of inertia I with the contact surface as the cross section is defined as the "divided stress value." The calculation device 20 calculates the safety factor R based on the value obtained by dividing the adhe- sion pressure by the divided stress value. The safety factor R is the value obtained by dividing the adhe- sion pressure at the gripping point by the sum of the bending stress and other tensile stresses that occur. The bending stress is calculated using the following formula (1).

[0044]

number

[0045] Fig. 8 is a schematic diagram for explaining distance data indicating the distance between the center of the contact surface and the center of gravity of the object, and Fig. 9 is a schematic diagram showing the state when the suction mechanism grips the object. In equation (1), σ(x) is the bending stress, M is the moment, I is the second moment of area, and x is the distance from the neutral axis. Figure 8 shows a model of a structure when bending stress acts on it. As shown in Figure 8, when bending stress acts on a structure, tensile stress and compressive stress are generated in the structure.

[0046] When bending stress acts on a structure, the structure will break if the maximum bending stress is greater than the tensile stress that the structure can withstand. In the case of suction, when the maximum bending stress is greater than the vacuum pressure of each suction pad 56a, it can be assumed that the suction pad 56a will peel off from the object O that it is suctioning. The safety factor R is calculated using the following equation (2).

[0047]

number

[0048] In equation (2), P is the vacuum pressure (suction pressure) of any holding method, σ is the bending stress, and Ts is other generated tensile stress. In this embodiment, it is represented by other generated tensile stress Ts. The safety factor R is the value obtained by dividing the suction pressure at the gripping point by the bending stress. If the tensile stress Ts is omitted, the safety factor R is calculated using the following equation (3).

[0049]

number

[0050] In equation (3), M is the moment determined by the distance L between the center K and the center of gravity G of the object. r is the shortest distance between the contour of the suction surface and the center of gravity G of the object. I is the second moment of area determined by any holding method. The example in FIG. 9 shows a case where the center of gravity G of the object is located outside the suction surface. In this case, the shortest distance r is the distance between the contour of the effective suction pad 56a1 closest to the center of gravity G of the object and the center of gravity G of the object. The effective suction pad 56a1 refers to the suction pad 56a of the multiple suction pads 56a that is used to hold the object O. The part of the suction surface that is most likely to peel off is the contour of the effective suction pad 56a1 closest to the center of gravity G of the object.

[0051] As described above, in order to calculate the safety factor R, it is necessary to calculate the moment of inertia I. In the case of a gripping unit 55 having multiple suction pads 56a, the number of combinations of effective suction pads 56a1 used for gripping is determined according to the number of suction pads 56a whose internal pressures can be controlled independently. When the number of suction pads 56a whose internal pressures can be controlled independently is N, the number Q of combinations of effective suction pads 56a1 used for gripping is calculated by the following formula (4). For example, when the internal pressures of five suction pads 56a can be controlled independently, the number Q is 31.

[0052]

number

[0053] The arrangement directions of the suction pads 56a are defined as a first arrangement direction and a second arrangement direction. The first arrangement direction and the second arrangement direction intersect with each other. The calculation device 20 calculates the geometric moment of inertia I about the first arrangement direction and the second arrangement direction while rotating the group of effective suction pads 56a1 by 1 degree up to 180 degrees along a plane parallel to the first arrangement direction and the second arrangement direction. The calculation device 20 performs this calculation for all combinations.

[0054] The computing device 20 calculates the above-mentioned safety factor R for each obtained second moment I. For example, if the number Q is 31, there are 31×181=5611 patterns of combinations of effective suction pads 56a1 and angles of the suction mechanisms 56. The computing device 20 calculates the second moment I and the safety factor R for each pattern. The computing device 20 selects the pattern with the highest safety factor R as the gripping point to be actually used for gripping.

[0055] Specific methods for calculating the safety factor when using the adsorption mechanism 56 are discussed, for example, in paragraphs

[0061] to

[0096] of Patent Publication 2021-037608.

[0056] An example of a method for calculating the safety factor when the clamping mechanism 57 is used will be described. Here, a method for calculating the safety factor when two support parts 57a hold the object O will be described. When the clamping mechanism 57 includes three or more support parts 57a, the calculation device 20 can approximate the holding state of the object O as a state in which the object O is held by two support parts 57a, and calculate the safety factor.

[0057] FIG. 10 is a schematic diagram showing how the clamping mechanism grips an object. The calculation device 20 calculates the safety factor by using multiple parameters. The multiple parameters include at least the diameter D, the distance d, the length L, and the gravity mg. The diameter D is a parameter related to the size of the area where the support portion 57a and the object O contact each other. Hereinafter, this area will be referred to as the contact area CR1. The diameter D is the diameter of a circle Ci1 inscribed in the outline of the contact area CR1. In this example, the shape of the contact area CR1 is approximated as the circle Ci1. In this case, the area of the contact area CR1 is estimated to be small, so the calculation device 20 can calculate the safety factor by taking into greater consideration the possibility of the object O falling.

[0058] Distance d is a parameter related to position P1 where the maximum bending stress (torsion stress) occurs in contact area CR1. Position P1 is the point on the contour of contact area CR1 that is farthest from the object's center of gravity G. Distance d is the distance between position P1 and center position P2 of contact area CR1.

[0059] Length L is a parameter related to the bending moment (torsion moment) that generates bending stress in contact area CR1. Length L indicates the length of the arm where the bending moment occurs. Length L is the distance between position P1 and a vertical line SL1 that passes through the center of gravity G of the object. Gravity mg is a parameter related to the weight of object O. Gravity mg is the product of the mass m of object O and the magnitude g of gravitational acceleration. Torque T, which is the bending moment generated at position P1, is expressed by the following equation (5).

[0060]

number

[0061] The polar moment of area Ip of the circle Ci1, which is an approximate shape of the contact area CR1, is expressed by the following equation (6).

[0062]

number

[0063] The bending stress τ generated by the torque T is expressed by the following equation (7).

[0064]

number

[0065] The friction pressure Fp generated in the contact area CR1 is expressed by the following equation (8).

[0066]

number

[0067] In equation (8), parameter f is the holding force of support portion 57a, parameter μ is the friction coefficient according to object O, and parameter A is the area of contact region CR1. Within contact region CR1, friction pressure Fp occurs in all directions. The safety factor R is expressed by the following equation (9).

[0068]

number

[0069] When vertical downward stress is taken into consideration, the safety factor R is expressed by the following equation (10).

[0070]

number

[0071] The calculation device 20 calculates the diameter D, the distance d, and the length L based on the object outer shape data and the object center of gravity data. The calculation device 20 acquires mass data indicating the mass m of the object O from the storage device 15. The calculation device 20 calculates the safety factor R according to the above formula.

[0072] The calculation device 20 repeatedly calculates the safety factor while changing the value of at least one of the multiple parameters. As a result, multiple safety factors corresponding to multiple states are calculated. The calculation device 20 selects the state that has the highest safety factor as the grip point.

[0073] Specific methods for calculating the safety factor when using the clamping mechanism 57 are discussed, for example, in paragraphs

[0052] to

[0107] of Patent Publication 2021-146434.

[0074] (Second method) FIG. 11 is a schematic diagram illustrating the second model. The second model used in the second method is machine-learned in advance. The second model includes a neural network. To obtain a better gripping point, the neural network is preferably a convolutional neural network (CNN) including a convolutional layer.

[0075] The second model 200 shown in FIG. 11 includes an input layer 210, an intermediate layer 220, and an output layer 230. An image of an object is input to the input layer 210. The intermediate layer 220 includes a convolutional layer. In the illustrated example, the intermediate layer 220 includes a ResNet 221 and a fully convolutional network (FCN) 222. The output layer 230 outputs a position (X, Y, Z) and an angle (φ, θ, ψ).

[0076] 11, the output layer 230 further outputs a gripping method. Specifically, when an object is gripped by the suction mechanism 56, the output layer 230 outputs a combination (pad pattern) of suction pads 56a used to grip the object. When an object is gripped by the clamping mechanism 57, the output layer 230 outputs a spacing (width) between the support portions 57a.

[0077] The second model 200 is machine-learned using a plurality of training data. Each training data includes a combination of an input image and training data. The training data indicates gripping points that can safely grip an object shown in the input image, the combination of suction pads 56a (or the spacing between support portions 57a), etc. The intermediate layer 220 is trained so that the training data is output for the input image.

[0078] Alternatively, multiple models may be used as the second model. For example, a first sub-model and a second sub-model are used as the second model. The first sub-model outputs the position of the gripper 55 when the object is gripped in response to an input of an image. The second sub-model outputs the angle of the gripper 55 when the object is gripped in response to an input of an image and the position output from the first sub-model.

[0079] (3rd method) 12, 13(a), 13(b), 14, and 15(a) to 15(c) are schematic diagrams showing examples of graphical user interfaces. When the third method is executed, the computing device 20 requests the terminal device 60 to specify a gripping point. The terminal device 60 displays a graphical user interface (GUI) for the user to specify the gripping point. For example, the terminal device 60 displays a GUI for specifying the position of an object and a GUI for specifying the position and angle of the gripping portion 55.

[0080] GUI 300 shown in FIG. 12 is displayed for specifying the position of an object. GUI 300 includes image 301, icons 302 to 305, and a message window 306. Image 301 displays an image acquired by sensor 30. Icon 302 is displayed for selecting an image to be displayed in image 301. For example, sensor 30 repeatedly and continuously acquires images. By clicking icon 302, the user can select an image to be displayed in image 301 from the multiple images acquired by sensor 30.

[0081] Icon 303 is displayed to select a method for annotating the range of an object from the image displayed in image 301. Icon 304 is clicked when specifying the perimeter of an object in the image displayed in image 301. Icon 305 is clicked when saving the taught position of the object. Message window 306 shows instructions to the user. The user inputs operations on GUI 310 according to the instructions displayed in message window 306.

[0082] Regarding the icon 303, annotation methods include manual annotation, annotation by image processing, and annotation that combines manual and image processing. The user can select which method to use by clicking the icon 303. When performing manual annotation, as shown in FIG. 13(a), the user operates the pointer 307 to sequentially specify the vertices of the object. After specifying all the vertices, the user clicks the icon 304. As shown in FIG. 13(b), the line segments connecting the specified vertices are used as the perimeter of the object, and the area inside the perimeter is enclosed. The area enclosed by the perimeter is taught as the position (range) of the object.

[0083] When annotation by image processing is selected, the terminal device 60 detects objects from the image and displays the range of each detected object. For example, edge detection is used to detect objects. The Canny method, key point matching, or the like can be used for edge detection. When annotation by combining manual and image processing is selected, the terminal device 60 first detects each object by image processing. The terminal device 60 displays each vertex of each detected object. The user adjusts the position of each vertex using the pointer 307, and then clicks the icon 304. As a result, the line segments connecting the specified vertices are used as the perimeter of the object.

[0084] GUI 310 shown in FIG. 14 is displayed for specifying the position and angle of gripper 55. GUI 310 includes image 311, icons 312 to 316, and a message window 317. Image 311 displays an image acquired by sensor 30. Image 311 also displays the position of the object instructed in GUI 300 using a dashed line. Icon 312 is displayed for selecting the image to be displayed in image 311. Icon 313 is clicked to select the object gripping method. When conveyance device 50 shown in FIG. 2 is used, gripper 55 has two gripping methods, suction and clamping. By clicking icon 313, the user can select which gripping method to use, suction or clamping. When gripper 55 has only one gripping method, display of icon 313 may be omitted.

[0085] When suction is used as the gripping method, icon 314 is displayed to select the number of suction pads to be used for gripping. When icon 314 is clicked, window 320 shown in FIG. 15(a) is displayed. FIG. 15(a) shows an example in which gripping unit 55 includes four suction pads. Grip unit 55 includes suction pads 0 to 3. The user can specify which suction pads to use for gripping by appropriately checking check boxes 321 to 324 in window 320.

[0086] When the user checks one or more of the check boxes 321 to 324, suction pads 311a to 311d and their center points 311e are displayed on the image 311 of the GUI 310, as shown in FIG. 15(b). The user can adjust the positions and angles of the displayed suction pads 311a to 311d by clicking on an icon 315. For example, the user can translate the suction pads 311a to 311d by operating the pointer 318 and dragging and dropping the center point 311e. The user can also rotate the suction pads 311a to 311d around the center point 311e by dragging and dropping any of the suction pads 311a to 311d.

[0087] Once the number, positions, and angles of the suction pads have been determined, the user clicks icon 316. This saves the number, positions, and angles of the suction pads specified by the user.

[0088] When clamping is selected as the gripping method, the user specifies the position of each support part of the clamping mechanism. FIG. 15(c) shows an example in which the clamping mechanism includes two support parts. The user specifies the position and angle of each support part on the image 311. For example, the user can place support parts 311f and 311g as shown in FIG. 15(c) by clicking pointer 318 on the image 311.

[0089] Once supports 311f and 311g are positioned, the user clicks icon 316. Supports 311f and 311g are saved as the positions of each support. Midpoint 311h between supports 311f and 311g is saved as the clamping point. The angle of the line segment from support 311f to support 311g with respect to the horizontal axis of the image coordinate system is saved as the rotation angle of the clamping mechanism.

[0090] A message window 317 displays instructions to the user. The user inputs operations on the GUI 310 in accordance with the instructions displayed in the message window 317.

[0091] When the position of the suction pad in suction or the position of the support part in clamping is specified by the GUI 310, the gripping point is derived. The angle of the gripping point is expressed as the normal vector of the surface where the suction pad contacts or the surface where the midpoints of the support parts contact. The tilt of the surface where the suction pad contacts or the surface where the midpoints of the support parts contact is calculated from the image. To improve the accuracy of the tilt, it is preferable that the sensor 30 be able to acquire a depth image.

[0092] FIG. 16 is a flowchart illustrating a processing method according to an embodiment. In processing method M1 shown in FIG. 16, first, processing system 1 receives an instruction for an object to be grasped from a higher-level system. When a container containing an object is transported in accordance with the instruction, sensor 30 acquires an image of the object to be grasped (step S1). Determination device 10 inputs the acquired image to determiner 11 (step S2). Determinator 11 outputs a method for acquiring a gripping point in response to the input image. Determination device 10 obtains the acquisition method output from the determiner (step S3). Arithmetic device 20 determines whether the acquisition method output from determination device 10 is one of the first to third methods (step S4).

[0093] When the acquisition method is the first method, the arithmetic device 20 calculates, from the image, contour data indicating the contour of the object (step S11). The arithmetic device 20 generates a plurality of combinations (patterns) of the parts to be grasped and the grasping points of the grasping unit 55 (step S12). The arithmetic device 20 calculates a score for each pattern (step S13). The arithmetic device 20 adopts the grasping points of the pattern with the best score as the grasping points to be actually used (step S14).

[0094] When the acquisition method is the second method, the arithmetic device 20 inputs the image into the second model (step S21). The arithmetic device 20 acquires the gripping points output from the second model (step S22).

[0095] If the acquisition method is the third method, the arithmetic device 20 requests the terminal device 60 to designate a gripping point (step S31). The terminal device 60 accepts the designation of the gripping point by the user (step S32). The arithmetic device 20 receives the gripping point from the terminal device 60 (step S33).

[0096] The control device 40 moves the gripping unit 55 to the gripping point acquired by any method (step S5). The control device 40 causes the gripping unit 55 to grip the object at the gripping point (step S6). The control device 40 causes the gripping unit 55 to transport the object to another container and place it there (step S7).

[0097] The advantages of the embodiment will be described. When a conveying device grasps an object, a grasp point is determined. The grasp point is represented by a three-dimensional position and a three-dimensional angle. The above-mentioned first and second methods can be considered as methods for obtaining the grasp point. In the first method, the object's outline data is calculated, and various parameters such as the center of gravity are calculated using the outline data. The grasp point is calculated using these parameters. In the second method, a second model outputs the grasp point. Therefore, there is no need to calculate the outline data or various parameters based on the outline data. According to the first method, a grasp point that can grasp an object more stably can be calculated compared to the second method. According to the second method, the grasp point can be obtained more quickly compared to the first method.

[0098] When only the first method is used to obtain the gripping point, more objects can be gripped more stably, which reduces the occurrence of errors such as gripping failure and dropping of the object during transportation.

[0099] When only the second method is used to obtain the gripping point, the gripping point can be obtained more quickly. This reduces the waiting time of the conveying device 50 before the gripping operation. For example, it is possible to avoid the conveying device 50 having to wait due to obtaining the gripping point. This increases the number of objects that can be gripped and conveyed per unit time, thereby further improving the processing efficiency of the conveyance.

[0100] Combining the first and second methods makes it possible to utilize the advantages of each method. According to the first method, an appropriate grasping point can be easily obtained even when an object in an image has not yet been learned or when the object is difficult to recognize in the image. As a result, occurrences of grasping failures and dropping of the object due to unstable grasping can be suppressed. According to the second method, calculation of contour data or parameters based on the contour data is unnecessary, and the time required for calculation can be reduced. In other words, when an appropriate grasping point can be obtained using the second method, the grasping point can be obtained more quickly by executing the second method. For example, when it is difficult to obtain a grasping point using the second method, the occurrence of errors such as grasping failures can be suppressed by executing the first method. As a result, an object can be grasped more stably while improving processing efficiency.

[0101] The efficiency of transportation depends on the time required to transport an object. The time required to transport an object includes the time it takes for the control device 40 to calculate the operation of the transport device 50, the operation time of the transport device 50, and the retry time if gripping fails. If the calculation time by the control device 40 is long, the transport device 50 will need to wait until the calculation is complete. In this regard, the calculation time can be shortened by obtaining a gripping point using the second method. Furthermore, by appropriately obtaining a gripping point using the first method, gripping failures can be reduced.

[0102] The inventor further verified the combination of the first and second methods and obtained the following findings. First, the combination of the first and second methods effectively improved transport efficiency by shortening the average calculation time and reducing the probability of gripping failure. Furthermore, it was found that the combination of the first and second methods required long retry times for the transport of some objects, which reduced transport efficiency. For example, gripping is likely to fail for objects that are easily deformed when gripped, transparent objects, or objects that easily reflect light. For gripping points for such objects, even when the first method is used, the gripping success rate is not sufficiently high. It was found that when gripping of such objects fails, retry operations result in another gripping failure, significantly increasing the retry time. Furthermore, it was found that when unexpected problems occur during the transport process, long retry times also occur, which also reduces transport efficiency.

[0103] In view of the above-mentioned problems, the inventors have devised a third method in addition to the first and second methods. In the third method, the gripping point is designated by a person. When the gripping point is designated by a person, an appropriate gripping point can be designated even for objects that are easily deformed, transparent, or reflective. Furthermore, a person can take more appropriate measures for unexpected problems. This reduces the probability of gripping failure. When the gripping point is designated by a person, in addition to the calculation time by the control device 40, human operation time is required. However, even when human operation time is added, it has been found that the occurrence of retry operations and repeated retry operations can be suppressed, thereby ultimately improving conveyance efficiency.

[0104] In the embodiment, which of the first, second, and third methods is to be used is output by the determiner 11. Therefore, the time required to determine the method for obtaining the gripping point can be shortened, and the transport efficiency can be further improved.

[0105] (First Modification) The determination device 10 may determine the method of obtaining the gripping point and may also determine the state of the object from the image. The state of the object is classified as either normal (no damage) or abnormal (damaged). "Damage" includes scratches, chips, cracks, folds, tears, etc. The determination device 10 inputs the image into a determiner for determining the state of the object. The determination device 10 acquires the state output from the determiner.

[0106] For example, a determiner for determining the state of an object may be provided separately from determiner 11. Preferably, determiner 11 has a function as a determiner for determining the state of an object. By having determiner 11 determine both the method for acquiring the grip point and the state of the object, the time required for determination can be shortened compared to when the method for acquiring the grip point and the state of the object are determined by separate determiners.

[0107] FIG. 17 is a flowchart showing a processing method according to a first modified example of the embodiment. Compared to the processing method shown in FIG. 16, processing method M2 shown in FIG. 17 includes steps S3a and S3b instead of step S3. Here, an example will be described in which determiner 11 outputs both the acquisition method and the state of the object. In step S3a, determination device 10 acquires the acquisition method and state of the object output from determiner 11. In step S3b, determination device 10 determines whether the state of the object is normal. If the state of the object is abnormal, the object is not grasped and the processing ends. If the state of the object is normal, step S4 is executed, as in processing method M1. If the object is determined to be abnormal due to damage, another object of the same type is prepared. Processing method M2 is executed again for the other object.

[0108] For example, an object transported by the transport device 50 is then shipped. The shipped object is required to be undamaged and in good condition. In order to ship a normal object, one method is to inspect whether the transported object is normal after the object is grasped and transported by the transport device 50. However, with this method, the inspection of the object is performed after transport, and if the object is found to be abnormal, a normal object must be transported. In other words, two or more transport operations are required for one object.

[0109] According to processing method M2, the state of the object is determined before gripping. If the object is determined to be abnormal, gripping is not performed. This reduces the possibility that an abnormal object will be gripped and transported. This reduces the number of times the transport operation is repeated due to an abnormality in the object, thereby improving the processing efficiency of the transport device 50.

[0110] The image used to acquire the gripping point is preferably used to determine the object's condition. The image acquired by the sensor 30 may contain noise. For example, inappropriate lighting conditions on the object, reflection of light from the object, or the texture of the object's surface can cause image distortion, blurring, whiteout, blackout, and other noise. When a noisy image is used to acquire the gripping point, the success rate of the gripping may decrease. To address this issue, it is effective to use the image used to acquire the gripping point to determine the object's condition. This method can determine whether an object depicted in the image is abnormal if the image contains noise that could affect the success rate of the gripping. In other words, it can determine whether the object depicted in the image is abnormal in addition to whether the object itself is actually abnormal. By determining whether an object depicted in the image is abnormal if the image contains noise that could affect the success rate of the gripping, the occurrence of gripping failures can be reduced. As a result, the processing efficiency of the conveyance device 50 can be further improved. The image used to acquire the gripping point and the image used to determine the state of the object do not have to be completely identical. For example, if the conditions for capturing the image used to acquire the gripping point and the conditions for capturing the image used to determine the state of the object are substantially the same and the time difference between the images is sufficiently short, the images can be considered to be substantially the same. The imaging conditions include the way light hits the object, the settings of the sensor 30, etc.

[0111] (study) FIG. 18 is a schematic diagram showing the configuration of a processing system according to a modified example of the embodiment. 18 further includes a learning device 70 compared to the processing system 1. The learning device 70 learns a first model and a second model in advance. The learning device 70 stores the learned first model and second model in the storage device 15.

[0112] Furthermore, the learning device 70 retrains the second model using the results obtained by the processing of the processing system 1a. When the third method is executed, a person specifies the object position, grip position, and grip angle for the image. The used image, the specified object position, the specified grip position, and the specified grip angle are stored in the storage device 15. The learning device 70 uses the image used in the third method as input data and the specified object position, the specified grip position, and the specified grip angle as training data to retrain the second model. This improves the success rate of grasping by the second model. Furthermore, for objects retrained by the second model, the classifier 11 may be retrained so as to increase the proportion of judgments made by the second method. This reduces the frequency with which the third method is used for judgments made on objects retrained by the second model, further improving transport efficiency.

[0113] The learning device 70 may be provided separately from the determination device 10 and the calculation device 20, or the determination device 10 or the calculation device 20 may have the function of the learning device 70.

[0114] FIG. 19 is a schematic diagram showing the hardware configuration. 19 is used as the determination device 10, the arithmetic device 20, or the control device 40. The computer 90 includes a CPU 91, a ROM 92, a RAM 93, a storage device 94, an input interface 95, an output interface 96, and a communication interface 97.

[0115] The ROM 92 stores a program that controls the operation of the computer 90. The ROM 92 stores a program necessary for causing the computer 90 to perform each of the above-described processes. The RAM 93 functions as a storage area in which the programs stored in the ROM 92 are expanded.

[0116] The CPU 91 includes a processing circuit. The CPU 91 uses a RAM 93 as a work memory and executes a program stored in at least one of a ROM 92 and a storage device 94. During program execution, the CPU 91 controls each component via a system bus 98 and executes various processes.

[0117] The storage device 94 stores data necessary for executing the program and data obtained by executing the program.

[0118] The input interface (I / F) 95 can connect the computer 90 to an input device 95a. The input I / F 95 is, for example, a serial bus interface such as USB. The CPU 91 can read various data from the input device 95a via the input I / F 95.

[0119] The output interface (I / F) 96 can connect the computer 90 and an output device 96a. The output I / F 96 is, for example, a video output interface such as a Digital Visual Interface (DVI) or a High-Definition Multimedia Interface (HPMI (registered trademark)). The CPU 91 can transmit data to the output device 96a via the output I / F 96 and cause the output device 96a to display an image.

[0120] The communication interface (I / F) 97 can connect the computer 90 to a server 97a external to the computer 90. The communication I / F 97 is, for example, a network card such as a LAN card. The CPU 91 can read various data from the server 97a via the communication I / F 97.

[0121] The storage device 94 includes one or more selected from a hard disk drive (HDD) and a solid state drive (SSD). The input device 95a includes one or more selected from a mouse, a keyboard, a microphone (voice input), and a touchpad. The output device 96a includes one or more selected from a monitor, a projector, a printer, and a speaker. A device having the functions of both the input device 95a and the output device 96a, such as a touch panel, may also be used.

[0122] The processing executed by any one of the determination device 10, the calculation device 20, or the control device 40 may be realized by one computer 90 or by cooperation of multiple computers 90. One computer 90 may function as two or more selected from the determination device 10, the calculation device 20, and the control device 40.

[0123] The various data processing operations described above may be recorded as a computer-executable program on a magnetic disk (such as a flexible disk or hard disk), an optical disk (such as a CD-ROM, CD-R, CD-RW, DVD-ROM, DVD±R, or DVD±RW), a semiconductor memory, or other non-transitory computer-readable storage medium.

[0124] For example, information recorded on a recording medium can be read by a computer (or an embedded system). The recording medium may have any recording format (storage format). For example, a computer reads a program from the recording medium and causes a CPU to execute instructions written in the program based on the program. The computer may acquire (or read) the program via a network.

[0125] Embodiments of the invention include the following features. (Feature 1) Acquire an image of the object to be grasped; inputting the image into a classifier including a trained first model; A determination device that obtains from the determiner a method for acquiring a gripping point at which a gripping unit grips the object, The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A determination device that outputs any one of the acquisition methods. (Feature 2) 2. The determination device according to Feature 1, wherein the determiner outputs the acquisition method and classifies the state of the object into either normal (no damage) or abnormal (damage). (Feature 3) the first model includes a neural network; 3. The determination device according to feature 2, wherein the first model outputs the acquisition method and a classification result of the state of the object. (Feature 4) A determination device according to any one of features 1 to 3; a calculation device that acquires the grip point by the acquisition method determined by the determination device; A processing system comprising: (Feature 5) a conveying device including the gripping unit; a control device that moves the gripping unit to the gripping point obtained by the calculation device; 5. The processing system according to claim 4, further comprising: (Feature 6) Further, a terminal device connected to the arithmetic device via a network is provided, The processing system described in feature 5, wherein, when the determination device determines that the third method is the acquisition method, the calculation device requests the terminal device to specify the gripping point and receives the gripping point specified by the terminal device. (Feature 7) The processing system according to feature 6, wherein the terminal device displays the image and accepts designation of the gripping point on the image. (Feature 8) further comprising a learning device that learns the second model; The processing system according to any one of Features 4 to 7, wherein the learning device trains the second model using the image and the gripping point specified by the third method. (Feature 9) a conveying device including a gripping unit that conveys an object by the gripping unit; a determination device that obtains an image of an object, inputs the image to a determiner including a trained first model, and obtains from the determiner a method for obtaining a gripping point at which a gripping unit grips the object; Equipped with The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A transport system that outputs any one of the acquisition methods. (Feature 10) On the computer, Acquire an image of the object to be grasped; inputting the image into a classifier including a trained first model; acquiring, from the determiner, a method for acquiring a gripping point for the gripping unit to grip the object; A determination method comprising: The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A determination method that outputs any one of the acquisition methods. (Feature 11) A program that causes the computer to execute the determination method according to feature 10. (Feature 12) A storage medium storing the program according to feature 11.

[0126] According to the embodiments described above, a determination device, a processing system, a transport system, a determination method, a program, and a storage medium are provided that can improve the efficiency of transport processing.

[0127] Although several embodiments of the present invention have been described above, these embodiments are presented by way of example only and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, substitutions, modifications, etc. can be made without departing from the spirit of the invention. These embodiments and their modifications are included within the scope and spirit of the invention, as well as within the scope of the invention and its equivalents as set forth in the claims. Furthermore, the above-described embodiments can be implemented in combination with each other. [Explanation of symbols]

[0128] 1, 1a: processing system, 10: determination device, 11: determiner, 15: storage device, 20: calculation device, 30: sensor, 40: control device, 50: conveyance device, 51: manipulator, 51a: link, 51b: rotation axis, 55: gripping unit, 56: suction mechanism, 56a: suction pad, 56a1: effective suction pad, 57: clamping mechanism, 57a: support unit, 58: switching mechanism, 60: terminal device, 70: learning device, 100: first model, 120: variable autoencoder, 121: encoder, 122: decoder, 125: latent variable, 130: clustering model, 200: second model, 210: input layer, 220: intermediate layer, 230: output layer, 301: image, 302 to 305: Icon, 306: Message window, 307: Pointer, 311: Image, 311a to 311d: Suction pad, 311e: Center point, 311f, 311g: Support part, 311h: Midpoint, 312 to 316: Icon, 317: Message window, 318: Pointer, 320: Window, 321 to 324: Check box, C1, C2: Container, CL1 to CL3: Cluster, M1, M2: Processing method

Claims

1. Acquire an image of the object to be grasped; inputting the image into a classifier including a trained first model; A determination device that obtains from the determiner a method for acquiring a gripping point at which a gripping unit grips the object, The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A determination device that outputs any one of the acquisition methods.

2. The determination device according to claim 1 , wherein the determiner outputs the acquisition method and classifies the state of the object into either a normal state with no damage or an abnormal state with damage.

3. the first model includes a neural network; The determination device according to claim 2 , wherein the first model outputs the acquisition method and a classification result of the state of the object.

4. The determination device according to any one of claims 1 to 3, a calculation device that acquires the grip point by the acquisition method determined by the determination device; A processing system comprising:

5. a conveying device including the gripping unit; a control device that moves the gripping unit to the gripping point obtained by the calculation device; The processing system of claim 4 further comprising:

6. Further, a terminal device connected to the arithmetic device via a network is provided, The processing system described in claim 5, wherein when the determination device determines that the third method is the acquisition method, the calculation device requests the terminal device to specify the gripping point and receives the gripping point specified by the terminal device.

7. The processing system according to claim 6 , wherein the terminal device displays the image and accepts designation of the gripping point on the image.

8. further comprising a learning device that learns the second model; The processing system according to claim 4 , wherein the learning device trains the second model using the image and the gripping point specified by the third method.

9. a conveying device including a gripping unit that conveys an object by the gripping unit; a determination device that obtains an image of an object, inputs the image to a determiner including a trained first model, and obtains from the determiner a method for obtaining a gripping point at which the gripping unit grips the object; Equipped with The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A transport system that outputs any one of the acquisition methods.

10. On the computer, Acquire an image of the object to be grasped; inputting the image into a classifier including a trained first model; acquiring, from the determiner, a method for acquiring a gripping point for the gripping unit to grip the object; A determination method comprising: The determiner, in response to the input of the image, a first method of calculating contour data of the object from the image and calculating the gripping point using the contour data; A second method of inputting the image into a trained second model and obtaining the gripping point output from the second model; or a third method for obtaining the designated gripping point; A determination method that outputs any one of the acquisition methods.

11. A program that causes the computer to execute the determination method according to claim 10.

12. A storage medium storing the program according to claim 11.

Images