System, device, program, and method

Abstract

To improve the accuracy in grasping an object by a predetermined quantity.SOLUTION: A grasp estimation apparatus according to an embodiment comprises a first mechanism and a controller. The first mechanism sends an object being grasped with a gripper toward the outside of the gripper. The controller controls the first mechanism so that the object grasped with the gripper is transferred to a destination by a target amount.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] Embodiments of the present invention relate to a system.

Background Art

[0002] Conventionally, in food factories such as bento factories, food is gripped from a food tray in a specified target amount and packed into a bento box. At this time, there are many types of food to be packed, and the types of food to be handled frequently change. Against this background, it is difficult to create dedicated machines for most foods, and the packing work is carried out manually.

[0003] The food to be packed is classified into foods managed by the number, such as 3 or 10, from the viewpoints of gripping and the size of the food, and flaky and granular foods managed by the weight, such as 60 g or 100 g. Granular foods are difficult to handle because they are irregular in shape due to being granular. When flaky foods are likely to become entangled with each other, they are difficult to handle. A gripping device for gripping flaky or granular foods may quantitatively grip them using a dedicated tool according to the type of food. At this time, in order to handle various foods and different gripping amounts, a great deal of labor such as design changes is required.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The problem to be solved by the invention is to improve the accuracy of quantitative gripping.

Means for Solving the Problems

[0006] The system according to this embodiment includes a first mechanism and a control unit. The first mechanism releases an object held by the gripper from the gripper. The control unit controls the first mechanism so that the object held by the gripper is transferred to the transfer destination by a target amount. [Brief explanation of the drawing]

[0007] [Figure 1] Figure 1 is a block diagram showing an example of the configuration of a system having a gripping estimation device according to an embodiment. [Figure 2] Figure 2 shows an example of the appearance of a gripping device according to an embodiment. [Figure 3] Figure 3 shows an example of a parallel gripper according to the embodiment. [Figure 4] Figure 4 shows an example of a parallel gripper according to the embodiment. [Figure 5] Figure 5 shows an example of a parallel gripper according to the embodiment. [Figure 6] Figure 6 shows an example of a parallel gripper according to the embodiment. [Figure 7] Figure 7 shows an example of a parallel gripper according to the embodiment. [Figure 8] Figure 8 is a flowchart illustrating an example of the procedure for gripping and releasing according to an embodiment. [Figure 9] Figure 9 shows an example of observational data according to this embodiment. [Figure 10] Figure 10 shows an example of an overview of the input and output for the gripping amount estimation model according to the embodiment. [Figure 11] Figure 11 is a schematic diagram illustrating an example of the procedure for an embodiment, including pre-gripping and gripping the object to be gripped after pre-gripping. [Figure 12] Figure 12 is a schematic diagram illustrating an example of an embodiment, showing an overview of how to adjust the gripping amount during post-grasping. [Figure 13] Figure 13 is a flowchart illustrating an example of the procedure for generating a gripping amount estimation model according to an embodiment. [Figure 14] Figure 14 shows an example of a cross-section of an end effector according to an application example of the embodiment. [Figure 15] Figure 15 is a cross-sectional view illustrating an example of the flow of operations related to post-grasping, relating to an application example of the embodiment. [Figure 16] Figure 16 is a block diagram showing an example of the hardware configuration of each device in the embodiment. [Modes for carrying out the invention]

[0008] The system and gripper according to the embodiment will be described below with reference to the drawings. Note that the configuration of the embodiment described below is merely an example and is not limited to the following description. Furthermore, the functions and effects brought about by this configuration are also merely an example and are not limited to the following description.

[0009] Figure 1 is a block diagram showing an example of the configuration of System 1 having a gripping estimation device 17. As shown in Figure 1, System 1 includes a gripping device 11, an observation device 13, a weighing device 15, and a gripping estimation device 17. Figure 2 is a diagram showing an example of the appearance of the gripping device 11.

[0010] The gripping device 11 includes an end - effector 111 that grips a gripping target 133, a plurality of links 113, joints 115 provided between two of the plurality of links, and a plurality of driving devices (such as motors) that drive the joints 115 and the end - effector 111, etc. The end - effector 111 is realized, for example, by a parallel gripper with a feeding mechanism. As shown in FIG. 2, the end - effector 111 has a gripper (gripping part) 123 and a first mechanism (hereinafter referred to as a feeding mechanism) 124. In this embodiment, as an example of the gripper 123, a parallel gripper that can be opened and closed in parallel along the base BR is used. The parallel gripper 123 has an opening 125. The parallel gripper 123 is opened and closed by a driving mechanism not shown. The plurality of links 113, the plurality of joints 115, and the plurality of driving devices (such as motors) that drive the plurality of joints 115 may constitute a second mechanism (vibration mechanism) that swings (vibrates) the parallel gripper 123 in a state of gripping the gripping target 133 once or multiple times.

[0011] Figures 3 to 7 show an example of a parallel gripper 123. Figure 3 shows two perspective views of the open parallel gripper 123 from two different directions. Figure 4 shows a front view of the open parallel gripper 123. Figure 5 shows a perspective view of the closed parallel gripper 123. The open state refers to the state in which the gripper is open to grip an object, and the closed state refers to the state in which the gripper is closed to grip an object. As shown in Figures 3 to 5, the parallel gripper 123 has, for example, a pair of tip portions 127, each having a comb-like set of branches 126. The opening 125 is formed by the ends of the branches 126 on the pair of tip portions 127. The branches 126 have a fork shape, as shown in Figure 3. The gap between the branches 126 on one tip portion 128 and the branches 126 on the other tip portion 129 of the pair of tip portions 127 is fitted together. In other words, at the pair of tip portions 127, the comb-shaped opposing branches 126 are fitted together without contacting each other when the parallel gripper 123 is closed. Another example of the parallel gripper 123 has a pair of opposing plate members instead of a pair of comb-shaped branches 126. This pair of plate members may have multiple grooves (recesses) carved into each plate member, and the multiple grooves carved into one plate member and the multiple grooves carved into the other plate member may be arranged alternately, or one or both plate members may have no grooves.

[0012] As shown in FIGS. 3 to 7, for example, the plurality of branches 126 may have protrusions 130 directed toward the gaps facing the plurality of branches 126. The protrusion 130 may be provided only on one of the pair of tip portions 127, and may be provided on at least a part of the surface (the surface contacting the object to be gripped) that grips the object to be gripped by the parallel gripper 123. The protrusion of the protrusion 130 is arranged toward the tip of the plurality of branches 126 in the direction in which the feeding mechanism 124 described later feeds out the object gripped by the parallel gripper 123. However, the direction in which the protrusion 130 protrudes is not limited to these. That is, the protrusion 130 may protrude perpendicularly to the gripping surface composed of the plurality of branches 126 toward the opposing gap (or groove), or may protrude in the direction toward the base portion 121 of the plurality of branches 126. Further, the protrusion 130 does not have to face the opposing gap (or groove) and may face in the lateral direction. Instead of the protrusion 130, a recessed portion in which individual protrusions are replaced with depressions may be employed. It is expected that the object to be gripped by the gripper is gradually loosened and sent out (dropped) little by little from the inside of the gripper to the outside in the gripping amount adjustment described later by the protrusion 130 or the recessed portion. Therefore, even when the gripper has a pair of plate members instead of the plurality of comb-shaped branches 126, the protrusion 130 or the recessed portion is provided on both or only one of the pair of tip portions 127, and it can be expected that the object to be gripped is gradually loosened and sent out (dropped) little by little from the inside of the gripper to the outside in the gripping amount adjustment described later. However, the gripping amount adjustment described later is also valid even in a configuration not provided with these protrusions 130 or recessed portions.

[0013] Figure 6 shows a front view of the open parallel gripper 123 when the pair of tip portions 127 are made of resin. As shown in Figure 6, the gap between the pair of tip portions 127 in the parallel gripper 123 may be tapered, for example, from the base of the multiple branches 126 to the tips of the multiple branches 126. That is, the multiple branches 126 extend at a predetermined angle inward relative to the opening 125 with respect to the direction perpendicular to the base BR of the multiple branches 126. The predetermined angle is, for example, +2°. Therefore, if the length of the projection 130 is constant, when viewing the parallel gripper 123 from the front, in the closed parallel gripper 123 shown in Figure 5, the gap between opposing projections 130 will widen towards the base BR. By tilting the fork shape, when the parallel gripper 123 is closed, the object to be gripped 133 is pulled in and can be securely grasped. Furthermore, if the pair of tip portions 127 are made of a material such as metal that does not easily bend when gripping an object, rather than a resin that easily bends when gripping the object, the pair of tip portions 127 may be parallel to each other, i.e., at a predetermined angle = 0°, or they may spread out towards the tip, i.e., at a predetermined angle = negative.

[0014] Furthermore, as shown in Figures 3 to 6, a force sensor may be installed, for example, at the base portion 121 of the parallel gripper 123. In this case, the force sensor detects the weight of the object to be gripped 133 held by the parallel gripper 123. The force sensor outputs the weight of the object to be gripped 133 to the gripping estimation device 17.

[0015] The delivery mechanism 124 includes a motor MT that rotates according to a control signal output from the controller 171, a pinion gear PG connected to the drive shaft of the motor MT and rotating in conjunction with the rotation of the drive shaft, and a rack gear RG that engages with the pinion gear PG and is provided on one of its tip ends 128. The delivery mechanism 124 delivers the object held by the parallel gripper 123 to the outside of the parallel gripper 123. The outside of the parallel gripper is, for example, the outside of the pair of tip ends 127, which corresponds to below the opening 125 shown in Figure 6.

[0016] Figure 7 shows an example of one end portion 128 being moved by the delivery mechanism 124. As shown in Figure 7, the delivery mechanism 124 moves one end portion 128 along the long axis of the multiple branches 126 relative to the other end portion 129. The control of the delivery mechanism 124 will be explained later.

[0017] The observation device 13 can be implemented using, for example, various cameras. For example, an RGB (Red, Green, Blue) camera can be implemented. The RGB camera is an optical camera that photographs multiple gripping objects 133 arranged on the tray 131. In this case, the observation data corresponds to an RGB image. Pixels in the RGB image have, for example, red, green, and blue color values ​​(0 to 255). A monochrome camera may be used instead of an RGB camera. Furthermore, the observation device 13 may acquire observation data of a different type than the above observation data. In this case, the observation device 13 can be implemented using a 3D measurement camera (hereinafter referred to as a D (Depth) camera). Alternatively, the observation device 13 may be implemented using an RGB camera and a D camera (hereinafter referred to as an RGB-D camera). Pixels in the image from the D camera (hereinafter referred to as a D image) have depth information. The depth information is, for example, the depth (cm) relative to the reference position. To make the explanation more specific, the observation device 13 will be assumed to be implemented using an RGB-D camera. The imaging range of the RGB-D camera includes the entire area of ​​the tray 131 on which the object to be grasped 133 is placed. The reference position is, for example, the position where the observation data is acquired, i.e., the installation position of the RGB-D camera.

[0018] The weighing device 15 is positioned, for example, on the underside of the tray 131. The weighing device 15 corresponds to a scale that measures the total weight of the tray 131 and the gripping object 133 placed on the tray 131. The weighing device 15 outputs weighing data associated with weighing the tray 131 to the gripping estimation device 17. The gripping estimation device 17 calculates the gripping amount from the difference in measurement data output from the weighing device 15 before and after gripping the gripping object 133, and uses this gripping amount, for example, in the generation (learning) of the gripping amount estimation model described later. Therefore, once the generation of the gripping amount estimation model is complete, the weighing device 15 does not need to be used in system 1. Also, if a force sensor is provided on the parallel gripper 123, the weighing device 15 does not need to be positioned on the underside of the tray 131. Also, if a force sensor is not provided on the parallel gripper 123, the weighing device 15 is positioned, for example, on the underside of the tray 131.

[0019] The grip estimation device 17 has a controller 171. As shown in Figure 1, the controller 171 has an acquisition unit 173, an estimation unit 175, a selection unit 179, and a control unit 181.

[0020] The acquisition unit 173 acquires observation data of the grasped object 133. As described above, the observation data includes image data of the grasped object 133. The observation data also includes distance data from the observation data acquisition position to the surface of the grasped object 133. That is, the distance data is data indicating the distance from the installation position of the D camera, which is the observation data acquisition position, to the surface of the grasped object 133. The grasped object 133 is, for example, a food item in the form of a small piece, such as shredded cabbage or julienned radish. The acquisition unit 173 may also acquire observation data of a different type than the observation data.

[0021] Specifically, the acquisition unit 173 acquires RGB images and D images. More specifically, the acquisition unit 173 acquires observation data in each of several sub-regions within the distribution area of ​​the gripping object 133. The distribution area of ​​the gripping object 133 corresponds, for example, to the area on the tray 131 where the gripping object can be placed. Each of the multiple sub-regions is a region of a predetermined size and corresponds to a region (cut-out region: patch) cropped from the RGB image and the D image. The center of the sub-region is sometimes called the gripping point. Hereinafter, a sub-region in the RGB image will be called an RGB patch image. Similarly, a sub-region in the D image will be called a D patch image. The acquisition unit 173 acquires weighing data from the weighing device 15, or acquires the weight of the gripping object 133 gripped by the end effector 111 from the force sensor. That is, the acquisition unit 173 acquires the quantity of the object gripped by the parallel gripper 123 from the weighing device 15 or the force sensor.

[0022] The estimation unit 175 outputs data related to the gripping amount of the gripping object 133 by the gripping device 11 (in one example, parameters of the probability distribution of the gripping amount) based on the depth to which the pair of tip portions 127 are provisionally inserted from the surface of the gripping object 133 (hereinafter referred to as the insertion depth) and the observed data. For example, if the output data is the parameters of a standard normal distribution, the output is the mean and standard deviation of the standard normal distribution. The mean corresponds to the average gripping amount (hereinafter referred to as the estimated gripping amount) at which the gripping object 133 is gripped by the gripping device 11. The estimated gripping amount is an estimated value of the gripping amount.

[0023] The standard deviation corresponds to an indicator of the uncertainty (hereinafter referred to as the uncertainty indicator) that only the estimated gripping amount is gripped by the gripping device 11. The uncertainty indicator may be, for example, an uncertainty degree that indicates the degree (or proportion) of uncertainty that the gripping device 11 will grip only the estimated gripping amount. In other words, the uncertainty indicator is an indicator for evaluating whether the actual gripped amount is close to the estimated gripping amount. Alternatively, instead of the uncertainty degree, the uncertainty indicator may be a degree (or proportion) that indicates the degree of certainty that the gripping device 11 will grip only the estimated gripping amount, i.e., a confidence level that the gripping device 11 will grip the estimated gripping amount.

[0024] The insertion depth is a preset length and is stored in the memory of the gripping estimation device 17. The insertion depth (cm) can be set to one of nine patterns, for example: +2.0, +2.25, +2.50, +2.75, +3.00, +3.25, +3.50, +3.75, or +4.0. Note that the above insertion depth patterns are just examples and can be set and adjusted arbitrarily.

[0025] The estimation unit 175 estimates the gripping amount of the object gripped by the parallel gripper 123 and an index of the uncertainty of whether the gripping amount is maintained for the area occupied by the object to be gripped. Specifically, the estimation unit 175 outputs an estimated gripping amount and an uncertainty index corresponding to the insertion depth pattern for each of the multiple sub-regions in the observed data. For example, the estimation unit 175 inputs the observed data and insertion depth information into a trained gripping amount estimation model that has been trained to output a probability distribution using the insertion depth information of the parallel gripper 123 (hereinafter referred to as insertion depth information) and observed data of the object to be gripped 133 as input, and estimates the probability distribution. The insertion depth information Ins[cm] corresponds to the value obtained by adding the variable val corresponding to the insertion depth to the median of the D-patch image (Ins=median+val).

[0026] More specifically, the estimation unit 175 inputs each of the multiple RGB patch images, each of the multiple D patch images, and each of the multiple insertion depth information corresponding to the insertion depth pattern into a trained gripping amount estimation model, and outputs the estimated gripping amount and standard deviation corresponding to each of the multiple sub-regions and each of the multiple insertion depths. In other words, the estimation unit 175 estimates a probability distribution based on the insertion depth and the observed data. The estimated probability distribution represents the probability distribution of the gripping amount of an object grasped by inserting the parallel gripper 123 to the insertion depth at the location of the RGB-D patch image (the horizontal axis corresponds to the gripping amount as a random variable, and the vertical axis corresponds to the probability density for the gripping amount).

[0027] For example, the probability of grasping a grasping amount X[g] is distributed by a standard normal distribution N(μ, σ). 2When modeled using a probability distribution represented by ), the parameters of the probability distribution output by the grasping amount estimation model are the mean μ and the standard deviation σ. That is, a standard normal distribution N(μ, σ) has the mean μ and standard deviation σ as parameters. 2 ) represents the probability distribution P of the gripping amount X [g] of the object gripped by the parallel gripper 123 when it is inserted according to the insertion depth information Ins [cm] at the location of the input RGB-D patch image. In other words, the probability P(X) that the gripping amount of the object gripped by the parallel gripper 123 at this location with this insertion depth information Ins [cm] is X [g] is expressed by the following formula.

number

[0028] In other words, the estimation unit 175 concatenates the RGB patch image and the D patch image for each of the multiple subregions and multiple insertion depths, and further concatenates the insertion depth information, and sequentially inputs these into a trained grasping amount estimation model to estimate multiple probability distributions corresponding to the multiple subregions and multiple insertion depths. Next, the estimation unit 175 outputs the parameters of each of the multiple probability distributions. Specifically, the estimation unit 175 outputs multiple estimated grasping amounts corresponding to the multiple subregions and multiple insertion depths, and multiple uncertainty indices (e.g., standard deviations) corresponding to the multiple estimated grasping amounts. In other words, the estimation unit 175 utilizes a grasping amount estimation model that, when the RGB-D patch image and insertion depth information are input to the grasping amount estimation model, outputs the estimated grasping amount at a location in the RGB-D patch image and the standard deviation corresponding to the estimated grasping amount.

[0029] In the explanations so far, the probability distribution for the gripping amount estimation model has been assumed to be a standard normal distribution, but the probability distribution is not limited to the standard normal distribution. For example, a mixture of normal distributions consisting of N standard normal distributions may be assumed as the probability distribution. When the probability of gripping a gripping amount X[g] is modeled by a probability distribution expressed by a mixture of normal distributions, the probability P(X) that the gripping amount of the object gripped by the parallel gripper 123 according to the insertion depth information Ins[cm] at the location of the RGB-D patch image input to the gripping amount estimation model is X[g] is expressed by the following formula.

number

[0030] Alternatively, the grasping amount estimation model may be a trained model that takes an RGB-D patch image and insertion depth information as input and outputs an estimated grasping amount μ. In this case, the grasping amount estimation model outputs an estimated grasping amount μ from the RGB-D patch image and insertion depth information. That is, the grasping amount estimation model does not output a standard deviation σ. In this case, the estimation unit 175 may input the RGB-D patch image and insertion depth information to multiple different grasping amount estimation models that do not output a standard deviation σ, and calculate the average and standard deviation of the multiple estimated grasping amounts output from these multiple models as the estimated grasping amount μ and standard deviation σ, respectively.

[0031] The gripping amount estimation model is obtained, for example, by inputting observation data and insertion depth information of the object to be gripped 133 by the gripping device 11 into the model, and updating the model's parameters based on the data output from the model and the measured gripping amount of the object to be gripped 133 by the gripping device 11. This procedure will be explained later.

[0032] The selection unit 179 determines whether the location satisfies predetermined conditions based on at least the estimated gripping amount and the estimated uncertainty index. The selection unit 179 identifies locations occupied by the object to be gripped that satisfy predetermined conditions allowing for gripping of more of the object than the target amount as locations where the object to be gripped will be gripped by the parallel gripper 123. For example, the selection unit 179 uses the estimated gripping amount μ and the standard deviation σ with respect to the estimated gripping amount μ in each of the multiple sub-regions to select a region from the multiple sub-regions in which the object to be gripped will be gripped by the gripping device 11. To make the explanation more concrete, let's assume that the target gripping amount set by the user (hereinafter referred to as the target gripping amount) is 52g. In this case, the selection unit 179 identifies regions that satisfy predetermined judgment conditions (hereinafter referred to as the condition-compliant regions) from the multiple sub-regions. The predetermined judgment condition is that the value obtained by the difference between the estimated gripping amount and n times the standard deviation (where n is a positive number) exceeds the target gripping amount, and is expressed as target gripping amount < μ - n * σ. The above n can be set arbitrarily. For example, when n=1 is set and the target gripping amount is 52g, the subregion PR shown in Figure 10 with mean μ=60g and standard deviation σ=7g satisfies the condition. The smaller n is, the higher the possibility that the gripping amount by the gripping device 11 will be less. In this case, the time required for post-grasping, described later, will be shortened according to the amount by which the gripping amount has been reduced.

[0033] On the other hand, the larger n is, the higher the likelihood that the gripping amount by the gripping device 11 will be. In this case, the time required for post-grasping increases in proportion to the increase in the gripping amount. Depending on the trade-off between the time required for post-grasping and the time required for retrying gripping when the gripped object does not reach the target gripping amount, the user can adjust and set n as appropriate. Next, the selection unit 179 calculates, for example, the absolute value of the difference between the average value μ for a plurality of specified sub-regions and the target gripping amount |target gripping amount - μ|.

[0034] The selection unit 179 selects the sub-region within the condition-compliant region where the judgment quantity is minimized as the region where the gripping device 11 will grip the object to be gripped (hereinafter referred to as the gripping region). Alternatively, the selection unit 179 may use the absolute value of the difference between the average value μ and the target gripping quantity |target gripping quantity-μ| for a plurality of identified sub-regions as the judgment quantity, and select the location where this judgment quantity is minimized as the gripping region. In this case, the selection unit 179 may calculate the judgment quantity as the value obtained by adding the standard deviation σ to the absolute value of the difference between the average value μ and the target gripping quantity |target gripping quantity-μ|. In other words, the smallest value obtained by adding the standard deviation σ to the absolute difference between the mean μ and the target grasp amount |target grasp amount - μ| means that the difference between the target grasp amount and the mean μ (|target grasp amount - μ|) is small, and the standard deviation σ, which represents the variability of the prediction, is also small.

[0035] Furthermore, when a mixture of normal distributions is used as the probability distribution, the selection unit 179 may specify the estimated gripping amount μ used to select the gripping region by, for example, one of the following three methods.

number

[0036] Furthermore, the selection unit 179 may select the gripping area without using the standard deviation σ. For example, the selection unit 179 identifies a location where the target gripping amount < μ - cons. (constant) is satisfied as a condition-compliant area. Then, from within the condition-compliant area, the selection unit 179 selects the location where |target gripping amount - μ| is minimized as the gripping area.

[0037] The control unit 181 controls the operation of the gripping device 11 (gripping, feeding, moving, and releasing operations of the object to be gripped 133). Specifically, the control unit 181 controls the gripping device 11 to grip the object to be gripped 133 at a selected gripping position and insertion depth as the gripping operation. As a result, the gripping device 11 grips the object to be gripped 133 at the gripping position and insertion depth. Specifically, the control unit 181 controls the parallel gripper 123 to grip the object to be gripped 133, lift it up, and then release it (hereinafter referred to as pre-grasping), and controls the parallel gripper 123 to grip the object to be gripped again after it has been released from the parallel gripper 123. As a result, the parallel gripper 123 grips the object to be gripped again, including the portion that is fed out by the feeding mechanism 124. Details of the above gripping operation will be explained later.

[0038] The control unit 181 controls the feeding mechanism 124 based on the amount of object gripped by the parallel gripper 123 and the target amount as part of the feeding operation. This causes the feeding mechanism 124 to feed the gripped object towards the opening 125. Specifically, the feeding mechanism 124 feeds the gripped object from the base of the multiple branches 126 towards the tips of the multiple branches 126 at one of the pair of tip ends 127, at the tip end 128. For example, the control unit 181 controls the speed at which the feeding mechanism 124 feeds the gripped object based on the amount of gripped object and the target gripping amount. The control unit 181 may also control the number of times the feeding mechanism 124 feeds the gripped object based on the amount of gripped object and the target gripping amount. Furthermore, the control unit 181 may perform both speed control and number of feeds.

[0039] In other words, the feeding operation corresponds to adjusting the gripping amount by the pair of tip ends 127 by moving one of the pair of tip ends 128 up and down using the rack and pinion mechanism (rack gear RG and pinion gear PG) in the feeding mechanism 124, thereby actively dropping (feeding out) the gripped object held by the pair of tip ends 127. Actively dropping does not mean widening the opening 125 of the parallel gripper 123 to allow the gripped object to fall naturally, but rather moving one of the tip ends 128 downwards so that the gripped object, which was gripped between the pair of tip ends 127 before the movement of the one tip end 128, is dropped without being gripped by the pair of tip ends 127, that is, feeding a part of the gripped object outside the pair of tip ends 127. Note that the rack and pinion mechanism that feeds out the gripped object by moving one of the pair of tip ends 128 up and down is just one example of a means for realizing the feeding operation, and the feeding operation may be realized by other known mechanisms. Based on these findings, the feeding operation is a post-grasping operation of the object to be gripped, and corresponds to adjusting the gripping amount of the object gripped by the pair of tip portions 127. Details of this feeding operation will be explained later.

[0040] The control unit 181 controls the gripping device 11 to move the gripped object 133 to a preset target position and release the gripped object 133 at this target position. Since known control methods are used for the movement and release operations, a detailed explanation is omitted.

[0041] The configuration of System 1 has been described above. Below, we will describe the series of processes from gripping to releasing by System 1 (hereinafter referred to as the gripping and releasing process). Figure 8 is a flowchart of an example of the procedure for the gripping and releasing process. To make the explanation more concrete below, the probability distribution for the gripping amount estimation model will be assumed to be a standard normal distribution.

[0042] (Grip release process) (Step S801) The selection unit 179 selects a sub-region to be pre-gripped. Specifically, the acquisition unit 173 acquires observation data from the observation device 13 for each of a plurality of sub-regions of a predetermined size relating to the gripping target 133. The predetermined size is pre-stored in the memory of the gripping estimation device 17 as the size that can be gripped by the parallel gripper 123. Figure 9 shows an example of observation data. As shown in Figure 9, the acquisition unit 173 acquires, for example, RGB images and D images of the food fragments, which are the gripping target 133, placed on the tray 131, for each of the plurality of sub-regions.

[0043] Next, the acquisition unit 173 acquires multiple RGB patch images and D patch images (hereinafter referred to as RGB-D patch images) of a predetermined size from the observation data at multiple locations. The multiple RGB-D patch images are sequentially input to the grasping amount estimation model. For example, multiple RGB-D patch images are set for the entire area of ​​the observation data, shifted by a predetermined width.

[0044] The estimation unit 175 sequentially inputs the RGB-D patch image and insertion depth information for each of the multiple subregions and multiple insertion depths into a trained grasping amount estimation model, thereby outputting multiple estimated grasping amounts and multiple standard deviations corresponding to the multiple subregions and multiple insertion depths.

[0045] Figure 10 shows an example of the input / output overview for the grasping amount estimation model. As shown in Figure 10, for each of the multiple subregions and multiple insertion depths, the RGB-D patch image and insertion depth information are input to the grasping amount estimation model. For a single RGB-D patch image, for example, the insertion depth information Ins[cm] obtained by applying each of the nine patterns of insertion depths—+2.0, +2.25, +2.50, +2.75, +3.00, +3.25, +3.50, +3.75, and +4.0—as the variable val is sequentially input to the grasping amount estimation model. The grasping amount estimation model outputs the parameters of the probability distribution P.

[0046] The parameters of the probability distribution P are, for example, the mean μ and standard deviation σ of a standard normal distribution, as shown in Figure 10. For example, the square frame PR in Figure 10 is a subregion corresponding to the RGB-D patch image in the observed data shown in Figure 10. As shown in the square frame PR in Figure 10, if the RGB-D patch image and the variable val as insertion depth are, for example, +2.0 cm, the grasping amount estimation model outputs 60 g as the mean (estimated grasping amount) and 7 as the standard deviation.

[0047] Next, the selection unit 179 identifies a condition-fitting region based on predetermined judgment conditions, using the parameters of the probability distribution output from the gripping amount estimation model and the target gripping amount. Then, the selection unit 179 calculates a judgment quantity using the target gripping amount and the estimated gripping amount (and possibly the standard deviation). Depending on the definition of the judgment quantity, the selection unit 179 selects the sub-region that minimizes the judgment quantity from the condition-fitting region as the gripping region. As a result, the sub-region to be pre-gripped is selected as the gripping region. At this time, insertion depth information related to the selected gripping region, i.e., insertion depth information output from the gripping amount estimation model with the parameters that minimize the judgment quantity, is transmitted from the selection unit 179 to the control unit 181.

[0048] (Step S802) The control unit 181 controls the gripping device 11 to perform pre-gripping in the selected gripping area. Specifically, the control unit 181 controls the drive devices for the operation of the multiple links 113 so that the end effector 111 is positioned directly above the gripping area, and more specifically, so that the center of the opening 125 is positioned at the center of the gripping area. At this time, if the opening width of the opening 125 is not open to a predetermined width, the control unit 181 controls the drive devices for opening and closing the pair of tip portions 127 (hereinafter referred to as opening / closing motors) so that the opening width of the opening 125 is widened to a predetermined width. As a result, the end effector 111 moves directly above the gripping area with the opening 125 open to a preset opening width.

[0049] Next, the control unit 181 controls the drive devices for the operation of the multiple links 113 according to the transmitted insertion depth information, so as to move the tips of the pair of tip portions 127 to the insertion depth (hereinafter referred to as the gripping depth) in the gripping area according to the transmitted insertion depth information. As a result, the tips of the pair of tip portions 127 are inserted to the gripping depth in the gripping area. Next, the control unit 181 controls the opening / closing motor to narrow the opening 125 to a preset closing width. For example, the control unit 181 monitors the load on the opening / closing motor and operates the opening / closing motor until the load exceeds a predetermined threshold. The predetermined threshold corresponds to a state in which a high load is generated on the opening / closing motor as the pair of tip portions 127 firmly grip the object to be gripped to a certain extent. As a result, the pair of tip portions 127 narrow the opening 125 to a preset closing width.

[0050] In the above explanation, the opening width and closing width were described as being pre-set, but are not limited to these. For example, the gripping amount estimation model may be further trained using the closing width. In this case, the closing width used will be the closing width input to the gripping amount estimation model for the selected gripping region. That is, the estimation unit 175 sequentially inputs the RGB-D patch image, insertion depth information, and closing width for each of the multiple sub-regions, each of the multiple insertion depths, and each of the multiple closing widths into the trained gripping amount estimation model, thereby outputting multiple estimated gripping amounts and multiple standard deviations corresponding to the multiple sub-regions and each of the multiple insertion depths. The same applies to the opening width.

[0051] The control unit 181 controls the drive device to move the end effector 111 towards directly above the gripping area while gripping the object to be gripped. The distance to move directly above the gripping area can be arbitrarily set, for example, to the gripping depth, a distance shorter than the gripping depth, or a distance longer than the gripping depth. As a result, the end effector 111 is moved upward while gripping the object to be gripped. Next, the control unit 181 controls the opening / closing motor to widen the opening 125 to a preset opening width. As a result, the pair of tip portions 127 widen the opening 125 to a preset opening width. With this, the pre-gripping is completed.

[0052] (Step S803) The control unit 181 controls the drive devices for the operation of the multiple links 113 according to the transmitted insertion depth information, so as to move the tips of the pair of tip portions 127 to the gripping depth in the gripping area. As a result, the tips of the pair of tip portions 127 are inserted to the gripping depth in the gripping area. Next, the control unit 181 controls the opening and closing motor to narrow the opening 125 to a preset closing width. As a result, the pair of tip portions 127 grip the object to be gripped. The insertion of the tips of the pair of tip portions 127 to the gripping depth and the gripping of the object by the pair of tip portions 127 are the same as the process in step S802, so a description is omitted.

[0053] (Step S804) The control unit 181 controls the second mechanism to swing the parallel gripper 123, which has re-gripped the object to be gripped, multiple times. For example, the control unit 181 controls the drive devices for the operation of multiple links 113 to shake off a portion of the object to be gripped by the end effector 111. As a result, the parallel gripper 123 swings under the control of the control unit 181. The second mechanism may also have a swinging mechanism that swings the parallel gripper 123, which has re-gripped the object to be gripped, and / or a swivel mechanism that rotates the parallel gripper 123, which has re-gripped the object to be gripped. Since known technologies such as multiple gears and rack and pinion can be applied to the swinging mechanism and swivel mechanism, a description is omitted. Note that the processing in this step is an optional process that can be added to the gripping release process, and can be omitted as appropriate. However, it can shake off gripped objects that have been unintentionally lifted, and improve the accuracy of adjusting the gripping amount in the subsequent gripping process.

[0054] Figure 11 is a schematic diagram showing an example of the procedure outline for steps S802 to S804, including pre-gripping and gripping of the object to be gripped after pre-gripping. The states in (1) to (4) in Figure 11 correspond to pre-gripping in step S802. As shown in (1) in Figure 11, the end effector 111 moves directly above the gripping area under the control of the control unit 181, and then inserts the pair of tip portions 127 to the insertion depth. Next, as shown in (2) in Figure 11, the end effector 111 narrows the opening 125 under the control of the control unit 181. As a result, the object to be gripped is grasped by the pair of tip portions 127.

[0055] Next, as shown in (3) in Figure 11, the end effector 111 moves (Unt) directly above the gripping area while gripping the object to be gripped, under the control of the control unit 181. This untangles the object gripped by the pair of tip ends 127 with the object gripped 133 on the tray 131, i.e., the entanglement of the food fragments. The area where the entanglement of the food fragments untangles is around the gripper, as shown by Ent in (3) in Figure 11. Also, GrO in (3) in Figure 11 shows the object gripped by the pair of tip ends 127, untangled from the object gripped on the tray. Furthermore, as shown in (4) in Figure 11, the end effector 111 widens the opening 125 under the control of the control unit 181. This releases the object gripped by the pair of tip ends 127.

[0056] Furthermore, the states in (5) and (6) in Figure 11 correspond to the processing in step S803. As shown in (5) in Figure 11, the end effector 111 inserts the pair of tip portions 127 to the insertion depth under the control of the control unit 181, narrowing the opening 125. As a result, the gripping target GrO is gripped again by the pair of tip portions 127. At this time, the end effector 111 may vibrate under the control of the control unit 181. As shown in (6) in Figure 11, the end effector 111 moves towards the area directly above the gripping area while gripping the gripping target under the control of the control unit 181. These actions reduce entanglement between the gripping target gripped by the pair of tip portions 127 and the gripping target 133 on the tray 131, that is, reduce the gripping of an unintended amount of food due to entanglement of food fragments, and enable gripping of an amount of gripping target close to the estimated gripping amount.

[0057] (Step S805) The acquisition unit 173 acquires the weight of the object gripped by the gripping device 11 (hereinafter referred to as the gripping weight) based on the output from the weighing device 15 before and after the processing in step S803. Alternatively, the acquisition unit 173 may acquire the gripping weight based on the output from the force sensor before and after the processing in step S803.

[0058] The control unit 181 controls the operation of the feed mechanism 124, i.e., the adjustment of the gripping amount, between the upper and lower limits of movement in the rack and pinion mechanism, based on the gripping weight and the target gripping amount, as a post-grasping function. For example, the control unit 181 determines a control amount related to the adjustment of the gripping amount based on the difference between the gripping weight and the target gripping amount. The control amount related to the adjustment of the gripping amount is related to the vertical movement of one of the tip portions 128, and is, for example, the width of the vertical movement, the execution time of the vertical movement, and the number of vertical movements. The method for determining the control amount by the control unit 181 can be implemented using either a deep learning-based method or a method using feedback control of sensor values.

[0059] For example, if the vertical movement of one end 128 is achieved by vertical movement in multiple steps, the amount of movement of one end 128 in each step (hereinafter referred to as the step movement amount) depends on the target gripping amount and the current gripping amount (gripping weight). Specifically, the control unit 181 increases the step movement amount as the difference between the target gripping amount (e.g., 38 [g]) and the current gripping amount (e.g., 60 [g]) increases. In this case, the movement speed of one end 128 becomes faster during the vertical movement of one end 128. Therefore, the larger the difference between the target gripping amount and the gripping weight, the greater the amount of gripped object (e.g., small pieces of food) that falls from the pair of end 127.

[0060] For example, if the difference is 30g or more, the control unit 181 controls the motor MT in the feeding mechanism 124 so that the step movement is 5cm. If the difference is less than 10g, the control unit 181 controls the motor MT in the feeding mechanism 124 so that the step movement is 1cm. If the difference is 10g or more but less than 30g, the control unit 181 controls the motor MT in the feeding mechanism 124 so that the step movement is a movement amount corresponding to the difference through linear interpolation from 1cm to 5cm. The control unit 181 repeatedly controls the operation of the feeding mechanism 124, i.e., the adjustment of the gripping amount, until the difference is less than or equal to a predetermined weight. The predetermined weight is, for example, 2g.

[0061] Furthermore, the control of the gripping amount adjustment is not limited to the control of the step movement amount, that is, the movement amount of one end portion 128 per step. For example, the control unit 181 may control the number of steps executed using a preset step movement amount (hereinafter referred to as the predetermined step movement amount) according to the difference between the target gripping amount and the gripping weight. For example, the control of the gripping amount adjustment will be described when the predetermined step movement amount is 1 cm. For example, if the difference is 30 g or more, the control unit 181 controls the motor MT with the number of steps set to 5. In this case, one end portion 128 moves 5 cm. If the difference is less than 10 g, the control unit 181 controls the motor MT with the number of steps set to 1. In this case, one end portion 128 moves 1 cm. If the difference is 10 g or more but less than 30 g, the control unit 181 controls the motor MT so that the movement amount corresponds to the difference by linear interpolation of the number of steps from 1 to 5.

[0062] Furthermore, in step 805, the control unit 181 may simultaneously control the adjustment of the gripping amount and the operation of shaking the parallel gripper 123 as described in step S804. Since the parallel gripper 123 is shaken along with the control of the gripping amount adjustment, the gripped object that is being held while protruding from the parallel gripper 123 can be efficiently shaken off. For this reason, when the parallel gripper 123 is shaken along with the control of the gripping amount adjustment, the linearity of the gripping amount adjustment can be improved.

[0063] Figure 12 is a schematic diagram showing an example of an overview of adjusting the gripping amount during post-gripping. As shown in (A) of Figure 12, the parallel gripper 123 narrows the opening 125 with the pair of tip portions 127 inserted to the insertion depth under the control of the control unit 181. As a result, the gripping target GrO is gripped again by the pair of tip portions 127. At this time, as shown in (A) of Figure 12, the gaps between the multiple branches of one tip portion 128 and the other tip portion 129 of the fork shape are in a fitted state. (A) of Figure 12 corresponds to step S803 and (5) of Figure 11.

[0064] As shown in Figure 12(B), the pinion gear PG in the rank and pinion mechanism R&P rotates along the arrow RA indicating rotation due to the rotation of the motor MT under the control of the control unit 181. In steps of rotation of the pinion gear PG, one end 128 moves downward DA. As a result, the object to be gripped GrO is fed out of the parallel gripper 123.

[0065] As shown in (C) of Figure 12, when the dispensing mechanism 124 is controlled by a control amount determined by the control unit 181, the gripping target DGO that has been sent out of the parallel gripper 123 falls into the tray 131. The gripping amount is adjusted by the repeated up-and-down movement of one of the tip portions 128 under the control of the control unit 181.

[0066] (Step S806) The control unit 181 controls the drive devices for the operation of the multiple links 113 so as to move the gripped object, after adjusting the gripping amount, to a transfer destination such as a lunch box. Specifically, the control unit 181 controls the drive devices so as to move a pair of tip portions 127 directly above the transfer destination. Next, the control unit 181 controls the opening and closing motor so as to widen the opening width of the opening 125 to a predetermined width. As a result, the gripped object held by the parallel gripper 123 is released to the transfer destination. With this, the gripping release process is completed. In this system 1, the gripping device 11 may further have a mechanism for scraping off objects attached to the surface of the gripper 123. This mechanism may be, for example, a mechanism that reciprocates a member up and down or left and right along the gripper surface, and this member may be shaped to comb the gaps between the branches 126 of the gripper 123. The control unit 181 may control the scraping mechanism at predetermined timings to scrape off any material adhering to the surface of the gripper 123. The predetermined timing may be, for example, each time the process in step S806 is completed, or each time the process in step S806 is completed a predetermined number of times.

[0067] The system 1 according to the embodiment described above acquires observation data of a gripping target 133, which is, for example, a piece of food, estimates the gripping amount of the gripping target 133 to be gripped by the gripping device 11 based on the acquired observation data, and identifies a gripping area based on the estimated gripping amount. Next, the system 1 includes a release mechanism 124 that sends the object being gripped by the parallel gripper 123 out of the parallel gripper 123, and controls the release mechanism 124 based on the amount of the object being gripped by the gripper (gripping weight) and the target amount (target gripping amount). For example, as a pre-gripping, the system 1 controls the parallel gripper 123 to grip the gripping target with the parallel gripper 123, lift it up, and then release it, and controls the parallel gripper 123 to grip the gripping target again after it has been released from the parallel gripper 123. Through the above pre-gripping, the system 1 can reduce the amount of gripping target (piece of food) that is lifted up in an entangled state, and can improve the accuracy of adjusting the gripping amount in post-gripping.

[0068] Based on these considerations, according to the system 1 of this embodiment, the user can estimate the desired gripping amount from the observation data, and select the region with the lowest degree of uncertainty among the estimated gripping amounts as the gripping area. In addition, the end effector 111 of this system 1 can appropriately grip small pieces of food that tend to get tangled. For this reason, with this parallel gripper 123, dedicated equipment and end effectors for gripping a fixed amount are unnecessary, and a great deal of effort such as design changes can be reduced.

[0069] Based on the above, System 1 can improve the accuracy of quantitatively grasping small food pieces that tend to tangle or stick together, such as shredded cabbage and finely chopped green onions.

[0070] Furthermore, in the system 1 according to the embodiment described above, the control unit 181 controls the dispensing mechanism 124 based on the amount of object being gripped by the gripper 123 and the target amount, thereby gradually releasing the object from the gripper 123 while loosening it. Focusing on this effect of "gradually releasing the object while loosening it," the system 1 can also be used in the following way. That is, in system 1, the control unit 181 can control the gripping device 11 to cause the gripper 123 to grip the object, move the gripper 123 that is gripping the object directly above the transfer destination of the object, and control the dispensing mechanism 124 to release only the target amount of the object from the gripper 123 to the transfer destination. In this case, the control unit 181 may acquire the amount of the object that actually falls from the gripper 123 (fall amount) and the target amount of the object to be dropped, and stop the delivery of the object by the delivery mechanism 124 based on the fall amount and the target amount. Alternatively, a weighing device may be installed at the transfer destination, and the control unit 181 may detect when the fall amount measured by the weighing device has reached the target amount, and stop the delivery of the object by the delivery mechanism 124 based on the detection signal. The fall amount may be calculated from the difference between the gripping amount of the gripper 123 at the start of delivery by the delivery mechanism 124 and the current gripping amount, or it may be measured by a weighing device installed at the transfer destination.

[0071] The following describes the process of generating the gripping amount estimation model. The gripping amount estimation model generation process corresponds to the learning process. At this time, System 1 or gripping estimation device 17 shown in Figure 1 functions as a model learning device. Figure 13 is a flowchart showing an example of the procedure for generating the gripping amount estimation model (hereinafter referred to as the estimation model generation process). Note that the estimation model generation process is executed according to the type of end effector 111.

[0072] (Estimation model generation process) (Step S131) The acquisition unit 173 acquires observation data from the observation device 13 for a predetermined-sized sub-region of the gripping object 133. The position of the sub-region on the tray 131 is set in advance. The acquisition unit 173 outputs the position of the sub-region to the control unit 181. The acquisition unit 173 also stores the position of the sub-region in memory. The estimation unit 175 inputs the RGB-D patch image and insertion depth information related to the set sub-region into the model to be trained.

[0073] (Step S132) The estimation unit 175 estimates the parameters of the probability distribution of the grasping amount corresponding to the input RGB-D patch image and insertion depth information using the model under training. If the probability distribution is a standard normal distribution, the parameters of the probability distribution are the mean μ and the standard deviation σ. If the probability distribution is a mixture of normal distributions, the parameters of the probability distribution are the mixing ratio α of the k-th standard normal distribution for N standard normal distributions. k , the mean μ of the k-th standard normal distribution k , the standard deviation σ of the k-th standard normal distribution k There are 3N values ​​(k=1~N). If a probability distribution is not used, the estimation unit 175 outputs the estimated grasping amount μ. The estimated grasping amount μ corresponds to the mean value among the parameters of the probability distribution.

[0074] (Step S133) The control unit 181 controls the gripping device 11 to perform gripping of the gripping object 133 in a set partial area. The gripping device 11 grips the gripping object 133 in the partial area under the control of the control unit 181. The acquisition unit 173 acquires the measured value of the gripping amount by the gripping device 11, i.e., the actual gripping amount, based on the output from the weighing device 15 or the force sensor. When calculating the measured value based on weighing data, which is the output from the weighing device 15, the acquisition unit 173 acquires the measured value by, for example, subtracting the weighing data before and after the gripping of the gripping object 133 by the gripping device 11. If the force sensor is mounted on the end effector 111, the acquisition unit 173 acquires the measured value by converting the output from the force sensor into weight.

[0075] (Step S134) The estimation unit 175 calculates the probability that the grasped quantity is grasped in the estimated probability distribution. Since known methods can be applied to the calculation of the probability, the explanation is omitted. Note that this step is unnecessary if a probability distribution is not used.

[0076] (Step S135) The estimation unit 175 calculates the loss based on the calculated probabilities. If the probability distribution is a standard normal distribution and the actual gripping amount obtained in step S133 is X[g], the estimation unit 175 calculates the model loss (-logP(X)) using the following formula.

number

[0077] Furthermore, if the probability distribution is a mixture of standard normal distributions and the actual amount grasped in step S133 is X[g], the estimation unit 175 calculates the model loss (-logP(X)) using the following formula as P(X).

number

[0078] Furthermore, if a probability distribution is not used, the estimation unit 175 calculates the loss based on the difference between the estimated gripping amount μ and the measured value. For example, the estimation unit 175 calculates the loss (squared error) by applying this difference to the loss function.

[0079] (Step S136) The estimation unit 175 updates the parameters of the model to be trained to reduce the loss calculated in step S135. For example, the estimation unit 175 updates the parameters of the model to be trained to reduce the loss using a known method such as backpropagation or stochastic gradient descent.

[0080] (Step S137) If the number of parameter updates is a predetermined number (Yes in step S137), the process in step S139 is executed. If the number of parameter updates is not a predetermined number (No in step S137), the process in step S138 is executed. The predetermined number is set in advance by the user or the like. Note that the process in this step is not limited to comparing the number of parameter updates with the predetermined number. For example, the processes in steps S131 to S138 may be repeated until the learning of the model to be learned converges. Convergence of learning of the model to be learned means, for example, when the difference between the estimated quantity and the measured value becomes less than or equal to a predetermined value. The predetermined number or predetermined value is stored in advance in the memory of the gripping estimation device 17.

[0081] (Step S138) The acquisition unit 173 sets a sub-region in the observation data that is different from the sub-region in step S131. The acquisition unit 173 outputs the position of the other sub-region to the control unit 181.

[0082] (Step S139) The estimation unit 175 sets the model to be trained as the gripping amount estimation model. The estimation unit 175 stores the gripping amount estimation model in the memory of the gripping estimation device 17. With this, the estimation model generation process is completed.

[0083] According to the estimation model generation process of the embodiment described above, the gripping amount estimation model used in the gripping estimation device 17 is generated by inputting observation data of the object to be gripped 133 by the gripping device 11 before gripping and information on the insertion depth of the parallel gripper 123 into the model to be learned, calculating a loss based on the parameters of the probability distribution output from the model to be learned and the measured value of the gripping amount of the object to be gripped 133 by the gripping device 11, and training the model to be learned to reduce the loss.

[0084] (First variation) In this modified example, the controller 171 (A) performs pre-grasping on multiple sub-regions, (B) outputs an estimated gripping amount and standard deviation for each sub-region where pre-grasping has been performed using a gripping amount estimation model, (C) selects a sub-region from which an estimated gripping amount close to the target gripping amount and a small standard deviation have been obtained as a gripping region, and (D) controls the end effector 111 to grip the object to be gripped from that gripping region. In process (B), the acquisition unit 173 acquires observation data for each of the multiple sub-regions from which pre-grasping has been performed. Next, the estimation unit 175 concatenates the acquired RGB-D patch images and insertion depth information and sequentially inputs them into a trained gripping amount estimation model, thereby outputting multiple estimated gripping amounts and multiple standard deviations for each of the multiple insertion depths for each sub-region. Subsequently, in process (C), the selection unit 179 calculates a judgment amount based on the outputted multiple estimated gripping amounts and multiple standard deviations, and selects a gripping region from among the multiple sub-regions based on this judgment amount. At this time, the selection unit 179 transmits insertion depth information to the control unit 181, similar to step S801 described above.

[0085] Then, in process (D), after selecting the gripping region, the control unit 181 controls the drive devices and opening / closing motors for the operation of the multiple links 113 so that the object to be gripped is gripped in this gripping region. As a result, the pair of tip portions 127 grip the object to be gripped. The insertion of the tips of the pair of tip portions 127 to the gripping depth and the gripping of the object by the pair of tip portions 127 are the same as in the process in step S802, so the explanation is omitted. Subsequently, the processes from step S804 onwards are executed. Note that in this modified example, the processes in steps S804 and S805 are optional.

[0086] System 1 according to the first modified example acquires observation data after pre-grasping and selects the gripping area. Therefore, System 1 according to the first modified example can further improve the accuracy of quantitative gripping of food fragments. In addition, System 1 according to the first modified example can reduce or eliminate the time required for post-grasping, thereby reducing the processing time for quantitative gripping of food fragments, i.e., improving the throughput of quantitative gripping of food fragments. Other effects are the same as in the embodiment, so their explanation will be omitted.

[0087] (Second variation) In this modified example, the controller 171 (A) sets one sub-region as a gripping region and performs pre-gripping on this gripping region, (B) outputs an estimated gripping amount and standard deviation for each of multiple insertion depth information for this gripping region where pre-gripping has been performed using a gripping amount estimation model, (C) identifies the insertion depth information input to the model when an estimated gripping amount close to the target gripping amount and a small standard deviation are obtained, and (D) controls the end effector 111 to grip the object to be gripped from the gripping region based on the identified insertion depth information. In process (B), the acquisition unit 173 acquires an RGB-D patch image in the gripping region, concatenates the acquired RGB-D patch image and the insertion depth information and sequentially inputs them to the trained gripping amount estimation model, thereby outputting an estimated gripping amount and standard deviation for each of multiple insertion depths with respect to the gripping region. Subsequently, in process (C), the estimation unit 175 identifies the insertion depth in the gripping region based on the estimated gripping amount and standard deviation for each insertion depth. Specifically, the estimation unit 175 identifies the insertion depth information from among multiple insertion depth information, which has been output from the gripping amount estimation model as the estimated gripping amount and standard deviation that minimizes the judgment amount. The estimation unit 175 then identifies the insertion depth in that insertion depth information as the insertion depth to be used for the gripping area.

[0088] Then, in process (D), the control unit 181 controls the drive devices and opening / closing motors for the operation of the multiple links 113 so that they grip the object to be gripped at a specified insertion depth in the gripping region. As a result, the pair of tip portions 127 grip the object to be gripped. The insertion of the tips of the pair of tip portions 127 to the specified gripping depth, and the gripping of the object by the pair of tip portions 127 are the same as in the process in step S802, so the explanation is omitted. Subsequently, the processes from step S804 onwards are executed. Note that in this modified example, the processes in steps S804 and S805 are optional.

[0089] The second modified system 1 acquires an RGB-D patch image in the gripping region after pre-grasping, and uses the acquired RGB-D patch image and a gripping amount estimation model to determine the insertion depth in the gripping region. The effects of the second modified system are the same as those of the embodiment and the first modified system, so a description is omitted.

[0090] (Examples of application) This application differs from the end effector 111 in the embodiment. Figure 14 shows an example of a cross-section of the end effector 112 in this application. The end effector 112 shown in Figure 14 is mounted on the gripping device 11 in System 1. The end effector 112 is designed to grip paste-like foods (hereinafter referred to as paste foods). Paste foods are foods processed into a fluid, semi-solid state, such as potato salad.

[0091] The end effector 112 shown in Figure 14 comprises a base 140, a scooping gripper 143, a wall gripper 145, and an extrusion plate 147 provided on the delivery mechanism 124. A force sensor may be mounted on the base portion of the base 14. The base 140 supports the scooping gripper 143, the wall gripper 145, and the extrusion plate 147. The base 140 supports the scooping gripper 143 and the wall gripper 145 so that the opening 125 can be opened and closed. The movement mechanism between the scooping gripper 143 and the wall gripper 145 is realized, for example, by a motor that rotates according to a control signal output from a controller 171, a pinion gear connected to the drive shaft of the motor and rotating with the rotation of the drive shaft, and a rack gear that is fitted with the pinion gear and provided on the base 140 along the opening and closing direction of the opening 125. The wall gripper 145 may also be fixed to the base 140.

[0092] The scooping gripper 143 is capable of scooping paste foods and is composed of multiple surface shapes. The base of the scooping gripper 143 is mounted on a rack gear. The rack gear is mounted on the base 140 via a linear guide. The extrusion plate 147 is attached to the base of the wall gripper 142. The extrusion plate 147 is movable along the scooping gripper 143 and the wall gripper 145 by a dispensing mechanism 124. For example, the extrusion plate 147 is mounted perpendicular to the tooth direction of the rack gear RG. The rack gear RG is engaged with a pinion gear PG mounted on the rotating shaft of the motor MT. As a result, when the motor MT rotates according to a control signal output from the controller 171, the pinion gear PG rotates in conjunction with the rotation of the drive shaft of the motor MT. As the pinion gear PG rotates, the rack gear RG moves, causing the push plate 147 to move while its end contacts the scooping gripper 143 and the wall gripper 145.

[0093] The following describes the post-grasping operation by the end effector 112 shown in Figure 14. Figure 15 is a cross-sectional view showing an example of the operation flow related to post-grasping. As shown in Figure 15(1), the scooping gripper 143 scoops up the paste food PO in the tray 131, and the paste food PO is gripped by the end effector 112. Next, as shown in Figure 15(2), under the control of the control unit 181, the wall gripper 145 and the scooping gripper 143 move to close the opening 125. As a result, as shown in Figure 15(2), the paste food PO is sandwiched between the scooping gripper 143 and the wall gripper 145 and formed into a triangular prism shape (triangular in cross-section). Note that any food that spills out of both grippers when the scooping gripper 143 and the wall gripper 145 are completely closed is likely to fall into the tray 131.

[0094] As shown in Figures 15(3) and (4), the dispensing mechanism 124 is driven under the control of the control unit 181, and the extrusion plate 147 pushes the paste food PO to be gripped out of the end effector 112. As the extrusion plate 147 moves, the scooping gripper 143 and the wall gripper 145 move away from each other under the control of the control unit 181. At this time, the extrusion plate 147 scrapes off the paste food PO adhering to the scooping gripper 143 and the wall gripper 145 and pushes it out of the end effector 112 (gripper). During this extrusion, the control unit 181 determines the amount of movement of the extrusion plate 147 according to the amount actually gripped and the target gripping amount so that when the paste food PO pushed out of the end effector 112 is dropped into the tray 131, the target gripping amount of paste food PO is gripped inside the gripper. The control unit 181 then controls the dispensing mechanism 124 to move the extrusion plate 147 in the reverse direction while closing both grippers. As a result, if the end effector 112 grips an amount of paste food PO exceeding the target gripping amount, the control unit 181 can drop the excess paste food PO into the tray 131. The effects in this application example are the same as in the embodiment, so a description is omitted.

[0095] The grasping amount estimation model in the embodiment described above accepts a D-patch image as input, but it may be configured to accept the mean and standard deviation of the depths of the pixels constituting the D-patch image as input instead of the D-patch image itself. In this case, when training and using the model, the mean and standard deviation of the depths in the D-patch image should be input to the model instead of the D-patch image.

[0096] In the previously described embodiment, the gripping amount estimation model received both the D patch image and the RGB patch image as input for patch observation data, and output data regarding the gripping amount of the object grasped by that patch. However, the gripping amount estimation model may also be one that does not require the RGB patch image as input. Such a model can be obtained by performing an estimation model generation process that does not use the RGB patch image, and can be used in the gripping release process without the RGB patch image. Furthermore, the gripping amount estimation model that does not use the RGB patch image may be trained to accept the mean and standard deviation of the depth in the D patch image as input instead of the D patch image, and then used.

[0097] The gripping amount estimation model in the embodiment described above took patch observation data and insertion depth information as input and output data regarding the gripping amount of the object gripped by that patch. However, the gripping amount estimation model may not require patch observation data as input, nor may it require insertion depth information as input.

[0098] A gripping amount estimation model that outputs data on gripping amount from insertion depth information without patch observation data can be obtained, for example, by the following estimation model generation process. The estimation model generation process measures the actual gripping amount by gripping an object from the food surface of the patch at multiple insertion depths, inputs the insertion depth into the model to estimate the mean μ and standard deviation σ of the gripping amount, calculates the loss based on the mean μ, standard deviation σ and the actual gripping amount, and updates the model parameters in a direction that reduces the loss. At this time, the food surface of the patch can be the median or mean of the patch depth, and various patch sizes can be used. Note that if it is possible to output the mean μ and standard deviation σ of the estimated gripping amount from the insertion depth, a data table can be used as the gripping amount estimation model instead of a learning model, and this table can be generated by associating the mean and standard deviation of the measured gripping amount for each insertion depth with the insertion depth. On the other hand, the gripping release process inputs the insertion depth for each of the multiple insertion depths into the gripping amount estimation model to estimate the mean μ and standard deviation σ of the gripping amount, identifies the insertion depth corresponding to the mean μ and standard deviation σ that minimize the judgment amount, selects a pre-set patch as the gripping area, and performs the processes from step S802 onward.

[0099] A gripping amount estimation model that outputs data on gripping amount from patch observation data without insertion depth information can be obtained, for example, by the following estimation model generation process. The estimation model generation process measures the actual gripping amount by gripping an object at a predetermined insertion depth from the food surface of each patch for multiple patches, inputs the observation data of that patch into the model to estimate the mean μ and standard deviation σ of the gripping amount, calculates the loss based on the mean μ, standard deviation σ and the actual gripping amount, and updates the model parameters in a direction that minimizes the loss. On the other hand, the gripping release process inputs the observation data of each patch into the gripping amount estimation model for each of the multiple patches to estimate the mean μ and standard deviation σ of the gripping amount, selects the patch corresponding to the mean μ and standard deviation σ that minimize the judgment amount from among the multiple patches as the gripping area, and performs the processing from step S802 onwards.

[0100] In the embodiments described above, some or all of the devices (such as the gripping estimation device 17 or the controller 171) may be composed of hardware, or they may be composed of information processing by software (programs) executed by a CPU (Central Processing Unit), GPU (Graphics Processing Unit), etc. If the information processing is composed of software, the software that realizes at least some of the functions of each device (such as the gripping estimation device 17 and / or the controller 171) in the embodiments described above may be stored on a non-temporary storage medium (non-temporary computer-readable medium) such as a CD-ROM (Compact Disc-Read Only Memory) or USB (Universal Serial Bus) memory, and the information processing of the software may be executed by loading it into a computer. Alternatively, the software may be downloaded via a communication network. Furthermore, all or part of the processing of the software may be implemented in a circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array), so that the information processing by the software is executed by hardware.

[0101] The storage medium for storing the software may be a removable medium such as an optical disc, or a fixed storage medium such as a hard disk or memory. Furthermore, the storage medium may be located inside the computer (main memory, auxiliary storage) or outside the computer.

[0102] Figure 16 is a block diagram showing an example of the hardware configuration of each device (such as the grip estimation device 17 or controller 171) in the embodiment described above. Each device may be implemented as a computer 81, for example, comprising a processor 811, a main memory 812 (memory), an auxiliary memory 813 (memory), a network interface 814, and a device interface 815, all connected via a bus 816.

[0103] The computer 81 in Figure 16 has one of each component, but it may have multiple identical components. Also, although Figure 16 shows one computer 81, the software may be installed on multiple computers, and each of these multiple computers may execute the same or different parts of the software's processing. In this case, it may be a distributed computing configuration in which each computer communicates via a network interface 814 or the like to execute processing. In other words, each device in the above-described embodiment (such as the gripping estimation device 17 or the controller 171) may be configured as a system that realizes its function by having one or more computers execute instructions stored in one or more storage devices. Alternatively, it may be configured so that information transmitted from a terminal is processed by one or more computers located on the cloud, and the processing results are transmitted to the terminal.

[0104] The various calculations performed by each device (such as the gripping estimation device 17 or the controller 171) in the embodiments described above may be performed in parallel using one or more processors, or using multiple computers via a network. Alternatively, the various calculations may be distributed to multiple processing cores within a processor and performed in parallel. Furthermore, some or all of the processing and means of this disclosure may be implemented by at least one of a processor and a storage device located on a cloud that can communicate with computer 81 via a network. Thus, each device in the embodiments described above may be implemented in the form of parallel computing using one or more computers.

[0105] The processor 811 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.) that performs either control or calculations for a computer. The processor 811 may also be a general-purpose processor, a dedicated processing circuit designed to perform specific calculations, or a semiconductor device including both a general-purpose processor and a dedicated processing circuit. Furthermore, the processor 811 may include optical circuits or quantum computing-based calculation functions.

[0106] The processor 811 may perform calculations based on data and software input from various devices within the computer 81, and may output calculation results and control signals to these devices. The processor 811 may also control the various components of the computer 81 by executing the computer 81's OS (Operating System) or applications.

[0107] Each device in the above-described embodiment (such as the gripping estimation device 17 or the controller 171) may be implemented by one or more processors 811. Here, the processor 811 may refer to one or more electronic circuits arranged on one chip, or one or more electronic circuits arranged on two or more chips or two or more devices. When multiple electronic circuits are used, each electronic circuit may communicate by wire or wireless.

[0108] The main memory 812 may store instructions executed by the processor 811 and various data, and the information stored in the main memory 812 may be read by the processor 811. The auxiliary memory 813 is a memory other than the main memory 812. These memory devices refer to any electronic component capable of storing electronic information, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. In the embodiments described above, the memory for storing various data in each device (such as the grip estimation device 17 or the controller 171) may be implemented by the main memory 812 or the auxiliary memory 813, or by the built-in memory of the processor 811. For example, the memory in the grip estimation device 17 in the embodiments described above may be implemented by the main memory 812 or the auxiliary memory 813.

[0109] In the embodiments described above, if each device (such as the gripping estimation device 17 or the controller 171) consists of at least one storage device (memory) and at least one processor connected to (coupled with) this at least one storage device, then at least one processor may be connected to one storage device. Also, at least one storage device may be connected to one processor. Furthermore, the configuration may include at least one processor among a plurality of processors being connected to at least one storage device among a plurality of storage devices. This configuration may also be realized by storage devices and processors included in a plurality of computers. Moreover, the configuration may include a storage device integrated with a processor (for example, a cache memory including an L1 cache and an L2 cache).

[0110] The network interface 814 is an interface for connecting to the communication network 82 wirelessly or via a wired connection. The network interface 814 can be any appropriate interface, such as one conforming to existing communication standards. Information may be exchanged between the computer 81 and an external device 83a connected via the communication network 82 through the network interface 814. The communication network 82 may be a WAN (Wide Area Network), LAN (Local Area Network), PAN (Personal Area Network), or a combination thereof, as long as information is exchanged between the computer 81 and the external device 83a. An example of a WAN is the Internet; an example of a LAN is IEEE 802.11 or Ethernet®; and an example of a PAN is Bluetooth® or NFC (Near Field Communication).

[0111] The device interface 815 is an interface such as USB that connects directly to the external device 83b.

[0112] External device 83a is a device connected to computer 81 via network 82. External device 83b is a device directly connected to computer 81.

[0113] External device 83a or external device 83b may, for example, be an input device. The input device may be a camera, microphone, motion capture device, various sensors, keyboard, mouse, touch panel, etc., and will provide the acquired information to the computer 81. Alternatively, it may be a device equipped with an input unit, memory, and processor, such as a personal computer, tablet terminal, or smartphone.

[0114] Furthermore, the external device 83a or external device 83b may, for example, be an output device. The output device may be a display device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) panel, or it may be a speaker that outputs sound, etc. It may also be a device equipped with an output unit, memory, and a processor, such as a personal computer, tablet terminal, or smartphone.

[0115] Furthermore, external devices 83a and 83b may be storage devices (memory). For example, external device 83a may be network storage, and external device 83b may be storage such as an HDD.

[0116] Furthermore, the external device 83a or external device 83b may be a device that has some of the functions of the components of each device (such as the gripping estimation device 17 or the controller 171) in the embodiments described above. In other words, the computer 81 may transmit some or all of the processing results to the external device 83a or external device 83b, or receive some or all of the processing results from the external device 9A or external device 9B. For example, the external device 83a or external device 83b may be at least one of the gripping device 11, observation device 13, weighing device 15, and gripping estimation device 17 in the embodiments described above.

[0117] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, ab, ac, bc, or abc. Furthermore, any element may have multiple instances, such as aa, abb, aabbcc, etc. In addition, it is also possible to add other elements other than the enumerated elements (a, b, and c), such as abcd which has d.

[0118] In this specification (including the claims), when expressions such as "using data as input / based on data / according to / in accordance with data" (including similar expressions) are used, unless otherwise specified, this includes using the data itself or using data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of the data, etc.). Furthermore, when it is stated that some result is obtained "using data as input / based on data / according to / in accordance with data" (including similar expressions), unless otherwise specified, this includes cases where the result is obtained based solely on the data in question or where the result is influenced by other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output" (including similar expressions), unless otherwise specified, this includes cases where the data itself is used as output or where data that has been processed in some way (e.g., data with added noise, normalized data, features extracted from the data, intermediate representations of various types of data, etc.) is used as output.

[0119] In this specification (including the claims), the terms “connected” and “coupled” are intended to be non-restrictive terms that include any direct connection / coupling, indirect connection / coupling, electrical connection / coupling, communicative connection / coupling, operational connection / coupling, physical connection / coupling, etc. The terms should be interpreted as appropriate in the context in which they are used, but any form of connection / coupling that is not intentionally or naturally excluded should be interpreted non-restrictively as being included in the terms.

[0120] In this specification (including the claims), when the expression "A configured to B" is used, it may include that the physical structure of element A has a configuration capable of performing operation B, and that the permanent or temporary setting / configuration of element A is configured to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and that it is configured to actually perform operation B by the setting of a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor, dedicated arithmetic circuit, etc., it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0121] Wherever terms meaning "comprising" or "possessing" (e.g., "comprising / including," "having," etc.) are used herein, they are intended to be open-ended terms, including cases where the subject matter of such terms is not the object of the term. Where the object of such terms meaning "comprising" or "possessing" is an expression that does not specify a quantity or suggests a singular number (an expression with the article "a" or "an"), such expression should be interpreted as not being limited to a specific number.

[0122] In this specification (including the claims), even if expressions such as "one or more" or "at least one" are used in some places, and expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singularity (expressions using the articles a or an) should not necessarily be interpreted as not being limited to a specific number.

[0123] In this specification, if a particular configuration of an embodiment is described as providing a specific advantage or result, it should be understood, unless otherwise stated, that the same advantage or result can also be obtained from one or more other embodiments having that configuration. However, it should be understood that the presence or absence of such an advantage or result generally depends on various factors, conditions, and / or states, and that the configuration does not necessarily guarantee that the advantage or result can be obtained. The advantage or result can only be obtained from the configuration described in the embodiment when various factors, conditions, and / or states are met, and the advantage or result cannot necessarily be obtained in the claimed invention that defines that configuration or a similar configuration.

[0124] In this specification (including the claims), when multiple hardware components perform a predetermined process, each component may cooperate to perform the predetermined process, or some components may perform all of the predetermined process. Alternatively, some components may perform part of the predetermined process, while other components perform the remainder. In this specification (including the claims), when expressions such as "one or more hardware components perform a first process, and the one or more hardware components perform a second process" (including similar expressions) are used, the hardware component performing the first process and the hardware component performing the second process may be the same or different. In other words, it is sufficient that the hardware component performing the first process and the hardware component performing the second process are included in the one or more hardware components. Hardware may include electronic circuits, devices containing electronic circuits, etc.

[0125] In this specification (including the claims), when multiple memory devices store data, each of the multiple memory devices may store only a portion of the data or the entire data. Furthermore, a configuration in which some of the multiple memory devices store data is also included.

[0126] While embodiments of this disclosure have been described in detail above, this disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, and partial deletions are possible, provided that they do not depart from the conceptual idea and spirit of the present invention derived from the claims and their equivalents. For example, where numerical values ​​or mathematical formulas are used in the description of the embodiments described above, these are provided for illustrative purposes only and do not limit the scope of this disclosure. Similarly, the sequence of operations shown in the embodiments is also illustrative and does not limit the scope of this disclosure.

Claims

1. A gripping device having a gripper that opens and closes; a control unit that controls the gripping device so as to insert the gripper into food contained in a container to grip and lift up a portion of the food; having The control unit controls the gripping device to perform a shaking action of the gripper while lifting up and gripping the portion of the food product.

2. The control unit controls the gripping device to move the gripper, which has lifted and gripped the portion of the food product after the shaking action, to a transfer destination. The system of claim 1 .

3. The control unit controls the gripping device to perform the shaking action so as to shake the gripper over the container.

3. A system according to claim 1 or 2.

4. The gripper does not close completely during the gripping. A system according to any one of claims 1 to 3.

5. The gripper grasps and lifts the portion of the food with the portion protruding from the gripper. A system according to any one of claims 1 to 4.

6. The food product that protrudes from the gripper is shaken off by the shaking action. The system of claim 5.

7. In the process of gripping and lifting the portion of the food, the food that is lifted by the gripper separately from the portion is shaken off by the shaking action. A system according to any one of claims 1 to 6.

8. A measuring means for measuring the weight of the food held by the gripper after the shaking action is performed while the food is held by the gripper. A system according to any one of claims 1 to 7.

9. The measuring means is provided on the gripping device. The system of claim 8.

10. The measuring means is provided at a base portion of the gripper. The system of claim 9.

11. The control unit controls the gripping device to move the food held by the gripper to a transfer destination according to the weight measured by the measuring means. A system according to any one of claims 8 to 10.

12. The control unit controls the gripping device so that the food held by the gripper is sent out of the gripper by a first mechanism while still being held by the gripper, according to the weight measured by the measuring means. A system according to any one of claims 8 to 11.

13. The insertion is performed from above in the vertical direction. A system according to any one of claims 1 to 12.

14. The control unit determines the depth of insertion according to a weight of the food to be gripped by the gripper. A system according to any one of claims 1 to 13.

15. The insertion depth corresponds to a depth from a surface of the food contained in the container. The system of claim 14.

16. A system as described in any one of claims 1 to 15, wherein the control unit grasps the portion of the food by inserting the gripper into the food in an open state and closing it.

17. The open state is a state in which the gripper is opened to a preset opening width, The depth of insertion is determined according to the weight of the food to be gripped by the gripper.

17. The system of claim 16.

18. The control unit determines a position for inserting the gripper to grip the portion of the food based on a state of the food contained in the container. A system according to any one of claims 1 to 17.

19. The shaking motion is a motion that vibrates, rotates, or swings the gripper.

19. A system according to any one of claims 1 to 18.

20. The gripper is a parallel gripper that opens and closes horizontally.

20. A system according to any one of claims 1 to 19.

21. The food is a small piece of food that tends to get tangled or stick to each other.

21. A system according to any one of claims 1 to 20.

22. The food is a paste-like food or a semi-solid processed food.

22. A system according to any one of claims 1 to 21.

23. A first mechanism for sending an object held by the gripper out of the gripper; a control unit that controls the first mechanism so that a target amount of the object gripped by the gripper is transferred to a transfer destination; A system having

24. The control unit controls the first mechanism based on the amount of the object gripped by the gripper and the target amount.

24. The system of claim 23.

25. The control unit controls the gripper to grip the object with the gripper, lift it up, and then release it, and controls the gripper to re-grip the object that has been released from the gripper; the gripper re-grasping the object including the portion delivered by the first mechanism.

24. The system of claim 23.

26. The control unit controls a second mechanism to vibrate the gripper while gripping the object.

26. The system of claim 25.

27. The tip of the gripper has an opening, The first mechanism delivers the object toward the opening.

27. A system according to any one of claims 23 to 26.

28. the control unit controls a speed at which the first mechanism sends out the object based on an amount of the object gripped by the gripper and the target amount.

28. A system according to any one of claims 23 to 27.

29. the control unit controls the number of times the first mechanism sends out the object based on the amount of the object gripped by the gripper and the target amount.

29. A system according to any one of claims 23 to 28.

30. A selection unit is further provided that specifies, as a location where the object is gripped by the gripper, a location that is occupied by the object and satisfies a predetermined condition that allows the object to be gripped in a larger amount than the target amount.

30. A system according to any one of claims 23 to 29.

31. An estimation unit that estimates a gripping amount of the object by the gripper and an indicator of an uncertainty of the gripping amount for a portion occupied by the object, The system according to claim 30 , wherein the selection unit determines whether the location satisfies the predetermined condition based on at least the estimated gripping amount and an index.

32. 32. The system of claim 23, wherein the gripper has a protrusion or depression on a portion of a surface that grips the object.

33. The system described in claim 32, wherein the protrusion portion has a protrusion facing in a direction in which the first mechanism sends out the object.

34. A method for controlling a gripping device so as to insert a gripper of the gripping device into food contained in a container to grip and lift up a portion of the food, a control unit that controls the gripping device to perform a swinging action of the gripper in a state where the gripper is lifting up and gripping the portion of the food; An apparatus having the above configuration.

35. A computer comprising: Controlling the gripping device to insert a gripper of the gripping device into the food contained in the container to grip and lift up a portion of the food; Controlling the gripping device to perform a swinging motion of the gripper while lifting and gripping the portion of the food product; A program to make this happen.

36. A method for controlling a gripping device having a gripper that opens and closes, comprising: Controlling the gripping device to insert the gripper into food contained in a container to grip and lift up a portion of the food; controlling the gripping device to perform a swinging motion of the gripper while lifting and gripping the portion of the food product; method.

37. A method for controlling a first mechanism by which a control unit sends an object held by a gripper out of the gripper, comprising: The target amount of the object gripped by the gripper is transferred to the transfer destination.

1. Control the mechanism method.

Images