A system that outputs control information to control a robot hand that can change the insertion depth into a collection area for small, irregularly shaped ingredients.

Abstract

To provide a system capable of implementing the taking-out of highly accurate desirable amount of small-sized food material.SOLUTION: The system is configured to: acquire an image of accumulation place of small-sized food material; estimate analogy with a sample local area; determine at least one candidate local area from the image; at the same time, determine an inclination and an intercept of a primary function which exhibit a relation between an insertion depth to the small-sized food material of the robot hand which is associated with each candidate local area based on information of insertion depth to the small-sized food material of the robot hand and weight of the small-sized food material which can be held and be taken-out by the robot hand; and output the inclination and the intercept of the primary function associated with each candidate local area in association with coordinate data of each candidate local area as control information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a system for outputting control information for controlling a robot hand that can change the depth of insertion into a collection location for small, irregularly shaped food ingredients. [Background technology]

[0002] Conventionally, the task of placing small, irregularly shaped ingredients, such as green onions, in a bento box or the like has been carried out by a person removing the small, irregularly shaped ingredients by grasping them with their fingers from a storage area for small ingredients, such as a food tray, and placing them in the appropriate position in the bento box or the like.

[0003] However, labor shortages are expected to become more serious in the near future. For this reason, automation of each process by introducing robots is being considered, and automation of food plating work is also being considered as part of this.

[0004] For example, as shown in Patent Document 1, a robot hand (soft gripper) that grips ingredients has been developed to automate the process of plating ingredients. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2020-168691 [Non-patent literature]

[0006] [Non-Patent Document 1] Girshick, RB: Fast R-CNN, CoRR, Vol. abs / 1504.08083(2015). [Non-patent document 2] Ren, S., He, K., Girshick, R. and Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks, IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, No. 6, pp. 1137-1149 (2017). Summary of the Invention [Problem to be solved by the invention]

[0007] However, when trying to grasp and remove small, irregularly shaped ingredients using a robot hand (soft gripper) such as that shown in Patent Document 1, the stacking condition of the small, irregularly shaped ingredients piled up in the collection area is not constant (for example, the surface unevenness varies), and the stacking condition changes each time the ingredients are removed, making it difficult to always remove the desired amount of small ingredients with high precision.

[0008] The inventors of the present invention have been conducting extensive research into whether it is possible to always accurately remove the desired amount of small ingredients by analyzing images of the accumulation area of ​​small, irregularly shaped ingredients and estimating the loading status of the small ingredients based on the images.

[0009] As this research progressed, we discovered that in order to realize a highly flexible robotic hand system, it would be effective to (1) use a robotic hand that can change its "insertion depth" into the collection area for small, irregularly shaped ingredients, and (2) have a system that outputs control information for controlling such a robotic hand determine the slope and intercept of a linear function that shows the relationship between the insertion depth of the robotic hand into small ingredients and the weight of the small ingredients that can be grasped and removed by the robotic hand.

[0010] This means that even when the weight of the small ingredients to be removed changes relatively frequently, such as when producing a variety of bento boxes in small quantities, the variable "insertion depth" of the robot hand can be used to achieve more flexible ingredient removal control.

[0011] The present invention was made based on the above findings, and an object of the present invention is to provide a system that outputs control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients, and that can achieve more accurate removal of a desired amount of small ingredients. [Means for solving the problem]

[0012] The present invention is a system that outputs control information for controlling a robot hand that can change its insertion depth into a collection area for small, irregularly shaped ingredients, and acquires an image of the collection area for the small ingredients, evaluates the similarity with a sample local area to determine at least one candidate local area from the image, and determines the slope and intercept of a linear function that is associated with each candidate local area and shows the relationship between the insertion depth of the robot hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robot hand, based on information that is associated with the sample local area and the insertion depth of the robot hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robot hand, and links the slope and intercept of the linear function associated with each candidate local area to the coordinate data of each candidate local area and outputs it as control information.

[0013] The system of the present invention can determine a linear function between the insertion depth of the robot hand into the small foodstuffs and the weight of the small foodstuffs that can be picked up from an image of the accumulation area of ​​the small foodstuffs, which makes it easier to respond to changes in the weight of the small foodstuffs to be picked up.

[0014] In the present invention, the number of "sample local regions" may be one or more. "Evaluating the similarity with the sample local region" may mean evaluating the similarity by extracting some parameters using an image processing program or the like and comparing the parameters, or may directly evaluate the similarity (correlation) between the two images using an AI learning program or the like.

[0015] Furthermore, in the present invention, "information associated with the sample local region relating to the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" may refer to the "slope and intercept of a linear function that shows the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" if such information is provided in advance for the sample local region. For example, if there are multiple "sample local regions" that can be determined to have some common characteristics, it is possible to obtain the "slope and intercept of the linear function" by the least squares method based on the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of these "sample local regions."

[0016] Furthermore, in the present invention, "determining the slope and intercept of a linear function that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, which is associated with each candidate local area, based on information on the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, which is associated with the sample local area" means "determining the slope and intercept of a linear function that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, which is associated with the sample local area." If the "information on the weight of the small food ingredient that can be grasped and removed by the robot hand" is the "slope and intercept of a linear function," this includes determining the "slope and intercept of the linear function" as is as the "slope and intercept of the linear function" that corresponds to (is linked to) each "candidate local area," or performing a correction process on the "slope and intercept of the linear function" based on the similarity (degree of correlation) between the "sample local area" and each "candidate local area," and determining the result as the "slope and intercept of the linear function" that corresponds to (is linked to) each "candidate local area."

[0017] Alternatively, in the present invention, "determining the slope and intercept of a linear function associated with each candidate local area that shows the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, based on information associated with the sample local area regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" includes, in a case where "the information associated with the sample local area regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand" is not provided as "the slope and intercept of a linear function," but is provided, for example, as pairs (sets) of the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of a plurality of sample local areas, using, for example, machine learning model data (and an AI program) constructed by using these multiple pairs (sets) as training data to directly calculate the "slope and intercept of a linear function" that corresponds to (is associated with) "each candidate local area."

[0018] In the present invention, an image may be a D image (distance image, depth image), an RGB image, or an RGBD image that combines the features of both.

[0019] Furthermore, in the present invention, when determining or calculating the "slope and intercept of a linear function" corresponding to (linked to) "each candidate local region," it is preferable to also evaluate the "likelihood" of such determination or calculation.

[0020] The "likelihood" may be calculated or evaluated based on the similarity (degree of correlation) between the "sample local region" and "each candidate local region," or may be evaluated directly by the AI ​​learning program.

[0021] In such a case, the present invention can be defined as follows: That is, one aspect of the present invention is a system that outputs control information for controlling a robotic hand that can change its insertion depth into a collection area for small, irregularly shaped ingredients, the system acquiring an image of the collection area for small ingredients, evaluating similarity with a sample local area to determine at least one candidate local area from the image, and determining the slope, intercept, and likelihood of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand based on information associated with the sample local area that indicates the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, and outputting the slope, intercept, and likelihood of the linear function associated with each candidate local area as control information, linked to coordinate data for each candidate local area.

[0022] In this aspect, if there are multiple candidate local regions, it is more preferable to determine the candidate local region with the highest likelihood as the final candidate local region, and to link the slope and intercept of the linear function associated with the final candidate local region to the coordinate data of the final candidate local region and output them as control information.

[0023] The present invention can also be categorized as a method. That is, the method invention is a method for outputting control information for controlling a robotic hand capable of changing the insertion depth into a collection area for small, irregularly shaped ingredients, the method comprising the steps of: acquiring an image of the collection area for small ingredients; determining at least one candidate local area from the image by evaluating similarity with a sample local area; and determining, based on information associated with the sample local area relating to the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, a slope and an intercept of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand; and linking the slope and intercept of the linear function associated with each candidate local area to coordinate data of each candidate local area and outputting them as control information.

[0024] When "likelihood" is also evaluated, the method invention is a method for outputting control information for controlling a robotic hand that can change its insertion depth into a collection location for small, irregularly shaped ingredients, the method comprising the steps of: acquiring an image of the collection location for small ingredients; determining at least one candidate local area from the image by evaluating the similarity with a sample local area; and determining the slope, intercept, and likelihood of a linear function associated with each candidate local area that indicates the relationship between the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand, based on information associated with the sample local area regarding the insertion depth of the robotic hand into the small ingredient and the weight of the small ingredient that can be grasped and removed by the robotic hand; and linking the slope, intercept, and likelihood of the linear function associated with each candidate local area to the coordinate data of each candidate local area, and outputting them as control information.

[0025] The present invention can also be defined as a program for outputting control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients, characterized in that the program can implement the above-mentioned method when executed by a computer (such programs are also protected by this application). [Effects of the Invention]

[0026] According to the present invention, a linear function between the insertion depth of the robot hand into the small foodstuffs and the weight of the small foodstuffs that can be picked up can be determined from an image of the accumulation area of ​​the small foodstuffs, which makes it easier to respond to changes in the weight of the small foodstuffs to be picked up. [Brief explanation of the drawings]

[0027] [Figure 1] 1 is a schematic perspective view of a robot hand system including a system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a schematic perspective view showing a state in which a robot hand of the robot hand system of FIG. 1 is positioned above a collection location. [Figure 3] FIG. 2 is a schematic perspective view showing a state in which a robot hand of the robot hand system of FIG. 1 is inserted into a small food ingredient in a collection location. [Figure 4] FIG. 2 is a perspective view of a soft gripper of the robot hand of FIG. 1. [Figure 5] 2 is a side view of the outer part of the scoop element on which the protruding element is erected in the soft gripper of FIG. 1. FIG. [Figure 6] 6 is a perspective view of an outer part of the scoop element on which the protruding element of FIG. 5 is erected. FIG. [Figure 7] FIG. 2 is a conceptual diagram illustrating the module configuration of the robot hand system of FIG. [Figure 8] FIG. 2 is a conceptual diagram illustrating the general structure of a neural network used in the system in FIG. [Figure 9]2 is a diagram illustrating how pseudo teacher data is assigned to an area where teacher data is unknown in the system in FIG. 1. FIG. [Figure 10] FIG. 2 is a diagram showing the distribution of insertion depth and grip weight in training data for the system in FIG. 1. [Figure 11A] 2 is a diagram showing an example of an RGB image obtained by photographing a small food ingredient (green onion) at a collection point with a camera in the system in FIG. 1. FIG. [Figure 11B] 2 is a diagram showing an example of a D image (depth image) obtained by photographing a small food ingredient (green onion) at a collection location with an RGBD camera in the system in FIG. 1. FIG. [Figure 12] FIG. 2 is a diagram showing an example in which multiple regions are proposed in the system in FIG. 1. [Figure 13] FIG. 2 is a diagram showing an evaluation of estimated weight relative to actual weight in an area where training data is known in the system of FIG. 1. [Figure 14] FIG. 10 is a conceptual diagram illustrating the module configuration of a robot hand system including a system according to a second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0028] Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

[0029] (Overall composition) FIG. 1 is a schematic perspective view of a robot hand system 2 including a system 1 according to a first embodiment of the present invention.

[0030] System 1 is a system that outputs control information for controlling a robot hand 8. The robot hand 8 is capable of changing the depth to which small, irregularly shaped ingredients 4 are inserted into a collection location 6 (see FIG. 10). Details of system 1 will be described later.

[0031] The robot hand system 2 is a robot hand system that picks up small, irregularly shaped ingredients 4 by grasping them from a collection area 6 for the small ingredients 4. For example, it may also be called a topping device. The robot hand system 2 includes a camera 10 that captures an image of the collection area 6 for the small ingredients 4 and transmits it to the system 1, and a robot hand 8 that is controlled based on control information output from the system 1.

[0032] The irregularly shaped small food ingredients 4 are, for example, finely chopped green onions, corn, etc., and the accumulation location 6 is, for example, a food container or the like.

[0033] The robot hand 8 is attached to a robot arm 12 (for example, duAro manufactured by Kawasaki Heavy Industries, Ltd.). The robot hand 8 is equipped with a soft gripper 14 (see FIG. 4) that can softly grasp small food ingredients 4. The soft gripper 14 is capable of softly (without damaging) grasping small food ingredients such as finely chopped green onions or corn, and is equipped with a base member 16 that is positioned above the object to be grasped (see Patent Document 1).

[0034] The base member 16 has a substantially cross-shaped member 16a and four quadrant arc plates 16b fixed to the underside of the substantially cross-shaped member 16a. A circular robot arm attachment portion 13 is attached to the upper surface of the substantially cross-shaped member 16a via four support columns 17.

[0035] A flexible silicone scoop element 20 hangs down vertically from the underside of each of the quadrant arc plates 16b of the base member 16. The four scoop elements 20 are arranged adjacent to each other in the circumferential direction in a plan view. Each scoop element 20 has a two-stage tapered shape.

[0036] 4 to 6, nine protruding elements 31 to 39 having air chambers 41 to 49 therein are symmetrically erected on the outer surface of each scoop element 20. More specifically, the scoop element 20 is configured by adhering the scoop element outer part 20a, on which the nine protruding elements 31 to 39 are erected, to the scoop element inner part 20b.

[0037] Each of the air chambers 41 to 49 passes through the scoop element outer portion 20a and extends into the inside of each of the protruding elements 31 to 39. The upper surface of the scoop element outer portion 20b to which the scoop element outer portion 20a is adhered is flat.

[0038] The air chambers 41 to 44 formed by the protruding elements 31 to 34 (and the scoop element outer portion 20a) are horizontal air chambers extending parallel to one another in the circumferential direction (horizontal direction), and the air chambers 45 to 49 formed by the protruding elements 35 to 39 (and the scoop element outer portion 20a) are vertical air chambers extending parallel to one another in the up-and-down direction in an area above the horizontal air chambers 41 to 44. All of the air chambers 41 to 49 are commonly connected by the vertical compressed air supply passage 20c and the horizontal compressed air supply passage 20d formed in the scoop element outer portion 20a.

[0039] The camera 10 is attached to the robot arm 12 so as to take pictures downward. Therefore, for example, by controlling the robot arm 12, the camera 10 is positioned directly above the center of the collection site 6 when taking pictures.

[0040] Camera 10 photographs the stacked state (stacked state) of small ingredients 4 in collection area 6 from above toward below. In this embodiment, camera 10 photographs from above collection area 6 vertically downward, but for example, camera 10 may photograph the stacked state of small ingredients 4 from above collection area 6 diagonally downward. Camera 10 in this embodiment is an RGBD camera, and acquires an RGB image corresponding to each pixel in the image and a distance image (depth image) to the small ingredients corresponding to each pixel in the image.

[0041] (Configuration of System 1 of First Embodiment) The system 1 will now be described in more detail with reference to FIG.

[0042] The system 1 is configured to realize so-called AI-based machine learning, and as shown in Fig. 7, includes a robot control unit 54, a recognition unit 56, and a learning unit 58. The robot control unit 54, the recognition unit 56, and the learning unit 58 are connected to one another via a wireless communication network such as the Internet (although they may also be connected by wire, or may be built on a single shared server).

[0043] In FIG. 7, the robot control unit 54, the recognition unit 56, and the learning unit 58 are illustrated separately, but some or all of these may form a common functional unit (control unit).

[0044] The robot control unit 54 includes, for example, an overall control node 60 that controls the entire system, an information recording node 62 that receives instructions to record data from the overall control node 60, a robot hand control node 64 that receives control commands for the robot hand 8 from the overall control node 60, a robot arm control node 66 that receives control commands for the robot arm 12 from the overall control node 60, an image recognition unit management node 68 that receives instructions for the target weight of small ingredients from the overall control node 60, and an RGBD camera control node 70 that controls the camera 10.

[0045] The overall control node 60 is connected to an input unit (not shown) and is configured to receive commands such as a target weight from the input unit. The overall control node 60 is also connected to an output unit (not shown) and is configured to be able to output (for example, display) control information of the overall control node 60 to the outside.

[0046] The robot hand control node 64 controls the robot hand 8 and the soft gripper 14, and also acquires information on their control states, control positions, and the like.

[0047] The robot arm control node 66 controls the robot arm 12 and acquires information such as its control state and control position.

[0048] The RGBD camera control node 70 is configured to receive control commands for the camera 10 (RGBD camera) from the overall control node 60. The RGBD camera control node 70 also controls the camera 10 and acquires information such as images captured by the camera 10.

[0049] The image recognition unit management node 68 is configured to receive an RGBD image (for example, 100 cm x 55 cm, 1280 pixels x 720 pixels in terms of pixel count) from the RGBD camera control node 70. The image recognition unit management node 68 also has functions such as preparing image data for image processing in the image recognition node 72 (described later) and converting image coordinates. The image recognition unit management node 68 is configured to synchronize data with the overall control node 60.

[0050] The recognition unit 56 includes, for example, an image recognition node 72 that mainly performs image processing.

[0051] The image recognition node 72 is configured to receive, for example, an RGBD image and a target weight from the image recognition unit management node 68 .

[0052] The image recognition node 72 then evaluates the similarity of the image received from the image recognition unit management node 68 with a pre-selected or set sample local area, and determines at least one candidate local area (for example, 5 cm x 5 cm, 90 pixels x 90 pixels in terms of pixel count, areas that are shifted from each other in pixel units, i.e., they may partially overlap each other) from the image, and based on information linked to the sample local area, which is information related to the insertion depth of the robot hand 8 into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand 8, determines the slope and intercept of a linear function linked to at least one candidate local area, which shows the relationship between the insertion depth of the robot hand 8 into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand 8.

[0053] The image recognition node 72 is provided to synchronize data with the image recognition unit management node 68 .

[0054] In this embodiment, the image recognition node 72 evaluates the likelihood of the slope and intercept of the linear function corresponding to each candidate local region when determining them (in this embodiment, these are evaluated directly by the AI ​​learning program).

[0055] In addition, in this embodiment, the image recognition node 72 is configured to transmit, for each of a plurality of candidate local regions, its coordinates, the insertion depth relative to the target weight calculated from the slope and intercept of the determined linear function, and its likelihood to the image recognition unit management node 68.

[0056] The image recognition unit management node 68 then selects the most likely candidate local region from the received candidate local regions and transmits its coordinates and the corresponding insertion depth to the overall control node 60.

[0057] The learning unit 58 includes, for example, a machine learning node 74 that performs machine learning.

[0058] The machine learning node 74 is configured to be able to acquire all or part of the information such as image data, image coordinates, and insertion depth held by the image recognition unit management node 68 as learning data.

[0059] In addition, the machine learning node 74 is capable of acquiring all or part of the information such as the coordinates, target weight, insertion depth, etc. of the robot hand 8 held by the overall control node 60 as learning data (training data) for updating.

[0060] Furthermore, the machine learning node 74 can transmit the machine learning model data (or the updated machine learning model data if there is an update) to the image recognition node 72, so that the image recognition node 72 can use the data for image processing.

[0061] The machine learning node 74 is set up so as to transmit and receive data asynchronously with each of the image recognition unit management node 68, the image recognition node 72, and the information recording node 62.

[0062] (System 1 control operation) Next, the control operation of the system 1 in this embodiment will be described.

[0063] The robot hand system 2 uses the camera 10 to acquire an RGBD image of the location (box) where the small ingredients (green onions) are collected.

[0064] The image recognition node 72 acquires an RGBD image from the RGBD camera control node 70 via the image recognition unit management node 68 .

[0065] Next, the image recognition node 72 of this embodiment receives machine learning model data from the machine learning node 74, evaluates the similarity (correlation) with images of multiple sample local regions (sample local regions (teacher data)) that have been set (stored) in advance, and determines at least one candidate local region from the image.The image recognition node 72 also calculates the "slope and intercept of a linear function" that directly corresponds to (is linked to) "each candidate local region" using the machine learning model data (and AI program) constructed by using pairs (sets) of the "insertion depth" and "grasping weight" of the robot hand that are actually measured for each of the multiple sample local regions as teacher data, and also calculates the likelihood thereof. (The "plurality of sample local regions" themselves may be set based on an NN (neural network) model constructed by machine learning (deep learning) using training data. The machine learning node 74 may store raw data (teacher data) of pairs (sets) of the "insertion depth" and "grasp weight" of the robot hand actually measured for each of the plurality of sample local regions, but it is sufficient for it to store only the machine learning model data constructed from the teacher data.)

[0066] Next, the image recognition node 72 transmits to the image recognition unit management node 68, for each of the multiple candidate local regions, its coordinates, the insertion depth relative to the target weight calculated from the slope and intercept of the determined linear function, and its likelihood.

[0067] Next, the image recognition unit management node 68 selects the most likely candidate local region from the received candidate local regions, and transmits its coordinates and the corresponding insertion depth to the overall control node 60 .

[0068] The control information output from the image recognition node 72 (e.g., coordinates of the determined candidate local region, information on the corresponding linear function, etc.) may be output to an output unit such as a monitor, either via or without the overall control node 60.

[0069] Next, the overall control node 60 lowers the soft gripper 14 of the robot hand 8, and causes it to perform an operation of gripping a green onion (a small food ingredient with an irregular shape).

[0070] During the operation of gripping the green onion, the tip 20t of each scoop element 20 of the soft gripper 14 of the robot hand 8 is inserted to a predetermined insertion depth D (see FIG. 10). The insertion depth D is, for example, the insertion depth from the surface (average height surface) of the green onion in the candidate local region.

[0071] After the tip 20t of the soft gripper 14 is inserted into the green onion, compressed air is supplied to the air supply path 52 of the compressed air inlet 51 of each scoop element 20 by a compressed air supply device (not shown). This causes each air chamber 41 to 49 to expand and deform by the compressed air, and in particular, the expansion of the lateral air chambers 41 to 44 causes each scoop element 20 to bend and deform, bringing the tip portions 20t of each scoop element 20 closer to each other. This allows even small, irregularly shaped ingredients such as green onions to be gripped (scooped up) without spilling and transported to a desired container (or other desired destination).

[0072] (Machine Learning) Next, a method for determining (proposing) candidate local regions using AI machine learning in the image recognition node 72 of the system 1 will be described in more detail.

[0073] (Machine learning: Relational equation between grasping motion and grasping weight) First, the relational expression between the gripping action and the gripped weight will be explained.

[0074] Let a be the parameter representing the movement of the robot hand when grasping a small food ingredient, and let w be the weight grasped as a result of that movement.

[0075] Furthermore, if the local conditions of the small food ingredients to be grasped are the same or similar, it is likely that the grasping actions also have the same or similar relationship (at least some correlation), and therefore the relationship between a and w can be expressed as follows: TIFF0007811524000001.tif6150 formula (1)

[0076] Here, I localis a variable that represents the local situation of small ingredients obtained from the RGBD image, and θ(I local ) is a function value (parameter) determined based on the local situation, and f is a function that takes the function value as one of its inputs.

[0077] In actual grasping operations, there are many factors that may affect the robot hand, such as subtle deformations of the robot hand, errors in the robot hand's operation itself, and the internal state of the food. local Since factors not expressed as a or a are also thought to affect the grip weight w, we group these together into the random variable ∈ It is expressed as:

[0078] From the RGBD image of the entire container at the collection site, θ(I local ), it is possible to determine the relationship between the grasping action a and the grasping weight w for that local area based on the above formula (1), and it is then possible to reverse-calculate the action a of the robot hand (e.g., the insertion depth of the robot hand) to achieve, for example, a target weight w in that local area.

[0079] Taking advantage of this, in this embodiment, machine learning is used to calculate θ(I local ) is estimated.

[0080] θ(I local ) should be consistent with the actual pair of grasping motion a and grasping weight w. ∈ is a probability distribution p( ∈ |I local ), we consider the likelihood as shown in the following equation (2). TIFF0007811524000002.tif6150 formula (2)

[0081] Then, based on the concept of maximum likelihood estimation, we determine the parameter θ(I local ) to train a machine learning model on the data.

[0082] (Machine learning: local loss functions) Next, the loss function for the local region will be described.

[0083] In the learning model of this embodiment, the target is grasping a green onion with a robot hand, and the only operating parameter of the robot hand when grasping a green onion in a certain local area is the insertion depth of the robot hand.

[0084] Regarding the correlation between insertion depth and grip weight, there is a tendency that the deeper the insertion depth, the greater the grip weight, so in the present invention, this is approximated by a linear function relationship (which is one of the main features of the present invention).

[0085] Based on this approximation, the two parameters θ0(I local ), θ1(I local ), the above equation (1) can be expressed as the following equation (3). The slope of such a linear function (θ1(I local )) and intercept (θ0(I local )+ ∈ ) is identified, then equation (3) is identified. TIFF0007811524000003.tif6150 formula (3)

[0086] On the other hand, in the calculation of the likelihood of Equation (2), ∈ In this embodiment, the mean is 0, but the standard deviation is different for each local region. local ) is assumed. According to this, Equation (2) becomes Equation (4) below. TIFF0007811524000004.tif11150 formula (4)

[0087] To maximize the likelihood calculated by equation (4), the loss function is the negative log-likelihood, which is minimized in training the machine learning model. That is, the loss function is expressed as equation (5) below. TIFF0007811524000005.tif11150 formula (5)

[0088] From the above, the machine learning model of this embodiment calculates θ0(I local ), θ1(I local ), σ(I local ), is estimated (output). When a pair (set) of actual grip weight w and grip operation parameter a (=insertion depth D) is given, if an estimated value can be obtained that minimizes the sum of the loss function calculated by equation (5) for each pair (set), the likelihood will be maximized and a model that well represents the relationship between the actual grip weight w and grip operation parameter a (=insertion depth D) will be obtained. In addition, θ0(I local ), θ1(I local ) as well as σ(I local ) can also be estimated to provide an index of the accuracy of the estimation.

[0089] (Machine Learning: Model Construction) Next, the configuration of the model will be described.

[0090] It is possible to perform a grasping operation in any local area inside the stacked box of onions, but since a single robot hand only actually grasps one location in each operation, there is no need to make an estimation for the entire area.

[0091] Therefore, in this embodiment, a region proposal model used in general object recognition etc. is used. Specifically, the configuration of Faster R-CNN (Non-Patent Document 2) proposed by Ren et al. is used.

[0092] Faster R-CNN (Faster Region-based convolutional neural networks) is a deep learning approach that generates region proposals similar to R-CNN (Region-based convolutional neural networks).

[0093] Faster R-CNN is a model that can output a rectangle enclosing each object in an RGB image and an object type label for that rectangle. In other words, the model can be roughly divided into a region proposal part that proposes a rectangle enclosing an object, and an object identification part that identifies the object within the proposed region.

[0094] In the method proposed in this embodiment, the number of input channels in the first layer is changed from 3 to 4 so that an RGBD image that also includes distance information from the camera can be input.

[0095] The structure of the region proposal unit is largely unchanged, but instead of object type labels for each region, the three parameters θ0(I local ), θ1(I local ), σ(I local ) was modified to output the relational expression (3) for each rectangular region (each local region).

[0096] By adopting this model, the image recognition node 72 can determine (propose) multiple candidate local regions, and also determine (output) the formula and likelihood of the linear function for each candidate local region.

[0097] Here, an overview of the neural network in the image recognition node 72 of this embodiment will be described with reference to FIG.

[0098] In FIG. 8, CNN (Convolutional Neural Networks) 76 indicates a convolutional neural network, RPN (Region Proposal Network) 78 indicates a network for extracting candidate image regions where an object may exist from an input image, Roll Pooling 80 indicates a function for extracting partial regions, and FC layers (Fully connected layers) 82, 84, 86, and 88 indicate fully connected layers.

[0099] As the output of these neural networks, a plurality of candidate local regions are determined (proposed) in the image recognition node 72, and the formula of the linear function and the likelihood for each candidate local region are determined (output).

[0100] The object identification unit based on Faster R-CNN adopted in this embodiment uses the structure proposed in "Fast R-CNN" shown in Non-Patent Document 1, and the loss function is as shown in the following equation (6) (equation (1) in Non-Patent Document 1). TIFF0007811524000006.tif6150 formula (6)

[0101] Equation (6) is the loss function for a region of interest (RoI), where p is an estimate of the probability distribution of each object class in the region, u is the true object class number, and t u are estimates of the coordinates of the bounding box (bbox) of the object, and v represents the coordinates of the true bbox.

[0102] In equation (6) for a certain RoI, the first term represents the loss function for the probability distribution of the estimated object class, and the second term represents the loss function for the estimated bbox.

[0103] In Fast R-CNN, the first term, L cls is the estimated probability p for the true object class u. u is used and is defined as follows: TIFF0007811524000007.tif6150 formula (7)

[0104] With reference to these, in this embodiment, instead of the vector p representing the probability distribution of the object class used in equation (7), three parameters θ0(I local ), θ1(I local ), σ(I local The object recognition part of Faster R-CNN has been modified to output the following:

[0105] Furthermore, instead of equation (7), equation (5) explained in "Loss function for local regions" is used as the loss function.

[0106] In Faster R-CNN, the loss function of the region proposal section includes a term that determines whether a bbox is an object region to be detected (foreground region) or a background region that does not contain the object to be detected, and the section is trained to propose regions that contain the object to be detected using this term.

[0107] In contrast, in this embodiment, which estimates the relationship (Equation (3)) between the gripping weight of the onion and the insertion depth D of the robot hand, the gripping operation can be performed in any area inside the box in which the onions are stacked, so it can be said that there is no ``background area.''

[0108] However, to calculate the loss function (equation (5)) that evaluates the parameters of equation (3), information (measured values) on the insertion depth D and the grip weight when actually gripping at that location is required. Therefore, for areas where the training data on these is unknown, the loss function cannot be calculated.

[0109] Therefore, in the learning program of this embodiment, such areas where the training data is unknown are treated as background areas in the sense that although grasping operations can be performed, the results are unpredictable, and the loss function of equation (5) is evaluated only for areas where the training data is known.

[0110] More specifically, the local area where actual grasping was performed and where training data for insertion depth and grip weight exists is defined as the "training foreground area," and the loss function is calculated for local areas where the IoU (Intersection over Union), an index showing overlap with the training area, is 0.7 or greater as the foreground area (other areas are considered background areas).

[0111] If sufficient training data is not available, in order to increase the areas that can be treated as foreground areas as described above when training a learning model, pseudo training data can be assigned to areas where grasping motions are possible but training data is unknown (for example, approximately 25% of those areas) by selecting, with equal probability, one of the sets of all (a, w) pairs in the training data as shown in Figure 10, as shown in Figure 9.

[0112] (Machine learning: training data) Next, the teacher data will be described.

[0113] In this embodiment, the training data is a pair (set) of the "insertion depth" and "grasping weight" of the robot hand measured for each area where the grasping operation was actually performed (this becomes the sample local area).

[0114] Specifically, using the system 1, the camera 10, and the robot hand 8 (soft gripper 14), a series of operations was performed for 57 pre-set local areas: "divide the RGBD image of the target local area from the RGBD image of the food container, insert the robot hand 8 (soft gripper 14) at an insertion depth D in the local area, grasp and remove the onion, and measure the weight of the grasped and removed onion."

[0115] The onions were then refilled into the box and the same procedure was repeated.

[0116] Through the above procedure, 105 sets of four pieces of information were obtained: "RGBD image of a local area before grasping," "coordinates of the local area," "insertion depth of the robot hand inserted in the local area," and "weight of the onion grasped and removed from the local area."

[0117] Fig. 11A shows an example of an RGB image obtained by photographing the onions at the collection site with camera 10. Fig. 11B shows an example of a D image (distance image, depth image) (colored to clearly indicate depth in analysis) obtained by photographing the onions at the collection site with camera 10. In Fig. 11B, the depth values ​​are displayed in color to clearly indicate depth, with lighter gray indicating closer to the camera and darker gray indicating closer to black indicating further away.

[0118] Furthermore, the acquired RGBD images may be adjusted by translation or by changing the brightness and saturation of the RGB images to increase the number of RGBD image data points available for training. For example, 105 sets of acquired RGBD image data examples were divided into 84 examples for training data and 21 examples for evaluation data (80:20 ratio). Then, random translation within a range of ±10 pixels in both the vertical and horizontal directions and random brightness changes within a range of ±20% (with a 50% probability of no brightness change) were applied to each set (example) of measured data (RGBD image information, etc.). Therefore, 84 × 30 = 2520 examples (2520 sets) of training data and 21 × 30 = 630 examples (630 sets) of training data were obtained.

[0119] In this manner, in this embodiment, 2520 sets of teacher data as training data and 630 sets of evaluation data were finally prepared.

[0120] (Machine Learning: Verification) Using the training data set described above, the neural network model described in "Model Construction" was trained. The loss function was minimized with a learning rate of 1.0 × 10 -6 Stochastic gradient descent (SGD) was used. At each stage of training, the value of the loss function for the evaluation dataset was calculated, and training was terminated at the 39th epoch when it was minimized. The following verification was then performed using the model at this point.

[0121] Using this model, estimation is performed on the RGBD image included in the evaluation data, and multiple candidate local regions and the parameter θ0(I local ), θ1(I local ), σ(I local ) was obtained.

[0122] 12 shows an example of candidate local regions proposed for the evaluation data. On the image of a green onion, multiple virtual square frames (square regions) are displayed as proposed regions.

[0123] Among these candidate local regions, we evaluated those that overlap with regions where the actual insertion depth and grip weight are already known, and whose IoU is 0.7 or higher. The results are shown in Figure 13.

[0124] In Fig. 13, the horizontal axis represents the actual grip weight, and the vertical axis represents the actual insertion depth and the estimated θ0 (I local ), θ1(I local ), σ(I local ) is the estimated grip weight obtained by substituting

[0125] In Figure 13, we can see that estimates close to the actual measured values ​​are obtained not only for the teacher data (training data) used in machine learning training, but also for the evaluation data not used in machine learning training.

[0126] Furthermore, the root mean square error (RMSE) for the deviation between each measured value and the estimated value of the training data was 0.84 [g], and the root mean square error for the deviation between each measured value and the estimated value of the evaluation data was 1.38 [g]. On the other hand, when all of the training data (see FIG. 10) was estimated collectively using a model that performed least-squares approximation using a linear function of the grip weight and insertion depth, the root mean square error (RMSE) for the deviation between the measured value and the estimated value for each data was 2.02 [g]. This verified that the learning model of this embodiment improved estimation accuracy.

[0127] (Operation and effect of the first embodiment) According to the system 1 of this embodiment, a candidate local area can be determined from an image of the food container in which the green onions (small food ingredients) are accumulated, and a linear function (approximation) between the insertion depth of the robot hand 8 in the candidate local area and the weight of the green onions being grasped can be estimated (determined) with high accuracy. This makes it easier to respond to changes in the weight of the green onions to be removed.

[0128] In particular, according to system 1 of this embodiment, the image recognition node 72 receives machine learning model data from the machine learning node 74, evaluates the similarity (correlation) with multiple sample local regions set (stored) in advance, and determines at least one candidate local region from all local regions. Furthermore, the image recognition node 72 calculates the "slope and intercept of a linear function" directly corresponding to (linked to) each candidate local region using the machine learning model data (and AI program) constructed by using pairs of the "insertion depth" and "grasp weight" of the robot hand actually measured for each of the multiple sample local regions as training data. This eliminates the need for the system user to be aware of or consider the specifics of the similarity (correlation) between the sample local region and the candidate local region.

[0129] Furthermore, according to system 1 of this embodiment, by using a loss function based on the relationship between the gripping action (insertion depth) and the gripping weight to learn the dependency on local regions within a deep learning framework, it is possible to determine candidate local regions and output (estimate) the "slope and intercept of a linear function" corresponding to each candidate local region, resulting in higher estimation accuracy.

[0130] Furthermore, according to the system 1 of this embodiment, it is possible to select and use the candidate local region with the highest "likelihood" from among a plurality of candidate local regions, thereby enabling more accurate control of the weight of the green onions to be taken out.

[0131] (Configuration of the second embodiment) Next, FIG. 14 is a schematic perspective view of a robot hand system 102 including a system 101 according to a second embodiment of the present invention.

[0132] The system 101 of the second embodiment is configured to determine the similarity between images using conventional image processing techniques for comparing images, rather than relying on AI machine learning.

[0133] In other respects, the configuration of the robot hand system 102 is generally similar to the configuration of the robot hand system 2. Therefore, for the second embodiment, only the differences from the first embodiment will be described, and similar parts will be assigned similar reference symbols and will not be described again.

[0134] The system 101 of the second embodiment includes, for example, an overall control node 160 that controls the entire system, and an image processing node 190 that performs processing such as image analysis.

[0135] The image processing node 190 is provided in the control unit of the system 101 together with the overall control node 160. The image processing node 190 is configured to acquire an RGBD image (which may be an RGB image or a D image) (for example, 100 cm x 55 cm, 1280 pixels x 720 pixels in terms of pixel count) from the RGBD camera control node 70.

[0136] The image processing node 190 compares a local area (for example, 5 cm x 5 cm, or 90 pixels x 90 pixels) of an image acquired from the RGBD camera control node 70 (these areas are offset from each other by a pixel unit, i.e., they may partially overlap each other) with a predetermined local area image stored in advance, selects a pair of local area images that are determined to have the highest similarity as a result of the comparison, and determines a local area image from the selected image pair that is a part of the image acquired from the RGBD camera control node 70 as a candidate local area.It also determines information on the slope and intercept of a linear function that shows the relationship between the insertion depth D of the robot hand 8 into a small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand 8, which was previously linked to a local area image from the selected image pair that was previously stored, as the slope and intercept of the linear function corresponding to the candidate local area, and sends (outputs) this determined data to the overall control node 160.

[0137] The image processing node 190 does not necessarily have to be implemented separately from the overall control node 160, but may be configured as a control unit integrated with the overall control node 160 (for example, an integrated control unit built into a robot). Furthermore, the image processing node 190 may be provided on a cloud, or may be connected to the overall control node 160 via an internet line.

[0138] As a preliminary step, the image processing node 190 stores a large number of comparison image data (predetermined local area images stored in advance). The comparison image data is an image of a collection location (e.g., a food container) of small, irregularly shaped ingredients (e.g., green onions) broken down into local areas, and is accompanied by coordinate data indicating their position information.

[0139] Then, under the conditions under which each of these comparative image data was photographed, a grasping experiment is conducted in which the robot hand 8 (soft gripper 14) grasps a small food material in the corresponding local area, and actual measurements of the insertion depth and grasping weight are obtained in advance. By repeating such grasping experiments many times, many data sets (sets) are obtained, each consisting of the comparative image data, the coordinate data corresponding to the comparative image data, the insertion depth data, and the grasping weight data.

[0140] The large number of comparison image data can be classified into a plurality of image groups (e.g., 10 types of image groups) based on, for example, the degree of unevenness of the surface of the small food ingredients (the degree of depth variation in the D images). The image processing node 190 similarly classifies the data sets according to the classification of the image groups, and then, for each image group, calculates, for example, using the least squares method, the slope and intercept of a linear function that indicates the relationship between the insertion depth D of the robot hand 8 into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand 8, and associates the slope and intercept with the image group.

[0141] (Operation and effect of the second embodiment) In substantially the same manner as in the first embodiment, the robot hand system 2 uses the camera 10 to acquire an RGBD image of the accumulation location (food container) of the small ingredients (green onions).

[0142] The image processing node 190 acquires an RGBD image from the RGBD camera control node 70. The image processing node 190 decomposes the RGBD image into local regions.

[0143] Next, the image processing node 190 of this embodiment compares each of the local region images, which are part of the RGBD image, with each of the comparison image data (predetermined local region images stored in advance), and selects a pair of local region images that can be determined to have the highest similarity (for example, similarity in the distribution of depth variations in the D image, etc.) as a result of the comparison.

[0144] Next, the image processing node 190 of this embodiment determines a local area image that is part of the image acquired from the RGBD camera node 70 from the selected image set as a candidate local area, and determines information on the slope and intercept of a linear function that has been pre-linked to comparison image data (pre-stored local area image) from the selected image set as the slope and intercept of the linear function corresponding to the candidate local area, and transmits (outputs) this determined data to the overall control node 160.

[0145] The control information output from the image processing node 190 (e.g., coordinates of the determined candidate local region, information on the corresponding linear function, etc.) may be output to an output unit such as a monitor, either via or without the overall control node 160.

[0146] Thereafter, for example, the overall control node 160 determines the insertion depth of the robot hand 8 based on the linear function and the target weight. Then, the overall control node 160 lowers the soft gripper 14 of the robot hand 8 to the determined insertion depth, and performs the operation of gripping the green onion.

[0147] (Other variations) As a further modification, the recognition unit 56 may be provided as an edge computing device in a terminal separate from the robot control unit or in a server provided in a relatively close proximity.

[0148] As yet another modified example, the recognition unit 56 may be provided on the cloud. In this case, the recognition unit 56 may be connected to multiple robot control units via an internet line, and a common recognition unit may provide the image recognition function to the multiple robot control units.

[0149] As a further modification, the learning unit 58 may be provided on the cloud and connected to the robot control unit 54 and the recognition unit 56 via an internet line. [Explanation of symbols]

[0150] 1 System 2. Robot Hand System 4 Small food items 6 Collection point 8. Robot Hand 10 Camera 12 Robotic Arm 13 Robot arm mounting part 14 Soft Gripper 16 Base material 16a Cross-shaped member 16b Quarter-arc plate 17 Posts 20 Scoop Elements 20a Scoop element outer part 20b Scoop element inner part 20c Vertical compressed air supply channel 20d Horizontal compressed air supply channel 20t tip 31~39 Protruding elements 41~44 Side air chamber 45~49 air chamber 51 Compressed air introduction section 52 Air supply line 54 Robot control unit 56 Recognition part 58 Learning Department 60 Overall Control Node 62 Information Recording Node 64 Robot Hand Control Node 66 Robot Arm Control Node 68 Image Recognition Unit Management Node 70 RGBD camera control nodes 72 Image Recognition Node 74 machine learning nodes 101 System 102 Robot Hand System 105 Actual measurement 160 Overall Control Node 190 Image Processing Nodes

Claims

1. A system that outputs control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients, Acquire an image of the accumulation location of the small food ingredients; determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand, based on information associated with the sample local region that indicates the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and removed by the robot hand; The slope and intercept of the linear function associated with each candidate local region are associated with the coordinate data of each candidate local region and output as control information. A system characterized by:

2. The image is a D image.

2. The system of claim 1.

3. The image is an RGB image.

2. The system of claim 1.

4. The image is an RGBD image.

2. The system of claim 1.

5. The determination of the at least one candidate local region and the determination of the slope and intercept of the linear function associated with each candidate local region are performed using a model constructed by machine learning using AI.

2. The system of claim 1.

6. The model is constructed by using, as training data, a plurality of pairs of actual measurements of the insertion depth of the robot hand into the small food ingredient and actual measurements of the weight of the small food ingredient grasped and removed by the robot hand at that time.

6. The system of claim 5.

7. A system that outputs control information for controlling a robot hand that can change the insertion depth into a collection location for small, irregularly shaped ingredients, Acquire an image of the accumulation location of the small food ingredients; determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, and their likelihoods, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand; The slope and intercept of the linear function associated with each candidate local region and their likelihood are linked to the coordinate data of each candidate local region and output as control information. A system characterized by:

8. When there are a plurality of candidate local regions, the candidate local region with the highest likelihood is determined as the final candidate local region; The slope and intercept of the linear function associated with the final candidate local region are associated with the coordinate data of the final candidate local region and output as control information.

8. The system of claim 7.

9. The image is a D image.

8. The system of claim 7.

10. The image is an RGB image.

8. The system of claim 7.

11. The image is an RGBD image.

8. The system of claim 7.

12. A robot hand system that takes out small ingredients of irregular shape by grasping them from a collection location of the small ingredients, A system according to any one of claims 1 to 11; a camera that captures an image of the collection location of the small food ingredients and transmits the image to the system; a robot hand that is controlled based on the control information output from the system; A robot hand system comprising:

13. A method for outputting control information for controlling a robot hand capable of changing the insertion depth of small ingredients of irregular shape into a collection location, comprising: acquiring an image of the accumulation location of the small food ingredients; determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand; a step of linking the slope and intercept of the linear function associated with each candidate local region to coordinate data of each candidate local region and outputting the result as control information; A method comprising:

14. A method for outputting control information for controlling a robot hand capable of changing the insertion depth of small ingredients of irregular shape into a collection location, comprising: acquiring an image of the accumulation location of the small food ingredients; determining at least one candidate local region from the image by evaluating the similarity with a sample local region, and determining the slope and intercept of a linear function associated with each candidate local region that indicates the relationship between the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand, and the likelihood of these, based on information associated with the sample local region regarding the insertion depth of the robot hand into the small food ingredient and the weight of the small food ingredient that can be grasped and taken out by the robot hand; a step of linking the slope and intercept of the linear function associated with each candidate local region and their likelihoods to the coordinate data of each candidate local region and outputting them as control information; A method comprising:

15. A program for outputting control information for controlling a robot hand that can change the insertion depth into a collection location for small ingredients of irregular shape, When the program is executed by a computer, the method according to claim 13 or 14 can be carried out. A program characterized by:

Images