Gripping device, gripping system and control method of gripping device

Abstract

To provide a gripping device that is configured to grip a food product using a robot or the like, which can grip a specified quantity of string-shape objects such as a noodle.SOLUTION: A gripping device comprises: an object recognizing part that recognizes an object that is a string-shape object; a gripping part comprising a plurality of claws, which grips the object on the basis of a control signal; a gripping learning part that learns motion of the gripping part to generate a learning parameter; and a control part that outputs the control signal, on the basis of the learning parameter. The control part outputs, to the gripping part, the control signal for making the gripping part grip the object recognized by the object recognizing part at a first place to place the object at a second place different from the first place.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a gripping device, a gripping system, and a method for controlling a gripping device. [Background technology]

[0002] Patent Document 1 describes a dispensing device that picks up and distributes a portion of food such as noodles, granules, thin plates, macaroni, etc., from a plated state. Specifically, a multi-fingered hand is controlled under arbitrary control conditions to learn the action of picking up a portion of food from the plated food. Based on the learning results, the dispensing device is controlled to pick up and distribute a portion of food from the plated food. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 7044366 Summary of the Invention [Problem to be solved by the invention]

[0004] When grabbing and distributing food from a plate, it is necessary to maintain a certain degree of precision and to dispense a fixed amount stably without inconsistency. However, when handling noodle-like foods, for example, the noodles can sometimes get tangled when being grabbed, making it difficult to dispense a fixed amount stably.

[0005] Therefore, an object of the present invention is to provide a gripping device, a gripping system, a control method for a gripping device, and a control program that can grip a predetermined amount of an object, even if it is a string-like object such as noodles, and place it in another location. [Means for solving the problem]

[0006] The gripping device of the present invention comprises an object recognition unit that recognizes an object, which is a string-like object, a gripping unit that has a plurality of claws and grips the object based on a control signal, a gripping learning unit that learns the operation of the gripping unit and generates learning parameters, and a control unit that outputs a control signal based on the learning parameters, and the control unit outputs a control signal to the gripping unit to grip the object recognized by the object recognition unit at a first location and place it at a second location different from the first location.

[0007] The gripping system of the present invention includes a gripping device, a simulator, and an image display device. The gripping device includes an object recognition unit that recognizes an object, which is a string-like object, a gripping unit that has multiple claws and grips the object based on a control signal, a gripping learning unit that learns the operation of the gripping unit and generates learning parameters, and a control unit that outputs a control signal based on the learning parameters, and the control unit outputs a control signal to the gripping unit to grip the object recognized by the object recognition unit at a first location and place it at a second location different from the first location. The simulator generates a 3D model of the gripping unit. The image display device includes a camera, and displays an image captured by the camera by combining the 3D model of the gripping unit generated by the simulator.

[0008] The gripping device control method of the present invention is a method for controlling a gripping device that grips an object which is a string-like object, and includes the steps of: capturing an image of the object; predicting the weight of the gripped object from the captured image of the object; calculating the weight of the gripped object by measuring the weight of the object before and after it is gripped; performing learning based on the predicted weight of the object and the calculated weight of the object to generate learning parameters; and controlling the gripping device to grip the object at a first location and place it at a second location different from the first location based on the learning parameters. Effect of the Invention

[0009] According to the present invention, even if an object is a string-like object, a predetermined amount of the object can be stably grasped and placed in another location. [Brief description of the drawings]

[0010] [Figure 1] Schematic diagram showing the configuration of a gripping device [Diagram 2] FIG. 1 shows the configuration of a grip control unit. [Diagram 3] Flow diagram showing the operation of the grip control unit [Figure 4] Block diagram of the grip learning unit [Diagram 5] A diagram explaining the flow of learning in the grip learning unit. [Figure 6] Diagram explaining cropping process [Figure 7] Flow diagram explaining the operation of the policy net [Figure 8] FIG. 1 shows an example of a residual learning block. [Figure 9] FIG. 13 shows an example of depth block processing. [Figure 10] Flow diagram of grip information generation in the grip learning unit [Figure 11] A diagram explaining the state of grasping after learning [Figure 12] Figure 1 shows the weight and frequency of grasps obtained by the grasp motion after learning. [Figure 13] FIG. 13 is a diagram showing an example in which the gripping portion grips a string-like object; [Figure 14] FIG. 13 is a diagram showing an example in which the gripping portion grips a solid object. [Figure 15] FIG. 13 is a diagram showing an example in which the gripping portion grips a low-rebound object. [Figure 16] A close-up of the claws used to grasp soft objects [Figure 17] FIG. 13 is a diagram showing the operation of the gripping part when gripping a low-rebound object. [Figure 18] Schematic diagram of a gripping device using AR [Figure 19] Diagram showing the configuration of a grasping system using AR [Figure 20] A diagram showing the operation flow of the grasping system using AR DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] An embodiment of the present invention will be described in detail below with reference to the drawings. The configuration, processing, flow, etc. described below are examples for explanation, and can be appropriately changed according to the specifications of the gripping device, gripping system, camera, robot, object, weighing scale, grip learning device, etc. In addition, in all drawings, the same reference numerals are given to corresponding elements, and duplicated explanations are omitted. The block diagram used to explain the embodiment classifies and organizes the configuration of the device by function. The blocks showing each function are realized by any combination of hardware or software. In addition, since the block diagram shows the functions, it can also be understood as a disclosure of the invention of a method and the invention of a program that realizes the method. In addition, the order of the flow may be changed unless there is a special description or constraint.

[0012] <1. Overview of the gripping device> First, a gripping device according to an embodiment of the present invention will be described.

[0013] FIG. 1 is a schematic diagram showing the configuration of a gripping device. The gripping device 1 includes a gripping control unit 100, a camera 200, and a robot 300. The gripping control unit 100 may be incorporated as a part of the robot 300. A plurality of cameras 200 may be provided. The robot 300 includes a gripping unit 301. The gripping unit 301 is, for example, an arm or a claw of the robot 300, and performs an action of actually gripping (grabbing) an object. The robot 300 may be a scalar type and an arm type, or may be a scalar type or an arm type. In this embodiment, the gripping unit 301 is provided as a part of the robot 300, but the entire robot may be defined as a gripping unit without distinguishing between the gripping unit and the robot.

[0014] On the tray 400, string-like objects such as spaghetti or low-resilience foods such as peaches are placed. In this embodiment, string-like objects such as noodles are used as the object to be grasped. In FIG. 1, noodles 500 are placed on the tray 400. The noodles 500 are, for example, noodle foods such as spaghetti, udon, or soba. They may also be various types of pasta such as macaroni. The tray 400 is placed on a weighing scale 510. The weight of the noodles remaining on the tray 400 is measured by the weighing scale 510.

[0015] In the gripping device configured as described above, the camera 200 captures an image of the vicinity of the tray 400. The gripping control unit 100 can estimate information about the gripped object, such as the weight and position of the gripped noodles, from the image captured by the camera 200. In addition, information from the weight meter 510 can also be used to estimate the weight of the gripped noodles.

[0016] The gripping learning unit 700 learns to grip a certain amount of noodles. Specifically, learning is performed using a deep reinforcement learning technique. A stable and highly accurate gripping operation is achieved by repeating the action of picking up a certain amount of noodles 500 from the tray 400 and learning using deep reinforcement learning. Details of the learning will be described later. Note that a gripping system that picks up a plurality of objects may be realized by combining a plurality of gripping devices 1.

[0017] <2. Operation of grip control unit> Next, the grip control section 100 will be described with reference to FIGS.

[0018] Fig. 2 is a diagram showing the configuration of the grip control unit in this embodiment. In Fig. 2, the grip control unit 100 includes a control unit 110, a storage unit 120, and an input / output unit 130. The control unit 110 includes an object recognition unit 111, a robot control unit 112, and a weight estimation unit 113. The storage unit 120 stores grip information 121. The input / output unit 130 includes an input unit 131 and an output unit 132.

[0019] The control unit 110 controls the robot 300. The control unit 110 may be a device that executes a program stored in a memory or the like and realizes each functional block. Examples of hardware that executes the program include a central processing unit (CPU), a micro processing unit (MPU), a controller, a large scale integration (LSI), an application specific integrated circuit (ASIC), a programmable logic device (PLD), and a field-programmable gate array (FPGA). The above-mentioned memory may be an independent memory or may be a part of the storage unit 120 (the same applies to the control unit 710 described later). The memory and storage unit 120 in which the program is stored are composed of, for example, a read-only memory (ROM) and a random access memory (RAM) (the same applies to the storage unit 720 described later).

[0020] The object recognition unit 111 recognizes the location, position, state, etc. of the object using the camera image output from the camera 200. The position refers to the position in the height direction, depth direction, and horizontal direction, and the object recognition unit 111 can measure the position of the object. The state refers to information such as whether the object is placed flat or in a mountain shape, and at which point in the tray 400 the object is placed in large numbers (thickly). The object to be grasped can be various foods such as fruits, vegetables, noodles, and other objects, but in this embodiment, noodles 500, which are string-like objects, are treated as the object.

[0021] The weight estimation unit 113 estimates the weight of the noodles 500 on the tray 400 based on the value measured by the weight meter 510.

[0022] The robot control unit 112 outputs a control signal based on information from the object recognition unit 111 and the weight estimation unit 113. Based on this control signal, the gripping unit 301 performs an operation to grip a predetermined amount of noodles 500 on the tray 400. At this time, the robot control unit 112 controls the gripping unit 301 based on gripping information 121 that has been learned in advance to grip a predetermined amount.

[0023] By performing optimized control through learning, when the gripping unit 301 grips the noodle 500, the gripping action is performed in a shallow range within a predetermined distance from the upper surface of the noodle 500. If the gripping action is performed in a range deeper than the predetermined distance from the upper surface, the noodle may become tangled and it may not be possible to grip the specified amount well. This gripping information 121 will be described later.

[0024] Next, the robot control unit 112 controls the robot to place the grasped noodles 500 from a predetermined location (first location) to another location (second location). In this embodiment, the first location refers to the tray 400. The second location refers to a location where a predetermined amount of noodles such as spaghetti or udon or other pasta should be placed, such as a predetermined area in a lunch box or dish.

[0025] Furthermore, the robot control unit 112 may predict or measure the weight of the object being gripped at that time based on the image captured by the camera 200, the measured value of the weight scale 510, etc., and change the grip strength accordingly. Changing the grip strength means strengthening or weakening the strength with which the object is gripped by the claws.

[0026] The memory unit 120 stores grip information 121. This grip information 121 is output by learning of the grip learning unit 700 described later. The grip information 121 is information on how the grip unit 301 grips an object. For example, the grip information 121 is information indicating how the grip unit 301 is to be moved and what action is to be performed at what position depending on the position and weight of the object and the shape of the claws of the grip unit 301. The grip information 121 also changes depending on the shape of the object to be gripped and the grip unit 301. Specific examples will be described later.

[0027] The input unit 131 receives (inputs) images captured by the camera 200, information transmitted from the robot, etc. The output unit 132 transmits (outputs) various information such as information required for controlling the camera 200 and the robot 300.

[0028] Fig. 3 is a flow diagram showing the operation of the gripping control unit. Here, the flow of the gripping operation will be described. As shown in Fig. 3, when the gripping operation starts, the position, state, weight, etc. of the target object are recognized from the image captured by camera 200, weight scale 510, etc. (S1). Then, based on gripping information 121 generated by learning, gripping unit 301 and robot 300 are controlled to grip the target object and place it in another location.

[0029] 3. Configuration of the Grasping Learning Unit Next, the grip learning unit 700 will be described with reference to Fig. 4 to Fig. 12. The grip learning unit 700 performs machine learning, particularly reinforcement learning, regarding gripping.

[0030] Fig. 4 is a block diagram of a grip learning unit that calculates grip information. In Fig. 4, the grip learning unit 700 includes a control unit 710, a storage unit 720, and an input / output unit 730. The control unit 710 includes a weight prediction unit 711, a weight calculation unit 712, a reward calculation unit 713, a buffer 714, a cropping unit 715, and an agent unit 716. The storage unit 720 stores a cropped image 721 generated by the cropping unit 715 and grip information 722 generated by the grip learning unit 700. The input / output unit 730 includes an input unit 731 and an output unit 732.

[0031] 5 is a diagram illustrating the flow of learning in the grip learning unit. The following description will be given with reference to FIGS.

[0032] When learning starts, a gripping operation is performed by gripping unit 301, and a part of the object is gripped by gripping unit 301. If the object is noodles 500, at this point in time, the gripped noodles 500 have not completely separated from the noodles 500 on tray 400. Weight prediction unit 711 predicts the weight of the gripped object using an image captured by camera 200, weight scale 510, etc. Then, weight prediction unit 711 compares the weight of the gripped object with a target value (threshold value), and if it is desired to increase the gripping weight, it strengthens the grip of gripping unit 301. On the other hand, if it is desired to decrease the gripping weight, it weakens the grip of gripping unit 301.

[0033] Weight calculation section 712 calculates the weight of the actual grasped object using the image captured by camera 200, weight scale 510, and the like.

[0034] The reward calculation unit 713 calculates the reward of the reinforcement learning performed by the grip learning unit 700. Reinforcement learning is a learning method in which a machine learns to take good actions by itself through trial and error. In reinforcement learning, the agent unit 716 learns "good actions" to take in each situation. In order to perform this learning, a definition of "good actions" is required. Therefore, in order to mathematically handle "goodness", a reward is defined. In reinforcement learning, the agent gets this reward every time it takes action in each situation. For example, if the grip unit 301 performs a good grip, it can get a positive reward, and if it performs a bad grip, it will be given a negative reward. Here, if a predetermined amount is gripped, it is judged to be a "good action" and a high reward is given. The reward calculation unit 713 calculates this reward every time the grip unit 301 grips.

[0035] The buffer 714 stores the transition amount for each operation. The transition amount includes information such as state, action, and reward. This transition amount represents the amount of operation of the gripping unit 301, that is, the movement path. The transition amount is accumulated and stored in the buffer every time the gripping unit 301 performs gripping. That is, the buffer 714 stores not only the most recent state but also past states. Therefore, this information can be used for learning. For example, data can be randomly extracted from the data accumulated in the buffer 714, and learning can be performed based on this data. This allows learning not only in a time-series environment, but also in a time-discrete environment.

[0036] The cropping unit 715 performs cropping processing on the image captured by the camera 200. Cropping processing refers to image conversion in which a partial image of only the target area is cut out from the image captured by the camera 200, which is the original digital image, and an image is created from which unnecessary peripheral parts have been removed. This is a so-called image extraction processing, and the cropping unit may be expressed as an image extraction unit.

[0037] FIG. 6 is a diagram for explaining the cropping process in the grip learning unit 700. In FIG. 6, 600 is an RGB image captured by the camera 200, and 601 is a depth image. A depth image is an image that represents the depth of an object or surface in space. If the camera 200 is an RGBD camera, the depth image is output from the camera 200. If the camera 200 is an RGB camera, the depth image may be generated based on RGB images obtained from multiple cameras. 602 is an RGB domain image in which only the noodle region is extracted from the RGB image 600. 603 is a depth domain image in which only the noodle region is extracted from the depth image 601. Then, a cropped image 604 is generated from the RGB domain image 602 and the depth domain image 603. By generating an image of the object from the RGB image and the depth image, the state of the noodle 500 can be grasped in detail, and the accuracy of the gripping amount can be improved.

[0038] Next, a description will be given of the agent unit 716. The data output by the buffer 714 and the image data output by the cropping unit 715 are input to the agent unit 716. The agent unit 716 has a learning processing unit 7161 and a policy net unit 7162.

[0039] The learning processing unit 7161 updates the learning parameters of the policy net unit 7162 based on the reward obtained by the reward calculation unit 713. The policy net unit 7162 is composed of a convolutional neural network.

[0040] The reinforcement learning performed by the learning processing unit 7161 may use a standard reinforcement learning method, but in this embodiment, a method called Soft Actor-Critic is used. Soft Actor-Critic is a method based on maximum entropy reinforcement learning. The purpose of standard reinforcement learning is to "perform an action that is expected to give the most reward", but in addition to this, this method also aims to "maximize the entropy (disorder) of the policy". This enables more diverse searches than standard reinforcement learning, and can improve the robustness of the policy net. Next, the operation of the policy net unit 7162 will be described in detail with reference to Figs. 7 to 9.

[0041] 7 is a flow diagram illustrating the operation of the policy net unit 7162. First, the RGB image processing (S11 to S16) shown in the flow on the left side will be described.

[0042] RGB data is extracted from an image of noodles 500 captured by camera 200 (S11). Then, a convolution operation is performed on the RGB data (S12). Convolution in an image refers to a process in which each pixel in the image is multiplied by a specific coefficient and the neighboring pixels (3x3 or 5x5, etc.) are summed up. Convolution operation is a method often used in image processing, and is a process in which lattice-like numerical data called a kernel (or filter) and numerical data of a partial image (called a window) of the same size as the kernel are converted into a single numerical value by calculating the sum of products for each element. This conversion process can be performed by shifting the window little by little to convert it into small lattice-like numerical data (i.e., tensors).

[0043] Next, Max Pooling is performed (S13). Max Pooling is a process of selecting the maximum value within a given region. Pooling, also known as downsampling, reduces the number of dimensions, which can reduce the number of parameters input to the neural network.

[0044] Next, the process of the residual learning block will be explained (S14). Here, a residual learning model, that is, a model that combines a convolution layer and shortcut connections, is used. The configuration is shown in Figure 8.

[0045] FIG. 8 is a diagram showing an example of a residual learning block. This is a configuration in which a shortcut connection in which input values ​​are added as is is added to a general convolutional neural network (CNN). In the figure, SN stands for Spectral Normalization, which is an effective method in reinforcement learning tasks. ELU stands for Exponential Linear Unit, and is a function in which when the input value to the function is 0 or less, the output value is a value between "0.0" and "-α" (the value of α is basically 1.0), and when the input value is greater than 0, the output value is the same as the input value.

[0046] There are shortcut connections in the learning process, and learning (weight update) is performed by learning the difference between what has been shortcut and what has passed through the residual block, called the "residual". As a result, a model can be constructed that is deeper than previous models, and high accuracy can be achieved. Finally, processing is performed by a multilayer perceptron (MLP) (S15).

[0047] Next, the depth image processing (S21 to S22) shown in the flow on the right side of Fig. 7 will be described. First, depth data is extracted or generated from an image of noodles 500 captured by camera 200 (S21). Then, depth block processing is performed (S22). An example of depth block processing is shown in Fig. 9.

[0048] FIG. 9 is a diagram showing an example of depth block processing. Usually, the depth image output from an RGBD camera often has low accuracy. However, in order to improve the accuracy of learning and the accuracy of operation, it is necessary to grasp the state of the noodles 500 in detail. Therefore, in order to increase the accuracy of the depth image, depth block processing is performed. The accuracy of the depth image can be increased by the depth block processing shown in FIG. 9. The following processing from S23 to S26 is similar to the processing from S12 to S15 described above, and therefore description thereof will be omitted.

[0049] As a result of the above, the learning parameters of the policy net unit 7162 are determined and stored as the grasping information 722. This grasping information 722 is sent to the storage unit 120 of the grasping control unit 100 and held as the grasping information 121.

[0050] <4. Grasping learning flow> Next, a flow of object grip learning will be described with reference to FIG.

[0051] FIG. 10 is a flow diagram of grip information generation in the grip learning unit. First, the gripping unit 301 is caused to grip at least a part of the noodles 500 based on the specified parameters (S31). When the object is noodles 500, the gripped noodles 500 are not completely separated from the noodles 500 on the tray 400 at this point. The weight prediction unit 711 described in FIG. 4 and FIG. 5 predicts the weight of the gripped object using the image captured by the camera 200, information from the weighing scale 510, etc. (S32). Then, the weight of the gripped object is compared with a threshold value (target value) (S33). When the weight of the gripped object is greater than the threshold value, the grip of the gripping unit 301 is weakened (S34). On the other hand, when the weight of the gripped object is less than the threshold value, the grip of the gripping unit 301 is strengthened (S35). Then, using the image captured by camera 200, weight scale 510, or the like, the weight of the grasped object after the grip is tightened or loosened is calculated (S36).

[0052] After the weight calculation (S36), cropping (S37) and reward calculation (S38) are performed. In the cropping process (S37), the image captured by the camera 200 is cropped. In the reward calculation (S38), a reward in reinforcement learning is calculated. Each time the gripping unit 301 grips an object, the reward is calculated.

[0053] Then, learning is performed (S39) using the results of cropping (S37) and reward calculation (S38). Learning corresponds to the processing of the agent unit 716. If learning is sufficiently performed and completed (S40: YES), learning is completed. If learning is not yet sufficient (S40: NO), the process returns to S31 and is repeated.

[0054] FIG. 11 is a diagram for explaining the state of gripping after learning. FIG. 11(a) shows the state of gripping before learning, and FIG. 11(b) shows the state of gripping when the result of learning is reflected. In this embodiment, spaghetti with a diameter of about 2 mm was used as an example of the noodle 500. Learning was performed with the goal of gripping 50 g of the noodle 500. When observing the movement before learning, it was found that the noodles frequently became tangled when the gripping movement was performed at a depth of 3 to 5 mm or more from the upper surface of the noodle 500. In other words, the noodle was trying to grip a layer deeper than 3 mm from the surface of the spaghetti, which is the target object. FIG. 11(a) shows the state of tangled noodles. When the noodles become tangled, it is difficult to grip noodles of the desired weight.

[0055] On the other hand, when observing the behavior after learning, it was found that the gripping unit 301 often gripped the noodle 500 at a depth of about 3 mm or less from the upper surface. In other words, it was trying to grip a layer of the target object, the pile of spaghetti, that is, 3 mm or less from the surface. However, the surface of the target object, the pile of spaghetti, may have an uneven shape such as an irregular shape. In this case, the gripping control unit 100 outputs a control signal so that the gripping unit 301 performs a gripping operation that conforms to the surface shape of the spaghetti. In other words, by moving the gripping unit 301 up and down in accordance with the surface shape of the noodle 500, it is possible to grip a layer of the spaghetti that is within 3 mm from the surface.

[0056] Furthermore, when gripping the noodles, the gripping unit 301 changes its claws from an open state to a closed state, and so tries to grip a layer of the spaghetti that is deeper than 3 mm from the surface. Therefore, the gripping control unit 100 outputs a control signal to raise the gripping unit 301 in response to the opening and closing of the claws. Specifically, when gripping, the gripping unit 301 itself is gradually raised while changing the claws from an open state to a closed state. A control is performed to realize a so-called tracing operation. That is, the gripping control unit 100 outputs a control signal to move the gripping unit 301 upward while operating the claws from an open state to a closed state. As a result, it became possible to grip the noodles at a depth of about 3 mm or less, and it was found that the frequency of the noodles getting tangled was reduced.

[0057] FIG. 12 is a diagram showing the weight and frequency of gripping operations after learning. As described above, the gripping amount was set to 50 g and learning was performed. After learning, the operation was performed multiple times and the weight actually gripped was measured. As a result, it was found that the number of times was the highest for a weight of about 50 g, and gripping was performed well. In this way, gripping was performed well when the gripping operation was performed within a specified distance from the upper surface of the object. This is a method suitable for gripping string-like objects that tend to get tangled. Even in the case of string-like objects such as noodles 500 that tend to get tangled and whose amount is difficult to adjust, it was found that the frequency approaches a normal distribution when the gripping operation is performed within a specified distance from the upper surface of the object. This means that it is close to the case of gripping by human hand.

[0058] <5.Gripping part> Next, the shape of the grip portion 301 will be described with reference to FIGS.

[0059] Fig. 13 is a diagram showing an example in which the gripping unit grips a string-like object. The object is a string-like object, namely, noodles. From the left, the states before gripping, during gripping, and after gripping are shown. The gripping unit 301 has relatively long claws. The number of claws is three. However, the present invention is not limited to this, and the number of claws may be three or more.

[0060] FIG. 14 is a diagram showing an example of the gripping unit gripping a solid object. The object used is fried chicken, which is a rounded solid object compared to a string-like object. From the left, the states before gripping, during gripping, and after gripping are shown. In this way, when the object to be gripped is an object with a relatively large width, the claws of the gripping unit 301 may be relatively short. Also, the number of claws is six. However, this is not limited to this, and a different number of claws may be used.

[0061] FIG. 15 is a diagram showing an example in which the gripping portion grips a low-resilience object. The object is a tomato. It is a food item that is relatively large, soft, and easily damaged, and must be gripped gently. From the left, the states before gripping, during gripping, and after gripping are shown. In this case, the claws of the gripping portion 301 are soft, elastic, and easily deformed. The number of claws is three. However, this is not limited to this, and the number of claws may be three or more.

[0062] FIG. 16 is an enlarged view of a claw used to grasp a soft object. FIG. 16(a) is a photograph of a claw having a shape similar to that of the claw used in FIG. 15, and FIG. 16(b) is a schematic diagram of the shape of the claw. In this embodiment, an isosceles triangular claw with a base of about 3 cm and a height of about 7 cm, assuming that the shortest side of the triangle is the base, is used. The apex angle of the triangle is about 24 to 25 degrees. A plurality of buffer parts are provided inside the triangle at approximately equal intervals. The buffer parts may be horizontal or oblique to the base of the triangle. The claw shown on the left side of FIG. 16(a) or FIG. 16(b) is provided with a buffer part horizontal to the base of the triangle. The claw shown on the right side of FIG. 16(a) or FIG. 16(b) has an inclination of about 20 degrees to the base of the triangle. When the angle is 0 to 20 degrees, it is easy to grasp a soft object. In addition, the angle of the apex of the triangle should be about 20 degrees, and if it is 10 to 30 degrees, it is possible to grasp soft objects.

[0063] However, the present invention is not limited to this, and the buffering portion may be a combination of a horizontal direction and an oblique direction. The thickness of the buffering portion may be configured to be substantially the same as the thickness of each side of the triangle, or may be configured to be different thicknesses. The buffering portions may be disposed at substantially equal intervals, or may be disposed at different intervals. In the present embodiment, the shape of the claw is described as an isosceles triangle, but the three sides may have different lengths. In the description, the shortest side of the three sides of the triangle is the base, but the other sides may be the base.

[0064] FIG. 17 is a diagram showing the operation of the gripping unit when gripping a low-resilience object. A marshmallow was used as the object. FIG. 17(a) shows the state before gripping. FIG. 17(b) shows the gripping state when the buffer unit uses a claw that is horizontal to the base of the triangle, and FIG. 17(c) shows the gripping state when the buffer unit uses a claw that is inclined at about 20 degrees to the base of the triangle. As shown in FIG. 17, the claw of the gripping unit 301 deforms from a triangle to the shape of the marshmallow, gently wrapping and gripping the marshmallow. This is because the rigidity of the material that constitutes the triangular claw is very low. In other words, when the inside of the claw gently touches the object, the inside of the claw, which has low rigidity, immediately deforms, and the outside of the triangle also begins to deform through the buffer unit. This deformation wraps the object as shown in FIG. 17(b) and (c). The rigidity and elasticity of the claw are such that they do not damage the objects, which are tomatoes, peaches, and marshmallows.

[0065] In this embodiment, the material used for the claws is the stereolithography elastic resin F80. Its properties are as follows: Shore hardness is 50-60A, tensile strength is 3.8MPa, tear strength is 9.75KN / m, elongation at break is 159%, viscosity (25 degrees Celsius) is 2300mpa.s, and elasticity is 1.8MPa. By using such a flexible material, which is highly elastic, it is possible to grip even soft objects.

[0066] <6. Grasping system using AR> Next, a gripping system using AR will be described with reference to FIGS.

[0067] FIG. 18 is a schematic diagram of the gripping device in combination with AR. FIG. 18(a) shows a state before the robot is installed in a factory or production site. Here, a scene 1801 of the production site photographed by a smartphone, for example, is displayed on a display unit 1800. FIG. 18(b) is a schematic diagram of the gripping device in combination with an AR system. By using the AR system, a 3D model of the robot generated by computer graphics or the like is synthesized on the smartphone. Here, a state is shown in which a virtual robot 1802 expressed by a 3D model is synthesized with a scene 1801 of a production site such as a factory.

[0068] By using such an AR system, it is possible to verify whether there are any problems with the placement position of the robot before actually placing the robot. In addition to checking the position, it is also possible to perform a simulation of the robot 300 actually moving according to the grip information learned by the grip learning unit 700. This makes it possible to prevent installation or placement errors in advance, such as the robot hitting a window glass or blocking the movement of workers when actually moving the robot.

[0069] The AR system can also be used after the robot 300 has been installed. For example, a simulation of the robot 300 performing a new action based on new grip information learned by the grip learning unit 700 is possible. In this case, too, installation or placement errors such as the robot hitting a window glass or blocking the movement of workers when actually moving the robot can be prevented in advance. Note that, although the present embodiment has been described using a smartphone, any image display device capable of displaying images may be used.

[0070] 19 is a diagram showing the configuration of a gripping system using AR. When AR is used, the gripping system further includes a smartphone 800. The grip control unit 100 includes a control unit 110 that controls the robot 300, a robot photographing camera 140, and a robot simulator 150. The smartphone 800 includes an operation unit / sensor 810, a background photographing camera 820, a synthesis unit 830, and a display unit 840. The smartphone 800 may be any other electronic device, such as a tablet or a personal computer, as long as it is capable of displaying images.

[0071] When an instruction to start AR processing is received from the smartphone 800, the control unit 110 starts controlling the robot 300. The robot photographing camera 140 captures an image of the robot 300 and generates image data (including still images and videos). An actual image of the robot 300 is output.

[0072] On the other hand, the robot simulator 150 generates a 3D model of the entire robot 300 from image data of the robot 300, 3D data of the robot 300, and operation information of the robot 300. Alternatively, a 3D model of the gripping part 301 may be generated. Furthermore, the robot simulator 150 can cause this 3D model to perform various operations in a virtual space.

[0073] The smartphone 800 is provided with an operation unit / sensor 810. Various operation instructions input by the user are transmitted to the grip control unit 100 via the operation unit / sensor 810. The operation unit / sensor 810 also detects the position, angle, etc. of the smartphone 800, and transmits this information to the grip control unit 100.

[0074] The background photographing camera 820 mounted on the smartphone 800 photographs the location where the robot 300 is to be placed. For example, the location is a factory or a workshop where the robot 300 is to be placed. The photographed image is displayed on the display unit 840, for example, as shown in FIG. 18(a). The composition unit 830 generates an image by combining the background image photographed by the background photographing camera 820 and the 3D model of the robot 300 generated by the robot simulator 150. The composite image is then displayed on the display unit 840. This composite image is, for example, an image as shown in FIG. 18(b). Since the 3D model of the robot 300 is generated by the robot simulator 150, it can be freely moved in a virtual space based on an operation instruction from the user. This allows various simulations to be performed on the display unit 840 before the robot 300 is actually placed.

[0075] In the above description, the synthesis unit 830 synthesizes a 3D model generated by the robot simulator 150, but image data of the actual robot 300 captured by the robot photographing camera 140 may be synthesized. When synthesizing image data of the actual robot 300, a more realistic image can be synthesized by changing the shooting angle of the robot photographing camera 140 based on the position and angle information of the display unit 840. The synthesis unit 830 may be implemented as an application that runs on the smartphone 800. The display unit 840 may be a general external display.

[0076] 20 is a diagram showing an operation flow of the gripping system using AR. First, the robot 300 is imaged and image data (including still images and videos) is generated (S51). Then, a 3D model of the robot 300 is generated from the image data of the robot 300, the 3D data of the robot 300, and operation information of the robot 300 (S52). Here, the 3D model can be made to perform a desired operation based on an operation instruction from the user.

[0077] Next, the smartphone 800 uses the background photographing camera 820 to photograph the location where the robot 300 is to be placed (S53). Then, an image is generated by combining the background image photographed by the background photographing camera 820 with the 3D model of the robot 300 generated by the robot simulator 150 (S54). The composite image is then displayed (S55). Note that, although it has been described that a 3D model is synthesized in S54, an image of the actual robot 300 photographed in S51 may be synthesized.

[0078] <7. Summary> The gripping device 1 includes an object recognition unit 111 that recognizes an object, which is a string-like object, and a gripping unit 301 that has a plurality of claws and grasps the object recognized by the object recognition unit 111 at a predetermined place (first place) and places it at a different place (second place) from the predetermined place, and the gripping unit 301 grasps the object with the claws within a predetermined distance from the upper surface of the object. This makes it possible to stabilize the distribution of the amount of grasping in repeated grasping in a gripping device that grasps an object such as a string-like object at a predetermined place and places it at a different place using a robot or the like.

[0079] In addition, the gripping device 1 may use machine learning to calculate the path of the gripping operation of the gripping unit 301. This makes it possible to more accurately and stably distribute the gripped amounts in the repeated gripping operations.

[0080] The gripping device 1 may further include a camera 200 that is a positioning means for measuring the position of the object, and a robot control unit 112 that is used to move the object from the predetermined location to the different location. This makes it possible to more accurately and stably distribute the gripped amounts in repeated gripping operations.

[0081] The gripping device 1 may be at least a part of a scalar robot or an arm robot, which can more accurately and stably distribute the gripped amounts when gripping is performed repeatedly.

[0082] Also, the number of claws is preferably three or more. This allows the object to be gripped stably, and therefore the distribution of gripped amounts during repeated gripping can be stabilized with higher accuracy.

[0083] The predetermined distance is preferably 3 to 5 mm, and more preferably 3 mm or less deep from the surface. In particular, in a gripping device that grips an object such as a string-like object at a predetermined location and places it at another location, the object can be gripped stably, and the distribution of gripped amounts during repeated gripping can be stabilized with higher accuracy.

[0084] In addition, the string-like object may be a noodle. In particular, in a gripping device that grips an object such as a string-like object at a predetermined location and places it at another location, the object can be gripped stably, and the distribution of the gripped amount in repeated gripping can be stabilized with higher accuracy.

[0085] The gripping system also includes a robot simulator 150 that outputs a 3D model of the gripping device, a background photographing camera 820 that captures an image of the environment in which the gripping device is used, and a display unit 840 that displays an image obtained by combining the 3D model and the image of the environment in which the gripping device is used captured by the background photographing camera 820. This makes it possible to perform a simulation using AR of the gripping device 1, particularly the robot 300, when it is installed or positioned, before installing the gripping device 1 or the like in a factory or production site, or before changing the position or the like.

[0086] The gripping system also includes a gripping device and a grip learning unit 700 that calculates the path of the gripping operation of the gripping unit 301 included in the gripping device. This makes it possible to stabilize the distribution of gripped amounts in repeated gripping operations in a gripping device that uses a robot or the like to grip an object such as a string-like object at a predetermined location and place it at another location.

[0087] The gripping method recognizes a target object, which is a string-like object, and uses multiple claws to grip the recognized target object at a predetermined location and place it at a different location from the predetermined location, and the claws perform a gripping operation within a predetermined distance from the upper surface of the target object. This makes it possible to stabilize the distribution of gripped amounts in repeated gripping operations in a gripping method in which a robot or the like is used to grip a target object such as a string-like object at a predetermined location and place it at a different location.

[0088] The gripping program causes the computer to recognize a target object, which is a string-like object, grip the recognized target object at a predetermined location using multiple claws, place the target object at a different location from the predetermined location, and perform a gripping operation with the claws within a predetermined distance from the upper surface of the target object. This makes it possible to stabilize the distribution of gripped amounts in repeated gripping operations in a gripping program that uses a robot or the like to grip a target object such as a string-like object at a predetermined location and place it at a different location. [Explanation of symbols]

[0089] 1 Gripping device 100 Grip control section 110 Control section 111 Object Recognition Unit 112 Robot control unit 113 Weight estimation section 120 Storage section 121 Grip information 130 Input / output section 131 Input section 132 Output section 140 Robot Camera 150 Robot Simulator 1800 Display 1801 Landscape 1802 Virtual Robot 200 Cameras 300 Robot (Grasp) 301 Gripping part 400 Trays 500 Noodles 510 Weight scale 600 RGB images 601 Depth Images 602 RGB domain images 603 Depth Region Image 604 Cropping Images 700 Grasping learning section 710 Control Unit 711 Weight Prediction Department 712 Weight calculation section 713 Remuneration Calculation Department 714 Buffer 715 Cropping Section 716 Agent Division 7161 Learning Processing Unit 7162 Policy Network Department 720 Storage section 721 Cropping Images 722 Grip information 730 Input / output section 731 Input section 732 Output section 800 Smartphones 810 Control unit / sensor 820 Background Shooting Camera 830 Synthesis Section 840 Display section

Claims

1. An object recognition unit that recognizes an object that is a string-like object; a gripping unit having a plurality of claws and configured to grip the object based on a control signal; A gripping learning unit that learns the motion of the gripping unit and generates learning parameters; a control unit that outputs the control signal based on the learning parameter, The control unit outputs the control signal to the gripping unit to grip the object recognized by the object recognition unit at a first location and place it at a second location different from the first location. gripping device.

2. A camera for capturing an image of the object; an image extraction unit that extracts a predetermined area from an image captured by the camera; a weight prediction unit that predicts a weight of the object gripped by the gripping unit from the image extracted by the image extraction unit; A weight calculation unit that measures the weight of the object before and after it is gripped and calculates the weight of the object gripped by the gripping unit based on the measured weight, The gripping device according to claim 1 , wherein the gripping learning unit generates the learning parameters by performing learning based on an output of the weight prediction unit and an output of the weight calculation unit.

3. The gripping device according to claim 2 , wherein the gripping learning unit performs learning by deep reinforcement learning.

4. The gripping device according to claim 1 , wherein the string-like object is a noodle.

5. The gripping device according to claim 1 , wherein the control unit further outputs a second control signal to the gripping unit to operate the tip of the claw within a predetermined distance from an upper surface of the object.

6. The gripping device according to claim 5 , wherein the control unit further outputs a third control signal for moving the gripping unit in a vertical direction in accordance with a surface shape of the object placed at the first location.

7. The gripping device according to claim 6, wherein the control unit further outputs a fourth control signal for opening and closing the claws, operates the claws from an open state to a closed state based on the fourth control signal, and moves the gripping portion upward based on the third control signal.

8. The gripping device according to claim 5 , wherein the predetermined distance is 3 mm.

9. An object recognition unit that recognizes an object that is a string-like object; a gripping unit having a plurality of claws and configured to grip the object based on a control signal; A gripping learning unit that learns the motion of the gripping unit and generates learning parameters; a control unit that outputs the control signal based on the learning parameter, the control unit outputs the control signal to the gripping unit to grip the object recognized by the object recognition unit at a first location and place the object at a second location different from the first location; A simulator for generating a 3D model of the gripping portion; an image display device that displays an image, A gripping system, wherein the image display device is equipped with a camera, and displays an image captured by the camera combined with a 3D model of the gripping part generated by the simulator.

10. 1. A method for controlling a gripping device that grips an object, which is a string-like object, comprising: Taking an image of the object; Predicting a weight of the grasped object from the captured image of the object; Calculating a weight of the grasped object by measuring the weight of the object before and after it is grasped; generating a learning parameter by performing learning based on the predicted weight of the object and the calculated weight of the object; A method for controlling a gripping device, the method controlling the gripping device to grip the object at a first location and place it at a second location different from the first location, based on the learning parameters.

11. A control program for controlling a gripping device that grips an object that is a string-like object, Taking an image of the object; Predicting a weight of the grasped object from the captured image of the object; Calculating a weight of the grasped object by measuring the weight of the object before and after it is grasped; generating a learning parameter by performing learning based on the predicted weight of the object and the calculated weight of the object; A control program that controls the gripping device to grip the object at a first location and place it at a second location different from the first location based on the learning parameters.

Images