Methods to mitigate the effects of posture

Abstract

To provide a packing style influence alleviation method which allows for easily gripping an object when a part of the object cannot be seen due to influence of internal material like a buffer material or a packing style.SOLUTION: A packing style influence alleviation method comprises the steps of: recognizing an object from image data including one or more imaged objects; inputting, when a part of the object is hidden because of an internal material, an object image of the object a part of which is hidden, into a self-supervised learning model to generate an object image where an area unseen due to influence of the internal material is reduced; calculating a gripping point of the recognized object; gripping, by a robot, the object at the gripping point; and causing a deep reinforcement learning model to learn the gripping point to enhance a gripping success rate on the basis of gap information between the objects, a force sensor value, and joint information, and causing the deep reinforcement learning model to re-learn, when an image data including one or more objects is imaged next time, the gripping point again to enhance the gripping success rate.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to a method for mitigating the influence of loading posture.

Background Art

[0002] In the manufacturing process, there is a process in which a robot equipped with a two-finger hand grips and takes out an object such as a part from a storage container or the like. It is desired that the robot appropriately grips and takes out the parts in the storage container.

[0003] Techniques for determining the feasibility of gripping a gripping object by image processing have been devised (see, for example, Patent Document 1). Patent Document 1 discloses a technique for analyzing the distance between a hand and a gripping object, determining the gripping success rate, generating a discriminator from a feature vector calculated based on the distance from the gripping object, and using this discriminator to determine the feasibility of gripping the gripping object.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, in the conventional technology, due to the influence of internal materials such as cushioning materials and loading posture, when a part of a part is not visible, the following problems exist. · When most of the parts are not visible due to the influence of internal materials such as cushioning materials and loading posture, the gripping points of the parts cannot be calculated. · Incorrect gripping points are calculated, resulting in interference between the part and the hand and damage to the part. · The parts that can be gripped are limited. · It is necessary to change the loading posture, increasing the cost.

[0006] In view of the above problems, the present invention provides a technology that makes it easier to grasp an object even when part of the object is not visible due to the influence of internal materials such as cushioning materials or the packaging. [Means for solving the problem]

[0007] In view of the above problems, the present invention provides a method for mitigating the effects of packaging when a robot equipped with a two-fingered hand grasps an object captured by an imaging device, comprising the steps of: recognizing an object from image data and point cloud data including one or more captured objects; if there is an object partially hidden by internal material, inputting an object image of the partially hidden object to a self-supervised learning model; the self-supervised learning model generating an object image in which the area invisible due to the effect of internal material is smaller than the object image; recognizing the object from image data including the object image generated so that the area invisible due to the effect of internal material is reduced; calculating a grasping point for the re-recognized object; and the robot performing the grasping of the object at the grasping point. The process includes the steps of inputting gap information between objects in image data, including an object image generated to minimize areas obscured by the effect, force sensor values ​​detected by the robot when grasping an object, and joint information into the deep reinforcement learning model, and the deep reinforcement learning model learning the gripping point to increase the success rate of grasping. The next time image data including one or more objects is captured, the deep reinforcement learning model instructs the robot to use a gripping point calculated based on the learning results, inputs gap information between objects in the next image data, force sensor values ​​detected by the robot when grasping an object at that gripping point, and joint information into the deep reinforcement learning model, and the deep reinforcement learning model learns the gripping point again to increase the success rate of grasping. [Effects of the Invention]

[0008] This technology makes it easier to grasp an object even if part of it is not visible due to internal materials such as cushioning or the packaging. [Brief explanation of the drawing]

[0009] [Figure 1]This figure shows an example of multiple parts stored in a storage container. [Figure 2] This is an example of a flowchart illustrating the processing flow performed by a parts gripping device. [Figure 3] This is an example of a functional block diagram that explains the functions of a parts gripping device by dividing them into blocks. [Figure 4] This diagram schematically illustrates the process performed by the component gripping device of this embodiment. [Figure 5] This is an example of a flowchart illustrating the process by which the part gripping device of this embodiment recognizes a part and has a two-fingered hand robot grasp it. [Figure 6] This figure shows an example of the component with the largest area in the image data. [Modes for carrying out the invention]

[0010] The following describes an example of an embodiment for carrying out the present invention: a parts gripping device and a method for mitigating the effects of the packaging performed by the parts gripping device.

[0011] <Example of parts inside the storage container> First, let's refer to Figure 1 and explain the components inside the storage container. Figure 1 shows multiple components stored in a storage container. Note that the components do not need to be in a storage container; they just need to be placed where they can be grasped. Also, there may be only one component. Figure 1 shows four components stored, but for example, component 101, the second from the left, is mostly hidden by internal materials such as cushioning material. Mostly means, for example, 50-60% or more. The component grasping device that grasps and removes this component images the storage container with an imaging device, so the following inconvenience occurs when component 101 is mostly hidden by internal materials. The part gripping device cannot correctly recognize the size of the part. • The gripping robot's hand interferes with areas of the part that are not visible. In this case, there is a high possibility that the part with a design surface will be damaged. It is necessary to determine if there are invisible areas, and if so, to generate images of the components within those invisible areas.

[0012] In addition, as a situation where an area that is not visible in the component occurs, in addition to the inner material, there may be cases due to the loading state such as the way the components are packed.

[0013] <Regarding terms> A robot equipped with a two-finger hand may be any robot that grasps an object. Also, the robot may have three or more fingers. Further, in this embodiment, the robot grasps the grasping point, but it may adsorb the adsorption point or the adsorption point and the grasping point may coexist.

[0014] The inner material is a material for preventing damage, dents, deformation, etc. that may occur in the components due to impacts during movement between processes or transportation. The inner material may be called a cushioning material. Examples of the inner material include cloth, air cushions, various protectors, cushion paper, styrofoam, etc. The loading state refers to the state in which the components are packed, and is the appearance of the components in the state where the components are packed and stored in a storage container.

[0015] That a part of the component is hidden due to the influence of the inner material or the loading state does not include the case where the entire component is not visible at all. When the inner material is transparent, even if the entire component is covered with the inner material, it is not said that the entire component is not visible at all. Also, even in a state where the entire component is not visible at all, if the inner material is in close contact with the component and the shape of the component is clear enough to estimate the grasping point, there may be a case where the entire component is not visible at all.

[0016] <Outline of the processing of this embodiment> Therefore, in this embodiment, as follows, a component image of an area that is not visible is generated by a self-supervised learning model, and various information such as gap information between components (an example of between objects) is input into a deep reinforcement learning model and re-learned to find an accurate grasping point.

[0017] FIG. 2 is a flowchart for explaining the flow of processing performed by the component grasping device. It is assumed that the component grasping device has an imaging device and a two-finger hand robot. S1: The parts gripping device images one or more parts stored in a storage container and recognizes the parts from the captured image data and point cloud data. The point cloud data is a distance image or 3D point cloud corresponding to the image data and is detected by a stereo camera or Lidar (Light Detection And Ranging). S2: If there are parts that cannot be recognized due to the internal materials or packaging (it is not necessary to know if they are parts at this point), the part gripping device inputs image data into a self-supervised learning model. S3: The part gripping device uses a self-supervised learning model to generate part images (an example of an object image) of parts that are partially obscured due to the influence of internal materials or packaging in the input image data. S4: The part gripping device recognizes the part again using a part image generated to eliminate the influence of internal material and packaging. S5: The part gripping device analyzes the parts recognized by the image data, which have been replaced with the generated part images, calculates the gripping point, and performs part picking (removal). S6: The part gripping device inputs gap information between parts (including gaps with internal materials) calculated from image data including part images that are free from the influence of internal materials and packaging, force sensor values ​​detected during picking, and information on each joint into a deep reinforcement learning model. S7: The part grasping device learns the grasping point that maximizes the success rate of grasping using a deep reinforcement learning model. In the next imaging session, the grasping point calculated based on the learning results is assigned to the two-fingered hand robot. S8: The part gripping device reacquires the gripping success rate, force sensor values, gap information between parts, and joint information when it operates at the assigned gripping point. S9: The part gripping device retrains itself using a deep reinforcement learning model based on the acquired information, and then instructs the two-fingered hand robot on the gripping point based on the retrained results.

[0018] According to the parts gripping device of this embodiment, even if a part is largely invisible due to internal materials such as cushioning material or the packaging, the intended gripping point can be calculated. Interference between the part and the gripping hand, which could damage the part, can be suppressed. Furthermore, the number of parts that can be gripped can be increased without changing the packaging.

[0019] <Example of Functional Configuration> Figure 3 is an example of a functional block diagram that explains the functions of the part gripping device 10 by dividing them into blocks. The part gripping device 10 includes, for example, a control unit 11, an imaging device 12, and a two-fingered hand robot 13. The control unit 11 may be a microcontroller, computer, or SOC having a CPU, RAM, ROM, HDD (or SSD), input / output unit, ASIC, FPGA, etc. The control unit 11 realizes the functions shown in Figure 3 by having the CPU execute a program stored in the ROM or HDD (or SSD). The control unit 11 includes a part recognition unit 14, an image generation unit 15, a gripping point calculation unit 16, a self-supervised learning unit 17, and a deep reinforcement learning unit 18.

[0020] The part recognition unit 14 may be an image recognition model that recognizes parts from image data of one or more parts captured by the imaging device 12. The image recognition model is constructed using a learning method such as deep learning. The part recognition unit 14 also recognizes point cloud data such as depth images and 3D point clouds corresponding to the image data. The depth images and 3D point clouds may be detected by a stereo camera or LiDAR.

[0021] The image generation unit 15 uses a self-supervised learning model to generate a part image from the part image contained in the image data, reducing the area that is hidden due to the internal material or packaging. The part recognition unit 14 then recognizes the part again using the generated part image.

[0022] The gripping point calculation unit 16 calculates a gripping point for the recognized part. For example, the gripping point calculation unit 16 estimates the center of gravity from the point cloud data of the recognized part and determines the gripping point that is closest to the estimated center of gravity and is also the closest (closest) to the center of gravity (allowing for a deeper grip). Furthermore, once the deep reinforcement learning model has completed training, the gripping point calculation unit 16 can use the deep reinforcement learning model constructed by the deep reinforcement learning unit 18 to calculate the gripping point using gap information between parts, force sensor values, and joint information as input data. The calculated gripping point is transmitted to the two-fingered hand robot 13.

[0023] The self-supervised learning unit 17 generates a self-supervised learning model (image generation unit 15) that generates part images with smaller hidden areas from part images with large hidden areas due to the influence of internal materials and packaging in the image data. Self-supervised learning is a learning method that uses a large dataset of unlabeled data to perform pre-training to solve a pretext task (an alternative task in which pseudo-labels are automatically generated).

[0024] The deep reinforcement learning unit 18 generates a deep reinforcement learning model (grasp point calculation unit 16) that outputs the gripping point that maximizes the gripping success rate, using gap information between parts calculated from image data including part images from which the influence of internal materials and packaging has been eliminated, force sensor values, and joint information. Reinforcement learning is a learning method in which there is no correct answer in the training data, but the system searches for the optimal action to obtain a set "reward (score)". In this embodiment, the reward can be set to increase as the gripping success rate increases. Deep reinforcement learning is a learning method that combines reinforcement learning and deep learning, but it refers to an approximation in which the action-value function (Q function) in reinforcement learning is replaced with deep learning. Note that the learning method is just one example, and other learning methods may be adopted.

[0025] The two-fingered hand robot 13 is connected to the end of an industrial robot and is a robot that grasps, deforms, processes, etc., parts in place of a human hand. In this embodiment, the number of fingers is two, but it may be three or more. The two-fingered hand robot 13 may be mounted on a mobile body and be mobile, or the two-fingered hand robot 13 may be a humanoid robot. The two-fingered hand robot 13 has a sensor 21 and a joint control unit 22.

[0026] Sensor 21 is a force sensor or pressure sensor embedded in the surface that contacts the components of the two-fingered hand. The force sensor value may be calculated from the current value for gripping. The joint control unit 22 controls the state of the joints that make up the fingers of the two-fingered hand robot 13. The joint control unit 22 can acquire information on the joint angle, velocity, acceleration, and torque as the state of the joints. Generally, one finger has six joints, but the number of joints is not limited to six.

[0027] <Processing or control flow> Figure 4 is a schematic diagram illustrating the processing performed by the part gripping device 10 of this embodiment. First, the deep reinforcement learning model 201 will be explained. The deep reinforcement learning model 201 receives force sensor values ​​31 detected by the sensor 21, joint information 32 acquired by the joint control unit 22, and gap information 33 between parts (including gap information between the inner material and the part) recognized by the part recognition unit 14. The deep reinforcement learning model 201 also receives gap information 33 between parts (including gap information between the inner material and the part) in image data 41 generated by the image generation unit 15, which includes a part image with the invisible area reduced. The image data 41 may also be the original part image (if the part is not hidden by the inner material, etc.).

[0028] As an example of a deep reinforcement learning model, we will explain using Deep Q-Network (DQN). First, we will explain reinforcement learning.

[0029]

number

[0030] This section explains deep reinforcement learning. Deep reinforcement learning utilizes deep learning to train the Q-table. Specifically, in deep reinforcement learning, the state (information about the gaps between parts, force sensor values, and information about each joint) is input to the input layer of the neural network, and the reward for each action that can be taken in that state is output from the output layer via a hidden layer. The output layer has nodes corresponding to each grasping point, and each node outputs a probability that correlates with the success rate of grasping. In other words, if the action is grasping, the probability of which grasping point to use to achieve a high success rate of grasping (corresponding to the reward) for a given gap information between parts, force sensor values, and information about each joint is output for each grasping point.

[0031] This section explains how neural network weights are learned in deep reinforcement learning. In the Q-table update of equation (1), when the second term on the right-hand side approaches zero (when the update amount decreases), the Q-table update also ends. Therefore, in the learning phase of deep reinforcement learning, a loss function (Loss) is defined as shown in equation (2), and weights are learned between the neural networks that infer rewards. Considering that the rewards in the Q-table are set to increase the success rate of grasping, the weights of the neural networks are adjusted to increase the success rate of grasping through backpropagation of the loss function (Loss).

[0032]

number

[0033] Next, we will describe the self-supervised learning model 202. Self-supervised learning is a supervised learning method that mechanically creates its own labels (training data) from the data itself and uses these labels for learning. In this embodiment, the self-supervised learning is trained to generate part images with smaller invisible areas from part images with large invisible areas. Specifically, the learning process is carried out in the following manner. • Pretext is automatically created by splitting a single part image into multiple parts and performing color and shape distortion processing. In this process, two different transformations (positive example and counterexample) are applied to each image. Next, the transformed images are processed using two encoders, one for positive examples and one for counterexamples, to obtain features. Next, the loss is calculated using a loss function that allows comparison of the features obtained from the two encoders, and the encoder weights are trained to minimize the loss. During training, the neural network weights are trained so that the two features obtained from the same source image become similar.

[0034] The self-supervised learning model 202 includes a generation unit 37, a real image holding unit 38, a generated image holding unit 39, and a reverse generation unit 40. The generation unit 37 receives a part image 42, which is cropped from the image data and has a large area (above a threshold) that is not visible due to the influence of internal materials or packaging. The generation unit 37 generates a part image 43 in which the area not visible due to the influence of internal materials or packaging is smaller. The part image 42 in the original image data 41 is replaced with this part image 43.

[0035] The real image storage unit 38, the generated image storage unit 39, and the inverse generation unit 40 are used to train the self-supervised learning model 202. The real image storage unit 38 stores various pre-prepared component images. The generated image storage unit 39 stores generated component images. The inverse generation unit 40 learns the weights of the neural network so that two feature quantities obtained from two images become closer together, and feeds this back to the generation unit 37.

[0036] Image data 41, which includes a part image 43 with a smaller invisible area, generated by the self-supervised learning model 202, is input to the part recognition process 34. Image data 41 is used for part recognition, and after recognition, the gripping point search process 35 is executed.

[0037] <Detailed processing flow> Figure 5 is a flowchart illustrating the process by which the part gripping device 10 of this embodiment recognizes a part and has the two-fingered hand robot 13 grip it.

[0038] First, the imaging device 12 images one or more parts stored in the storage container. The part recognition unit 14 recognizes the parts from the image data and point cloud data of the one or more parts (S11).

[0039] Next, the component recognition unit 14 identifies the component with the largest area in the image data (S12). Figure 6 shows an example of the component 203 with the largest area in the image data.

[0040] Next, the component recognition unit 14 acquires the surface irregularities information of the component 203 with the largest area (S13). The surface irregularities information can be the difference between the maximum and minimum distances, or the variance of the distances, when the point cloud data of the component is considered as the distance from the imaging device 12.

[0041] Next, the component recognition unit 14 compares the area of ​​the component with the largest area with the area of ​​an object that appears to be another component, and the surface irregularities of the component with the largest area with the object that appears to be another component (the target component) (S14). The target component may be any component other than the component with the largest area, the component with the smallest area, or the component with the next largest area.

[0042] As an example of a comparison method, "Area of ​​the largest component × 0.9 > Area of ​​the target component" AND "Roughness information of the most prominent component × 0.9 > Roughness information of the target component" The system determines whether the following conditions are met. This determination is made by slightly underestimating the area or surface irregularities of the largest component (by multiplying by 0.9), and if it is still larger than other objects that appear to be components, then it is determined that there are components hidden inside the material or packaging (the area is not uniform).

[0043] If the decision in step S14 is Yes, the process proceeds to step S15; otherwise, it proceeds to step S20.

[0044] In step S15, the image generation unit 15 inputs the image data into the self-supervised learning model 202 (S15).

[0045] The image generation unit 15 inputs the part image with the largest area into the self-supervised learning model 202 and generates a part image in which areas hidden by internal materials or packaging are reduced (S16). The generated part image replaces the part image in the captured image data.

[0046] The control unit 11 determines whether the number of loop executions matches "number of parts - 1" (S17). That is, it determines whether part images have been generated for all parts in the image data. If the determination in step S17 is Yes, the process proceeds to step S19; otherwise, it proceeds to step S14. In other words, if step S14 is satisfied, part images with smaller areas hidden by internal materials or packaging are generated in steps S15 and S16, but if step S14 is no longer satisfied, the process moves to the picking operation. Therefore, part images with smaller areas hidden by internal materials or packaging may be generated for only some of the parts in the image data.

[0047] In step S19, the part recognition unit 14 recognizes the object again using image data that includes a part image generated to eliminate the influence of internal materials and packaging (S19).

[0048] Next, the gripping point calculation unit 16 analyzes the point cloud data of the recognized parts to calculate the optimal gripping point. The control unit 11 then instructs the two-fingered hand robot 13 to pick (remove) the parts at the gripping point (S20). As a result, all parts are removed. If the deep reinforcement learning model has already been trained, the gripping point may be calculated using the deep reinforcement learning model.

[0049] Next, the control unit 11 uses the information obtained from the component extraction to train a deep reinforcement learning model.

[0050] First, the deep reinforcement learning unit 18 inputs the gap information between parts (including the gap with the internal material) calculated from the part image or the original part image, which has been freed from the influence of the internal material and packaging, the force sensor values ​​obtained in step S20, and the joint information into the deep reinforcement learning model 201 (S21).

[0051] The deep reinforcement learning unit 18 uses the deep reinforcement learning model 201 to learn the gripping point that maximizes the success rate of gripping (S22). The control unit 11 instructs the two-fingered hand robot 13 to use the gripping point calculated based on the learning results for the next time it images a part.

[0052] During the next imaging session, the control unit 11 acquires the gripping success rate when gripping at the assigned gripping point, force sensor values, gap information between parts, and joint information from the two-fingered hand robot 13 based on the learning results (S23).

[0053] The control unit 11 retrains the deep reinforcement learning model 201 using the acquired information, and then issues instructions to the two-fingered hand robot 13 based on the retrained results (S24).

[0054] <Main effects> According to the parts gripping device of this embodiment, even if a part is largely invisible due to internal materials such as cushioning material or the packaging, the intended gripping point can be calculated. Interference between the part and the gripping hand, which could damage the part, can be suppressed. Furthermore, the number of parts that can be gripped can be increased without changing the packaging. [Explanation of Symbols]

[0055] 10. Part gripping device 11 Control Unit 12 Imaging device 13 Two-fingered hand robot

Claims

[Claim 1] A method for mitigating the influence of the packaging when a robot equipped with a two-fingered hand grasps an object imaged by an imaging device, A step of recognizing an object from image data and point cloud data that include one or more captured objects, If there are objects that are partially hidden due to internal materials or packaging, the process involves inputting an image of the partially hidden object into a self-supervised learning model. The self-supervised learning model generates an object image in which the area that is not visible due to the influence of internal materials or packaging is smaller than that of the aforementioned object image, The step of re-recognizing the object from image data that includes an object image generated in such a way that the areas not visible due to the internal material or packaging are minimized, The steps include: calculating the gripping point for the recognized object again, The robot performs the step of grasping an object at the grasping point, The steps include inputting gap information between objects in image data, including an object image generated in such a way that the area not visible due to the influence of internal materials or packaging, force sensor values ​​detected by the robot when grasping an object, and joint information into a deep reinforcement learning model, The deep reinforcement learning model includes a step of learning the gripping point in such a way that the gripping success rate is high, Next time, when image data containing one or more objects is captured, the deep reinforcement learning model will instruct the robot to use the gripping point calculated based on its learning results. The gap information between objects in the next image data, the force sensor value detected when the robot grasps the object at the grasping point, and the joint information are input to the deep reinforcement learning model. A method for mitigating the effects of the packaging, wherein the deep reinforcement learning model relearns the gripping points in a way that increases the success rate of gripping.

Images