Information processing device and picking device

Abstract

The present application provides an information processing device and a picking device capable of effectively inferring the posture of an object by contact. The information processing device includes a storage unit that stores object information indicating the shape and size of an object (Q1); a force sensor that is capable of acquiring detection information of a contact point in the case of contact with an arbitrary surface; a movement control unit that moves the force sensor; and an inference unit that infers the posture of the object (Q1) based on the detection information obtained through multiple contacts of the force sensor with the object and the object information. Furthermore, the movement control unit moves the force sensor in such a manner that the force sensor contacts different surfaces (s1, s2) of the object in the multiple contacts.

Description

Technical Field

[0001] This invention relates to an information processing device and a picking device. Background Technology

[0002] Patent document 1 discloses a system for identifying the posture of an object based on an image of the object and measurement information of the contact position with the object.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2017-136677 Summary of the Invention

[0006] The technical problem to be solved by the invention

[0007] For transparent or black objects, objects with mirrored surfaces, or objects covered by cloth, it is difficult to determine the posture of an object based solely on an image. In the system of Patent Document 1, the position of an object wrapped in a plastic bag or cushioning material can be identified by simultaneously using measurement information based on contact with the object. However, in devices that infer the posture of an object through contact, there is room for improvement in the manner of contact with the object for effective inference.

[0008] The purpose of this invention is to provide an information processing device and a picking device that can effectively infer the posture of an object using detection information obtained through contact.

[0009] means for solving technical problems

[0010] The information processing apparatus involved in this invention is an information processing apparatus for inferring the posture of an object, which includes:

[0011] The storage unit stores object information representing the shape and size of the object;

[0012] Force sensors acquire detection information at contact points through contact.

[0013] The movement control unit moves the force sensor; and

[0014] The inference unit infers the posture of the object based on the detection information obtained through multiple contacts between the force sensor and the object, and the object information.

[0015] The movement control unit moves the force sensor in such a way that the force sensor contacts different surfaces of the object during the multiple contacts.

[0016] The picking device according to the present invention includes:

[0017] The aforementioned information processing device; and

[0018] The extraction mechanism extracts the object using the inference result from the inference unit.

[0019] Invention Effects

[0020] According to the present invention, an information processing device and a picking device are provided that can effectively infer the posture of an object by means of contact. Attached Figure Description

[0021] Figure 1 This is a block diagram illustrating the picking device according to the first embodiment of the present invention.

[0022] Figure 2 This is a flowchart illustrating an example of the picking process sequence performed by the control department.

[0023] Figure 3A This is an explanatory diagram of the first step in the first example of processing to infer the pose of an object.

[0024] Figure 3B This is an explanatory diagram of the second process in the first example above.

[0025] Figure 3C This is an explanatory diagram of the third process in the first example above.

[0026] Figure 3D This is an explanatory diagram of the fourth process in the first example above.

[0027] Figure 3E This is an explanatory diagram of the fourth process in the first example above.

[0028] Figure 4A This is an explanatory diagram of the first step in the second example of processing to infer the pose of an object.

[0029] Figure 4B This is an explanatory diagram of the second process in the second example above.

[0030] Figure 4C This is an explanatory diagram of the third process in the second example above.

[0031] Figure 4D This is an explanatory diagram of the fourth process in the second example above.

[0032] Figure 4E This is an explanatory diagram of the fifth step in the second example above.

[0033] Figure 4F This is an explanatory diagram of the sixth process in the second example above.

[0034] Figure 5This is a block diagram illustrating the functions of the control device based on the second embodiment.

[0035] Figure 6A This is a floor plan showing an example of the permitted range of movement.

[0036] Figure 6B It is a cross-sectional view including the rotation center axis of the movable arm.

[0037] Figure 7 This is a flowchart illustrating the processing performed by the movement range limiting unit and the arm control unit.

[0038] Figure 8A This is a plan view representing a candidate example of the movement path of the end effector of a movable arm.

[0039] Figure 8B It is a cross-sectional view including the rotation center axis of the movable arm.

[0040] Figure 9A This is a plan view representing an example of the total range of motion that a movable arm can move.

[0041] Figure 9B It is a cross-sectional view including the rotation center axis of the movable arm.

[0042] Figure 10 This is a diagram showing an image displayed in the display section of a robotic arm based on another embodiment.

[0043] Figure 11 This is a block diagram of a system based on yet another embodiment.

[0044] Figure 12 This is a block diagram of a picking device based on the third embodiment.

[0045] Figure 13A It is a drawing that shows the CAD data of the shape of a hexagonal nut, defined as an example of an object, on a plane.

[0046] Figure 13B It is a diagram that shows multiple reference points p located on the surface of the CAD model on a plane.

[0047] Figure 14 This is a schematic diagram representing multiple representative reference points p_i.

[0048] Figure 15 This is a schematic diagram illustrating an example of coordinate transformation used in pre-learning.

[0049] Figure 16 This is a flowchart showing the sequence of additional learning performed by the position and pose inference additional learning unit.

[0050] Figure 17This is a flowchart illustrating how the contact path determines the order in which the reinforcement learning department performs reinforcement learning.

[0051] Figure 18 This is a flowchart illustrating the sequence in which the contact path determines the imitation learning process performed by the imitation learning unit.

[0052] Figure 19 This is a diagram illustrating an example of graphics and images output to an output device during imitation learning.

[0053] Figure 20 This is a flowchart illustrating the actions of the picking device in the additional learning application mode.

[0054] Figure 21 This is a flowchart illustrating the actions of the picking device in learning mode. Detailed Implementation

[0055] (First Embodiment)

[0056] Hereinafter, the first embodiment of the present invention will be described in detail with reference to the accompanying drawings. Figure 1 This is a block diagram illustrating a picking device according to the first embodiment of the present invention. The picking device 1 according to the first embodiment is a device that infers the posture of an object with a known shape and size and removes the object from a receiving portion by grasping or the like. The picking device 1 includes: a camera unit 21 for acquiring an image of the object; a force sensor 22 for acquiring detection information of the contact point by contacting the object; a drive unit 23 for moving the force sensor 22; a robotic arm 24 as a picking mechanism for grasping and transporting the object; and a control unit 10 for controlling the above-mentioned parts. The picking device 1 is an example of an information processing device according to the present invention.

[0057] The camera unit 21 has an imaging element and an imaging lens, and acquires images of digital signals. The camera unit 21 only needs to acquire an image that allows us to determine the approximate position of an object. Alternatively, the camera unit 21 can also be a depth sensor that acquires two-dimensional images and obtains depth information of each point on the image.

[0058] When the contact point of force sensor 22 is in contact with any surface, force sensor 22 acquires detection information of the contact point. The detection information includes the position information of the contact point and information indicating the orientation of the reaction force applied to the contact point from the contact point (i.e., information including the orientation of the surface of the contact point (the normal to the surface)). Force sensor 22 stops when subjected to a very small reaction force, thus acquiring the detection information of the contact point without moving the object being contacted.

[0059] The drive unit 23 is a three-dimensional drive unit capable of moving the force sensor 22 (specifically, its contact) along any path. The drive unit 23 that moves the force sensor 22 can also be used as the drive unit of the robotic arm 24.

[0060] The control unit 10 includes a CPU (Central Processing Unit); RAM (Random Access Memory) for CPU data processing; a storage device storing the control program executed by the CPU; and an interface 16 for transmitting and receiving signals with the camera unit 21, force sensor 22, drive unit 23, and robotic arm 24. In the control unit 10, the CPU executes the control program to implement multiple functional modules. These modules include: a camera control unit 12 for controlling the camera unit 21; an object information storage unit 11 for storing object information representing the shape and size of an object; a motion control unit 13 for controlling the drive unit 23 to move the force sensor 22; an inference unit 14 for inferring the object's posture; and a robotic arm control unit 15 for driving and controlling the robotic arm 24.

[0061] The object information may include, for example, position data (relative position data) of multiple points uniformly distributed on the surface of the object. In addition to the position data of each point, the object information may also include normal data for each point. Normal data represents the direction of the normal to the surface including the point and passing through that point. Alternatively, the normal data for each point may not be provided in advance but may be calculated by the control unit 10 based on the position data of the multiple points.

[0062] The inference unit 14 infers the approximate position of an object based on the image acquired by the imaging unit 21. The image-based inference can be a relatively low-precision inference that allows the force sensor 22 to make contact with a certain part of the object. The specific inference method is not limited, but for example, the inference unit 14 can perform image recognition processing based on the image to infer the center point of the object or the point where the object is located with a relatively large allowable error.

[0063] Furthermore, the inference unit 14 infers the object's posture based on the detection information of the contact points obtained from the force sensor 22 contacting the object once or multiple times. Here, the object's posture is a concept that includes both the object's position and orientation; however, the posture inference involved in this invention may also be an inference of orientation only. In the stage where the number of contacts of the force sensor 22 is small enough to prevent narrowing down the object's posture to a single one, the inference unit 14 infers multiple candidates for posture based on the contact point detection information. Then, as the number of contacts of the force sensor 22 increases, the candidates for posture are narrowed down, thereby enabling the inference unit 14 to finally infer a single posture.

[0064] The motion control unit 13 determines the contact path when the force sensor 22 comes into contact with the object and moves the force sensor 22 along the determined contact path so that the inference unit 14 can effectively infer the posture of the object. The method for determining the contact path will be described in detail later.

[0065] The robotic arm control unit 15 drives the robotic arm 24 based on the posture information of the object inferred by the inference unit 14, so that the robotic arm 24 grasps and transports the object with that posture.

[0066] <Methods for inferring posture>

[0067] Next, a specific example of the inference method for inferring the pose of an object by the inference unit 14 will be described. However, the inference method for inferring the pose of an object by the inference unit 14 is not limited to the example described below. Here, a method for the inference unit 14 to infer the pose of an object or multiple candidate poses using the cost function G(r,t) of equation (1) will be described.

[0068] [Formula 1]

[0069]

[0070] The variables in the formula are as follows.

[0071] r: Represents the rotational expression of Rodrigues.

[0072] t: Represents the translation vector

[0073] K: Total number of contacts at the inference time

[0074] k: Index of contact frequency

[0075] M: Represents the set of multiple points contained in the object information (hereinafter referred to as "point group M").

[0076] m: Represents the index and position of each point contained in point group M (object coordinate system)

[0077] r p c，k Location of the contact point (robot coordinate system)

[0078] R(r): Rotation matrix with r as an element

[0079] w: Adjustment coefficient

[0080] r f k The direction of the reaction force contained in the detection information of force sensor 22 (manipulator coordinate system).

[0081] n m The direction of the normal at point m (object coordinate system)

[0082] The aforementioned robotic arm coordinate system is a coordinate system based on the base of the robotic arm 24, meaning that the origin and three axes of the coordinate system are fixed when the base of the robotic arm 24 does not move. The object coordinate system is a coordinate system based on the object, meaning that if the position and orientation of the object change, the origin and three axes of the coordinate system rotate and translate relative to the reference coordinate system (e.g., the Earth's coordinate system) according to that change. Rotation r and translation t are quantities representing the position and orientation of the object in the robotic arm coordinate system. For example, the difference between the origin of the robotic arm coordinate system and the origin of the object coordinate system can be expressed as translation t, and the difference between the three axes of the robotic arm coordinate system and the three axes of the object coordinate system in the rotation direction can be expressed as rotation r.

[0083] The above adjustment coefficients are the coefficients for the degree to which the first and second items within the brackets on the right side of adjustment formula (1) affect the cost function. The first item is as follows: if the point group M representing the surface of the object includes a point m close to the contact point, the cost function is reduced; the greater the distance between the point m closest to the contact point in the point group M and the contact point, the greater the cost function. The second item is as follows: if the point m in the point group M has a normal direction with the same orientation as the surface of the contact point, the cost function is reduced; the more different the orientation of the normal at the point m with the orientation of the surface closest to the contact point is from the orientation of the surface of the contact point, the greater the cost function. The absolute value symbol indicates the length of the vector, and "<a,b>" indicates the inner product of vectors a and b. "min[]" indicates the minimum value among the values ​​within the brackets at each point m. Thus, the value within the plus sign Σ becomes as follows: if the point group M includes a point close to the contact point and the normal direction of that point is close to the orientation of the surface of the contact point, it moves away from zero in the positive direction if such a point is not included.

[0084] The inference unit 14 searches for the value of the cost function G(r, t) to be a small value of the independent variable (translation r, rotation t) near zero, thereby inferring the searched rotation and translation {r, t} as the pose of the object or a candidate pose of the object.

[0085] <Inference Order>

[0086] First, the inference unit 14 sets an initial region for searching the object's pose based on the image obtained from the imaging unit 21 and object information representing the object's size and shape. The initial region is set to have a large margin, making the possibility of the object exceeding this region almost zero. For example, the inference unit 14 calculates the circumsphere of the object whose center is located at the center point based on the object's center point obtained through image recognition from the image, increases the diameter of the circumsphere by an error amount corresponding to the inference accuracy of the image recognition, and uses the inner side of the circumsphere as the initial region.

[0087] Next, the inference unit 14 sets all possible positions and orientations of objects contained in the initial region as candidates for the initial position. Each position is represented by a combination of rotation r and translation t. Therefore, the initial candidates for the position become a set of combinations of rotation and translation {r, t}. However, if very small differences in translation or rotation are considered as other positions, the number of candidates for the initial position becomes very large. Therefore, the inference unit 14 appropriately sets the minimum difference Δt in translation and the minimum difference Δr in rotation, and creates a set I0 = {(r, t)} of multiple candidates representing the initial position by treating positions where the difference in translation or rotation is greater than or equal to the minimum difference Δt or Δr as different positions. i , i = 1, 2, ...}.

[0088] Furthermore, when there are elements such as a workbench that restrict the posture of the object, the inference unit 14 can perform a process to exclude posture candidates that would not arise due to the aforementioned restrictions from a plurality of initial posture candidates (set I0). This process reduces the number of candidates, thereby reducing the computational load of subsequent inference processes, and narrowing the candidates down to a single one with fewer interactions. An example without this exclusion is shown below.

[0089] If the force sensor 22 makes contact for the first time, detection information indicating the position of the contact point and the direction of the reaction force (including the orientation of the surface of the contact point) is sent to the inference unit 14. The inference unit 14 applies this information to the set I0 = {(r, t)}. i The cost function G(r,t) is calculated for all elements of the group i = 1, 2, ..., i. Then, elements in the set I0 whose cost function G(r,t) is close to zero are extracted as candidates for pose. If the candidates for pose consist only of consecutive elements falling within a constant width corresponding to the aforementioned threshold, the element that minimizes the cost function G(r,t) is extracted from these consecutive elements as the final inference result of the object's pose. Consecutive elements mean multiple elements consecutive with the minimum difference Δt in translation or the minimum difference Δr in rotation. On the other hand, if the candidates for pose are not only consecutive elements, it means that multiple candidates for pose remain, so the inference unit 14 continues the inference process, waiting for the next contact of the force sensor 22.

[0090] If the force sensor 22 makes contact again, the inference unit 14 performs the same process as during the first contact (replacing the initial candidate with the candidate whose posture was reduced in the first contact), thereby further reducing the candidate posture of the object. Then, this process is repeated, so that the inference unit 14 can obtain the posture of the object reduced to one as the final inference result.

[0091] <Conditions for contact path>

[0092] During the inference process performed by the inference unit 14 and before the force sensor 22 comes into contact with the object, the motion control unit 13 moves the force sensor 22 (specifically, its contact) from any orientation to any position and makes it come into contact with the object, or calculates its path as a contact path. The contact path is determined based on the following first to third contact conditions.

[0093] The first contact condition is as follows: in the case of multiple contacts, contact occurs with a surface different from the previously contacted surface. A different surface in an object refers to the surfaces defined by peaks or V-grooves contained within the object. If the object does not have curved surfaces, then each plane becomes a different surface. The contact path corresponding to the first contact condition can be determined as a path that proceeds along a direction intersecting (e.g., orthogonal) the normal to the contact point detected in previous contacts.

[0094] Alternatively, the contact path corresponding to the first contact condition can also be determined as a path that proceeds along a direction intersecting the previously uncontacted surface to the part of the object's posture that the inference unit 14 infers at the current stage.

[0095] The second contact condition is as follows: If an asymmetrical part exists in the object, contact with the asymmetrical part is highly probable. An asymmetrical part of the object refers to a part that is point-symmetric, axially symmetric, or has a low degree of surface symmetry relative to the object's center point, central axis, or central plane. For example, in the case of a bolt, the head edge and the shaft end are considered asymmetrical parts. Furthermore, if it is a pin component whose diameter decreases at a specified location other than the center in the length direction, then that specified location is considered an asymmetrical part. Such asymmetrical parts can be calculated by the movement control unit 13 based on the object information described above, or they can be pre-included in the object information.

[0096] Regarding the contact path corresponding to the second contact condition, before contact with the object, the inference unit 14 can infer the posture from the image obtained by the photography unit 21 and the information of the aforementioned asymmetric surface.

[0097] Regarding the contact path corresponding to the second contact condition, during a phase where one or more contacts have occurred, the location of the asymmetric part with a high probability of being located can be calculated based on multiple candidates of the posture inferred at the current stage, and the path toward that location can be determined.

[0098] The third contact condition is as follows: during one or more contact phases, the object makes contact with a surface with a high degree of uncertainty in the position of the surface of one of the multiple candidates of the posture obtained by the inference unit 14.

[0099] The surface with high positional uncertainty is not particularly limited, but it can be determined, for example, as follows: First, the motion control unit 13 calculates the distance between the center point of the object inferred from the image and each point included in the multiple candidate point groups M of the pose, and selects n representative points with long and dispersed distances. Then, the motion control unit 13 sets a cylinder of the same shape and volume with the normal of each of the selected n representative points as the central axis as a local region. Moreover, the motion control unit 13 extracts all points included in each local region from all point groups M included in the multiple candidates of the pose, and calculates the average value and dispersion value of the position of these points in the axial direction (along the direction of the central axis of the cylinder). Then, the motion control unit 13 determines the local region with large dispersion value from the n local regions as the part with high positional uncertainty of the surface, and determines the surface included in the local region as the surface with high positional uncertainty.

[0100] Furthermore, in the above example, n local regions were defined based on n representative points that are farthest from the center point of the object inferred from the image. However, the method for defining n local regions is not limited to the above example. For example, n local regions can also be defined based on randomly selected n representative points, n representative points that are isotropically dispersed, etc. Also, in the above example, a cylindrical region was defined as a local region and the axial positional dispersion of the points contained within the local region was calculated. However, a local region can also be any shape, such as a sphere, and the dispersion is not limited to axial positional dispersion. For example, it can also be the dispersion of multiple points contained within the local region from the central position, and the direction of dispersion is not limited. Furthermore, in the above example, the dispersion value of the points contained in each local region was calculated. However, various methods can be used to calculate the degree of uncertainty. For example, the distance between the two points furthest apart along the axis in each local region can be calculated, and the region with the largest distance between these two points can be defined as the region containing the surface with a high degree of uncertainty. Furthermore, in the above example, multiple local regions were set, and the values ​​representing uncertainty were compared among these local regions to find the regions that include surfaces with high uncertainty. However, local regions can also be omitted. As long as the parts that include surfaces with high uncertainty can be found, any method can be used, such as finding the parts with high dispersion of all points M of multiple candidate poses as the parts that include surfaces with high uncertainty.

[0101] Regarding the direction of the contact path corresponding to the third contact condition, it is sufficient to calculate the path that proceeds towards the part containing the surface with high uncertainty, as determined in the above manner, along the direction intersecting with the surface contained in that part. This direction can be set as the normal direction of the representative point contained in the local region with high uncertainty, or it can be set as the direction calculated based on the normal directions of all points contained in that local region (such as the average direction).

[0102] <Methods for Determining Contact Path>

[0103] Sometimes multiple contact paths can be obtained that satisfy all of the first to third contact conditions. Alternatively, sometimes a contact path can be obtained that corresponds to any one or two of the first to third contact conditions but does not correspond to the other contact conditions. Therefore, the motion control unit 13 appropriately selects a contact path from the contact paths obtained based on the first to third contact conditions and determines the selected contact path as the contact path for moving the force sensor 22 next.

[0104] One method for selecting a contact path is to assign priorities to each contact path and select the one with the highest priority. The priority can be set to different values ​​based on the first through third contact conditions, or it can be a value obtained by multiplying the probability of matching each contact condition by that value. Alternatively, any method can be used to select a contact path, such as random selection.

[0105] In addition, in the above example, the motion control unit 13 determines the contact path based on all the first to third contact conditions. However, the motion control unit 13 may also omit one or two contact conditions and determine the contact path based on the remaining two or one contact conditions.

[0106] <Sorting Processing>

[0107] Figure 2 This is a flowchart illustrating an example of the picking process performed by the control unit 10. Picking refers to retrieving an object. Figure 2 The picking process begins when the object to be picked is within the field of view of the camera unit 21.

[0108] If the picking process begins, firstly, the camera control unit 12 acquires an image via the camera unit 21 and sends the acquired image to the inference unit 14 (step S1). The inference unit 14 performs image recognition processing based on the image to infer the center position of the object, and sets the aforementioned initial region R0 (refer to) that sufficiently includes the object based on the center position and object information. Figure 3A (Step S2).

[0109] If an initial region R0 is set, the control unit 10 initiates a loop process (steps S3 to S13) to reduce the object's posture by contacting the object. Specifically, the movement control unit 13 determines whether it is the first contact (step S3). If it is the first contact, it calculates a contact path that matches the second contact condition (contact with asymmetric parts is highly probable) and assigns a priority to that contact path (step S6). As this contact path, a path is calculated based on the candidate posture of the object inferred at that moment and the calculation results of the asymmetric parts of the object calculated from the object information, moving towards the surface located at that part in a direction intersecting that surface. The priority assigned to the calculated contact path can be a value preset for the second contact condition to differentiate the priorities of the first to third contact conditions. Alternatively, the priority can be a value obtained by multiplying the preset value for the second contact condition by a value representing the probability (e.g., probability) that the contact path matches the contact condition.

[0110] On the other hand, if the determination process in step S3 determines that the contact is a second or subsequent contact, the movement control unit 13 calculates a contact path that matches the first contact condition (and the contact with a surface different from the surface that was previously contacted) based on the candidate poses of the object inferred at that moment (step S4). Furthermore, the movement control unit 13 performs a process of assigning priority to this contact path (step S4). Assuming there are j candidate poses, a path is calculated that extracts the surface that was inferred not to have been contacted from the previous contact in each candidate and moves towards one or more points within that surface along a normal direction associated with that point. The direction of the contact path can be calculated as the direction in which the normal of the previously contacted surface intersects. The priority assigned to this contact path can be a value preset for the first contact condition to distinguish the priorities of the first to third contact conditions. Alternatively, the priority can be a value obtained by multiplying the preset value for the first contact condition by a value representing the probability (e.g., probability) that the contact path matches the contact condition.

[0111] Next, the motion control unit 13 calculates a contact path that matches the third contact condition (contact with a surface with high positional uncertainty) based on the candidate posture of the object inferred at that moment, and performs a process of assigning priority to the contact path (step S5). The method for obtaining this contact path is as described above. The priority can be a value preset for the third contact condition to distinguish the priorities of the first to third contact conditions. Alternatively, the priority can be a value obtained by multiplying the preset value of the third contact condition by a value (e.g., probability) representing the possibility that the path matches the contact condition. If the calculation in step S5 is completed, the motion control unit 13 performs the contact path calculation and priority assignment process in step S6 described above.

[0112] Then, the motion control unit 13 determines whether a contact path has been calculated (step S7). If it is "yes", the contact path with the highest priority among the calculated contact paths is extracted (step S8). During this extraction, the motion control unit 13 determines whether the multiple calculated contact paths can be combined into a path that simultaneously satisfies multiple contact conditions. If they can be combined, the priorities of the multiple contact paths can be added together and used as a contact path with the added result having an additional priority.

[0113] On the other hand, if the determination result of step S7 is "no", then the motion control unit 13 calculates any contact path that can make contact with the object (step S9). For example, the motion control unit 13 calculates a contact path from the initial region R0 ( Figure 3A The contact path from any point on the sphere toward the center.

[0114] If a contact path is determined, the movement control unit 13 moves the force sensor 22 along the contact path, thereby bringing the force sensor 22 (its contact point) into contact with the object (step S10). This contact is a weak contact that will not cause the object to move. Then, the inference unit 14 obtains information about the position of the contact point and the orientation of the surface from the force sensor 22 (step S11) and performs inference processing to infer the pose of the object (step S12). As described above, the inference processing is performed using a cost function G(r, t).

[0115] Then, if the inference result of the object's pose is obtained, the inference unit 14 determines whether the pose has been reduced to one (step S13). If it is "no", the process returns to step S3. On the other hand, if the pose has been reduced to one, the control unit 10 exits the loop process of inferring the pose (steps S3 to S13) and the process proceeds to the next step.

[0116] If the loop process is exited, the robot control unit 15 calculates the object's position and orientation, as well as the sequence of actions for the robot arm 24 to grasp the object, based on the inferred object's posture (parameters {r, t} representing rotation and translation) (step S14). Then, the robot control unit 15 drives the robot arm 24 according to the calculated sequence of actions to remove the object (step S15). Thus, one picking process is completed.

[0117] <The first case of posture inference processing>

[0118] Next, a specific example of the object orientation inference process performed in the above-described picking process will be explained. Figures 3A to 3D This is an explanatory diagram illustrating steps 1 through 4 of the first example of this deductive process. Figure 3EThis is an explanatory diagram showing the fourth process from another angle. In the first example, the object Q1 whose posture is inferred is a nut and it is placed on the worktable.

[0119] Figure 3A This indicates the initial stage where the force sensor 22 has not yet made contact with the object Q1. In this stage, in step S2, the inference unit 14 sets an initial region R0 based on the image, and in step S6, the movement control unit 13 determines the surface of the object Q1 with asymmetry based on the object Q1 information and calculates the contact path h1. In the case of a nut, the rotational symmetry around the central axis of the screw hole is below a threshold, and the outer peripheral surface of the nut is calculated as the surface with asymmetry. On the other hand, in the first example, the inference accuracy based on the image-based posture inference is low, so a path to contact the asymmetric surface cannot be calculated. In step S9, the movement control unit 13 calculates an arbitrary contact path h1. The arbitrarily selected contact path h1 can be a path from a point on the outer peripheral surface of the spherical initial region R0 towards the center of the initial region R0.

[0120] Figure 3B This indicates the stage where, after the force sensor 22 moves along the contact path h1 in steps S10-S12, the posture of object Q1 is inferred based on the first contact. Through this contact, the position of contact point p1 and the orientation (normal k1) of the surface s1 including contact point p1 are detected. The inference unit 14 then uses this detection information to narrow down the candidates for translation t and rotation r, making the cost function G(r, t) near zero. Figure 3B In the diagram, a double-dotted line represents the candidate range including the narrowed pose. The range includes all poses where the contact point p1 corresponds to a point on one of the faces of object Q1, and the face containing that point is perpendicular to the normal k1. The inference unit 14 identifies multiple poses occupying this range as candidates. Additionally, candidates whose poses extend beyond the initial region R0 can be excluded from the candidate poses of object Q1, but... Figures 3A-3E Examples that are not excluded are shown in the text.

[0121] Figure 3CThis indicates the stage in steps S4 to S8 where the contact path h2 for the second contact is calculated. In this stage, the movement control unit 13 calculates contact paths corresponding to the first contact condition, the second contact condition, and the third contact condition based on candidate postures of the object Q1 obtained from the previous contact (represented by double-dotted lines), and extracts the contact path h2, which has a higher priority. Contact path h2 is a path that travels along the direction intersecting the normal to the first contact surface s1. Therefore, it has a high probability of contacting a surface different from the first contact surface s1, and the probability of the contacting surface being the outer peripheral surface of the nut, which has an asymmetry, is higher than other contact paths c1. Furthermore, among the candidate postures of object Q1, surfaces that may intersect contact path h2 are scattered within a wide range X1, while surfaces that may intersect other contact paths c1 are scattered within a narrow range X2. Therefore, as contact path h2, the path with a high probability of contacting a surface with a high degree of positional uncertainty is calculated.

[0122] Figure 3D and Figure 3E This indicates the stage where, after the force sensor 22 moves along the contact path h2 in steps S10-S12, the posture of object Q1 is inferred based on the second contact. Through the second contact, the position of contact point p2 and the orientation (normal k2) of the surface s2 including contact point p2 are detected. The inference unit 14 narrows down the candidates for translation t and rotation r to make the cost function G(r, t) near zero based on the detection information obtained from the first and second contacts. Figure 3E As shown by the double-dotted line in the plan view, through the second contact, the number of contact points p1 and p2 is reduced to multiple pose candidates contained in the two surfaces s1 and s2 of object Q1, and the nut is offset in a direction parallel to both surfaces s1 and s2. Figure 3E (represented by a double-dotted line).

[0123] Thus, as the number of contacts increases, the candidates for the pose of object Q1 are narrowed down, and the inference unit 14 is eventually able to infer the pose of object Q1 as one.

[0124] <Example 2 of posture inference processing>

[0125] Figures 4A to 4F This is an explanatory diagram illustrating steps 1 to 6 of the second example of the inference process for inferring the posture of an object. In the second example, an example is shown where the object Q2 whose posture is inferred is a bolt.

[0126] exist Figure 4AIn the initial stage before the force sensor 22 contacts the object Q2, the inference unit 14 sets an initial region R0 based on the image (step S2), and the movement control unit 13 determines the asymmetric surface on the outer surface of the object Q2 based on the object Q2 information, and calculates the contact path h11 corresponding to the second contact condition (step S6). In the case of a bolt, the degree of symmetry of the surface symmetrical about the center plane that divides the bolt into two in the axial direction is below a threshold, and the side surface s21 of the bolt head and the side surface s11a of the bolt shaft end are determined as the asymmetric parts. On the other hand, in Figure 4A In the example, the initial inference accuracy of the pose of object Q2 based on the image is low, so the contact path h11 towards the side facing the center of the shaft that is detached from the asymmetric face is calculated.

[0127] exist Figure 4B In this process, the force sensor 22 moves along the contact path h11 (step S10), thereby performing detection based on the first contact (step S11) and inference of the pose of the object Q2 based on the detection (step S12). Through this contact, the position of the contact point p11 and the orientation (normal k11) of the surface s11 including the contact point p11 are detected. The inference unit 14 calculates the cost function G(r,t) based on this detection information and searches for multiple pose candidates for the object Q2. Here, as Figure 4B As shown by the double-dotted line, the candidates for the reduced pose include all poses of the bolt head side, head upper surface, shaft end face, or shaft side, as well as any one of these points, where the surface s11 and contact point p11 are the bolt head side. Additionally, candidates for the pose of object Q2 that exceed the initial region R0 can be excluded from the candidates for the pose of object Q2; however, in... Figures 4A to 4F The text shows examples that are not excluded.

[0128] exist Figure 4C In this process, the motion control unit 13 calculates the contact path h12 with higher priority based on the first to third contact conditions (steps S4 to S8). The contact path h12 is a path that achieves contact with a surface different from the surface s11 that made the first contact and with a surface with a high degree of positional uncertainty. For example, among the candidates for the posture of object Q2 (represented by double-dotted lines), the surfaces that may intersect with the contact path h12 are scattered above the length of the bolt's shaft. On the other hand, the surfaces that may intersect with other contact paths c12 are scattered within a range comparable to the width of the bolt head. The former has a larger dispersion value and a higher degree of uncertainty.

[0129] Then, as Figure 4DAs shown, by performing a second contact (step S10), the position of the contact point p12 and the orientation (normal k12) of the surface s12 including the contact point p12 are detected (step S11). Based on these detection results, the inference unit 14 further narrows down the candidates for the pose of object Q2 (step S12). Figure 4D As shown by the double-dotted line, through the second contact, the candidate is reduced to two postures with the bolt head and shaft end reversed. In addition, as long as the first contact is performed on the bolt head side s21 or shaft end side s11a which has asymmetry, the inference unit 14 can reduce the posture of object Q2 to one during the second contact stage.

[0130] exist Figure 4E In this process, the motion control unit 13 calculates a contact path h13 with a high probability of achieving the first to third contact conditions (steps S4 to S8). Among the candidates for the posture at this moment, the bolt head side s21 or the shaft end side s11a is a surface with a high degree of positional uncertainty, and both the bolt head side s21 and the shaft end side s11a are surfaces with asymmetrical parts. Therefore, contact path h13 becomes a path corresponding to the second and third contact conditions. Furthermore, contact path h13 becomes a path with the possibility of contacting a surface different from the first contact surface s11 and the second contact surface s12 (bolt head side s21), and also becomes a path corresponding to the first contact condition. Contact path h13 actually contacts the same surface as the first contact surface s11 (i.e., the shaft end side s11a), but for the candidates for the posture inferred at this stage, the probability of contacting the bolt head side s21 is approximately 50%.

[0131] Then, as Figure 4F As shown, by making a third contact (step S10), the position of the contact point p13 and the orientation of the surface s11a including the contact point p13 are detected (step S11), and the candidates for the pose of object Q2 are narrowed down to one based on these detection results (step S12).

[0132] As described above, in the picking device 1 according to the first embodiment, the inference unit 14 infers the posture of the object based on the detection information and object information obtained by the force sensor 22 making multiple contacts with the object. Therefore, even for objects whose posture is difficult to infer from images alone, such as objects covered by cloth, transparent or black objects, or objects with mirror surfaces, the posture of the object can be inferred.

[0133] Furthermore, according to the picking device 1 of the first embodiment, the movement control unit 13 calculates the contact path of the force sensor 22 so that the force sensor 22 makes multiple contacts with the object on different surfaces. Therefore, the inference unit 14 can use detection information based on fewer contacts to reduce the candidates for the object's posture to fewer candidates, thereby enabling efficient posture inference.

[0134] Furthermore, according to the picking device 1 of the first embodiment, information about the position of the contact point and information including the orientation of the surface of the contact point are obtained as detection information through the contact of the force sensor 22. Moreover, the movement control unit 13 calculates a contact path at the nth contact, causing the force sensor 22 to move along a direction intersecting the normal to the surface of the contact point of the previous nth contact. By using this method of determining the forward direction of the contact path, the movement control unit 13 can calculate the contact path of the force sensor 22, causing multiple contacts between the force sensor 22 and the object to occur on different surfaces of the object.

[0135] Furthermore, according to the picking device 1 of the first embodiment, the movement control unit 13 calculates from a plurality of candidates for the posture of the object being reduced at that moment a contact path (the contact path corresponding to the third contact condition) with a high probability of contact with a surface having a high degree of positional uncertainty. Therefore, the inference unit 14 can reduce the candidates for the object's posture to fewer candidates based on the detection information obtained by moving the force sensor 22 along the contact path. Thus, the inference unit 14 can reduce the object's posture to one using detection information based on fewer contacts, thereby achieving efficient posture inference.

[0136] Furthermore, according to the picking device 1 of the first embodiment, the movement control unit 13 calculates the contact path (the contact path corresponding to the second contact condition) of the surface contact with the asymmetrical part of the object. Therefore, the orientation of the object can be effectively determined based on the detection information of the asymmetrical surface of the object obtained by the force sensor 22 moving along the contact path. Therefore, the inference unit 14 can reduce the object's posture to one using detection information based on fewer contacts, thus enabling efficient posture inference.

[0137] Furthermore, the picking device 1 according to the first embodiment includes a camera unit 21 that acquires images of objects, and an inference unit 14 that infers the approximate position of the object based on the image acquired by the camera unit 21, making first contact possible. That is, the inference unit 14 infers the position where contact with the object can be made based on the image. Therefore, even when the position of the object is unclear over a wide range, the position for contact with the object can be inferred from the image, and the posture of the object can be inferred through contact. Thus, even when the position of the object is unclear over a wide range, the object can be picked up.

[0138] The first embodiment of the present invention has been described above. However, the present invention is not limited to the first embodiment described above. For example, in the first embodiment described above, a method for inferring the pose of an object using information from a point set M uniformly distributed on the outer surface of the object and a cost function G(r,t) has been described, but the invention is not limited to the above method. Any method can be used as long as the pose of the object can be inferred from information detected by contact. Furthermore, as described above, in the first embodiment described above, an example of inference processing for inferring the pose without considering the presence of elements that restrict the pose of the object, such as a workbench on which the object is placed. However, if elements that restrict the pose of the object exist, processing such as excluding candidates for poses that do not match those elements can be added.

[0139] Furthermore, in the first embodiment described above, an example is shown where the object information storage unit 11 (a storage unit that stores object information representing the shape and size of an object) is located within the device. However, the object information storage unit 11 may also be configured to be installed on a server computer connected via a communication network and to send object information to the device via communication. Also, in the first embodiment described above, an example is shown where the present invention is applied to a picking device 1 for retrieving objects from shelves or the like. However, the information processing device of the present invention may also be applied to various devices for inferring the posture of objects without the control and action of retrieving the object. Furthermore, the detailed structure shown in the embodiments can be appropriately modified without departing from the spirit of the present invention.

[0140] (Second Implementation)

[0141] When automating human-operated tasks using robotic arms such as industrial or collaborative robots, it is necessary to teach the robotic arm the movements it should perform. The robotic arm learns by imitation based on the taught movements. For example, a deep learning model can be constructed based on the taught movements (see Japanese Patent Application Publication No. 2020-110920, etc.). The robotic arm determines the motion plan of the movable arm (e.g., the motion path of the movable arm's tip) based on the constructed deep learning model and performs the movements of the movable arm according to the determined motion path. Thus, the taught movements can be reproduced.

[0142] If unexpected situations occur during teaching, there is a possibility that the movable arm may perform unexpected movements. For example, if the movable arm stops abruptly for some reason and needs to restart from the stopped position, there is a possibility that the movable arm may perform unexpected movements. For example, if the motion data before the movable arm stopped is lost and it starts moving from a state different from the starting point at the time of restarting, or if it performs a reset at the origin before stopping the motion, it may sometimes move along an unexpected path. Furthermore, when the movement path of the movable arm of a robot is generated through machine learning such as imitation learning, if an input different from the input shown in the teaching is applied, an unexpected movement path may sometimes be generated. For example, when an image of an object C, which is different from the object A shown in the teaching, is input to a robot that has learned to generate a movement path B based on an image including object A, the movable arm may sometimes move along an unexpected path (e.g., see S. Levine et al., End-to-End Trading of Deep Visuomotor Policies, 2016). Even if the movable boom makes an unexpected movement, to avoid the risk of collision with workers or other equipment, it is preferable to designate the area where the movable boom might move as a restricted area. Furthermore, it is preferable not to install any other equipment within this area. This results in a decrease in the efficiency of space utilization within the factory.

[0143] In the second embodiment, a control device and system for a robotic arm are provided that do not lead to a decrease in space utilization efficiency and can avoid danger even if unexpected situations occur during teaching.

[0144] refer to Figures 5 to 9B The control device for the robotic arm based on the second embodiment of the present invention will be described. The control device for the robotic arm is an example of the information processing device involved in the present invention.

[0145] Figure 5 This is a block diagram illustrating the function of the control device 110 based on the second embodiment. The control device 110 based on the second embodiment controls the movement of the movable arm 120. The movable arm 120 is, for example, a multi-joint robotic arm with six degrees of freedom, which performs a picking action of taking a workpiece (e.g., a bolt) from a receiving part and moving it to a predetermined position. The robotic arm also includes an input unit 121, a display unit 122, and a camera 123.

[0146] The control device 110 includes a trajectory information acquisition unit 111, an imitation learning unit 112, an allowable range determination unit 113, a movement range limiting unit 114, an arm control unit 115, and an interface unit 116. The functions of the trajectory information acquisition unit 111, the imitation learning unit 112, the allowable range determination unit 113, the movement range limiting unit 114, and the arm control unit 115 are implemented, for example, by executing a program by a central processing unit (CPU). The interface unit 116 has the function of inputting and outputting data or commands between the movable arm 120, the input unit 121, the display unit 122, and the camera 123 and the control device 110.

[0147] Motors are installed at each joint of the movable arm 120. The control device 110 drives the motors at the joints, thereby moving the movable arm 120. Furthermore, an encoder for detecting the rotation angle of the motor and a torque sensor for detecting the torque generated at the joint are installed at each joint. The detection results from the encoder and the torque sensor are input to the control device 110.

[0148] An end effector (acting part) 120A is mounted on the front end of the movable arm 120. The end effector 120A is capable of holding the workpiece. Alternatively, machining tools or the like can be mounted as the end effector 120A.

[0149] The input unit 121 is an operating device used by workers to operate the movable arm 120 during teaching operations, and is also called a teach pendant. Workers can also directly operate the movable arm 120 to perform teaching operations. This teaching method is called direct teaching. In addition to operating devices, the input unit 121 may also include a keyboard, a positioning device, etc.

[0150] Under the control of the control device 110, the display unit 122 graphically displays the information shown, the permissible range of movement of the movable arm 120, etc.

[0151] Camera 123 takes pictures of the movable arm 120 and the receiving part of the object to be picked up (i.e., the workpiece). The captured image data is input to the control device 110.

[0152] Next, the functions of each part of the control device 110 will be explained.

[0153] The trajectory information acquisition unit 111 acquires trajectory information representing the movement trajectory of the movable arm 120 provided through teaching. The acquired trajectory information includes information representing the movement trajectories of multiple joints and end effectors 120A. Teaching of the motion can be performed, for example, by a worker directly operating the movable arm 120 (direct teaching method), by teaching the motion of each axis of the movable arm 120 one by one, or by remote operation. The positions of each joint of the movable arm 120 or end effector 120A during teaching can be determined based on the detection results of angle sensors installed at each joint.

[0154] The imitation learning unit 112 associates the image of the workpiece captured by the camera device with the movement trajectory of the movable arm 120 provided through teaching, stores the association, and performs machine learning. For example, deep learning methods using neural networks can be used for machine learning. The imitation learning unit 112 stores the machine learning results as teaching data. Alternatively, instead of the image of the workpiece captured by the camera device, the detection results of the workpiece based on various sensors such as LiDAR scanners, distance cameras, or millimeter-wave sensors can be associated with the movement trajectory of the movable arm 120 provided through teaching, stored, and then used for machine learning.

[0155] The allowable range determination unit 113 determines the allowable range of movement of the movable arm 120 based on the trajectory information acquired by the trajectory information acquisition unit 111. The allowable range of movement is determined, for example, based on the convex hull of the set of movement trajectories of multiple joints and end effectors 120A provided during teaching. The convex hull refers to the smallest convex set including the provided set. In the second embodiment, the "provided set" corresponds to the set of points representing the trajectories of the multiple joints and end effectors 120A of the movable arm 120. The allowable range determination unit 113 automatically determines the allowable range of movement using an algorithm that calculates the convex hull containing the set of multiple movement trajectories. Here, "automatically determined" means determined without user intervention.

[0156] As an example, the allowable movement range is determined in a manner consistent with the convex hull of the movement trajectory of the movable arm 120. Alternatively, the allowable movement range can be defined as including the convex hull of the movement trajectory of the movable arm 120. For example, the surface of the convex hull after moving outward by a predetermined distance can be used as the surface of the allowable movement range. Furthermore, the allowable movement range can also be defined by at least one basic graphic configured to include the set of movement trajectories provided in the teaching. Basic graphics include, for example, cubes, cuboids, spheres, cylinders, etc. The allowable movement range determination unit 113 automatically determines the allowable movement range using an algorithm that determines the allowable movement range consisting of at least one basic graphic including the set from a set of multiple movement trajectories.

[0157] Figure 6A This is a floor plan illustrating an example of the permitted movement range. Figure 6B This is a cross-sectional view including the rotation center axis of the movable arm 120. The base of the movable arm 120 is mounted on the base 125. The movable arm 120 rotates about the rotation center axis fixed to the base 125. Moreover, the movable arm 120 extends and retracts with respect to the base 125, thereby changing its posture in three-dimensional space. Figure 6A The curve in the figure represents an example of the movement trajectory 131 of the end effector 120A during teaching.

[0158] The permissible range of motion 130 is determined based on the convex hull of the movement trajectory 131 of the end effector 120A during teaching and the movement trajectories of other joints. For example, as... Figure 6A As shown, the shape when viewed from above within a 130° range of motion is a fan-shaped structure with a central angle near the center of rotation. (As shown...) Figure 6B As shown, the shape of the vertical cross-section is, for example, a pentagon formed by cutting one corner of a rectangle with sides parallel to the horizontal and vertical directions into a triangle. The corner cut into the triangle is located diagonally opposite the base of the movable arm 120. Furthermore, Figure 6A and Figure 6B The shape of the allowable movement range 130 shown is an example, and the allowable movement range 130 can take various other shapes.

[0159] Permissible range determination section 113 ( Figure 5 The determined permissible range of motion 130 and the movable arm 120 are graphically displayed on the display unit 122 in a manner that allows identification of their positional relationship. Operators can view the display unit 122 to obtain information related to the position, shape, and size of the currently set permissible range of motion 130.

[0160] The movement range limiting unit 114 determines the movement path of the movable arm 120 based on image data acquired by the camera 123 and the results of imitation learning. The arm control unit 115 causes the movable arm 120 to move along the movement path determined by the movement range limiting unit 114. More specifically, the rotation angle of the motors driving each joint of the movable arm 120 is calculated based on the coordinates on the movement path, and each motor is driven.

[0161] Figure 7 This is a flowchart illustrating the processes performed by the movement range limiting unit 114 and the arm control unit 115. First, the movement range limiting unit 114 acquires image data of the workpiece captured by the camera 123 (step SS1). Based on the acquired image data and the data obtained by the imitation learning unit 112 (…),… Figure 5 The learning model constructed determines the candidates for the movement path of the movable boom 120 (step SS2).

[0162] The movement range restriction unit 114 determines whether the candidate movement path falls within the allowable range determination unit 113. Figure 5 Within the determined permissible movement range of 130 ( Figure 6A and Figure 6B (Step SS3). As an example, the proposed alternative movement path is the movement path of the end effector 120A. If all positions of the end effector 120A and the plurality of joints fall within the allowable movement range 130 when the end effector 120A moves along this movement path, it is determined that the alternative movement path falls within the allowable movement range 130. If at least one position of the end effector 120A and the plurality of joints exceeds the allowable movement range 130, it is determined that the alternative movement path does not fall within the allowable movement range 130.

[0163] If no candidate for a movement path falls within the allowable movement range 130, it is determined whether there are other candidates for movement paths (step SS4). If there are other candidates for movement paths, they are used as candidates for movement paths (step SS2). If there are no other candidates for movement paths, an error is reported and processing stops (step SS5). That is, the movement range limiting unit 114 has the function of determining the movement path of the movable arm 120 while limiting the movement range of the movable arm 120 within the allowable movement range 130. Errors are reported, for example, by displaying error information on the display unit 122.

[0164] If, in step SS3, a candidate for a movement path is determined to fall within the allowable movement range 130, the arm control unit 115 ( Figure 5 The movable arm 120 is moved according to the candidate movement path (step SS6). This removes the workpiece from the receiving area and moves it to the designated position. If any unprocessed workpiece remains, the process from step SS1 is repeated (step SS7). If no unprocessed workpiece remains, the process ends.

[0165] Next, refer to Figures 8A to 9B The superior effects of the second embodiment described above will be explained.

[0166] Figure 8A This is a plan view showing an example of a candidate 132 representing the movement path of the end effector 120A of the movable boom 120. Figure 8BThis is a cross-sectional view including the rotation center axis of the movable arm 120. When the preconditions for determining the candidate movement path differ significantly from the conditions anticipated during teaching, the candidate movement path 132 may deviate considerably from the intended path, potentially resulting in the movement not falling within the allowable movement range 130. In this case, in the second embodiment described above, the movable arm 120 will not move along the candidate movement path 132. In other words, it is possible to prevent the movable arm 120 from moving outside the allowable movement range 130. Therefore, by prohibiting personnel from entering the allowable movement range 130, collisions between the movable arm 120 and personnel can be avoided.

[0167] Figure 9A This is an example of the total range 133 that the movable arm 120 can move, compared with the allowable range 130 of movement determined by the method based on the second embodiment. Figure 6A and Figure 6B The floor plans were compared. Figure 9B This is a cross-sectional view including the rotation center axis of the movable arm 120. A roughly hemispherical region centered on the base of the movable arm 120 and with the longest length of the movable arm 120 as its radius corresponds to the total range 133 of movement of the movable arm 120. Even without a function limiting the range of movement of the movable arm 120, the movable arm 120 can move freely within its range of motion. Figure 8A and Figure 8B In the event of unforeseen actions, to avoid collisions between workers and the movable arm 120, it is preferable to pre-define the total movable range 133 as a prohibited area. Furthermore, it is preferable to keep the robotic arm away from obstacles, for example, by not placing any obstacles (e.g., walls, ceilings, etc.) within the total movable range 133.

[0168] In contrast, in the second embodiment described above, the range of motion of the movable arm 120 is limited to a permissible range of motion 130. The permissible range of motion 130 is greater than the total range of motion 133 that the movable arm 120 can move. Figure 9A and Figure 9B The space is narrow. Therefore, it is possible to reduce the area that is off-limits to workers. Moreover, even within the total movable range 133, obstacles can be placed outside the permitted movement range 130. In other words, the robot can be placed close to obstacles. For example, even if the ceiling at a certain location in the factory is lower than the top of the total movable range 133 of the movable arm 120, as long as it is higher than the permitted movement range 130, the robot can be placed at that location. Therefore, space within the factory can be utilized effectively.

[0169] Furthermore, in the second embodiment described above, as long as the movement allowable range 130 is fixedly set as a prohibited area, it is not necessary to configure sensors or the like for detecting staff entry near the movable boom 120 to avoid collisions between staff and the movable boom 120.

[0170] If at least one basic graphic is used to define the allowable movement range 130, it is possible to reduce whether the candidate movement path in step SS3 falls within the allowable movement range 130. Figure 6A and Figure 6B The computational cost when determining ).

[0171] Next, refer to Figure 10 The control device for a robotic arm based on another embodiment will be described below. References will be omitted hereafter. Figures 5 to 9B The description follows the same structure as the second embodiment. In this embodiment, the user can modify the allowable range determination unit 113 ( Figure 5 The defined allowable movement range is 130 ( Figure 6A and Figure 6B ).

[0172] Figure 10 This indicates that it is displayed on display unit 122 ( Figure 5 The image in the diagram. The permissible range determination unit 113, based on the movement trajectory of the movable arm 120 during teaching, displays the permissible movement range 130A determined at the current moment and the movable arm 120 on the display unit 122 in a manner that allows for understanding the positional relationship between the two. For example, it displays the positional relationship between the permissible movement range 130A and the movable arm 120 when viewed from above. Figure 10 (Left side diagram) and positional relationships in the vertical section ( Figure 10 (Right side image).

[0173] The user can operate the input unit 121, such as the positioning device, to correct the allowable movement range 130A, thereby resetting the corrected allowable movement range 130B. This correction can be performed, for example, by dragging the outer perimeter of the allowable movement range 130A. Figure 10 The example shown is an example of stretching the uncorrected allowable range of movement 130A along a radial direction centered on the axis of rotation.

[0174] If the user modifies the allowed movement range 130A and then clicks or touches the "Confirm Allowed Range" button, the allowed range determination unit 113 will set the modified allowed movement range 130B as the new allowed movement range 130. If the user clicks or touches the "Back" button, the allowed range determination unit 113 will not modify the allowed movement range 130A and will end the modification process.

[0175] Next, the superior effects of this embodiment will be explained.

[0176] In this embodiment, the user can adjust the permissible movement range 130A determined based on the movement trajectory of the movable arm 120 during teaching, according to the surrounding conditions of the movable arm 120. For example, if an area wider than the original permissible movement range 130A is to be designated as a prohibited area, the wider range than the permissible movement range 130A determined based on the movement trajectory of the movable arm 120 during teaching can be set as the permissible movement range 130B. Furthermore, if the ceiling of the factory where the movable arm 120 is installed is sufficiently high that even if the movable arm 120 is extended upwards to its maximum extent, it will not reach the ceiling, the height restriction of the permissible movement range 130B can be lifted.

[0177] If the allowable movement range of 130 is widened, then in step SS3 ( Figure 7 The probability that a candidate movement path falls within the allowed movement range of 130 increases during the decision-making process. This reduces the frequency of re-searching for candidate movement paths and significantly reduces the frequency of errors.

[0178] Conversely, if other devices are newly installed near the movable arm 120, the permissible range of movement 130 can be narrowed so that the movable arm 120 does not collide with the newly installed devices.

[0179] Next, a variation of the second embodiment described above will be explained.

[0180] In the second embodiment described above, the controlled object is the movable arm 120 of the collaborative robotic arm. Figure 5 However, other movable booms can also be used as the controlled objects. For example, the boom, stick, and auxiliary devices (actuators) of an automatically operated excavator can be used as the controlled objects. In this case, the boom, stick, and auxiliary devices are equivalent to the movable boom 120 of the second embodiment described above.

[0181] Next, refer to Figure 11 A system for controlling a robotic arm based on yet another embodiment will be described. Hereinafter, references to other systems will be omitted. Figures 5-10 The description is of the same structure as the implementation method described.

[0182] Figure 11 This is a block diagram of a system based on this embodiment. The system based on this embodiment includes: multiple robotic arms 140, each containing a movable arm 120 and a control device 110; a network 160 such as a LAN; and a control server 150. The multiple robotic arms 140 are connected to the control server 150 via the network 160. Figure 5In the illustrated embodiment, a portion of the functions of the control device 110 are implemented by the control server 150. Both the control device 110 and the control server 150 have the function of sending and receiving various instructions or data via the network 160.

[0183] The control devices 110 of the robotic arm 140 include a trajectory information acquisition unit 111, an arm control unit 115, and an interface unit 116. The functions of the trajectory information acquisition unit 111, the arm control unit 115, and the interface unit 116 are respectively related to... Figure 5 The trajectory information acquisition unit 111, arm control unit 115, and interface unit 116 of the control device 110 in the illustrated embodiment have the same functions.

[0184] The control server 150 includes a trajectory information receiving unit 151, an imitation learning unit 152, an allowable range determination unit 153, a movement range restriction unit 154, a movement path sending unit 155, and a display unit 156. The trajectory information receiving unit 151 receives trajectory information representing the movement trajectory during teaching, acquired by the trajectory information acquisition unit 111 of the robot arm 140, via a network 160. The functions of the imitation learning unit 152, the allowable range determination unit 153, the movement range restriction unit 154, and the display unit 156 are respectively related to… Figure 5 The control device 110 shown has the same functions as the imitation learning unit 112, the allowable range determination unit 113, the movement range restriction unit 114, and the display unit 122 of the robot arm 140.

[0185] That is, the movement range restriction unit 154 performs. Figure 7 The flowchart shown outlines steps SS1 to SS5 in sequence. In step SS1, image data captured by the camera 123 of the robotic arm 140 is acquired. If, in step SS3, it is determined that a candidate movement path falls within the permissible movement range, the movement path sending unit 155 sends information indicating the candidate movement path to the arm control unit 115 of the control device 110 via the network 160. The arm control unit 115 controls the movable arm 120 based on the received information indicating the candidate movement path.

[0186] Next, regarding Figure 11 The superior effects of the illustrated embodiments will be explained.

[0187] exist Figure 11 In the embodiments shown, it is also consistent with Figures 5 to 9B Similarly, the implementation shown can effectively utilize the space within the factory.

[0188] Next, variations of the above-described embodiments will be described. The control server 150 can also replace the one based on... Figure 11The system of the illustrated embodiment utilizes a robotic arm 140 to construct a system for controlling an automatically operated excavator. That is, the control server 150 can be used to control construction machinery such as the robotic arm 140 and the automatically operated excavator. Furthermore, the functional sharing between the control device 110 and the control server 150 is not limited to... Figure 11 The implementation shown is as follows. For example, a portion of the functions of the control device 110 can be implemented by the control server 150, or a portion of the functions of the control server 150 can be implemented by the control device 110.

[0189] The above embodiments are examples; naturally, parts of the structure shown in different embodiments can be substituted or combined. The same effects based on the same structure in multiple embodiments are not mentioned one by one in each embodiment. Furthermore, the present invention is not limited to the above embodiments. For example, various changes, improvements, combinations, etc., can be made, which will be obvious to those skilled in the art.

[0190] (Summary of the second embodiment)

[0191] The outline of the apparatus and system of the second embodiment is as follows.

[0192] [Summary of the Second Embodiment 1]

[0193] A control device comprising:

[0194] The trajectory information acquisition unit acquires trajectory information representing the movement trajectory of the movable arm provided through teaching;

[0195] The allowable range determination unit determines the allowable range of movement of the movable boom based on the trajectory information acquired by the trajectory information acquisition unit; and

[0196] The movement range limiting unit restricts the movement range of the movable arm to the allowable movement range determined by the allowable range determining unit.

[0197] [Summary of the Second Embodiment 2]

[0198] According to the control device described in Summary 1 of the second embodiment, wherein,

[0199] The movable arm includes at least one joint and an action part located at the front end.

[0200] The allowable range determination unit determines the allowable range of movement based on the set of movement trajectories of the respective action part and the joint.

[0201] [Summary of the Second Embodiment 3]

[0202] The control device according to the summary 1 or 2 of the second embodiment further includes a display unit.

[0203] The allowable range determination unit displays information indicating the determined allowable movement range on the display unit.

[0204] [Summary of the Second Embodiment 4]

[0205] The control device according to any one of the summaries 1 to 3 of the second embodiment further includes an input section for user operation.

[0206] If the user operates the input unit and corrects the movement allowable range determined at the current moment, the allowable range determination unit will redetermine the corrected range as the movement allowable range.

[0207] [Summary of the Second Embodiment 5]

[0208] The control device according to any one of the summaries 1 to 4 of the second embodiment, wherein,

[0209] The allowable range determination unit determines the allowable movement range based on the convex hull of the movement trajectory provided by teaching, or by using at least one basic graphic containing the movement trajectory provided by teaching.

[0210] [Summary of the Second Embodiment 6]

[0211] According to the control device described in Summary 5 of the second embodiment, wherein,

[0212] The permissible range determination unit determines the permissible movement range without user intervention.

[0213] [Summary of the Second Embodiment 7]

[0214] The control device according to any one of the summaries 1 to 6 of the second embodiment further includes an imitation learning unit that performs machine learning after establishing a correlation between the detection results of the sensor based on the detected object and the movement trajectory of the movable arm provided through teaching.

[0215] [Summary of the Second Embodiment 8]

[0216] A system that possesses:

[0217] The trajectory information receiving unit receives trajectory information from the construction machinery, which represents the movement trajectory of the movable boom provided through teaching.

[0218] The allowable range determination unit determines the allowable range of movement of the movable arm based on the trajectory information received by the trajectory information receiving unit;

[0219] The movement range limiting unit determines the movement path of the movable arm while limiting its movement range to the allowable movement range determined by the allowable range determining unit; and

[0220] The movement path sending unit sends information representing the movement path determined by the movement range limiting unit to the construction machinery.

[0221] (Third Implementation)

[0222] Japanese Patent Application Publication No. 2017-136677 discloses a system for identifying the posture of an object and performing picking based on an image of the object and a measurement of the contact position with the object. In the system disclosed in the aforementioned publication, by simultaneously using an image of the object and measurement information based on the contact with the object, the position of an object wrapped in a plastic bag or cushioning material can be identified.

[0223] Japanese Patent Application Publication No. 2020-82322 discloses a picking device that uses a simulator to generate a learning dataset for machine learning and uses the learning dataset to perform machine learning.

[0224] The data input into the learning model used in actual picking operations is collected from real-world contact with objects in a real environment. It is difficult to reproduce, in a training dataset generated using a simulator, the friction and other factors that occur during contact with objects in a real-world environment. Therefore, it is difficult to improve the accuracy of the learning model.

[0225] In the third embodiment, a picking device and a learning device are provided that can improve the accuracy of the learning model.

[0226] refer to Figures 12-21 The picking device based on the third embodiment of the present invention will be described.

[0227] Figure 12 This is a block diagram of a picking device based on the third embodiment. The picking device based on the third embodiment includes a control device 210, a picking mechanism 240, an output device 251, an input device 252, and a camera device 253. The picking mechanism 240 includes a multi-joint robotic arm 241, a gripping part 242 mounted at its front end, and a force sensor 243. Alternatively, the force sensor 243 can also be mounted on a different robotic arm than the multi-joint robotic arm 241 that moves the gripping part 242. The combination of the control device 210 and the force sensor 243 corresponds to an example of the information processing device according to the present invention.

[0228] The camera device 253 captures images of the selected object to obtain image data. The approximate location of the object can be determined based on this image data. The image data captured by the camera device 253 is input to the control device 210. Alternatively, a device that acquires candidate information for determining the position or orientation of the object in a non-contact manner can be used instead of the camera device 253. For example, a depth sensor that acquires two-dimensional images and obtains depth information from the images can be used.

[0229] Input device 252 is used to input various data or instructions to control device 210, such as through a keyboard, pointing device, touch panel, communication device, or removable media reader. Data or instructions input to input device 252 are then input to control device 210. Output device 251 is used to output various data, images, notification information, etc., under the control of control device 210, such as through a display, communication device, or removable media writer.

[0230] Under the control of the control device 210, the multi-joint robotic arm 241 can support the gripping part 242 in any posture and move the gripping part 242 along any path. The gripping part 242 performs actions of grasping and releasing objects under the control of the control device 210. Furthermore, under the control of the control device 210, the multi-joint robotic arm 241 can support the force sensor 243 in any posture and move the force sensor 243 along any path.

[0231] The force sensor 243 has a contact point that acquires information about the reaction force from the contact point when it comes into contact with the surface of an object. The contact information includes information determining the position of the contact point and the direction of the reaction force from that point. The direction of the reaction force is approximately equal to the normal direction of the surface in contact with the contact point. The force sensor 243 stops when subjected to a very small reaction force, thus acquiring contact information without moving the object. The contact information can be used as basic information to infer the position and orientation of the object.

[0232] The control device 210 is a computer equipped with a central processing unit (CPU), RAM, non-volatile memory, and an interface unit. The non-volatile memory stores programs for execution by the CPU. The various functions of the control device 210, described later, are implemented by the CPU executing these programs.

[0233] Next, the various functions of the control device 210 will be explained.

[0234] [Shape Definition Data Acquisition Department]

[0235] The shape definition data acquisition unit 211 acquires shape definition data of the object that is defined as the selected object. The shape definition data is, for example, CAD data, which is input from the input device 252. The shape definition data acquisition unit 211 stores the acquired shape definition data in RAM.

[0236] Figure 13A A CAD model is a drawing on a plane that represents the shape of a hexagonal nut, defined as an example of an object. The surface of the hexagonal nut is represented by multiple triangular features. The three-dimensional shape defined by CAD data is called a CAD model.

[0237] [Pre-Study Department]

[0238] Pre-learning Department 212 Figure 12 The position and pose inference learning model 231 is learned using a simulator. Specifically, the position and pose inference learning model 231 is learned based on the shape definition data acquired by the shape definition data acquisition unit 211. If contact information is input from the force sensor 243, the position and pose inference learning model 231 outputs the result of inferring the position and pose of the object. The function of the pre-learning unit 212 will be explained below.

[0239] The pre-learning unit 212 defines multiple reference points p located on the surface of the CAD model based on the shape definition data. For example, thousands of reference points p are defined.

[0240] Figure 13B This is a diagram that displays multiple reference points p defined on the surface of the CAD model on a plane. Reference points p are represented by black dots. The pre-learning unit 212 extracts representative reference points p_i from the multiple reference points p. Here, i is the sequence number labeled on the representative reference point. For example, there are approximately 5 to 210 representative reference points p_i.

[0241] Figure 14 This is a schematic diagram representing multiple representative reference points p_i. The pre-learning unit 212 calculates the normal vector f_i of the surface to which each representative reference point p_i belongs. A data set {p_i, f_i | i = 1, 2, ..., k} including multiple representative reference points p_i and normal vectors f_i can be obtained. Here, k is the number of representative reference points p_i. The obtained data set is then transformed using vectors representing translation t and rotation r. The position and pose of the object are determined by translation t and rotation r. In this specification, the vectors defining translation t and rotation r are referred to as the position and pose vectors {t, r}.

[0242] Figure 15This is a schematic diagram illustrating an example of coordinate transformation used in pre-learning. By transforming the representative reference point p_i and the normal vector f_i using the position and pose vector defined by translation t and rotation r, the transformed representative reference point p'_i and the transformed normal vector f'_i can be obtained. The transformed representative reference point p'_i and the transformed normal vector f'_i are represented, for example, by a manipulator coordinate system based on the base of the multi-joint manipulator 241.

[0243] Pre-learning Department 212 Figure 12 The transformed dataset {p'_i, f'_i | i = 1, 2, ..., k} and the position / pose vector {t, r} defined by translation t and rotation r used for the transformation are used as a training dataset to generate multiple training datasets. In this case, the position / pose vector {t, r} is randomly generated. As an example, the number of generated training datasets is set to approximately one million.

[0244] Pre-learning is performed using multiple generated learning datasets. For example, a position and pose inference learning model 231 is learned by taking the transformed dataset {p'_i, f'_i|i=1,2,……k} as input and the position and pose vector {t, r} as output. Specifically, the parameters of the neural network of the position and pose inference learning model 231 are determined. Each element of the transformed dataset {p'_i, f'_i|i=1,2,……k} corresponds to the contact position and the direction of the reaction force obtained from the contact information obtained by the force sensor 243.

[0245] [Position and Posture Candidate Output Department]

[0246] The position and pose candidate output unit 213 acquires image data of the object captured by the camera device 253 and analyzes the image data to output multiple candidates for the position and pose of the object. Each of the multiple candidates for position and pose is determined by a position and pose vector {t, r}. The output multiple candidates are stored in RAM or the like.

[0247] [Contact Path Determination Section]

[0248] The contact path determination unit 214 uses multiple candidates output by the position and posture candidate output unit 213 as input to the contact path determination learning model 232 to determine the contact path of the force sensor 243 for making the force sensor 243 contact the object. The contact path includes information specifying the path for moving the force sensor 243 and the posture of the force sensor 243.

[0249] While the learning of the contact path determination learning model 232 is not yet complete, the contact path determination unit 214, for example, finds multiple candidate minimum bounding spheres of the position and posture of the object, and determines multiple paths toward the center of the minimum bounding sphere as contact paths.

[0250] [Contact Information Acquisition Department]

[0251] The contact information acquisition unit 215 acquires contact information when the force sensor 243 comes into contact with an object. The contact information includes information about the contact position and information indicating the direction of the reaction force received by the force sensor 243 from the object. The action that causes the force sensor 243 to come into contact with the object is called a contact action.

[0252] [Position and Posture Inference Department]

[0253] The position and pose inference unit 216 uses multiple contact information acquired by the contact information acquisition unit 215 through multiple contact actions as input to the position and pose inference learning model 231 and infers the position and pose of the object.

[0254] [Picking Control Department]

[0255] The picking control unit 217 controls the picking mechanism 240 based on the position and posture of the object inferred by the position and posture inference unit 216, thereby picking the object using the gripping unit 242. Furthermore, the picking control unit 217 outputs information indicating whether the object was successfully picked. The success or failure of the picking can be determined based on the gripping unit 242 after the picking action. Figure 12 The relative positional relationship of the multiple jaws is used to determine the position.

[0256] [Additional Learning Section for Position and Pose Deduction]

[0257] The position and pose inference additional learning unit 218 uses information obtained through actual picking actions to perform additional learning on the position and pose inference learning model 231. Whether to perform additional learning in actual applications can be indicated by the user.

[0258] Figure 16 This is a flowchart illustrating the sequence of additional learning performed by the position and pose inference additional learning unit 218. The position and pose inference additional learning unit 218 determines whether the picking action is successful (step SA1). If the picking action fails, no additional learning is performed on the position and pose inference learning model 231. If the picking action is successful, the multiple contact information used by the position and pose inference unit 216 as input to the position and pose inference learning model 231, and the position and pose vector {t, r} output when that contact information is input, are stored as a new learning dataset (step SA2).

[0259] If the amount of the stored learning dataset is less than the baseline value, the position and pose inference supplementary learning ends (step SA3). If the amount of the stored learning dataset exceeds the baseline value, the position and pose inference learning model 231 is supplemented by learning the contact information of the stored learning dataset as input and the position and pose vector {t, r} as output.

[0260] [Contact Path Determination Reinforcement Learning Department]

[0261] Contact path determination reinforcement learning department 219 ( Figure 12 Reinforcement learning is then performed on the contact path determination learning model 232. Specifically, the parameters of the neural network for the contact path determination learning model 232 are generated. The following refers to... Figure 17 The sequence of reinforcement learning performed by the reinforcement learning unit 219 for determining the contact path is explained.

[0262] Figure 17 This is a flowchart illustrating the sequence of reinforcement learning performed by the contact path determination reinforcement learning unit 219. First, the contact path determination reinforcement learning unit 219 obtains multiple candidates for the position and orientation of the object from the position and orientation candidate output unit 213 (step SB1). Then, it determines an arbitrary contact path based on these multiple candidates (step SB2). For example, it finds the smallest bounding sphere of the region where multiple candidates can exist, and determines an arbitrary path toward the center of this smallest bounding sphere as the contact path.

[0263] If a contact path is determined, the force sensor 243 is moved along that contact path to perform a contact action (step SB3). Based on the result of the contact action, candidates are narrowed down, and a reward is calculated (step SB4). For example, if a candidate in terms of position and posture is located in an area that the force sensor 243 did not contact during the contact action, that candidate is discarded. Furthermore, candidates located outside a sphere centered on the contact point and with the maximum size of the object as its radius are also discarded. The number of candidates discarded through a single contact action is taken as the reward.

[0264] Steps SB2 to SB4 are repeated a predetermined number of times (step SB5). Then, the contact path determination learning model 232 is learned by taking the multiple candidate sets obtained in step SB1 as input and multiple contact paths with increased total rewards as output (step SB6). That is, the contact path determination reinforcement learning unit 219 performs reinforcement learning by setting the multiple candidate sets of the object's position and posture as "states," the contact actions based on multiple contact paths as "behaviors," and the reduction in the number of candidate positions and postures of the object as "rewards."

[0265] [Contact Path Determination and Imitation Learning Department]

[0266] The contact path determination imitation learning unit 220 performs imitation learning of the contact path determination learning model 232. (See below for reference.) Figure 18 and Figure 19 The sequence of imitation learning performed by the contact path determination imitation learning unit 220 will be explained.

[0267] Figure 18 This is a flowchart illustrating the sequence of imitation learning performed by the contact path determination imitation learning unit 220. First, the contact path determination imitation learning unit 220 acquires multiple candidates for the position and posture of the object, as well as image data that serves as the source of the output candidate, from the position and posture candidate output unit 213 (step SC1). After acquiring this data, the graphics and images of the multiple candidates are overlaid and displayed on the output device 251 (step SC2).

[0268] Figure 19 This diagram illustrates an example of graphics and images output to the output device 251. An actual image 260 of the object is displayed, and multiple candidates 261 are shown in a dashed line overlapping the image 260. Furthermore, a contact path specifying arrow 262 is displayed to specify the contact path of the force sensor 243. The user specifies the contact path by moving the pointer 263 to adjust the position and direction of the contact path specifying arrow 262. Additionally, by operating the pointer 263, the image 260, the graphics representing the multiple candidates 261, and the contact path specifying arrow 262 can be rotated three-dimensionally. Once the contact path specifying arrow 262 has been adjusted, the user clicks or touches the confirmation button 264 to specify the contact path (step SC3).

[0269] If a contact path is specified, the contact path determination imitation learning unit 220 performs imitation learning of the contact path determination learning model 232 based on multiple candidates of the position and posture of the object and the contact path specified by the user (step SC4).

[0270] [Position and Pose Inference Learning Model Evaluation Department]

[0271] Evaluation of Position and Pose Inference Learning Model 221 Figure 12 The position and pose inference learning model 231 is evaluated. Specifically, the position and pose inference unit 216 uses the position and pose inference learning model 231 to infer the position and pose of the object, and calculates the frequency of successful or unsuccessful picking when the picking control unit 217 performs a picking action based on the inference results. If the frequency of picking failures exceeds a reference value, the user is notified to perform additional learning based on the position and pose inference additional learning unit 218. This notification is, for example, made by displaying information on the output device 251.

[0272] [Contact Path Determination Learning Model Evaluation Department]

[0273] The contact path determination learning model evaluation unit 222 evaluates the contact path determination learning model 232. Specifically, for each object, the contact path determination learning model evaluation unit 222 counts the number of times the force sensor 243 contacts the object until successful picking. If the number of contacts exceeds a reference value, the user is notified to proceed with learning the contact path determination learning model 232. This notification is, for example, made by displaying information on the output device 251.

[0274] The picking device based on the third embodiment operates in either an application mode or a learning mode. The application mode includes two modes: an additional learning application mode and a general application mode. The user instructs the control device 210 on which mode the picking device should operate in via the operation input device 252.

[0275] [Application Mode]

[0276] refer to Figure 20 The operation of the picking device in the additional learning application mode is explained. Figure 20 This is a flowchart illustrating the actions of the picking device in the additional learning application mode.

[0277] First, the user determines whether the selected object is a new object (step SD1). If the object is a new object, the user operates the input device 252 ( Figure 12 The control device 210 is instructed to perform pre-learning. If pre-learning is instructed, the shape definition data acquisition unit 211 acquires shape definition data. Figure 13A (Step SD2). Pre-learning section 212 ( Figure 12 Pre-learning of position and pose inference learning model 231 is performed based on the acquired shape definition data (step SD3).

[0278] If the selected object is not a new object and the pre-learning of the position and pose inference learning model 231 has been completed, or if the pre-learning in step SD3 has ended, the position and pose candidate output unit 213 ( Figure 12 Acquire image data of the object and perform image analysis to output multiple candidate positions and poses of the object (step SD4).

[0279] If multiple candidates are output, then the contact path determination unit 214 ( Figure 12 The force sensor 243 determines multiple predetermined contact paths, and the contact information acquisition unit 215 performs multiple predetermined contact actions (step SD5). The number of determined contact paths and the number of contact actions are preset, for example, three times. Afterwards, the position and posture inference unit 216... Figure 12 Based on multiple contact information obtained through contact actions, the position and pose of the object are inferred using the position and pose inference learning model 231 (step SD6).

[0280] Next, the picking control unit 217 controls the picking mechanism 240 based on the inferred position and posture, thereby performing the picking action (step SD7). If the picking is successful, the position and posture inference additional learning unit 218 performs... Figure 16 The additional learning shown is step SD9. Additionally, Figure 16 Step SA1 and Figure 20 The steps are the same as SD8.

[0281] If the picking fails in step SD7, the contact action in step SD5 is executed again. This time, the candidate positions and orientations of the object have been narrowed down using the previous contact actions, so the contact path determination unit 214 determines multiple contact paths that are different from the contact paths of the previous contact actions. The contact information acquisition unit 215 performs multiple contact actions based on the newly determined multiple contact paths and acquires contact information. In step SD6, the position and orientation inference unit 216 uses all the contact information acquired through the previous contact actions and this contact action to infer the position and orientation of the object. Because more contact information is used, the accuracy of the object's position and orientation inference is increased.

[0282] When the picking device is used in a normal application mode, additional learning is not performed (step SD9).

[0283] [Learning Mode]

[0284] Next, refer to Figure 21 The operation of the picking device in learning mode is explained. Figure 21 This is a flowchart illustrating the actions of the picking device in learning mode.

[0285] The order of steps SD1 to SD4 and the append-learning execution mode ( Figure 20 The order of steps SD1 to SD4 is the same. After step SD4, the control device 210 performs one of reinforcement learning and imitation learning according to the learning method indicated by the user. The user pre-selects which learning method to perform and instructs the control device 210 on the selected learning method from the input device 252.

[0286] With reinforcement learning selected, the contact path is determined by the reinforcement learning department 219 ( Figure 12 )according to Figure 17 The reinforcement learning is performed in the sequence shown (step SE1). Additionally, in Figure 17In step SB1 shown, the contact path determination reinforcement learning unit 219 acquires... Figure 21 Step SD4 Position and orientation candidate output unit 213 ( Figure 12 The output position and posture are the candidates.

[0287] With imitation learning selected, the contact path is determined by the imitation learning section 220. Figure 12 )according to Figure 18 The imitation learning is performed in the sequence shown (step SE2). Figure 18 In step SC1, the contact path determination imitation learning unit 220 acquires the information in... Figure 21 The image data acquired by the position and pose candidate output unit 213 in step SD4, and the position and pose candidates output by the position and pose candidate output unit 213.

[0288] The control device 210 executes and adds learning application modes after performing reinforcement learning or imitation learning. Figure 20 The order of steps SD5 to SD9 is the same.

[0289] Next, the superior effects of the third embodiment described above will be explained.

[0290] In the third embodiment described above, the pre-learning unit 212 ( Figure 12 )use Figure 13A The CAD data shown generates multiple training datasets. Therefore, a position and pose inference learning model can be learned without object preparation or actual picking actions.231 Figure 12 ).

[0291] Furthermore, the position and pose inference additional learning unit 218 ( Figure 12 By using a dataset (contact information and inferred position and posture) obtained from actual picking actions to supplement the learning model 231 for position and posture inference, a learning model 231 that conforms to the actual environment can be constructed. For example, when the force sensor 243 is in contact with the surface of an object, the direction of the reaction force received by the force sensor 243 from the object may sometimes be affected by friction and deviate from the normal direction of the surface. In pre-learning, such as... Figure 14 As shown, the position and pose inference learning model 231 was learned assuming the direction of the reaction force is the normal direction of the object's surface, thus failing to reproduce friction and other phenomena generated in the actual environment. Therefore, when using the position and pose inference learning model 231, which is learned solely through pre-learning, the inference accuracy sometimes decreases. In the third embodiment described above, additional learning is performed using a dataset obtained from the actual environment, thereby improving the inference accuracy of the position and pose inference learning model 231.

[0292] The contact path determination unit 214 uses a contact path determination learning model 232, learned through at least one of reinforcement learning and imitation learning, to determine the contact path of the force sensor 243. Therefore, optimal contact information for enabling the position and posture inference unit 216 to infer position and posture with high accuracy can be provided to the position and posture inference unit 216. Furthermore, the number of contacts required until successful picking can be reduced. This improves throughput.

[0293] Moreover, through learning mode ( Figure 21 The picking device is then operated by a contact path determination learning unit, including a contact path determination reinforcement learning unit 219 and a contact path determination imitation learning unit 220, which learn a contact path determination learning model 232. Therefore, the contact path determination learning model 232 can be used to determine a preferred contact path that further reduces the number of contacts until successful picking.

[0294] The position and pose inference learning model evaluation unit 221 evaluates the inference accuracy of the position and pose inference learning model 231 and notifies the user to perform additional learning if the inference accuracy is low. Specifically, if the frequency of picking failures exceeds a baseline value, the user is notified to perform additional learning. Therefore, the user can easily determine whether to perform additional learning on the position and pose inference learning model 231.

[0295] Furthermore, the contact path determination learning model evaluation unit 222 evaluates the appropriateness of the contact path determined using the contact path determination learning model 232. Specifically, if the number of contacts until the picking is completed exceeds a benchmark value, the appropriateness of the determined contact path is determined to be low. In this case, the contact path determination learning model evaluation unit 222 notifies the user to learn the contact path determination learning model 232. Thus, the user can easily determine whether to learn the contact path determination learning model 232.

[0296] The third embodiment described above is an example, and the present invention is not limited to the third embodiment described above. For example, various changes, improvements, combinations, etc., can be made, which will be obvious to those skilled in the art.

[0297] (Summary of the third embodiment)

[0298] The outline of the apparatus in the third embodiment is as follows.

[0299] [Summary of the Third Embodiment 1]

[0300] A picking device comprising:

[0301] Force sensors come into contact with objects and obtain contact information that forms the basis for inferring the position and orientation of the objects.

[0302] Picking mechanism, which picks up objects; and

[0303] The control device controls the picking mechanism based on contact information obtained from the force sensor.

[0304] The control device includes:

[0305] The pre-learning department uses a learning dataset generated using a simulator to pre-learn a learning model for picking actions;

[0306] The picking control unit uses a pre-learned learning model to control the force sensor and the picking mechanism to perform picking actions; and

[0307] An additional learning unit is added, which uses a dataset obtained through actual picking actions to learn a learning model for picking actions.

[0308] [Summary of the Third Embodiment 2]

[0309] According to the picking device described in Summary 1 of the third embodiment, wherein,

[0310] The pre-learning unit pre-learns a position and posture inference learning model based on shape definition data that defines the shape of an object.

[0311] The control device further includes a position and posture inference unit, which uses the contact information obtained from the force sensor as input to the position and posture inference learning model to infer the position and posture of the object.

[0312] The picking control unit controls the operation of the picking mechanism based on the position and posture inferred by the position and posture inference unit.

[0313] The additional learning unit performs additional learning of the position and posture inference learning model based on the contact information obtained from the force sensor and the success or failure of the picking mechanism's operation based on the picking control unit.

[0314] [Summary of the Third Embodiment 3]

[0315] According to the picking device described in Summary 2 of the third embodiment, wherein,

[0316] The control device also includes a position and pose inference learning model evaluation unit. If the frequency of failures in the operation of the picking mechanism exceeds a baseline value, the position and pose inference learning model evaluation unit notifies the user to perform additional learning based on the additional learning unit.

[0317] [Summary of the Third Embodiment 4]

[0318] According to any one of the summaries 1 to 3 of the third embodiment, the picking device wherein,

[0319] The control device further includes:

[0320] The position and pose candidate output unit outputs multiple candidates for the position and pose of the object based on information obtained from the object in a non-contact manner.

[0321] The contact path determination unit uses a contact path determination learning model that takes multiple candidates for the position and posture of an object as input and the contact path of the force sensor as output, and determines the contact path of the force sensor based on the multiple candidates output from the position and posture candidate output unit; and

[0322] The contact path determination learning unit uses multiple candidates for the position and posture of the object output from the position and posture candidate output unit, and contact paths determined for the multiple candidates, to learn the contact path determination learning model.

[0323] [Summary of the Third Embodiment 5]

[0324] According to the picking device described in summary 4 of the third embodiment, wherein,

[0325] The contact path determination learning unit performs at least one of reinforcement learning and imitation learning. The reinforcement learning sets a set of multiple candidates for the position and posture of the object as a "state", sets a contact action based on multiple contact paths as a "behavior", and sets the reduction of the number of candidates for the position and posture of the object as a "reward". The imitation learning is based on multiple candidates for the position and posture of the object and the contact path specified by the user.

[0326] [Summary of the Third Embodiment 6]

[0327] According to the picking device described in summary 4 or 5 of the third embodiment, wherein,

[0328] The control device also includes a contact path determination learning model evaluation unit, which counts the number of times the force sensor contacts each object pair until successful picking, and notifies the user when the number of contacts exceeds a benchmark value to execute the learning of the contact path determination learning model.

[0329] [Summary of the Third Embodiment 7]

[0330] The picking device according to any one of the summaries 1 to 6 of the third embodiment, wherein,

[0331] The control device also has a shape definition data acquisition unit, which accepts input shape definition data of the shape of the object being defined.

[0332] [Summary of the Third Embodiment 8]

[0333] A learning device that uses a learning dataset generated using a simulator to pre-learn a learning model of picking actions, and uses the dataset to further learn the learning model of picking actions, wherein the dataset is obtained by using the pre-learned learning model to control the picking actions performed by a force sensor that obtains contact information by contacting an object and a picking mechanism that grasps the object.

[0334] The first to third embodiments have been described above. The structures of the first, second, and third embodiments can be combined with each other.

[0335] Industrial availability

[0336] This invention can be used in information processing devices and picking devices.

[0337] Symbol Explanation

[0338] 1- Picking device (information processing device), 10- Control unit, 11- Object information storage unit, 12- Photography control unit, 13- Movement control unit, 14- Inference unit, 15- Robotic arm control unit, 16- Interface, 21- Photography unit, 22- Force sensor, 23- Drive device, 24- Robotic arm (removal mechanism), Q1, Q2- Object, R0- Initial area, h1, h2, h11~h13- Contact path, p1, p2, p11~p13- Contact point, s11a, s21- Surface with asymmetry, 110- Control device, 111- Trajectory information Information Acquisition Unit, 112 Imitation Learning Unit, 113 Allowable Range Determination Unit, 114 Movement Range Limiting Unit, 115 Arm Control Unit, 116 Interface Unit, 120 Movable Arm, 120A End Effector, 121 Input Unit, 122 Display Unit, 123 Camera, 125 Base, 130 Allowable Movement Range, 130A Allowable Movement Range Before Correction, 130B Allowable Movement Range After Correction, 131 Movement Trajectory During Teaching, 132 Candidate Movement Path, 133 Total Movement Range of the Movable Arm, 140 Robotic Hand, 150 Control Unit The system comprises: a control server, a trajectory information receiving unit, an imitation learning unit, a permissible range determination unit, a movement range limiting unit, a movement path sending unit, a display unit, a network, a control device, a shape definition data acquisition unit, a pre-learning unit, a position and posture candidate output unit, a contact path determination unit, a contact information acquisition unit, a position and posture inference unit, a picking control unit, a position and posture inference supplementary learning unit, and a contact path determination reinforcement learning unit. 220 - Contact path determination imitation learning unit; 221 - Position and posture inference learning model evaluation unit; 222 - Contact path determination learning model evaluation unit; 231 - Position and posture inference learning model; 232 - Contact path determination learning model; 240 - Picking mechanism; 241 - Multi-joint robotic arm; 242 - Holding unit; 243 - Force sensor; 251 - Output device; 252 - Input device; 253 - Camera device; 260 - Image of the object; 261 - Candidate for position and posture; 262 - Arrow for contact path specification; 263 - Pointer; 264 - Confirm button.

Claims

1.An information processing apparatus that estimates a posture of an object, the information processing apparatus characterized by comprising: a storage unit that stores object information indicating a shape and a size of the object; a force sensor that acquires detection information of a contact point by contact; a movement control unit that moves the force sensor; and an estimation unit that estimates the posture of the object based on the detection information obtained by a plurality of contacts of the force sensor with the object and the object information, the estimation unit estimating a plurality of candidates for the posture of the object before the plurality of contacts is completed, the movement control unit moving the force sensor so that the force sensor contacts different faces of the object in the plurality of contacts, and further contacts a face included in the plurality of candidates with a higher degree of uncertainty in position than other faces included in the plurality of candidates. 2.The information processing apparatus according to claim 1, characterized in that the detection information includes information of a position of the contact point and a normal line of a face including the contact point, the movement control unit moves the force sensor in a direction intersecting a normal line of a face including a contact point of a contact prior to an nth contact in the plurality of contacts at the nth contact. 3.The information processing apparatus according to claim 1 or 2, characterized by comprising: an imaging unit that acquires an image of an object, the estimation unit estimating a position capable of contacting the object based on the image. Further comprising: a trajectory information acquisition unit that acquires trajectory information indicating a movement trajectory of a movable arm provided by teaching; an allowable range determination unit that determines a movement allowable range of the movable arm based on the trajectory information acquired by the trajectory information acquisition unit; and a movement range restriction unit that restricts a movement range of the movable arm within the movement allowable range determined by the allowable range determination unit. 5.The information processing apparatus according to claim 1, characterized by further comprising a learning apparatus that pre-learns a learning model of a picking motion using a learning dataset generated using a simulator, and additionally learns the learning model of the picking motion using a dataset obtained by a picking motion of a picking mechanism that grasps an object and the force sensor that acquires contact information by controlling contact with the object using the pre-learned learning model. Further comprising: the information processing apparatus according to any one of claims 1 to 3; and a taking-out mechanism that takes out the object using an estimation result of the estimation unit. ​ ​ ​ ​ ​ ​ ​ 4. The information processing apparatus according to claim 1, characterized in that, ​ ​ ​ ​ ​ ​ 6. A picking device, characterized in that ​ ​ ​

Images