Control device, robot system, control method, and program

Abstract

This control device comprises: a limitation means that, via an expression using a direction in which a target object is held and a direction specifying the attitude of the target object, sets a condition of a surface of the target object, wherein said condition is included among limitation conditions in determining an attitude of the target object and a movement path of the target object, and said condition relates to holding the target object, terminating the holding of the target object, or changing the manner of holding the target object; and a control means that controls at least one among a first holding mechanism and a second holding mechanism so that holding the target object, terminating the holding of the target object, or changing the manner of holding the target object is carried out using the surface determined on the basis of the condition set by the limitation means.

Description

Technical Field

[0001] The present disclosure relates to a control device, a robot system, a control method, and program .

Background Art

[0002] Robots are used in various fields such as logistics. Some robots operate autonomously. Patent Document 1 discloses a technique related to a device that generates a trajectory plan for the tip of a robot arm to move from a starting point to an ending point. Also, Patent Documents 2 and 3 disclose techniques related to a robot system that grasps an object.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

Problems to be Solved by the Invention

[0004] By the way, in a robot system, it is desired to appropriately control a robot arm according to the state of an object.

[0005] Each aspect of the present disclosure aims to provide a control device, a robot system, a control method, and program that can solve the above problems.

Means for Solving the Problems

[0006] To achieve the above object, according to one aspect of the present disclosure, a control device is The system includes: constraint means for setting conditions on the surface of the object relating to gripping, releasing, or changing the grip of the object, which are included in the constraint conditions for determining the orientation of the object and the movement path of the object, by expressing the product of the norm of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis for defining the orientation of the object; and control means for controlling at least one of the first gripping mechanism and the second gripping mechanism so that the object is gripped, released, or changed using the surface determined based on the conditions set by the constraint means. .

[0007] To achieve the above objective, according to another aspect of this disclosure, the robot system comprises a first gripping mechanism, a second gripping mechanism, and the control device described above.

[0008] To achieve the above objective, according to another aspect of this disclosure, the control method is: The constraints for determining the orientation of the object and the movement path of the object include conditions on the object's surface relating to gripping, releasing, or changing the object's grip. These conditions are set using an expression representing the product of the norms of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis defining the object's orientation. The system controls at least one of the first gripping mechanism and the second gripping mechanism so that the object is gripped, released, or changed using the surface determined based on the set conditions. .

[0009] To achieve the above objectives, according to another aspect of this disclosure, the program is: The computer is made to perform the following: setting the conditions of the object's surface related to gripping, releasing, or changing the grip of the object, which are included in the constraints for determining the object's posture and the object's movement path, by expressing the product of the norm of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis for defining the object's posture; and controlling at least one of the first gripping mechanism and the second gripping mechanism so that the object is gripped, released, or changed using the surface determined based on the set conditions. .

[0010] To achieve the above objective, according to another aspect of the present disclosure, the control device comprises: determination means for determining the direction in which the second gripping mechanism grips an object based on the orientation of the surface of the object being gripped by the first gripping mechanism; and control means for controlling the operation of the second gripping mechanism to grip the object from the direction determined by the determination means.

[0011] To achieve the above objective, according to another aspect of the present disclosure, the control device includes determination means for determining the operation by which the first gripping mechanism and the second gripping mechanism grip an object based on the direction in which the first gripping mechanism grips, the direction in which the second gripping mechanism grips, and the direction of the surface of the object, and control means for controlling the first gripping mechanism and the second gripping mechanism to perform the determined operation. [Effects of the Invention]

[0012] According to each aspect of this disclosure, the robot arm can be appropriately controlled in the robot system according to the state of the object. [Brief explanation of the drawing]

[0013] [Figure 1] This figure shows an example of the configuration of a robot system according to one embodiment of the present disclosure. [Figure 2] This figure shows an example of the configuration of a control device according to one embodiment of the present disclosure. [Figure 3]It is a diagram showing an example of the configuration of a generation unit according to an embodiment of the present disclosure. [Figure 4] It is a diagram showing an example of the position coordinates and axis vectors of a robot hand in an embodiment of the present disclosure. [Figure 5] It is a diagram showing an example of the position coordinates and axis vectors of an object in an embodiment of the present disclosure. [Figure 6] It is the first diagram for explaining the constraint conditions in an embodiment of the present disclosure. [Figure 7] It is the second diagram for explaining the constraint conditions in an embodiment of the present disclosure. [Figure 8] It is the third diagram for explaining the constraint conditions in an embodiment of the present disclosure. [Figure 9] It is the fourth diagram for explaining the constraint conditions in an embodiment of the present disclosure. [Figure 10] It is the fifth diagram for explaining the constraint conditions in an embodiment of the present disclosure. [Figure 11] It is a diagram showing an example of each process and the movement path of an object in an embodiment of the present disclosure. [Figure 12] It is a diagram showing an image of a solution obtained by using the Lagrange multiplier method. [Figure 13] It is the first diagram for explaining a method of efficiently obtaining a desired solution in an embodiment of the present disclosure. [Figure 14] It is the second diagram for explaining a method of efficiently obtaining a desired solution in an embodiment of the present disclosure. [Figure 15] It is a diagram showing an example of a sequence of an initial plan generated by a generation unit according to an embodiment of the present disclosure. [Figure 16] It is a diagram showing an example of a control signal of an initial plan generated by a control unit according to the first embodiment of the present disclosure. [Figure 17] It is a diagram showing an example of a processing flow of a robot system according to an embodiment of the present disclosure. [Figure 18] It is a diagram showing an example of the configuration of a robot according to another embodiment of the present disclosure. [Figure 19] This figure shows an example of the configuration of a minimal control device according to an embodiment of the present disclosure. [Figure 20] This figure shows an example of the processing flow of a minimal control device according to an embodiment of the present disclosure. [Figure 21] This is a schematic block diagram showing the configuration of a computer according to at least one embodiment. [Modes for carrying out the invention]

[0014] The embodiments will be described in detail below with reference to the drawings. <Embodiment> A robot system 1 according to one embodiment of this disclosure is a system for moving an object M located at one position to another position. For example, the robot system 1 determines the posture and path of the object M during its movement, within the range of constraints that will be described later. The robot system 1 generates a control signal to move the object M along the determined posture and path. Then, the robot system 1 uses the generated control signal to control a robot arm that grasps the object M. By performing these processes, the robot system 1 moves the object M located at one position to another position. The robot system 1 will be described below.

[0015] (Robot system configuration) Figure 1 shows an example of the configuration of a robot system 1 according to one embodiment of the present disclosure. As shown in Figure 1, the robot system 1 comprises a control device 2, robots 40a and 40b, and an imaging device 50.

[0016] As shown in Figure 1, the robot 40a comprises a robot arm 401a, a base 402a, and a robot hand 403a (an example of a first gripping mechanism). The robot arm 401a is connected to the base 402a. The robot hand 403a is connected to the end of the robot arm 401a opposite to the end connected to the base 402a. The robot hand 403a comprises, for example, two or more pseudo-finger-like appendages that mimic the fingers of a human or animal, or a vacuum. The robot hand 403a grips the object M in accordance with the control signal output by the control device 2. The robot arm 401a moves the object M from the source to the destination in accordance with the control signal output by the control device 2.

[0017] Furthermore, as shown in Figure 1, the robot 40b includes a robot arm 401b, a base 402b, and a robot hand 403b (an example of a second gripping mechanism). The robot arm 401b is connected to the base 402b. The robot hand 403b is connected to the end of the robot arm 401b opposite to the end connected to the base 402b. The robot hand 403b includes, for example, two or more pseudo-finger-like appendages that mimic the fingers of a human or animal, or a vacuum. The robot hand 403b grips the object M in accordance with the control signal output by the control device 2. The robot arm 401b moves the object M from the source to the destination in accordance with the control signal output by the control device 2.

[0018] In each embodiment of this disclosure, "gripping" includes "suction," which involves sucking an object M with a vacuum or the like, and "pinching," which involves gripping an object with two or more pseudo-finger-like fingers that mimic the fingers of a human or animal. Hereinafter, robots 40a and 40b will be collectively referred to as robot 40. Robot arms 401a and 401b will be collectively referred to as robot arm 401. Bases 402a and 402b will be collectively referred to as base 402. Robot hands 403a and 403b will be collectively referred to as robot hand 403.

[0019] The imaging device 50 captures the state of the object M. The imaging device 50 is, for example, a depth camera and can determine the state (i.e., position and orientation) of the object M. The image captured by the imaging device 50 is represented, for example, by colored point cloud data and contains three-dimensional information of the captured object. The imaging device 50 outputs the captured image to the generation unit 202.

[0020] Figure 2 shows an example of the configuration of a control device 2 according to one embodiment of the present disclosure. As shown in Figure 2, the control device 2 comprises an input unit 201, a generation unit 202, and a control unit 203.

[0021] The input unit 201 inputs the work objective and constraints to the generation unit 202. Examples of work objectives include information indicating the type of object M, the quantity of object M to be moved, the source of the object M, and the destination of the object M. Examples of constraints include areas where the object M cannot be entered when moving the object M, areas that deviate from the robot 40's range of motion, and conditions on the surface of the object M regarding gripping, releasing, or changing the grip of the object M. The input unit 201 may also receive input from the user as a work objective, such as "move 3 units of product A from cardboard box C to tray T," and specify that the type of object M to be moved is product A, the quantity of object M to be moved is 3 units, the source of the object M is cardboard box C, and the destination of the object M is tray T, and input this specified information to the generation unit 202. Furthermore, the position of the object M identified in the image captured by the imaging device 50 may be used as the source of movement for the object M. The input unit 201 may also, for example, receive the positions of obstacles between the source and destination of the object M from the user as constraints indicating no-entry zones, and input this information to the generation unit 202. Alternatively, a file indicating the constraints may be stored in a storage device, and the input unit 201 may input the constraints indicated in that file to the generation unit 202, or the fourth processing unit 202d of the generation unit 202 (described later) may directly read the constraints from that file, or both. In short, any method of acquisition is acceptable as long as the generation unit 202 can obtain the necessary work objectives and necessary constraints. Details on how the constraints are defined will be described later.

[0022] Figure 3 shows an example of the configuration of a generation unit 202 according to one embodiment of the present disclosure. As shown in Figure 3, the generation unit 202 comprises a first processing unit 202a, a second processing unit 202b, a third processing unit 202c, a fourth processing unit 202d (an example of a constraint means), and a fifth processing unit 202e (an example of a control means).

[0023] The first processing unit 202a recognizes the robot 40. For example, the first processing unit 202a recognizes the robot model using CAD (Computer Aided Design) data. This CAD data includes information indicating the shape of the robot 40 and information indicating the range of motion, such as the reach range of the robot arm 401. The shape includes dimensions. CAD data is, for example, drawing data designed using CAD.

[0024] Furthermore, the first processing unit 202a recognizes the environment surrounding the robot 40. For example, the first processing unit 202a acquires images captured by the imaging device 50. The images captured by the imaging device 50 include information captured by the camera and information in the depth direction. This information in the depth direction corresponds to the aforementioned colored point cloud data. The first processing unit 202a recognizes the position and shape of obstacles from the acquired images. Obstacles here refer to all objects other than the object M that the robot 40 moves to its destination, which are within the imaging range of the imaging device 50. As mentioned above, the imaging device 50 can acquire three-dimensional information of objects within its imaging range. Therefore, the first processing unit 202a can recognize the environment surrounding the robot 40, including the position and shape of obstacles. Note that the first processing unit 202a is not limited to recognizing the environment surrounding the robot 40 from images captured by the imaging device 50. For example, the first processing unit 202a may recognize the environment surrounding the robot 40 using a 3D occupancy map (Octomap), CAD data, AR (Augmented Reality) markers, etc. This CAD data includes information indicating the shape of obstacles. The shape includes dimensions.

[0025] Furthermore, the first processing unit 202a recognizes the release position of the object M at its destination. For example, if the destination is a container (e.g., a tray T), the first processing unit 202a recognizes the release position by using machine learning with model-based matching. Model-based matching is a method for determining the position and orientation of an object by comparing image data obtained from a camera or the like with the shape and structural data of the object whose position and orientation are to be acquired (in this case, the container) with the shape and structural data of the object extracted from the image. Note that the first processing unit 202a is not limited to recognizing the release position by using machine learning with model-based matching. For example, the first processing unit 202a may recognize the release position using an AR marker.

[0026] Furthermore, the second processing unit 202b recognizes the base 402 of the robot 40, which will be described later. For example, the second processing unit 202b recognizes the base 402 by acquiring CAD data. This CAD data contains information indicating the shape of the base 402. The shape includes dimensions. As a result, the second processing unit 202b can recognize the Z coordinate of the upper surface of the base 402 in its coordinate system as the height of the base 402.

[0027] The third processing unit 202c recognizes the state (i.e., position and orientation) of the object M. For example, the third processing unit 202c recognizes the position of the object M by performing machine learning using model-based matching. Furthermore, the third processing unit 202c recognizes the orientation of the object M by using techniques to generate bounding boxes, such as AABB (Axis Aligned Bounding Box) or OBB (Oriented Bounding Box), for the object M whose position has been identified. Alternatively, the third processing unit 202c may classify the object M using a machine learning technique such as clustering on the image captured by the imaging device 50 and identify the state of the object M by using techniques to generate bounding boxes.

[0028] Furthermore, the third processing unit 202c acquires the height of the object M. For example, the third processing unit 202c recognizes the object M by acquiring CAD data. This CAD data contains information indicating the shape of the object M. The shape includes dimensions. As a result, the third processing unit 202c can recognize the Z coordinate of the object M in its coordinate system as the height of the object M. Alternatively, the third processing unit 202c may recognize the height of the object M by subtracting the Z coordinate of the base 402 from the Z coordinate of the top surface of the object M.

[0029] The fourth processing unit 202d acquires the constraint conditions. Then, the fourth processing unit 202d sets the acquired constraint conditions. Here, we will explain in detail how the constraint conditions are defined. Here, we will explain how the surfaces of the object M are described as constraint conditions related to the robot 40's gripping of the object M, release of gripping the object M, or changing the grip of the object M.

[0030] Figure 4 shows an example of the position coordinates and axis vectors of a robot hand 403 in one embodiment of the present disclosure. First, the position coordinates and axis vectors of the robot hand 403 at time k are defined using the notation in Figure 4. That is, the position coordinates r(h1,k) of the robot hand 403a at time k are expressed as shown in equation (1). The position coordinates r(h2,k) of the robot hand 403b at time k are expressed as shown in equation (2). Note that the position coordinates r(h1,k) and r(h2,k) represent the positions of the robot hands 403a and 403b at time k in the three-dimensional space R3 (for example, the three-dimensional space R3 represented by the x, y, and z axes in Figure 4) in which the robot 40, including the robot hands 403a and 403b, operates and the object M moves. Furthermore, the x-axis vector x(h1,k) of robot hand 403a at time k is expressed by equation (3), the y-axis vector y(h1,k) by equation (4), and the z-axis vector z(h1,k) by equation (5). Similarly, the x-axis vector x(h2,k) of robot hand 403b at time k is expressed by equation (6), the y-axis vector y(h2,k) by equation (7), and the z-axis vector z(h2,k) by equation (8). Note that for the three-dimensional shapes of robot hand 403a and robot hand 403b, uniquely determined x, y, and z axes are set independently of the three-dimensional space R3. The x-axis vector x(h1,k) indicates the direction of the x-axis set for the three-dimensional shape of robot hand 403a in the three-dimensional space R3 at time k. The y-axis vector y(h1,k) indicates the direction of the y-axis in 3D space R3 at time k, relative to the 3D shape of the robot hand 403a. The z-axis vector z(h1,k) indicates the direction of the z-axis in 3D space R3 at time k, relative to the 3D shape of the robot hand 403a. Similarly, the x-axis vector x(h2,k) indicates the direction of the x-axis in 3D space R3 at time k, relative to the 3D shape of the robot hand 403b. The y-axis vector y(h2,k) indicates the direction of the y-axis in 3D space R3 at time k, relative to the 3D shape of the robot hand 403b.The z-axis vector z(h2,k) indicates the direction of the z-axis in the 3D space R3 at time k, set for the 3D shape of the robot hand 403b.

[0031]

number

[0032]

number

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] Each of the axis vectors expressed by equations (3) to (8) is a unit vector. For convenience, we use unit vectors here, but axis vectors do not necessarily have to be unit vectors, and the length of the vector may vary depending on the direction of the axis vector.

[0040] Furthermore, the three-dimensional space R3 of the x, y, and z axes shown in Figure 4 will be represented as shown in equation (9).

[0041]

number

[0042] In this case, equation (10) holds true for the position coordinates of the robot hand 403. Also, equation (11) holds true for the axis vectors of the robot hand 403.

[0043]

number

[0044]

number

[0045] In other words, the position coordinates of each robot hand 403 at time k are elements of 3D space. Also, the axis vectors of each robot hand 403 at time k are elements of 3D space.

[0046] In this case, the position coordinates of the robot hand 403a can be defined as shown in equation (12). Similarly, the position coordinates of the robot hand 403b can be defined as shown in equation (13). Note that T in equations (12) and (13) represents the transpose operation.

[0047]

number

[0048]

number

[0049] Figure 5 shows an example of the position coordinates and axis vectors of an object M in one embodiment of the present disclosure. Here, the position coordinates and axis vectors of the object M at time k are defined using the notation in Figure 5. That is, the position coordinates r(obj,k) of the object M at time k are expressed as shown in equation (14). The x-axis vector x(obj,k) of the object M at time k is expressed as shown in equation (15), the y-axis vector y(obj,k) as shown in equation (16), and the z-axis vector z(obj,k) as shown in equation (17). Note that the position coordinates r(obj,k) represent the position of the object M at time k in the three-dimensional space R3 (for example, the three-dimensional space R3 represented by the three axes x, y, and z in Figure 4). Furthermore, for the object M, similar to the robot hands 403a and 403b, uniquely determined x, y, and z axes are set for the three-dimensional shape, independently of the three-dimensional space R3. The x-axis vector x(obj,k) indicates the direction of the x-axis in 3D space R3 at time k, set relative to the 3D shape of object M. The y-axis vector y(obj,k) indicates the direction of the y-axis in 3D space R3 at time k, set relative to the 3D shape of object M. The z-axis vector z(obj,k) indicates the direction of the z-axis in 3D space R3 at time k, set relative to the 3D shape of object M.

[0050]

number

[0051]

number

[0052]

number

[0053]

number

[0054] In other words, the position coordinates of object M at time k are an element of 3D space. Furthermore, the axis vectors of object M at time k are also an element of 3D space.

[0055] (In the case of a robot hand that uses suction to pick up objects) First, we will explain how the surfaces of the object M are described as constraints for a robot hand 403 that uses suction to grasp an object M, release the object M, or change the grip of the object M.

[0056] (When gripping or releasing an object) Figure 6 is a first diagram illustrating constraints in one embodiment of the present disclosure. Figure 6 illustrates constraints when the position coordinates and axis vectors of the robot hand 403 and the position coordinates and axis vectors of the object M are defined as shown in equations (1) to (17) above. As can be seen from Figure 6, when gripping or releasing the object M, the direction perpendicular to the suction surface of the object M must coincide with the suction direction of the robot hand 403. The vector indicating this suction direction is an example of the first vector. In other words, when gripping or releasing the object M, the constraints for the robot hand 403a can be described by the product of the norms of the cross product, which is an operation on the angle between the first vector indicating the direction of gripping the object M and the second vector indicating the axis for defining the posture of the object M, as shown in equation (18). Furthermore, when gripping or releasing the object M, the constraints for the robot hand 403b can be described by the product of the norms of the cross product, as shown in equation (19).

[0057]

number

[0058]

number

[0059] Note that equations (18) and (19) are constraints that assume an object M with a shape like the one shown in Figure 6, which has two faces parallel to each of the x, y, and z axes, such as a cube or a cuboid or other hexahedron. For example, in the case of a tetrahedron, the description of the constraints is different from equations (18) and (19).

[0060] Figure 7 is a second figure illustrating constraints in one embodiment of the present disclosure. Figure 7 shows the normals of each face of the tetrahedron. When gripping or releasing an object M having such a tetrahedron shape, the constraints for the robot hand 403a can be described using the product of the norms of the cross product, as shown in equation (20), under the same considerations as for the hexahedron shown in Figure 6 (i.e., the direction perpendicular to the suction surface of the object M must coincide with the suction direction of the robot hand 403). Furthermore, when gripping or releasing the object M, the constraints for the robot hand 403b can be described using the product of the norms of the cross product, as shown in equation (21).

[0061]

number

[0062]

number

[0063] However, in the case of this tetrahedron, there are no faces that are parallel to each other. Therefore, unless the robot hand 403 is brought close to the face of the object M to be grasped from the correct direction, the object M cannot be grasped. Constraints expressed as the product of the norms of cross products using the cross product, as in equations (20) and (21), represent scalar quantities. Therefore, these constraints require that one of the normals of each face is parallel to the suction direction of the robot hand 403, but they do not require that one of the normals of each face coincides with the suction direction of the robot hand 403. Thus, in order for the constraints to require that one of the normals of each face coincides with the suction direction of the robot hand 403, it is necessary to add a new constraint that indicates which normal's direction coincides with the suction direction of the robot hand 403. An example of this is the constraint added to robot hand 403a by equation (22), and the constraint added to robot hand 403b by equation (23).

[0064]

number

[0065]

number

[0066] The constraints expressed by equations (22) and (23) require that the magnitude of the dot product, which is the calculation of the angle between the vectors indicated by the normals of each face and the vector indicating the suction direction of the robot hand 403, be 1, that is, that the directions of these vectors coincide. Therefore, for an object M having a shape in which no parallel faces exist, such as a tetrahedron, it becomes possible to have the robot hand 403 grasp the object M by adding constraints such as equations (22) and (23), which require that the magnitude of the dot product between the vectors indicated by the normals of each face and the vector indicating the suction direction of the robot hand 403 be 1.

[0067] Furthermore, in the case of a cube or cuboid, which is a hexahedron, having two faces parallel to each of the x, y, and z axes, the norm of the cross product is duplicated. Therefore, equations (18) and (19) show the calculation of one norm of the cross product for each of the x, y, and z axes. However, equations (22) and (23) are used to calculate the dot product for each face. In other words, for each face of the object M, by using equations (22) and (23) to calculate an operation based on the angle between two directions, i.e., the dot product, it becomes possible to define equations that show constraints for an object M of any shape. Note that the constraints based on the equations that calculate the dot product for each face encompass the constraints based on the equations that calculate the cross product. Therefore, when using equations (22) and (23) to calculate the dot product for each face, constraints based on the equations that calculate the cross product are unnecessary. However, if a constraint condition using an equation that calculates the cross product is used instead of being limited to the dot product, then only a constraint condition indicating which surface normal direction coincides with the robot hand 403's suction direction needs to be added.

[0068] (When switching the object) Figure 8 is a third figure illustrating the constraints in one embodiment of the present disclosure. Figure 8 is an illustrative diagram of the processes performed by robot hands 403a and 403b. Part 8(a) shows the process in which robot hand a grasps the object M. Part 8(b) shows the process of transferring the object M from robot hand 403a to robot hand 403b (i.e., handing over the object M). Part 8(c) shows the process in which robot hand 403b releases the grasp of the object M. The description of the constraints in the processes shown in parts 8(a) and (c) is as described above. Here, we will explain the description of the constraints in the process shown in part 8(b) (i.e., the process of transferring the object M).

[0069] When transferring the object M from robot hand 403a to robot hand 403b, each robot hand 403 must satisfy the constraints for gripping or releasing the object M. Therefore, as shown in Figure 8, the constraints when the object M is a cube or a cuboid (a hexahedron) require equations (18) and (19) simultaneously. Note that the constraints when the object M does not have parallel faces, such as a tetrahedron, require equations (20) to (23) simultaneously.

[0070] (In the case of a robot hand that grips an object) Next, we will explain how the surfaces of the object M are described as constraints for gripping, releasing, or changing the grip of the object M in a robot hand 403 of the type that grips the object M.

[0071] Figure 9 is a fourth figure illustrating the constraints in one embodiment of the present disclosure. The direction of v in Figure 9 coincides with the direction in which the robot hand 403 grips the object M. Figure 10 is a fifth figure illustrating the constraints in one embodiment of the present disclosure. n1, n2, and n3 in Figure 10 correspond to the y-axis vector, x-axis vector, and z-axis vector shown in Figure 5. In the case of a robot hand 403 that grips an object M, the direction in which the object M is gripped, i.e., the direction of v in Figure 9, should be aligned with the suction direction of a robot hand 403 that sucks up an object M, and the constraints can be considered in the same way as for a robot hand 403 that sucks up an object M.

[0072] (When gripping or releasing an object) The constraints for gripping or releasing the object M shown in Figure 10 for the robot hand 403 in Figure 9 and the object M shown in Figure 10 can be expressed as shown in equation (24) for each of the robot hands 403, by aligning the direction of gripping the object M with the direction of suction of the robot hand 403 that sucks up the object M.

[0073]

number

[0074] Furthermore, the constraints for the robot hand 403 shown in Figure 9 and the object M shown in Figure 10 when switching the grip of the object M require that equation (24) be simultaneously satisfied for each of the robot hands 403.

[0075] The fourth processing unit 202d sets various constraints, including the constraints described above.

[0076] The fifth processing unit 202e generates an initial plan sequence showing the flow of robot 40's movements based on the work objectives determined by the processing of the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c, and the constraint conditions set by the processing of the fourth processing unit 202d. For example, the fifth processing unit 202e obtains the work objectives from the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c. The fifth processing unit 202e also obtains the constraint conditions from the fourth processing unit 202d. The fifth processing unit 202e adds the constraint conditions obtained from the fourth processing unit 202d to the constraint conditions input from the input unit 201. Then, the fifth processing unit 202e generates information that is necessary for the control unit 203 to generate a control signal to control the robot 40, which includes the state of the robot 40 at each time step from the state of the object M at its source to the state of the object M at its destination (including the type of object M, the position and orientation of the robot 40, the grip strength of the object M, and the operation of the robot 40 (for example, an approach operation to approach the object M (corresponding to the processing of the approach process in Figure 11 described later), a pick operation to grip the object M (corresponding to the processing of the pick process in Figure 11), a carry operation to move the arm to correctly move the gripped object M to the transport destination (corresponding to the processing of the carry process in Figure 11), a place operation to release the grip of the object M (corresponding to the processing of the place process in Figure 11), etc.)). In other words, the sequence is the information that indicates each state of the robot 40 at each time step, from the state of the object M at the source of movement to the state of the object M at the destination of movement, which is necessary for the control unit 203 to generate control signals for controlling the robot 40.

[0077] Figure 11 is a diagram showing an example of each step and the movement path of the object M in one embodiment of the present disclosure. The steps and movement paths of the object M in the sequence for moving the object M shown in Figure 11 are determined by the fifth processing unit 202e performing a simulation with objective functions such as minimizing the amount of energy consumed by the robot 40, minimizing the trajectory of the robot arm 401, and minimizing the movement path of the object M, given that a work objective and various constraints are set. As shown in Figure 11, an example of a step for moving the object M from the source to the destination is an approach step in which the robot arms 401a and 401b approach the object M, a pick step in which the object M is grasped, a carry step in which the object M is moved, and a place step in which the object M is released from being grasped.

[0078] Here, we will explain the method by which the fifth processing unit 202e determines the attitude and movement path of the object M at each time step through simulation. For simplicity, we will explain the method for minimizing the trajectory of the robot arm 401. In this case, variables (e.g., x, y) that affect the trajectory of the robot arm 401 are defined, and the objective function is to minimize the function f(x,y) that represents the trajectory of the robot arm 401. Also, for example, due to constraints on the range of motion of the robot arm 401, a constraint condition such as x+y=0 is set. In such a case, the Lagrangian function L is given by equation (25) using a positive λ.

[0079]

number

[0080] The fifth processing unit 202e then finds the solution by using the Lagrange multiplier method. Figure 12 shows an image of the solution obtained using the Lagrange multiplier method. In this case, the fifth processing unit 202e finds the desired solution (an example of an optimal solution) indicated by the asterisk in Figure 12, where the objective function f(x,y) is minimized, by repeatedly searching for a region that satisfies the constraint x+y=0 and where differentiating the Lagrange function L yields a local minimum. Note that the Lagrange function L expressed by equation (25) is just one example, and any Lagrange function commonly used in continuous optimization can be used. For example, when using an optimization algorithm based on a gradient method called the prim-dual interior-point method, the Lagrange function L has a value of zero in the region that does not violate the constraint, and takes on a value of infinity as soon as the region that violates the constraint enters; this is called a barrier function.

[0081] However, when using the derivative of a function to find a solution under certain constraints as described above, a problem generally known as a "non-convex constraint" may arise, where the searchable region of the solution is limited by the constraints, making it impossible to search for a local minimum.

[0082] Here, we will describe a method for efficiently finding the desired solution even when the problem of non-convex constraints arises. Here, the shape of the object M is assumed to be a hexahedron with parallel faces, such as a cube or a rectangular prism. The robot hand 403 is assumed to be of the type that grips the object M. When the shape of the object M is a hexahedron with parallel faces and the robot hand 403 is of the type that grips the object M, equation (24) holds. Figure 13 is the first figure illustrating a method for efficiently finding the desired solution in one embodiment of the present disclosure. In equation (24), the vector ni indicating the suction direction of the robot hand 403 is represented in Figure 13 by vectors ex, ey, and ez.

[0083] If we assume the magnitude of vector v is 1, then in Figure 13, the only feasible points for vector v are the six points indicated by the asterisks. That is, all points on the sphere in Figure 13 other than the six points indicated by the asterisks are infeasible. In this case, if an infeasible point is chosen as the initial search point, the fifth processing unit 202e will be unable to search any points other than that initial search point. In other words, the non-convex constraint problem arises.

[0084] In such cases, the fifth processing unit 202e uses, for example, the SA (Simulated Annealing) method. Specifically, the fifth processing unit 202e expands the search points by relaxing the constraints and finds a local minimum (an example of a local optimum). Figure 14 is a second figure illustrating a method for efficiently finding a desired solution in one embodiment of the present disclosure. Figure 14 shows only one-eighth of the sphere shown in Figure 13. Furthermore, parts (a), (b), and (c) in Figure 14 show differences in the infeasible region depending on the degree of relaxation of the constraints using tk, which is taken as a positive value. Furthermore, part (d) in Figure 14 shows the infeasible region in the case of the original constraint (tk=0). Note that if the shape of the object M is a hexahedron with parallel faces, such as a cube or a rectangular prism, due to the symmetry of the vectors, an infeasible region similar to the infeasible region shown in Figure 14 occurs in parts other than one-eighth of the sphere shown in Figure 14.

[0085] In the example of tk=1 shown in part (a) of Figure 14, all regions become feasible. In the example of tk=0.3 shown in part (b) of Figure 14, some regions become infeasible. In the example of tk=0.25 shown in part (c) of Figure 14, the infeasible regions divide the feasible regions into multiple discontinuous regions. As shown in part (c) of Figure 14, when an infeasible region divides a feasible region into multiple discontinuous regions, the fifth processing unit 202e becomes unable to search for other feasible regions from one feasible region. In other words, if the desired solution does not exist in a feasible region, the fifth processing unit 202e will not be able to obtain the desired solution. Therefore, as shown in part (c) of Figure 14, if an infeasible region divides a feasible region into multiple discontinuous regions, the fifth processing unit 202e does not continue the same search, but instead adjusts the degree of relaxation of the constraints and changes the search conditions, such as randomly changing the search points, to search for a solution. Note that adjusting the degree of relaxation of the constraints and changing the search conditions are examples of changing the content of the relaxation.

[0086] Therefore, even when the problem of non-convex constraints arises, an efficient method for finding the desired solution is to first determine, through simulation or other means, the constraints that divide the infeasible region into multiple discontinuous regions (an example of a condition under which a local optimum cannot be found), as shown in part (c) of Figure 14. The fifth processing unit 202e stores these predetermined constraints. Then, as shown in part (a) of Figure 14, the fifth processing unit 202e relaxes the constraints so that all regions become feasible regions. With the constraints relaxed, the fifth processing unit 202e finds the local minimum of the Lagrangian function L by differentiating the Lagrangian function L. Then, the fifth processing unit 202e checks whether this local minimum satisfies the original constraints. If the original constraints are satisfied, the fifth processing unit 202e determines that the local minimum is the desired solution. Furthermore, if the original constraints are not satisfied, the fifth processing unit 202e finds a local minimum of the Lagrangian function L by differentiating the Lagrangian function L with a reduced degree of relaxation of the constraints. The fifth processing unit 202e then checks whether this local minimum satisfies the original constraints. The fifth processing unit 202e repeats this process until a local minimum that satisfies the original constraints is found, or until the reduced degree of relaxation of the constraints becomes the previously determined constraint (i.e., a constraint that divides the infeasible region into multiple discontinuous regions). Also, if the reduced degree of relaxation of the constraints becomes the previously determined constraint that divides the infeasible region into multiple discontinuous regions, the fifth processing unit 202e changes the search conditions by adjusting the degree of relaxation of the constraints and searches for a solution. In this way, the control device 2 can efficiently find the desired solution even when a non-convex constraint problem occurs.

[0087] The fifth processing unit 202e outputs the generated sequence to the control unit 203. The fifth processing unit 202e may be implemented using artificial intelligence (AI) technologies, including temporal logic, reinforcement learning, and optimization techniques.

[0088] Figure 15 shows an example of an initial plan sequence TBL1 generated by a generation unit 202 according to one embodiment of the present disclosure. For example, as shown in Figure 15, the initial plan sequence TBL1 generated by the generation unit 202 is a sequence that shows the state of the robot 40 at n time steps from the source to the destination of the object M.

[0089] The control unit 203 generates control signals to control the robot 40 based on the sequence generated by the generation unit 202. Specifically, it generates control signals that realize the posture of the object M and the movement path of the object M according to the sequence generated by the generation unit 202. The control unit 203 outputs the generated control signals to the robot 40.

[0090] Figure 16 shows an example of an initial planning control signal Cnt generated by the control unit 203 according to the first embodiment of this disclosure. For example, the initial planning control signal Cnt generated by the control unit 203 is, as shown in Figure 16, a control signal for each time step n from the source to the destination of the object M.

[0091] Figure 17 is a diagram showing an example of the processing flow of a robot system 1 according to one embodiment of the present disclosure. Here, the process of generating a sequence to be performed by the robot system 1 and controlling the robot 40 will be described with reference to Figure 17. Here, it is assumed that the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c each perform the processing described above.

[0092] The fourth processing unit 202d sets various constraints (step S1). For example, the fourth processing unit 202d sets the surface conditions of the object M related to gripping, releasing, or changing the grip of the object M, which are included in the constraints for determining the orientation and movement path of the object M, by expressing them using the cross product of a vector indicating the direction in which the object M is gripped (an example of a first vector) and an x-axis vector (an example of a second vector), a y-axis vector (an example of a second vector), and a z-axis vector (an example of a second vector) for defining the orientation of the object M.

[0093] The fifth processing unit 202e generates an initial plan sequence showing the flow of robot 40's movements based on the work objectives determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c, and the constraint conditions set by the processing by the fourth processing unit 202d (step S2). For example, the fifth processing unit 202e obtains the work objectives from the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c. The fifth processing unit 202e also obtains the constraint conditions from the fourth processing unit 202d. The fifth processing unit 202e adds the constraint conditions obtained from the fourth processing unit 202d to the constraint conditions input from the input unit 201. Then, the fifth processing unit 202e generates information that is necessary for the control unit 203 to generate a control signal to control the robot 40, which includes the state of the robot 40 at each time step from the state of the object M at its source to the state of the object M at its destination (including the type of object M, the position and orientation of the robot 40, the grip strength of the object M, and the operation of the robot 40 (for example, an approach operation to approach the object M (corresponding to the processing of the approach step in Figure 11), a pick operation to grip the object M (corresponding to the processing of the pick step in Figure 11), a carry operation to move the arm to correctly move the gripped object M to the transport destination (corresponding to the processing of the carry step in Figure 11), a place operation to release the grip of the object M (corresponding to the processing of the place step in Figure 11), etc.)) which is necessary for the control unit 203 to generate a control signal to control the robot 40.

[0094] For example, the fifth processing unit 202e determines the orientation and movement path of the object M at each time step through simulation. Specifically, the fifth processing unit 202e can find the solution for the Lagrangian function L by using the method of Lagrange multipliers. More specifically, the fifth processing unit 202e identifies the desired solution that minimizes the objective function f(x,y) by repeatedly searching for a region that satisfies the constraints and where the Lagrangian function L has a local minimum when differentiated.

[0095] Furthermore, even when a non-convex constraint problem arises, the fifth processing unit 202e uses the SA method to efficiently find the desired solution. Specifically, the fifth processing unit 202e stores the constraint condition that the previously determined infeasible region divides the feasible region into multiple discontinuous regions (step S3). Then, as shown in part (a) of Figure 14, the fifth processing unit 202e relaxes the constraint condition so that all regions become feasible regions (step S4). With the constraint condition relaxed, the fifth processing unit 202e finds the local minimum of the Lagrangian function L by differentiating the Lagrangian function L (step S5). Then, the fifth processing unit 202e determines whether this local minimum satisfies the original constraint condition (step S6). If the fifth processing unit 202e determines that the local minimum satisfies the original constraint condition (YES in step S6), it sets that local minimum as the desired solution (step S7). Then, the control unit 203 generates a control signal to control the robot 40 based on the sequence generated by the fifth processing unit 202e of the generation unit 202 (step S8). The control unit 203 outputs the generated control signal to the robot 40 (step S9).

[0096] Furthermore, if the fifth processing unit 202e determines that the local minimum does not satisfy the original constraint (NO in step S6), it determines whether the constraint has become one of the pre-stored constraints (step S10). If the fifth processing unit 202e determines that the constraint has not become one of the pre-stored constraints (NO in step S10), it reduces the degree of relaxation of the constraint (step S11). Then, the fifth processing unit 202e returns to the process in step S5.

[0097] Furthermore, if the fifth processing unit 202e determines that the constraint conditions have become the constraint conditions that have been stored in advance (YES in step S10), it changes the search conditions by adjusting the degree of relaxation of the constraint conditions (step S12). Then, the fifth processing unit 202e returns to the process in step S5.

[0098] As described above, the fifth processing unit 202e generates an initial plan sequence showing the flow of robot 40's movements based on the work objectives determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c, and the constraints set by the processing by the fourth processing unit 202d. For example, in the process of changing the grip of the object M shown in part (b) of Figure 8, the fifth processing unit 202e (an example of a determination means) determines the direction in which the robot hand 403b (an example of a second gripping mechanism) grips the object M based on the direction of the surface of the object M being gripped by the robot hand 403a (an example of a first gripping mechanism). Furthermore, the control unit 203 (an example of a control means) controls the movement of the robot hand 403b to grip the object M from the direction determined by the fifth processing unit 202e.

[0099] Furthermore, as described above, the fifth processing unit 202e generates an initial plan sequence showing the flow of robot 40's movements based on the work objectives determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c, and the constraints set by the processing by the fourth processing unit 202d. For example, in the process of changing the grip of the object M shown in part (b) of Figure 8, the fifth processing unit 202e (an example of a determination means) also determines the first direction in which the robot hand 403a (an example of a first gripping mechanism) grips the object M, and the second direction in which the robot hand 403b (an example of a second gripping mechanism) grips the object M, based on the orientation of the surface of the object M. Furthermore, the control unit 203 (an example of a control means) also controls the movement of the robot hand 403a so that the robot hand 403a grasps the object M from the first direction, and controls the movement of the robot hand 403b so that the robot hand 403b grasps the object M from the second direction.

[0100] (advantage) The robot system 1 according to one embodiment of the present disclosure has been described above. In the robot system 1, the fourth processing unit 202d (an example of a constraint means) sets the conditions of the surface of the object M related to gripping, releasing, or changing the grip of the object M, which are included in the constraint conditions for determining the posture of the object M and the movement path of the object M, by expressing them using the cross product of a vector indicating the direction of gripping the object M (an example of a first vector) and an x-axis vector (an example of a second vector), a y-axis vector (an example of a second vector), and a z-axis vector (an example of a second vector) for defining the posture of the object M. The control unit 203 (an example of a control means) controls at least one of the robot hand 403a (an example of a first gripping mechanism) and the robot hand 403b (an example of a second gripping mechanism) so that the object M is gripped, released, or changed using the surface of the object M determined based on the constraint conditions set by the fourth processing unit 202d.

[0101] By doing so, the robot arm in robot system 1 can be appropriately controlled according to the state of the object.

[0102] Furthermore, by doing so, constraints can be easily set in the robot system 1, for example, using cross product representation.

[0103] In one embodiment of this disclosure, the robot arm 401a and robot hand 403a are provided by robot 40a, and the robot arm 401b and robot hand 403b are provided by robot 40b. However, in another embodiment of this disclosure, the robot arm 401a, robot hand 403a, robot arm 401b, and robot hand 403b may all be provided by a single robot. Figure 18 shows an example of the configuration of robot 40c according to another embodiment of this disclosure. For example, robot 40c includes robot arm 401a, robot arm 401b, base 402c, robot hand 403a, and robot hand 403b, as shown in Figure 18. Then, the control device 2 can control the robot arm 401a, robot hand 403a, robot arm 401b, and robot hand 403b of robot 40c, in the same way that the control device 2 controlled the robot arm 401a and robot hand 403a of robot 40a and the robot arm 401b and robot hand 403b of robot 40b in one embodiment of the present disclosure.

[0104] Next, a minimal configuration control device 2 according to an embodiment of the present disclosure will be described. Figure 19 is a diagram showing an example of the configuration of a minimal configuration control device 2 according to an embodiment of the present disclosure. As shown in Figure 19, the minimal configuration control device 2 according to an embodiment of the present disclosure comprises a fourth processing unit 202d (an example of a constraint means) and a control unit 203 (an example of a control means). The fourth processing unit 202d is included in the constraint conditions for determining the posture of an object and the movement path of the object, and sets the conditions of the surface of the object relating to gripping the object, releasing the grip of the object, or changing the grip of the object, using an expression that uses the direction for gripping the object and the direction that defines the posture of the object. The fourth processing unit 202d can be realized, for example, using the functions of the fourth processing unit 202d illustrated in Figure 3. The control unit 203 controls at least one of the first gripping mechanism and the second gripping mechanism so that the object is gripped, released, or the object is changed using the surface determined based on the conditions set by the fourth processing unit 202d. The control unit 203 can be implemented, for example, by using the functions of the control unit 203 illustrated in Figure 2.

[0105] Next, the processing of the minimally configured control device 2 according to the embodiments of this disclosure will be described. Figure 20 is a diagram showing an example of the processing flow of the minimally configured control device 2 according to the embodiments of this disclosure. Here, the processing of the minimally configured control device 2 will be described with reference to Figure 20.

[0106] The fourth processing unit 202d (an example of a constraint means) sets the conditions of the surface of the object relating to gripping, releasing, or changing the grip of the object, which are included in the constraint conditions for determining the posture of the object and the movement path of the object, using an expression that uses the direction for gripping the object and the direction that defines the posture of the object (step S101). The control unit 203 (an example of a control means) controls at least one of the first gripping mechanism and the second gripping mechanism so that the object is gripped, released, or changed using the surface determined based on the conditions set by the fourth processing unit 202d (step S102).

[0107] The minimum configuration of the control device 2 according to the embodiments of this disclosure has been described above. This control device 2 enables the robot arm to be appropriately controlled in a robot system according to the state of the object.

[0108] In addition, the order of processing in the embodiments of this disclosure may be changed, as long as appropriate processing is performed.

[0109] Although embodiments of this disclosure have been described, the robot system 1, control device 2, input unit 201, generation unit 202, control unit 203, robot 40, imaging device 50, and other control devices described above may have a computer device inside. The process described above is stored in the form of a program on a computer-readable recording medium, and the above process is performed when the computer reads and executes this program. A specific example of a computer is shown below.

[0110] Figure 21 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in Figure 21, the computer 5 includes a CPU (Central Processing Unit) 6, main memory 7, storage 8, and interface 9. For example, the robot system 1, control device 2, input unit 201, generation unit 202, control unit 203, robot 40, imaging device 50, and other control devices are each implemented in the computer 5. The operation of each of the above-mentioned processing units is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8 and loads it into the main memory 7, and executes the above processing according to the program. The CPU 6 also allocates memory areas in the main memory 7 corresponding to each of the above-mentioned storage units according to the program.

[0111] Examples of storage 8 include HDDs (Hard Disk Drives), SSDs (Solid State Drives), magnetic disks, magneto-optical disks, CD-ROMs (Compact Disc Read Only Memory), DVD-ROMs (Digital Versatile Disc Read Only Memory), and semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if this program is distributed to computer 5 via a communication line, computer 5, upon receiving the program, may expand it into main memory 7 and execute the above processing. In at least one embodiment, storage 8 is a tangible storage medium that is not temporary.

[0112] Furthermore, the above program may implement some of the functions described above. Moreover, the above program may be a file that can implement the above functions in combination with a program already recorded on the computer device, a so-called differential file (differential program).

[0113] While several embodiments of this disclosure have been described, these embodiments are illustrative and do not limit the scope of the disclosure. These embodiments may be modified in various ways, without departing from the gist of the disclosure.

[0114] Furthermore, some or all of the above embodiments may also be described as follows, but are not limited to these.

[0115] (Note 1) A constraint means that includes constraints in determining the orientation of an object and the movement path of the object, and sets conditions on the surface of the object relating to gripping, releasing, or changing the grip of the object, using expressions that define the direction of gripping the object and the direction that defines the orientation of the object, A control means for controlling at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the conditions set by the constraint means, release the grip of the object, or change the grip of the object, A control device equipped with the following features.

[0116] (Note 2) The aforementioned restricting means is If no local optimum exists in the search for the optimal solution of the aforementioned surface, the conditions are relaxed so that a local optimum exists, and the local optimum is sought for the relaxed conditions. If the found local optimum is not the desired solution, the degree of relaxation of the conditions is reduced, and the local optimum is sought for the reduced degree of relaxation. The control device described in Appendix 1.

[0117] (Note 3) The aforementioned restricting means is By using the Simulated Annealing (SA) method, the conditions are relaxed so that a local optimum exists, the local optimum is found for the relaxed conditions, and if the found local optimum is not the desired solution, the degree of relaxation of the conditions is reduced, and the local optimum is found for the reduced degree of relaxation. The control device described in Appendix 2.

[0118] (Note 4) The aforementioned restricting means is If the relaxed conditions make it impossible to obtain the previously determined local optimum, the content of the relaxation is changed, and the local optimum is sought for the relaxed conditions with the changed content. If the obtained local optimum is not the desired solution, the degree of relaxation for the conditions is reduced, and the local optimum is sought for the conditions with the reduced degree of relaxation. The control device described in Appendix 2 or Appendix 3.

[0119] (Note 5) First gripping mechanism, The second gripping mechanism, A control device described in any one of the appendices 1 to 4, A robotic system equipped with the following features.

[0120] (Note 6) The constraints in determining the orientation of the object and the movement path of the object include conditions on the surface of the object relating to gripping, releasing, or changing the grip of the object, which are set using expressions that define the direction in which the object is gripped and the direction that defines the orientation of the object. Control at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set conditions, release the grip on the object, or change the grip on the object. Control method.

[0121] (Note 7) The constraints in determining the orientation of the object and the movement path of the object include setting the conditions of the object's surface regarding gripping, releasing, or changing the grip of the object using expressions that define the direction in which the object is gripped and the direction that defines the orientation of the object. Control at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set conditions, release the grip on the object, or change the grip on the object. A recording medium that stores a program that causes a computer to execute a program.

[0122] (Note 8) A determination means for determining the direction in which the second gripping mechanism grips the object based on the direction of the surface of the object being gripped by the first gripping mechanism, Control means for controlling the operation of the second gripping mechanism so as to grip the object from the direction determined by the determination means, A control device equipped with the following features.

[0123] (Note 9) A determination means for determining the operation by which the first gripping mechanism and the second gripping mechanism grip an object, based on the direction in which the first gripping mechanism grips, the direction in which the second gripping mechanism grips, and the direction of the surface of the object, Control means for controlling the first gripping mechanism and the second gripping mechanism to perform the determined operation. A control device equipped with the following features. [Industrial applicability]

[0124] According to each aspect of this disclosure, the robot arm can be appropriately controlled in the robot system according to the state of the object. [Explanation of Symbols]

[0125] 1. Robot System 2. Control device 5. Computers 6..CPU 7. Main Memory 8. Storage 9. Interface 40... Robots 50... Imaging device 201...Input section 202...Generation section 202a...First Processing Unit 202b...Second Processing Unit 202c...3rd Processing Unit 202d...Fourth Processing Unit 202e...5th Processing Unit 203... Control Unit C...cardboard F...Floor surface M...Target object T...tray

Claims

1. A constraint means that includes constraints in determining the orientation of an object and the movement path of the object, and sets the conditions of the object's surface regarding gripping, releasing, or changing the grip of the object by expressing the product of the norm of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis for defining the orientation of the object, A control means for controlling at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the conditions set by the constraint means, release the grip on the object, or change the grip on the object, A control device equipped with the following features.

2. The aforementioned restricting means is If no local optimum exists in the search for the optimal solution of the aforementioned surface, the conditions are relaxed so that a local optimum exists, and the local optimum is sought for the relaxed conditions. If the found local optimum is not the desired solution, the degree of relaxation of the conditions is reduced, and the local optimum is sought for the reduced degree of relaxation. The control device according to claim 1.

3. The aforementioned restricting means is By using the SA (Simulated Annealing) method, the conditions are relaxed so that a local optimum exists, the local optimum is found for the relaxed conditions, and if the found local optimum is not the desired solution, the degree of relaxation of the conditions is reduced, and the local optimum is found for the reduced degree of relaxation. The control device according to claim 2.

4. The aforementioned restricting means is If the relaxed conditions make it impossible to obtain the previously determined local optimum, the content of the relaxation is changed, and the local optimum is sought for the relaxed conditions with the changed content. If the obtained local optimum is not the desired solution, the degree of relaxation for the conditions is reduced, and the local optimum is sought for the conditions with the reduced degree of relaxation. The control device according to claim 2.

5. The control means is The control device according to claim 1, which controls the first gripping mechanism and the second gripping mechanism to change the grip of the object.

6. First gripping mechanism, The second gripping mechanism, The control device according to claim 1, A robotic system equipped with the following features.

7. The constraints in determining the orientation of the object and the movement path of the object include the conditions on the surface of the object relating to gripping, releasing, or changing the grip of the object, and these are set by expressing the product of the norm of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis for defining the orientation of the object. Control at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set conditions, release the grip on the object, or change the grip on the object. Control method.

8. The constraints in determining the orientation of the object and the movement path of the object include setting the surface conditions of the object relating to gripping, releasing, or changing the grip of the object by expressing the product of the norm of the cross product, which is an operation on the angle between a vector indicating the direction of gripping the object and a vector indicating the axis for defining the orientation of the object. Control at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set conditions, release the grip on the object, or change the grip on the object. A program that causes a computer to execute something.

Images