CONTROL DEVICE, ROBOT SYSTEM, CONTROL METHOD, AND PROGRAM

Abstract

This control device comprises: a limitation means that, via an expression using a direction in which a target object is held and a direction specifying the attitude of the target object, sets a condition of a surface of the target object, wherein said condition is included among limitation conditions in determining an attitude of the target object and a movement path of the target object, and said condition relates to holding the target object, terminating the holding of the target object, or changing the manner of holding the target object; and a control means that controls at least one among a first holding mechanism and a second holding mechanism so that holding the target object, terminating the holding of the target object, or changing the manner of holding the target object is carried out using the surface determined on the basis of the condition set by the limitation means.

Description

Control device, robot system, control method, and recording medium

[0001] The present disclosure relates to a control device, a robot system, a control method, and a recording medium.

[0002] Robots are used in various fields, such as logistics. Some robots operate autonomously. Patent Document 1 discloses, as related technology, a device that generates a trajectory plan for moving the tip of a robot arm from a start point to an end point. Patent Documents 2 and 3 also disclose, as related technology, technologies related to a robot system that grasps an object.

[0003] Japanese Patent Application Publication No. 2021-079482 Japanese Patent Application Publication No. 6996963 Japanese Patent Application Publication No. 2017-100214

[0004] In a robot system, it is desirable to appropriately control the robot arm depending on the state of the object.

[0005] One of the objectives of each aspect of the present disclosure is to provide a control device, a robot system, a control method, and a recording medium that can solve the above-mentioned problems.

[0006] In order to achieve the above object, according to one aspect of the present disclosure, a control device includes a constraint means for setting conditions on the surface of the object related to grasping, releasing the grip of the object, or changing the object, the conditions being included in constraint conditions for determining the posture of the object and the movement path of the object, by expressing the conditions using a direction for grasping the object and a direction that defines the posture of the object, and a control means for controlling at least one of the first gripping mechanism and the second gripping mechanism so that the object is grasped, released from the grip of the object, or changed. The conditions are determined based on the conditions set by the constraint means.

[0007] To achieve the above object, according to another aspect of the present disclosure, a robot system includes a first gripping mechanism, a second gripping mechanism, and the above control device.

[0008] In order to achieve the above object, according to another aspect of the present disclosure, a control method includes a constraint condition for determining the posture of an object and the movement path of the object, and sets conditions for the surface of the object related to grasping, releasing the grip of the object, or changing the object by expressing the direction of grasping the object and a direction that defines the posture of the object, and controls at least one of a first gripping mechanism and a second gripping mechanism so that the object is grasped, released from the grip of the object, or changed based on the set conditions.

[0009] In order to achieve the above object, according to another aspect of the present disclosure, a recording medium stores a program that causes a computer to execute the following steps: set conditions for the surface of an object related to grasping, releasing the grip of the object, or changing the object, the conditions being included in constraints for determining the posture of the object and the movement path of the object, by expressing the direction in which the object is grasped and the direction that defines the posture of the object; and control at least one of the first gripping mechanism and the second gripping mechanism so that the object is grasped, released from the grip of the object, or changed based on the set conditions.

[0010] In order to achieve the above object, according to another aspect of the present disclosure, the control device includes a determination means for determining the direction in which the second gripping mechanism will grip the object based on the direction of the surface of the object being gripped by the first gripping mechanism, and a control means for controlling the operation of the second gripping mechanism so as to grip the object from the direction determined by the determination means.

[0011] In order to achieve the above object, according to another aspect of the present disclosure, a control device includes a determination means for determining the operation of a first gripping mechanism and a second gripping mechanism to grip an object based on the gripping direction of the first gripping mechanism, the gripping direction of the second gripping mechanism, and the direction of the surface of the object, and a control means for controlling the first gripping mechanism and the second gripping mechanism to perform the determined operation.

[0012] According to each aspect of the present disclosure, in a robot system, it is possible to appropriately control a robot arm according to the state of an object.

[0013] FIG. 1 is a diagram illustrating an example of a configuration of a robot system according to an embodiment of the present disclosure. FIG. 2 is a diagram illustrating an example of a configuration of a control device according to an embodiment of the present disclosure. FIG. 3 is a diagram illustrating an example of a configuration of a generation unit according to an embodiment of the present disclosure. FIG. 4 is a diagram illustrating an example of a position coordinate and an axis vector of a robot hand according to an embodiment of the present disclosure. FIG. 5 is a diagram illustrating an example of a position coordinate and an axis vector of an object according to an embodiment of the present disclosure. FIG. 6 is a first diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 7 is a second diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 8 is a third diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 9 is a fourth diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 10 is a fifth diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 11 is a diagram illustrating an example of each process and a movement path of an object according to an embodiment of the present disclosure. FIG. 12 is a diagram illustrating an image of a solution obtained by using Lagrange's undetermined multiplier method. FIG. 13 is a first diagram illustrating a method for efficiently obtaining a desired solution according to an embodiment of the present disclosure. FIG. 14 is a second diagram illustrating a method for efficiently obtaining a desired solution according to an embodiment of the present disclosure. FIG. 15 is a diagram illustrating an example of a sequence of an initial plan generated by a generation unit according to an embodiment of the present disclosure. FIG. 16 is a diagram illustrating an example of a control signal for an initial plan generated by a control unit according to the first embodiment of the present disclosure. FIG. 17 is a diagram illustrating an example of a processing flow of a robot system according to an embodiment of the present disclosure. Fig. 1 is a diagram illustrating an example of a configuration of a robot according to another embodiment of the present disclosure. Fig. 2 is a diagram illustrating an example of a configuration of a control device with a minimum configuration according to an embodiment of the present disclosure. Fig. 3 is a diagram illustrating an example of a processing flow of a control device with a minimum configuration according to an embodiment of the present disclosure. Fig. 4 is a schematic block diagram illustrating the configuration of a computer according to at least one embodiment.

[0014] Hereinafter, an embodiment will be described in detail with reference to the drawings. <Embodiment> A robot system 1 according to an embodiment of the present disclosure is a system that moves an object M present at a certain position to another position. For example, the robot system 1 determines a posture and a path for moving the object M within a range that satisfies constraints described below. The robot system 1 generates a control signal for moving the object M along the determined posture and path. The robot system 1 then controls a robot arm that grasps the object M using the generated control signal. The robot system 1 performs these processes to move the object M present at a certain position to another position. The robot system 1 will be described below.

[0015] 1 is a diagram illustrating an example of a configuration of a robot system 1 according to an embodiment of the present disclosure. As shown in FIG. 1, the robot system 1 includes a control device 2, robots 40a and 40b, and an imaging device 50.

[0016] As shown in FIG. 1 , the robot 40a includes a robot arm 401a, a base 402a, and a robot hand 403a (an example of a first gripping mechanism). The robot arm 401a is connected to the base 402a. The robot hand 403a is connected to the end of the robot arm 401a opposite to the end connected to the base 402a. The robot hand 403a includes, for example, two or more artificial fingers that resemble the fingers of a human or an animal, or a vacuum. The robot hand 403a grips the object M in response to a control signal output by the control device 2. The robot arm 401a moves the object M from a source to a destination in response to a control signal output by the control device 2.

[0017] 1, the robot 40b includes a robot arm 401b, a base 402b, and a robot hand 403b (an example of a second gripping mechanism). The robot arm 401b is connected to the base 402b. The robot hand 403b is connected to the end of the robot arm 401b opposite to the end connected to the base 402b. The robot hand 403b includes, for example, two or more artificial fingers that resemble the fingers of a human or an animal, or a vacuum. The robot hand 403b grips the object M in response to a control signal output by the control device 2. The robot arm 401b moves the object M from the origin to the destination in response to the control signal output by the control device 2.

[0018] In each embodiment of the present disclosure, "grasping" includes "suction" in which an object M is sucked by a vacuum or the like, and "clamping" in which an object is pinched between two or more pseudo fingers that mimic the fingers of a human or animal. Hereinafter, the robots 40a and 40b will be collectively referred to as robot 40. The robot arms 401a and 401b will be collectively referred to as robot arm 401. The pedestals 402a and 402b will be collectively referred to as pedestal 402. The robot hands 403a and 403b will be collectively referred to as robot hand 403.

[0019] The image capturing device 50 captures an image of the state of the object M. The image capturing device 50 is, for example, a depth camera, and is capable of identifying the state (i.e., the position and posture) of the object M. The image captured by the image capturing device 50 is represented, for example, by colored point cloud data, and includes three-dimensional information of the captured object. The image capturing device 50 outputs the captured image to the generation unit 202.

[0020] 2 is a diagram illustrating an example of the configuration of the control device 2 according to an embodiment of the present disclosure. As shown in FIG. 2, the control device 2 includes an input unit 201, a generation unit 202, and a control unit 203.

[0021] The input unit 201 inputs task goals and constraint conditions to the generation unit 202. Examples of task goals include information indicating the type of object M, the number of objects M to be moved, the source of the object M, and the destination of the object M. Examples of constraint conditions include a no-entry area when moving the object M, an area that deviates from the range of motion of the robot 40, and further conditions on the surface of the object M related to gripping the object M, releasing the grip of the object M, or changing the object M. Note that the input unit 201 may accept an input from a user as a task goal, such as "move three items A from a cardboard box C to a tray T," and specify that the type of object M to be moved is item A, the number of items M to be moved is three, the source of the object M is cardboard box C, and the destination of the object M is tray T, and input the specified information to the generation unit 202. Alternatively, the position of the object M identified in the image captured by the image capture device 50 may be set as the origin of movement of the object M. The input unit 201 may, for example, receive from the user the position of an obstacle along the path of the object M from the origin to the destination as a constraint condition indicating a no-entry area, and input the information to the generation unit 202. Alternatively, a file indicating the constraint conditions may be stored in a storage device, and the input unit 201 may input the constraint conditions indicated by the file to the generation unit 202, or a fourth processing unit 202d (described later) of the generation unit 202 may read the constraint conditions directly from the file, or both. In other words, any acquisition method may be used as long as the generation unit 202 can acquire the necessary task goal and the necessary constraint conditions. Details of how to define the constraint conditions will be described later.

[0022] 3 is a diagram illustrating an example of the configuration of the generation unit 202 according to an embodiment of the present disclosure. As illustrated in FIG. 3, the generation unit 202 includes a first processing unit 202a, a second processing unit 202b, a third processing unit 202c, a fourth processing unit 202d (an example of a constraint unit), and a fifth processing unit 202e (an example of a control unit).

[0023] The first processing unit 202a recognizes the robot 40. For example, the first processing unit 202a recognizes a robot model using CAD (Computer Aided Design) data. This CAD data includes information indicating the shape of the robot 40 and information indicating the range of motion, such as the reach range of the robot arm 401. The shape includes dimensions. The CAD data is, for example, drawing data designed using CAD.

[0024] The first processing unit 202a also recognizes the environment around the robot 40. For example, the first processing unit 202a acquires images captured by the camera device 50. The images captured by the camera device 50 include information captured by the camera and depth information. This depth information corresponds to the colored point cloud data described above. The first processing unit 202a recognizes the position and shape of obstacles from the acquired images. Here, obstacles refer to all objects present within the camera range of the camera device 50, other than the target object M that the robot 40 is to move to its destination. As described above, the camera device 50 is capable of acquiring three-dimensional information of objects within the camera range. Therefore, the first processing unit 202a can recognize the environment around the robot 40, including the position and shape of obstacles. Note that the first processing unit 202a is not limited to recognizing the environment around the robot 40 from images captured by the camera device 50. For example, the first processing unit 202a may recognize the environment around the robot 40 using a three-dimensional occupancy map (Octomap), CAD data, AR (Augmented Reality) markers, etc. This CAD data includes information indicating the shape of obstacles. The shape includes dimensions.

[0025] The first processing unit 202a also recognizes the release position at the destination of the object M. For example, when the destination is a container (e.g., a tray T), the first processing unit 202a recognizes the release position by machine learning using model-based matching. Model-based matching is a technique for determining the position and orientation of an object by using image data obtained from a camera or the like and shape and structural data of the object (in this case, a container) whose position and orientation are to be acquired, and matching the shape and structural data with the object extracted from the image. Note that the first processing unit 202a is not limited to a device that recognizes the release position by machine learning using model-based matching. For example, the first processing unit 202a may also recognize the release position using an AR marker.

[0026] The second processing unit 202b also recognizes a pedestal 402 (described later) of the robot 40. For example, the second processing unit 202b recognizes the pedestal 402 by acquiring CAD data. This CAD data includes information indicating the shape of the pedestal 402. The shape includes dimensions. This allows the second processing unit 202b to recognize the Z coordinate of the top surface of the pedestal 402 in the coordinate system as the height of the pedestal 402.

[0027] The third processing unit 202c recognizes the state (i.e., position and posture) of the object M. For example, the third processing unit 202c recognizes the position of the object M by performing machine learning using model-based matching. Furthermore, the third processing unit 202c recognizes the posture of the object M by using a technique for generating a bounding box, such as an AABB (Axis Aligned Bounding Box) or an OBB (Oriented Bounding Box), for the object M whose position has been identified. Note that the third processing unit 202c may classify the object M using clustering, which is a machine learning technique, for the image captured by the image capture device 50, and identify the state of the object M by using a technique for generating a bounding box.

[0028] Furthermore, the third processing unit 202c acquires the height of the object M. For example, the third processing unit 202c recognizes the object M by acquiring CAD data. This CAD data includes information indicating the shape of the object M. The shape includes dimensions. This allows the third processing unit 202c to recognize the Z coordinate of the object M in that coordinate system as the height of the object M. Note that the third processing unit 202c may recognize the height of the object M by subtracting the Z coordinate of the base 402 from the Z coordinate of the top surface of the object M.

[0029] The fourth processing unit 202d acquires the constraint conditions. Then, the fourth processing unit 202d sets the acquired constraint conditions. Here, a method for defining the constraint conditions will be described in detail. Here, it will be described how the surfaces of the object M related to the robot 40 gripping the object M, releasing the grip of the object M, or changing the grip of the object M are described as constraint conditions.

[0030] FIG. 4 is a diagram illustrating an example of the position coordinates and axis vectors of the robot hand 403 according to an embodiment of the present disclosure. First, the position coordinates and axis vectors of the robot hand 403 at time k are defined using the notation in FIG. 4 . That is, the position coordinate r(h1, k) of the robot hand 403a at time k is expressed as in Equation (1). Furthermore, the position coordinate r(h2, k) of the robot hand 403b at time k is expressed as in Equation (2). Note that the position coordinates r(h1, k) and r(h2, k) respectively represent the positions of the robot hand 403a and the robot hand 403b at time k in a three-dimensional space R3 (e.g., a three-dimensional space R3 represented by three axes, i.e., the x-axis, the y-axis, and the z-axis in FIG. 4 ) in which the robot 40 including the robot hand 403a and the robot hand 403b operates and the target M moves. The x-axis vector x(h1,k) of the robot hand 403a at time k is expressed as equation (3), the y-axis vector y(h1,k) as equation (4), and the z-axis vector z(h1,k) as equation (5). The x-axis vector x(h2,k) of the robot hand 403b at time k is expressed as equation (6), the y-axis vector y(h2,k) as equation (7), and the z-axis vector z(h2,k) as equation (8). Uniquely defined x-, y-, and z-axes are set for each of the three-dimensional shapes of the robot hand 403a and the robot hand 403b, independent of the three-dimensional space R3. The x-axis vector x(h1,k) indicates the direction of the x-axis set for the three-dimensional shape of the robot hand 403a in the three-dimensional space R3 at time k. The y-axis vector y(h1, k) indicates the direction of the y-axis set for the three-dimensional shape of the robot hand 403a in the three-dimensional space R3 at time k. The z-axis vector z(h1, k) indicates the direction of the z-axis set for the three-dimensional shape of the robot hand 403a in the three-dimensional space R3 at time k. The x-axis vector x(h2, k) indicates the direction of the x-axis set for the three-dimensional shape of the robot hand 403b in the three-dimensional space R3 at time k. The y-axis vector y(h2, k) indicates the direction of the y-axis set for the three-dimensional shape of the robot hand 403b in the three-dimensional space R3 at time k.The z-axis vector z(h2, k) indicates the direction of the z-axis set for the three-dimensional shape of the robot hand 403b in the three-dimensional space R3 at time k.

[0031]

[0032]

[0033]

[0034]

[0035]

[0036]

[0037]

[0038]

[0039] Each of the axial vectors expressed by Equations (3) to (8) is a unit vector. Note that, for convenience, the axial vectors are expressed using unit vectors here, but they do not necessarily have to be unit vectors, and the length of the vector may change depending on the direction of the axial vector.

[0040] Also, a three-dimensional space R3 of the x-axis, y-axis, and z-axis shown in FIG. 4 is expressed as in equation (9).

[0041]

[0042] In this case, the formula (10) holds for the position coordinates of the robot hand 403. The formula (11) holds for the axis vector of the robot hand 403.

[0043]

[0044]

[0045] That is, the position coordinates of each robot hand 403 at time k are one element of the three-dimensional space. Also, the axis vector of each robot hand 403 at time k is one element of the three-dimensional space.

[0046] In this case, the position coordinates of the robot hand 403a can be defined as in equation (12). The position coordinates of the robot hand 403b can be defined as in equation (13). Note that T in equations (12) and (13) represents a transposition operation.

[0047]

[0048]

[0049] 5 is a diagram illustrating an example of the position coordinates and axis vectors of the object M according to an embodiment of the present disclosure. Here, the position coordinates and axis vectors of the object M at time k are defined using the notation in FIG. 5 . That is, the position coordinate r(obj, k) of the object M at time k is expressed as in Equation (14). Furthermore, the x-axis vector x(obj, k) of the object M at time k is expressed as in Equation (15), the y-axis vector y(obj, k) is expressed as in Equation (16), and the z-axis vector z(obj, k) is expressed as in Equation (17). The position coordinate r(obj, k) represents the position of the object M at time k in a three-dimensional space R3 (e.g., the three-dimensional space R3 represented by three axes, the x-axis, the y-axis, and the z-axis in FIG. 4 ). Similarly to the robot hands 403a and 403b, uniquely determined x-, y-, and z-axes are set for the three-dimensional shape of the object M, independent of the three-dimensional space R3. The x-axis vector x(obj, k) indicates the direction of the x-axis set for the three-dimensional shape of the object M in the three-dimensional space R3 at time k. The y-axis vector y(obj, k) indicates the direction of the y-axis set for the three-dimensional shape of the object M in the three-dimensional space R3 at time k. The z-axis vector z(obj, k) indicates the direction of the z-axis set for the three-dimensional shape of the object M in the three-dimensional space R3 at time k.

[0050]

[0051]

[0052]

[0053]

[0054] That is, the position coordinates of the object M at time k are one element of the three-dimensional space. Also, the axis vector of the object M at time k is one element of the three-dimensional space.

[0055] (In the case of a robot hand of the type that adsorbs an object) First, for a robot hand 403 of the type that adsorbs an object M, we will explain what constraints are described for the surface of the object M regarding grasping the object M, releasing the grasp of the object M, or changing the grip of the object M.

[0056] (When gripping or releasing an object) FIG. 6 is a first diagram illustrating constraints according to an embodiment of the present disclosure. FIG. 6 is a diagram illustrating constraints when the position coordinates and axis vectors of the robot hand 403 and the position coordinates and axis vectors of the object M are defined as in the above-described formulas (1) to (17). As can be seen from FIG. 6, when gripping or releasing the object M, the direction perpendicular to the suction surface of the object M must match the suction direction of the robot hand 403. The vector indicating this suction direction is an example of a first vector. In other words, when gripping or releasing the object M, the constraints on the robot hand 403a can be described by the product of the norm of a cross product, which is a calculation of the angle between a first vector indicating the direction in which the object M is gripped and a second vector indicating an axis for defining the posture of the object M, as in formula (18). Furthermore, when gripping or releasing the object M, the constraints on the robot hand 403b can be described by the product of the norms of cross products using cross products as in equation (19).

[0057]

[0058]

[0059] Note that equations (18) and (19) are constraints that assume that the object M has a shape as shown in Fig. 6, which has two faces parallel to each of the x-axis, y-axis, and z-axis directions, such as a hexahedron such as a cube or a rectangular parallelepiped. For example, in the case of a tetrahedron, the description of the constraints differs from equations (18) and (19).

[0060] FIG. 7 is a second diagram illustrating constraints according to an embodiment of the present disclosure. In FIG. 7 , normals of each face of a tetrahedron are shown. When gripping or releasing an object M having such a tetrahedron shape, the constraints for the robot hand 403a can be expressed by the product of the norms of cross products using cross products, as in Equation (20), based on the same concept as for the hexahedron shown in FIG. 6 (i.e., based on the concept that the direction perpendicular to the suction surface of the object M must coincide with the suction direction of the robot hand 403). Furthermore, when gripping or releasing the object M, the constraints for the robot hand 403b can be expressed by the product of the norms of cross products using cross products, as in Equation (21).

[0061]

[0062]

[0063] However, in the case of this tetrahedron, there are no parallel faces. Therefore, unless the robot hand 403 approaches the face of the object M to be grasped from the correct direction, the object M cannot be grasped. The constraints expressed by the product of the norms of cross products using cross products, such as in Equation (20) and Equation (21), indicate scalar quantities. Therefore, although the constraints require that one of the normals to each face be parallel to the suction direction of the robot hand 403, they do not require that one of the normals to each face be coincident with the suction direction of the robot hand 403. Therefore, in order to require that one of the normals to each face be coincident with the suction direction of the robot hand 403 using the constraints, it is necessary to add a new constraint indicating which face's normal direction coincides with the suction direction of the robot hand 403. Examples of such constraints are the constraints added to the robot hand 403a using Equation (22) and the constraints added to the robot hand 403b using Equation (23).

[0064]

[0065]

[0066] The constraints expressed by equations (22) and (23) require that the magnitude of the dot product, which is the calculation of the angle between the vector indicated by the normal to each face and the vector indicating the suction direction of the robot hand 403, be 1, i.e., that the directions of these vectors be the same. Therefore, for an object M having a shape without parallel faces, such as a tetrahedron, by adding a constraint, such as equation (22) or (23), which requires that the magnitude of the dot product between the vector indicated by the normal to each face and the vector indicating the suction direction of the robot hand 403 be 1, it is possible to cause the robot hand 403 to grasp the object M.

[0067] In addition, when a hexahedron such as a cube or rectangular parallelepiped has two faces parallel to each of the x-axis, y-axis, and z-axis directions, the norms of the cross product overlap, and thus, equations such as Equation (18) and Equation (19) are used, which indicate the calculation of one norm of the cross product for each of the x-axis, y-axis, and z-axis directions. However, equations such as Equation (22) and Equation (23) are used to calculate the inner product for each face. In other words, by using equations such as Equation (22) and Equation (23) for each face of the object M, which are calculations based on the angle between two directions, i.e., equations that calculate the inner product, it is possible to define equations that indicate constraints for an object M of any shape. Note that constraints based on equations that calculate the inner product for each face include constraints based on equations that calculate the cross product. Therefore, when equations such as Equation (22) and Equation (23) are used to calculate the inner product for each face, constraints based on equations that calculate the cross product are not necessary. However, the new constraint indicating which surface's normal direction coincides with the suction direction of the robot hand 403 is not limited to the dot product, and if a constraint based on an equation that calculates the cross product is used, it is sufficient to simply add a new constraint indicating which surface's normal direction coincides with the suction direction of the robot hand 403.

[0068] (When transferring an object) FIG. 8 is a third diagram for explaining constraints according to an embodiment of the present disclosure. FIG. 8 is an image diagram of processing performed by the robot hand 403a and the robot hand 403b. Part (a) of FIG. 8 shows processing in which the robot hand a grasps the object M. Part (b) of FIG. 8 shows processing in which the robot hand 403a transfers the object M to the robot hand 403b (i.e., handing over the object M). Part (c) of FIG. 8 shows processing in which the robot hand 403b releases its grip on the object M. The description of constraints in the processing shown in parts (a) and (c) of FIG. 8 is as described above. Here, the description of constraints in the processing shown in part (b) of FIG. 8 (i.e., processing in which the object M is transferred) will be described.

[0069] When transferring the object M from the robot hand 403a to the robot hand 403b, each of the robot hands 403 must satisfy the constraints for gripping or releasing the object M. Therefore, as shown in Fig. 8, when the object M is a hexahedron such as a cube or a rectangular parallelepiped, the constraints are constraints that simultaneously require the expressions (18) and (19). Note that when the object M does not have parallel faces, such as a tetrahedron, the constraints are constraints that simultaneously require the expressions (20) to (23).

[0070] (In the case of a robot hand that clamps an object) Next, for a robot hand 403 that clamps an object M, we will explain how the surface of the object M is described as a constraint condition regarding grasping the object M, releasing the grip of the object M, or changing the grip of the object M.

[0071] FIG. 9 is a fourth diagram for describing constraint conditions according to an embodiment of the present disclosure. The direction of v in FIG. 9 corresponds to the direction in which the robot hand 403 grips the target M. FIG. 10 is a fifth diagram for describing constraint conditions according to an embodiment of the present disclosure. n1, n2, and n3 in FIG. 10 correspond to the y-axis vector, x-axis vector, and z-axis vector shown in FIG. 5. In the case of a robot hand 403 that grips the target M, the suction direction of the robot hand 403 that suctions the target M can be aligned with the direction in which the target M is gripped, i.e., the direction of v in FIG. 9, and the constraint conditions can be considered to be the same as those for the robot hand 403 that suctions the target M.

[0072] (When grasping or releasing an object) The constraints for grasping or releasing the object M for the robot hand 403 shown in Figure 9 and the object M shown in Figure 10 can be expressed as equation (24) for each robot hand 403 by matching the suction direction of the robot hand 403 of the type that adsorbs the object M with the direction in which the object M is clamped.

[0073]

[0074] Furthermore, the constraints for transferring the object M for the robot hand 403 shown in FIG. 9 and the object M shown in FIG. 10 require that the equation (24) be satisfied simultaneously for each of the robot hands 403.

[0075] The fourth processing unit 202d sets various constraints including the constraints described above.

[0076] The fifth processing unit 202e generates an initial plan sequence indicating the flow of motion of the robot 40 based on the task goals determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c and the constraint conditions set by the processing by the fourth processing unit 202d. For example, the fifth processing unit 202e acquires the task goals from the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c. The fifth processing unit 202e also acquires the constraint conditions from the fourth processing unit 202d. The fifth processing unit 202e adds the constraint conditions acquired from the fourth processing unit 202d to the constraint conditions input from the input unit 201. Then, based on the acquired work goal and constraint conditions, the fifth processing unit 202e generates information indicating each state of the robot 40 at each time step from the state at the source of the object M to the state at the destination of the object M, which is necessary for the control unit 203 to generate a control signal for controlling the robot 40 (including, for example, the type of object M, the position and posture of the robot 40, the strength of the grip of the object M, the operation of the robot 40 (e.g., the approach operation to approach the object M (equivalent to the processing of the approach process in Figure 11 described later), the pick operation to grip the object M (equivalent to the processing of the pick process in Figure 11)), the carry operation to move the arm to correctly move the gripped object M to the destination (equivalent to the processing of the carry process in Figure 11), and the place operation to release the grip of the object M (equivalent to the processing of the place process in Figure 11)). In other words, the sequence is information indicating each state of the robot 40 at each time step from the state at the origin of the object M to the state at the destination of the object M, which is necessary for the control unit 203 to generate a control signal to control the robot 40.

[0077] 11 is a diagram illustrating an example of each process and a movement path of the object M according to an embodiment of the present disclosure. When a task goal and various constraints are set, the processes in the sequence for moving the object M illustrated in FIG. 11 and the movement path of the object M are determined by the fifth processing unit 202e executing a simulation using objective functions such as reducing the amount of energy consumed by the robot 40 as much as possible, shortening the trajectory of the robot arm 401 as much as possible, and shortening the movement path of the object M as much as possible. As illustrated in FIG. 11 , examples of processes for moving the object M from the origin to the destination include an approach process for moving the robot arms 401a and 401b toward the object M, a pick process for grasping the object M, a carry process for moving the object M, and a place process for releasing the grasp of the object M.

[0078] Here, a method in which the fifth processing unit 202e determines the posture and movement path of the object M for each time step through simulation will be described. For simplicity, a method in which the trajectory of the movement of the robot arm 401 is minimized will be described. In this case, variables (e.g., x, y) that affect the trajectory of the movement of the robot arm 401 are defined, and the objective function is to minimize a function f(x, y) that represents the trajectory of the movement of the robot arm 401. Furthermore, for example, a constraint condition such as x+y=0 is set due to a restriction on the movable area of ​​the robot arm 401. In such a case, the Lagrangian function L is given by equation (25) using positive λ.

[0079]

[0080] The fifth processing unit 202e may then use Lagrange's multiplier method to find a solution. FIG. 12 is a diagram illustrating an example of a solution found using Lagrange's multiplier method. In this case, the fifth processing unit 202e repeatedly searches for a region that satisfies the constraint condition x + y = 0 and that results in a local minimum value by differentiating the Lagrange function L, thereby identifying a desired solution (an example of an optimal solution) indicated by a star in FIG. 12 where the objective function f(x, y) is minimized. Note that the Lagrange function L expressed by equation (25) is just an example, and any Lagrange function generally used in continuous optimization may be used. For example, when using an optimization algorithm based on a gradient method known as the primal-dual interior point method, the Lagrange function L has a value of zero in a region where the constraints are not violated and takes on a value of infinity as soon as it enters a region where the constraints are violated, which is called a barrier function.

[0081] However, when finding a solution using the derivative of a function under certain constraints, as described above, a problem commonly known as "non-convex constraints" may arise, in which the searchable area for solutions limited by the constraints is limited and it is not possible to search for a local minimum.

[0082] Here, a method for efficiently finding a desired solution even when a problem with non-convex constraints occurs will be described. Here, it is assumed that the shape of the object M is a hexahedron with parallel faces, such as a cube or a rectangular parallelepiped. Furthermore, it is assumed that the robot hand 403 is a type that clamps the object M. When the shape of the object M is a hexahedron with parallel faces and the robot hand 403 is a type that clamps the object M, equation (24) holds. FIG. 13 is a first diagram for explaining a method for efficiently finding a desired solution according to an embodiment of the present disclosure. In equation (24), the vector ni indicating the suction direction of the robot hand 403 is represented by vectors ex, ey, and ez in FIG. 13.

[0083] If the magnitude of vector v is 1, then among the possible positions of vector v in FIG. 13, only the six points indicated by stars are feasible (feasible). That is, all points other than the six points indicated by stars on the sphere in FIG. 13 are infeasible (infeasible). In this case, if the infeasible points are set as initial search points, the fifth processing unit 202e will not be able to search for points other than the initial search points. That is, the problem of non-convex constraints occurs.

[0084] In such a case, the fifth processing unit 202e uses, for example, the SA (Simulated Annealing) method. Specifically, the fifth processing unit 202e first relaxes the constraints to expand the search points and find a local minimum (an example of a locally optimal solution). FIG. 14 is a second diagram for explaining a method for efficiently finding a desired solution in one embodiment of the present disclosure. FIG. 14 shows only one-eighth of the sphere shown in FIG. 13. Also, portions (a), (b), and (c) in FIG. 14 show differences in infeasible regions depending on the degree of relaxation of the constraints using tk, which is a positive value. Also, portion (d) in FIG. 14 shows an infeasible region in the case of the original constraints (tk = 0). In addition, when the shape of the object M is a hexahedron with parallel faces such as a cube or a rectangular parallelepiped, due to the symmetry of the vectors, infeasible areas similar to the infeasible areas shown in Figure 14 occur in parts other than one-eighth of the spherical surface shown in Figure 14.

[0085] In the example of tk = 1 shown in part (a) of Figure 14, all regions are feasible regions. Also, in the example of tk = 0.3 shown in part (b) of Figure 14, some regions are infeasible regions. Also, in the example of tk = 0.25 shown in part (c) of Figure 14, the infeasible regions divide the feasible regions into multiple discontinuous regions. As shown in part (c) of Figure 14, when the infeasible regions divide the feasible regions into multiple discontinuous regions, the fifth processing unit 202e cannot search for other feasible regions from a certain feasible region. In other words, if a desired solution does not exist in a certain feasible region, the fifth processing unit 202e will not be able to obtain the desired solution. 14(c), when an infeasible region divides the feasible region into multiple discontinuous regions, the fifth processing unit 202e searches for a solution by adjusting the degree of relaxation of the constraint conditions and changing the search conditions, such as randomly changing the search points, rather than continuing the same search. Note that adjusting the degree of relaxation of the constraint conditions and changing the search conditions are examples of changing the content of relaxation.

[0086] Therefore, even when a problem with non-convex constraints occurs, a method for efficiently obtaining a desired solution involves first determining, in advance by simulation or the like, constraint conditions under which an infeasible region divides the feasible region into multiple discontinuous regions (an example of a condition under which a local optimum solution cannot be obtained), as shown in part (c) of FIG. 14 . The fifth processing unit 202e stores the previously determined constraint conditions. Then, as shown in part (a) of FIG. 14 , the fifth processing unit 202e relaxes the constraint conditions so that all regions become feasible. With the constraint conditions relaxed, the fifth processing unit 202e obtains a minimum value of the Lagrange function L by differentiating the Lagrange function L. The fifth processing unit 202e then checks whether this minimum value satisfies the original constraint conditions. If the original constraint conditions are satisfied, the fifth processing unit 202e determines that this minimum value is the desired solution. Furthermore, if the original constraints are not satisfied, the fifth processing unit 202e reduces the degree of relaxation of the constraints and differentiates the Lagrange function L to obtain a minimum value of the Lagrange function L. Then, the fifth processing unit 202e checks whether this minimum value satisfies the original constraints. The fifth processing unit 202e repeats this process until a minimum value that satisfies the original constraints is obtained, or until the constraints with the reduced degree of relaxation become the previously determined constraints (i.e., the constraints in which the infeasible region divides the feasible region into a plurality of discontinuous regions). Furthermore, if the constraints with the reduced degree of relaxation become the constraints in which the previously determined infeasible region divides the feasible region into a plurality of discontinuous regions, the fifth processing unit 202e adjusts the degree of relaxation of the constraints to change the search conditions and search for a solution. In this way, the control device 2 can efficiently find a desired solution even when a problem with non-convex constraints occurs.

[0087] The fifth processing unit 202e outputs the generated sequence to the control unit 203. Note that the fifth processing unit 202e may be realized using artificial intelligence (AI) technology including temporal logic, reinforcement learning, optimization technology, and the like.

[0088] 15 is a diagram illustrating an example of a sequence TBL1 of an initial plan generated by the generator 202 according to an embodiment of the present disclosure. For example, the sequence TBL1 of the initial plan generated by the generator 202 is a sequence indicating each state of the robot 40 for each n time step from the origin to the destination of the object M, as shown in FIG.

[0089] The control unit 203 generates a control signal for controlling the robot 40 based on the sequence generated by the generation unit 202. That is, the control unit 203 generates a control signal that realizes the posture of the object M and the movement path of the object M according to the sequence generated by the generation unit 202. The control unit 203 outputs the generated control signal to the robot 40.

[0090] 16 is a diagram illustrating an example of the control signal Cnt of the initial plan generated by the control unit 203 according to the first embodiment of the present disclosure. For example, the control signal Cnt of the initial plan generated by the control unit 203 is, for example, each control signal for n time steps from the movement origin to the movement destination of the object M, as shown in FIG.

[0091] 17 is a diagram illustrating an example of a processing flow of the robot system 1 according to an embodiment of the present disclosure. Here, the processing performed by the robot system 1 to generate a sequence and control the robot 40 will be described with reference to FIG. 17. It is assumed here that each of the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c performs the above-described processing.

[0092] The fourth processing unit 202d sets various constraint conditions (step S1). For example, the fourth processing unit 202d sets the conditions of the surface of the object M related to grasping the object M, releasing the grasp of the object M, or changing the grip of the object M, which are included in the constraint conditions for determining the posture of the object M and the movement path of the object M, by an expression using the cross product of a vector (an example of a first vector) indicating the direction in which the object M is grasped, and an x-axis vector (an example of a second vector), a y-axis vector (an example of a second vector), and a z-axis vector (an example of a second vector) for defining the posture of the object M.

[0093] The fifth processing unit 202e generates an initial plan sequence indicating the flow of motion of the robot 40 based on the task goals determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c and the constraint conditions set by the processing by the fourth processing unit 202d (step S2). For example, the fifth processing unit 202e acquires the task goals from the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c. The fifth processing unit 202e also acquires the constraint conditions from the fourth processing unit 202d. The fifth processing unit 202e adds the constraint conditions acquired from the fourth processing unit 202d to the constraint conditions input from the input unit 201. Then, based on the acquired work goal and constraint conditions, the fifth processing unit 202e generates information indicating each state of the robot 40 at each time step from the state at the source of the object M to the state at the destination of the object M, which is necessary for the control unit 203 to generate a control signal for controlling the robot 40 (including, for example, the type of object M, the position and posture of the robot 40, the strength of the grip of the object M, the operation of the robot 40 (e.g., the approach operation to approach the object M (equivalent to the processing of the approach process in Figure 11), the pick operation to grip the object M (equivalent to the processing of the pick process in Figure 11), the carry operation to move the arm to correctly move the gripped object M to the destination (equivalent to the processing of the carry process in Figure 11), and the place operation to release the grip of the object M (equivalent to the processing of the place process in Figure 11)).

[0094] For example, the fifth processing unit 202e determines the posture and movement path of the object M for each time step through simulation. Specifically, the fifth processing unit 202e may find a solution by using Lagrange's method of undetermined multipliers for the Lagrangian function L. More specifically, the fifth processing unit 202e identifies a desired solution that minimizes the objective function f(x, y) by repeatedly searching for a region that satisfies the constraints and that becomes a local minimum by differentiating the Lagrangian function L.

[0095] Furthermore, even when a problem with non-convex constraints occurs, the fifth processing unit 202e uses the SA method to efficiently find the desired solution. Specifically, the fifth processing unit 202e stores constraints that divide a previously obtained infeasible region into multiple discontinuous regions (step S3). Then, as shown in part (a) of FIG. 14, the fifth processing unit 202e relaxes the constraints so that all regions become feasible (step S4). With the constraints relaxed, the fifth processing unit 202e differentiates the Lagrange function L to find the minimum value of the Lagrange function L (step S5). The fifth processing unit 202e then determines whether this minimum value satisfies the original constraints (step S6). If the fifth processing unit 202e determines that the minimum value satisfies the original constraints (YES in step S6), it determines the minimum value as the desired solution (step S7). Then, the control unit 203 generates a control signal for controlling the robot 40 based on the sequence generated by the fifth processing unit 202e of the generation unit 202 (step S8). The control unit 203 outputs the generated control signal to the robot 40 (step S9).

[0096] Furthermore, if the fifth processing unit 202e determines that the minimum value does not satisfy the original constraint condition (NO in step S6), it determines whether the constraint condition has become a pre-stored constraint condition (step S10). If the fifth processing unit 202e determines that the constraint condition has not become a pre-stored constraint condition (NO in step S10), it reduces the degree of relaxation of the constraint condition (step S11). Then, the fifth processing unit 202e returns to the processing of step S5.

[0097] Furthermore, if the fifth processing unit 202e determines that the constraint conditions have become the pre-stored constraint conditions (YES in step S10), it changes the search conditions by adjusting the degree of relaxation of the constraint conditions (step S12).Then, the fifth processing unit 202e returns to the processing of step S5.

[0098] As described above, the fifth processing unit 202e generates an initial plan sequence indicating the flow of motion of the robot 40 based on the task goals determined by the processing of the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c and the constraints set by the processing of the fourth processing unit 202d. Therefore, for example, in the process of transferring the object M shown in part (b) of FIG. 8, the fifth processing unit 202e (an example of a determining unit) determines the direction in which the robot hand 403b (an example of a second gripping mechanism) will grip the object M based on the direction of the surface of the object M gripped by the robot hand 403a (an example of a first gripping mechanism). Furthermore, the control unit 203 (an example of a control unit) controls the motion of the robot hand 403b so that the robot hand 403b grips the object M from the gripping direction determined by the fifth processing unit 202e.

[0099] As described above, the fifth processing unit 202e generates an initial plan sequence indicating the flow of operation of the robot 40, based on the task goal determined by the processing by the first processing unit 202a, the second processing unit 202b, and the third processing unit 202c and the constraints set by the processing by the fourth processing unit 202d. Therefore, for example, with respect to the processing of transferring the object M shown in part (b) of Figure 8, the fifth processing unit 202e (an example of a determining means) also determines a first direction in which the robot hand 403a (an example of a first gripping mechanism) grips the object M and a second direction in which the robot hand 403b (an example of a second gripping mechanism) grips the object M, based on the direction of the surface of the object M. In addition, the control unit 203 (an example of a control means) controls the operation of the robot hand 403a so that the robot hand 403a grasps the object M from the first direction, and also controls the operation of the robot hand 403b so that the robot hand 403b grasps the object M from the second direction.

[0100] (Advantages) The robot system 1 according to an embodiment of the present disclosure has been described above. In the robot system 1, the fourth processing unit 202d (an example of a constraint means) sets the conditions of the surface of the object M related to grasping, releasing the grip of the object M, or changing the grip of the object M, which are included in the constraint conditions for determining the posture of the object M and the movement path of the object M, using an expression that uses the cross product of a vector (an example of a first vector) indicating the direction in which the object M is grasped, an x-axis vector (an example of a second vector), a y-axis vector (an example of a second vector), and a z-axis vector (an example of a second vector) that define the posture of the object M. The control unit 203 (an example of a control means) controls at least one of the robot hand 403a (an example of a first gripping mechanism) and the robot hand 403b (an example of a second gripping mechanism) so that the object M is grasped, released from the grip of the object M, or changed over using the surface of the object M determined based on the constraint conditions set by the fourth processing unit 202d.

[0101] In this way, the robot system 1 can appropriately control the robot arm according to the state of the object.

[0102] Furthermore, by doing so, constraints can be set simply in the robot system 1 using, for example, cross product expressions.

[0103] In one embodiment of the present disclosure, the robot 40a is described as having the robot arm 401a and the robot hand 403a, and the robot 40b is described as having the robot arm 401b and the robot hand 403b. However, in another embodiment of the present disclosure, a single robot may have the robot arm 401a, the robot hand 403a, the robot arm 401b, and the robot hand 403b. FIG. 18 is a diagram showing an example of the configuration of a robot 40c according to another embodiment of the present disclosure. For example, as shown in FIG. 18, the robot 40c includes the robot arm 401a, the robot arm 401b, a base 402c, a robot hand 403a, and a robot hand 403b. The control device 2 may then control the robot arm 401a, robot hand 403a, robot arm 401b, and robot hand 403b provided on the robot 40c, in the same manner as it controls the robot arm 401a and robot hand 403a provided on the robot 40a, and the robot arm 401b and robot hand 403b provided on the robot 40b in one embodiment of the present disclosure.

[0104] Next, a control device 2 with a minimum configuration according to an embodiment of the present disclosure will be described. FIG. 19 is a diagram illustrating an example of the configuration of a control device 2 with a minimum configuration according to an embodiment of the present disclosure. As shown in FIG. 19 , the control device 2 with a minimum configuration according to an embodiment of the present disclosure includes a fourth processing unit 202d (an example of a constraint means) and a control unit 203 (an example of a control means). The fourth processing unit 202d sets the conditions of the surface of the object related to grasping, releasing the grip of the object, or changing the object, which are included in the constraint conditions for determining the posture of the object and the movement path of the object, by expressing the direction of grasping the object and the direction that defines the posture of the object. The fourth processing unit 202d can be realized, for example, using the function of the fourth processing unit 202d illustrated in FIG. 3. The control unit 203 controls at least one of the first gripping mechanism and the second gripping mechanism so that the object is grasped, released from the grip of the object, or changed. The conditions are included in the constraint conditions for determining the posture of the object and the movement path of the object. The control unit 203 can be realized, for example, by using the functions of the control unit 203 illustrated in FIG.

[0105] Next, a description will be given of processing by the control device 2 with the minimum configuration according to an embodiment of the present disclosure. Fig. 20 is a diagram showing an example of a processing flow of the control device 2 with the minimum configuration according to an embodiment of the present disclosure. Here, the processing by the control device 2 with the minimum configuration will be described with reference to Fig. 20.

[0106] The fourth processing unit 202d (an example of a constraint means) sets a condition of the surface of the object related to grasping, releasing, or changing the object, which is included in the constraint conditions for determining the posture of the object and the movement path of the object, by an expression using a direction for grasping the object and a direction that defines the posture of the object (step S101). The control unit 203 (an example of a control means) controls at least one of the first gripping mechanism and the second gripping mechanism so that the object is grasped, released, or changed using the surface determined based on the conditions set by the fourth processing unit 202d (step S102).

[0107] The above has described a control device 2 with a minimum configuration according to an embodiment of the present disclosure. With this control device 2, it is possible to appropriately control a robot arm in a robot system according to the state of an object.

[0108] The order of the processes in the embodiments of the present disclosure may be changed as long as the processes are performed appropriately.

[0109] Although the embodiments of the present disclosure have been described, the robot system 1, the control device 2, the input unit 201, the generation unit 202, the control unit 203, the robot 40, the image capture device 50, and other control devices may have a computer device inside. The above-described processing steps are stored in the form of a program on a computer-readable recording medium, and the above processing is performed by the computer reading and executing the program. Specific examples of computers are shown below.

[0110] FIG. 21 is a schematic block diagram showing the configuration of a computer according to at least one embodiment. As shown in FIG. 21 , the computer 5 includes a CPU (Central Processing Unit) 6, a main memory 7, a storage 8, and an interface 9. For example, the robot system 1, the control device 2, the input unit 201, the generation unit 202, the control unit 203, the robot 40, the imaging device 50, and other control devices described above are each implemented in the computer 5. The operation of each of the processing units described above is stored in the storage 8 in the form of a program. The CPU 6 reads the program from the storage 8, loads it into the main memory 7, and executes the above-described processing in accordance with the program. The CPU 6 also allocates storage areas in the main memory 7 corresponding to each of the storage units described above in accordance with the program.

[0111] Examples of storage 8 include a hard disk drive (HDD), a solid state drive (SSD), a magnetic disk, a magneto-optical disk, a compact disc read-only memory (CD-ROM), a digital versatile disc read-only memory (DVD-ROM), and a semiconductor memory. Storage 8 may be an internal medium directly connected to the bus of computer 5, or an external medium connected to computer 5 via interface 9 or a communication line. Furthermore, if the program is distributed to computer 5 via a communication line, computer 5 that receives the program may load the program into main memory 7 and execute the above-described processing. In at least one embodiment, storage 8 is a non-transitory tangible storage medium.

[0112] The program may also implement some of the functions described above. Furthermore, the program may be a file that can implement the functions described above in combination with a program already stored in the computer device, a so-called differential file (differential program).

[0113] Although several embodiments of the present disclosure have been described, these embodiments are merely examples and do not limit the scope of the disclosure. Various additions, omissions, substitutions, and modifications may be made to these embodiments without departing from the spirit of the disclosure.

[0114] Note that part or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes.

[0115] (Supplementary Note 1) A control device comprising: a constraint means for setting conditions on a surface of an object relating to grasping, releasing the grip of the object, or changing the object, the conditions being included in constraint conditions for determining the posture of the object and the movement path of the object, by an expression using a direction for grasping the object and a direction that defines the posture of the object; and a control means for controlling at least one of a first gripping mechanism and a second gripping mechanism so that the object is grasped, released from the grip of the object, or changed over using the surface determined based on the conditions set by the constraint means.

[0116] (Supplementary Note 2) The control device according to Supplementary Note 1, wherein the constraint means, if a local optimum solution does not exist in the search for an optimum solution for the surface, relaxes the conditions so that a local optimum solution exists, finds a local optimum solution for the relaxed conditions, and if the found local optimum solution is not a desired solution, reduces the degree of relaxation of the conditions, and finds a local optimum solution for the conditions with the reduced degree of relaxation.

[0117] (Supplementary Note 3) The control device according to Supplementary Note 2, wherein the constraint means uses a simulated annealing (SA) method to relax the conditions so that the local optimum solution exists, finds a local optimum solution for the relaxed conditions, and if the found local optimum solution is not a desired solution, reduces the degree of relaxation of the conditions, and finds a local optimum solution for the conditions with the reduced degree of relaxation.

[0118] (Supplementary Note 4) The control device according to Supplementary Note 2 or Supplementary Note 3, wherein the constraint means, when the relaxed conditions become conditions under which the previously determined local optimum solution cannot be obtained, changes the content of the relaxation and obtains a local optimum solution for the relaxed conditions with the changed content, and when the obtained local optimum solution is not a desired solution, reduces the degree of relaxation of the conditions and obtains a local optimum solution for the conditions with the reduced degree of relaxation.

[0119] (Supplementary Note 5) A robot system comprising: a first gripping mechanism; a second gripping mechanism; and the control device according to any one of Supplementary Notes 1 to 4.

[0120] (Supplementary Note 6) A control method comprising: setting conditions for a surface of an object relating to grasping, releasing, or changing the object, the conditions being included in constraints for determining the posture of the object and the movement path of the object, by expressions using a direction for grasping the object and a direction that defines the posture of the object; and controlling at least one of a first gripping mechanism and a second gripping mechanism so that the object is grasped, released, or changed using the surface determined based on the set conditions.

[0121] (Supplementary Note 7) A recording medium storing a program that causes a computer to execute the following steps: setting conditions of the surface of the object related to grasping, releasing the grip of the object, or changing the object, which conditions are included in constraints in determining the posture of the object and the movement path of the object, by expressions using a direction in which the object is grasped and a direction that defines the posture of the object; and controlling at least one of a first gripping mechanism and a second gripping mechanism so that the object is grasped, released from the grip of the object, or changed over using the surface determined based on the set conditions.

[0122] (Supplementary Note 8) A control device comprising: a determination means for determining a direction in which a second gripping mechanism will grip an object based on a direction of a surface of the object being gripped by a first gripping mechanism; and a control means for controlling the operation of the second gripping mechanism so as to grip the object from the direction determined by the determination means.

[0123] (Supplementary Note 9) A control device comprising: a determining means for determining an operation of a first gripping mechanism and a second gripping mechanism to grip an object based on a gripping direction of the first gripping mechanism, a gripping direction of the second gripping mechanism, and a direction of a surface of the object; and a control means for controlling the first gripping mechanism and the second gripping mechanism to perform the determined operation.

[0124] According to each aspect of the present disclosure, in a robot system, it is possible to appropriately control a robot arm according to the state of an object.

[0125] REFERENCE SIGNS LIST 1 robot system 2 control device 5 computer 6 CPU 7 main memory 8 storage 9 interface 40 robot 50 imaging device 201 input unit 202 generation unit 202a first processing unit 202b second processing unit 202c third processing unit 202d fourth processing unit 202e fifth processing unit 203 control unit C cardboard F floor M object T tray

Claims

1. a constraint means for setting a condition of a surface of the object related to gripping, releasing, or changing the grip of the object, the condition being included in a constraint condition for determining a posture of the object and a moving path of the object, by an expression using a direction for gripping the object and a direction that defines the posture of the object; a control means for controlling at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the condition set by the constraint means, release the grip of the object, or change the grip of the object; and A control device comprising:

2. The restricting means is If a local optimum solution does not exist in the search for an optimum solution for the surface, the conditions are relaxed so that a local optimum solution exists, and a local optimum solution is obtained for the relaxed conditions. If the obtained local optimum solution is not a desired solution, the degree of relaxation of the conditions is reduced, and a local optimum solution is obtained for the conditions with the reduced degree of relaxation. The control device according to claim 1 .

3. The restricting means is By using a simulated annealing (SA) method, the conditions are relaxed so that the local optimum solution exists, a local optimum solution is obtained for the relaxed conditions, and if the obtained local optimum solution is not a desired solution, the degree of relaxation of the conditions is reduced, and a local optimum solution is obtained for the conditions with the reduced degree of relaxation. The control device according to claim 2.

4. The restricting means is If the relaxed conditions result in a condition under which the previously determined local optimum solution cannot be obtained, the content of the relaxation is changed, and a local optimum solution is obtained for the relaxed conditions with the changed content, and if the obtained local optimum solution is not a desired solution, the degree of relaxation of the conditions is reduced, and a local optimum solution is obtained for the conditions with the reduced degree of relaxation. The control device according to claim 2.

5. The control means controlling the first gripping mechanism and the second gripping mechanism to re-grasp the object; The control device according to claim 1 .

6. A first gripping mechanism; A second gripping mechanism; The control device according to claim 1 ; A robot system comprising:

7. a constraint condition for determining the posture of the object and the movement path of the object, the constraint condition for determining the posture of the object and the movement path of the object, the condition of the surface of the object related to the gripping of the object, the release of the gripping of the object, or the change of the object is set by an expression using a direction for gripping the object and a direction that specifies the posture of the object; controlling at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set condition, release the grip of the object, or change the grip of the object; Control methods.

8. A condition of the surface of the object related to gripping, releasing, or changing the grip of the object, which is included in a constraint condition in determining the posture of the object and the movement path of the object, is set by an expression using a direction for gripping the object and a direction that specifies the posture of the object; Controlling at least one of the first gripping mechanism and the second gripping mechanism so as to grip the object using the surface determined based on the set condition, release the grip of the object, or change the grip of the object; A program that causes a computer to execute the following.

9. a determining means for determining a direction in which the second gripping mechanism grips the object based on a direction of a surface of the object gripped by the first gripping mechanism; a control means for controlling an operation of the second gripping mechanism so as to grip the object from the direction determined by the determination means; A control device comprising:

10. a determination means for determining an operation of a first gripping mechanism and a second gripping mechanism gripping an object based on a gripping direction of the first gripping mechanism, a gripping direction of the second gripping mechanism, and a direction of a surface of the object; a control means for controlling the first gripping mechanism and the second gripping mechanism to perform the determined operation; A control device comprising: