Robot system and control method

Abstract

This robot system, which is for changing a first orientation indicating a position and an angle of an object in an environment to a predetermined orientation, comprises: a manipulator having one end fixed to the environment and connected with an end effector for gripping the object; a sensor system for detecting, by means of a sensor mechanically connected to the manipulator, a second orientation of the object with respect to the sensor; and a control unit for controlling the orientation of the manipulator in the environment so as to change the first orientation to a predetermined orientation on the basis of the second orientation of the object detected by the sensor system. Before changing the first orientation of the object, the control unit changes an angle of the orientation of the sensor in the environment by driving the manipulator, and brings the sensor into contact with the object fixed to the environment for each angle change to thereby acquire detection results detected by the sensor system. The control unit calculates an orientation for changing the first orientation of the object on the basis of the detection results obtained by the sensor system acquired by changing the angle of the sensor.

Description

Robot system and control method

[0001] The present disclosure relates to a robot system for moving an object and a control method thereof.

[0002] Patent Document 1 discloses a robot device equipped with a robot arm mechanism having multiple joints. The robot arm mechanism supports an end effector for performing work on a work object, and the end effector is provided with an imaging unit and three contact sensors that are sensitive in the imaging direction of the imaging unit. A control device of the robot device controls the robot arm mechanism to move the end effector in a direction substantially perpendicular to the imaging direction using an image captured by the imaging unit, and to move the end effector in a direction substantially parallel to the imaging direction, based on the position at which the end effector contacts the work object as determined by the contact output of the contact sensor. In this way, Patent Document 1 aims to control the movement of the end effector relative to the work object without using an external camera that captures images of the work object.

[0003] JP 2016-101632 A

[0004] R. Li et al., "Localization and manipulation of small parts using GelSight tactile sensing," 2014 IEEE / RSJ International Conference on Intelligent Robots and Systems, Chicago, IL, USA, 2014, pp. 3988-3993, doi: 10.1109 / IROS.2014.6943123.

[0005] The present disclosure provides a robot system that can move an object with high precision and a control method thereof.

[0006] A robot system according to one aspect of the present disclosure is a robot system for changing a first attitude indicating the position and angle of an object in an environment to a predetermined attitude, the robot system including a manipulator, a sensor system, and a control unit. One end of the manipulator is fixed to the environment and an end effector for grasping the object is connected to the manipulator. The sensor system detects a second attitude of the object relative to the sensor using a sensor mechanically connected to the manipulator. The control unit controls the attitude of the manipulator in the environment to change the first attitude of the object to the predetermined attitude based on the second attitude of the object detected by the sensor system. Before changing the first attitude of the object, the control unit changes the angle of the sensor in the environment by driving the manipulator, and acquires a detection result by the sensor system by bringing the sensor into contact with an object fixed to the environment each time the angle of the sensor attitude is changed. The control unit calculates an attitude for changing the first attitude of the object based on multiple detection results by the sensor system acquired by changing the angle of the sensor.

[0007] These general and specific aspects may be realized by a system, a method, and a computer program, as well as combinations thereof.

[0008] According to the robot system and the control method thereof disclosed herein, the posture of an object can be changed to a predetermined posture, thereby moving the object with high precision.

[0009] FIG. 1 is a diagram for explaining a robot system according to the first embodiment; FIG. 2 is a block diagram illustrating the configuration of a robot, a control device, and a terminal device in the robot system; FIG. 3 is a diagram for explaining the operation of the robot system;

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. However, more detailed explanation than necessary may be omitted. For example, detailed explanation of well-known matters or redundant explanation of substantially the same configuration may be omitted. This is to avoid unnecessary redundancy in the following explanation and to facilitate understanding by those skilled in the art.

[0011] The applicant provides the accompanying drawings and the following description to enable those skilled in the art to fully understand the present disclosure, and does not intend for them to limit the subject matter described in the claims.

[0012] First Embodiment 1. Configuration The configuration of a robot system according to a first embodiment will be described with reference to FIGS. 1 and 2. FIG.

[0013] 1-1. Overview of the Robot System Fig. 1 is a diagram illustrating a robot system 1 according to this embodiment. The robot system 1 of this embodiment includes a robot 2, a control device 4, and a terminal device 5. In this system 1, the robot 2, the control device 4, and the terminal device 5 are connected to each other so that data can be communicated with each other.

[0014] The present system 1 is applied to an application in which the operation of a robot 2 is controlled by control commands transmitted from a control device 4 in an environment 10, such as a factory. The robot 2 operates in accordance with the control commands by driving a manipulator 2b including multiple joints, with a base 2a fixed to the environment 10 as a base point. In the example of FIG. 1 , the robot 2 moves a connector 6 of a USB terminal and inserts it into a socket 7 fixed to the environment 10. The orientation of the connector 6 that fits into the recess of the socket 7 in such environment 10 is an example of a predetermined orientation in this embodiment, and is specified by a user using a terminal device 5, for example.

[0015] In the present disclosure, the posture of an object represents the position and angle (also referred to as "orientation" or "direction") of the object. The position indicates the coordinates of the object in a Cartesian coordinate system represented by, for example, the X-axis, Y-axis, and Z-axis, and the angle (orientation) indicates the angle (orientation) of the object expressed by rotation around each axis of the coordinate system. The angle may be expressed as an Euler angle, or as a quaternion or rotation matrix. The format of the angle expression is not limited to the above example as long as it can identify the orientation of the object.

[0016] In the robot 2 illustrated in FIG. 1 , an end effector 2c is connected to the other end (also referred to as the "hand") of a manipulator 2b, which is different from the end fixed to the base 2a. The end effector 2c includes, for example, a parallel two-fingered gripper, and is capable of gripping an object by controlling the opening width of the two fingers of the gripper. In this embodiment, an example of the object is a connector 6. In addition, in the present system 1, the manipulator 2b is connected to a tactile sensor 8 via the end effector 2c. The tactile sensor 8 is attached, for example, to the fingertips of the gripper, and comes into contact with the object as the gripper grips the object. The tactile sensor 8 detects the object, for example, mechanically or optically, in response to contact with the object, and generates sensor data.

[0017] In the present system 1, for example, the control device 4 uses sensor data from the tactile sensor 8 to perform an operation for moving the connector 6 with high precision to the above-described pre-determined posture.

[0018] 1-2. Configuration of Robot, Control Device, and Terminal Device The configuration of the robot 2, control device 4, and terminal device 5 in this system 1 will be explained using Fig. 2. Fig. 2 is a block diagram illustrating the configuration of the robot 2, control device 4, and terminal device 5 in the robot system 1.

[0019] As shown in Fig. 2, the robot 2 includes a control unit 20 and a communication interface 22b in the manipulator 2b. Hereinafter, interface will be abbreviated as "I / F." In the example of Fig. 2, the robot 2 also includes a communication I / F 22c and a communication I / F 82 in the end effector 2c and the tactile sensor 8, respectively. The control unit 20 is composed of various processors such as a CPU, and controls the overall operation of the robot 2.

[0020] Each of the communication I / Fs 22b, 22c, and 82 is a circuit for performing data communication in accordance with a predetermined communication standard. For example, the predetermined communication standard may include USB, IEEE 1395, IEEE 802.3, IEEE 802.11a / 11b / 11g / 11ac, Wi-Fi (registered trademark), and Bluetooth (registered trademark). That is, the communication I / Fs 22b, 22c, and 82 may be configured to include a connection terminal or the like for connecting to an external device, and may communicate with the external device via a communication network or directly.

[0021] The control unit 20 receives control commands for operating the robot 2 from the control device 4, for example, via the communication I / F 22b, and drives each joint of the manipulator 2b based on the control commands. The robot 2 is equipped with a drive device such as a motor that drives each joint. The control unit 20 can change the rotation angle of each joint to an angle according to the control command, using the detection results of an angle detection sensor, for example, an encoder or resolver, provided in the drive device.

[0022] The control unit 20 may calculate a hand posture indicating the posture of the tip of the manipulator 2b from, for example, rotation angles detected sequentially during the operation of the robot 2, and transmit the calculated posture to an external device such as the control device 4 via the communication I / F 22b. The robot system 1 of this embodiment manages the hand posture using, for example, a first coordinate system based on the base 2a fixed to the environment 10. The first coordinate system is represented by X １ , Y １ , Z １ It is expressed as:

[0023] In the robot system 1 of this embodiment, the hand posture is calculated as the relative posture between the first coordinate system and a manipulator tip coordinate system fixed to the tip of the manipulator 2b, i.e., the rear end of the end effector 2c. The manipulator tip coordinate system is represented by X Ｅ , Y Ｅ , Z Ｅ In the present system 1, the orientation of the connector 6 to be moved is given as the relative orientation between the first coordinate system and a second coordinate system fixed to the connector 6 gripped by the end effector 2c, for example. The second coordinate system is represented by X2, Y2, and Z2 in FIG.

[0024] In this embodiment, the tactile sensor 8 detects an object using known technology, such as that described in Non-Patent Document 1. The tactile sensor 8 irradiates an object in contact with, for example, a gel-like surface with three colors of light (R, G, and B) from multiple directions using a light source such as a light-emitting diode (LED), and captures an image of the object using a camera based on the reflected light. The tactile sensor 8 generates, for example, image data representing the image and outputs it as sensor data via the communication I / F 82. If the obtained image data is separated into RGB channels, information equivalent to that obtained by capturing three images while changing the light source direction can be obtained.

[0025] In the present system 1, the control device 4 receives this sensor data and estimates the three-dimensional surface shape of the object that has come into contact with the tactile sensor 8 by image recognition using, for example, photometric stereo. In this way, the three-dimensional surface shape of the object that has come into contact can be detected by the tactile sensor 8 and the control device 4. The tactile sensor 8 and the control device 4 are an example of a sensor system in this embodiment.

[0026] In the example of Fig. 1, the orientation of the connector 6 relative to the tactile sensor 8 is calculated by estimating the three-dimensional surface shape of the connector 6 from the sensor data of the tactile sensor 8 that is in contact with the connector 6. The orientation of the connector 6 relative to the tactile sensor 8 is calculated as the relative orientation between, for example, a sensor coordinate system fixed to the tactile sensor 8 and a second coordinate system fixed to the connector 6. The sensor coordinate system is represented by X S , Y S , Z SIt is expressed as:

[0027] In this system 1, for example, the control device 4 calculates the hand posture of the manipulator 2b by calculating coordinate transformation between the coordinate systems described above, so as to position the connector 6 in a posture that will mate with the socket 7.

[0028] 2, the control device 4 includes a control unit 40, a storage unit 41, and a communication I / F 42. The control device 4 is configured as an information processing device including, for example, a robot controller that controls the robot 2.

[0029] The control unit 40 includes, for example, a CPU, and executes a program (software) to realize predetermined functions of the control device 4, such as the coordinate transformation described above. Instead of a CPU, the control unit 40 may include a processor configured with a dedicated electronic circuit designed to realize the predetermined functions. That is, the control unit 40 can be realized by various processors such as a CPU, MPU, GPU, DSU, FPGA, and ASIC. The control unit 40 may be configured with one or more processors.

[0030] The storage unit 41 is a storage medium that stores programs and data necessary to realize the functions of the control device 4, and is configured, for example, with an HDD or SSD. For example, in addition to the above programs, the storage unit 41 stores sensor data D1 from the tactile sensor 8 received via the communication I / F 42 and hand posture data D2 indicating the hand posture of the manipulator 2b. The storage unit 41 may also store, for example, annotation data D3 generated from the sensor data D1, as described below. The above programs may be provided via a communication network such as the Internet, or may be stored on a portable recording medium.

[0031] The storage unit 41 may include a RAM such as a DRAM or an SRAM, and may function as a buffer memory that temporarily stores (i.e., holds) data. The storage unit 41 may function as a working area for the control unit 40, and may include a storage area in the internal memory of the control unit 40.

[0032] The communication I / F 42 is a circuit for performing data communication in accordance with a predetermined communication standard, similar to, for example, the communication I / Fs 22b, 22c, and 82 of the robot 2. In the present system 1, the communication I / F 42 of the control device 4 is configured to be able to communicate with, for example, the communication I / Fs 22b, 22c, and 82 of the robot 2. The communication I / F 42 may constitute, in the control device 4, an acquisition unit that receives various information through communication with an external device, or an output unit that transmits various information, or may constitute an input unit that inputs the received information to the control unit 40.

[0033] The terminal device 5 includes a control unit 50, a storage unit 51, a communication I / F 52, an operation unit 53, and a display unit 54. The terminal device 5 is configured as an information processing device such as a personal computer (PC). The control unit 50 is configured as various processors such as a CPU, and realizes predetermined functions by executing programs stored in the storage unit 51, for example. The storage unit 51 is a storage medium that stores the programs and the like, and is configured as a non-volatile semiconductor memory, for example. The storage unit 51 may include a RAM such as a DRAM or an SRAM, and may function as a work memory for the control unit 50. The communication I / F 52 is a circuit for performing data communication in accordance with a predetermined communication standard, similar to the communication I / F 42 of the control device 4, for example.

[0034] The operation unit 53 is a general term for operation members operated by the user. The operation unit 53 is configured by, for example, any one of a keyboard, a mouse, a trackpad, a touchpad, buttons and switches, etc., or a combination thereof. The operation unit 53 may form a touch panel together with the display unit 54. The operation unit 53 acquires various information input by user operations. The operation unit 53 may include a connection terminal or the like for connecting an external operation member, and may accept remote operation of the robot 2 by, for example, a joystick, a 3D mouse, a master-slave master arm, and / or a VR controller, etc.

[0035] The display unit 54 is configured by, for example, a liquid crystal display or an organic EL display, and displays a GUI used for machine learning of the control model 30. The display unit 54 may display various icons for operating the operation unit 53, and various information such as information input from the operation unit 53.

[0036] The configuration of the robot system 1 is not limited to the above example, and for example, the control device 4 may be configured integrally with the robot 2, or may be configured integrally with the terminal device 5. In the above, an example has been described in which various data such as the sensor data D1 is stored in the storage unit 41 of the control device 4, but the data D1 to D3 may be stored in a storage device external to the control device 4, or may be stored in separate storage devices. In the above, an example has been described in which the terminal device 5 includes the display unit 54, but instead of or in addition to the display unit 54, a speaker or the like that outputs audio may be included.

[0037] The above describes an example in which the end effector 2c of the robot 2 includes the communication I / F 22c, and the tactile sensor 8 includes the communication I / F 82. For example, the end effector 2c and the tactile sensor 8 may be connected to the control unit 20 of the manipulator 2b, and may communicate with the control device 4, etc., via the communication I / F 22b of the manipulator 2b.

[0038] 2. Operation The operation of the robot system 1 and the control device 4 configured as above will be described below.

[0039] A robot system 1 of this embodiment controls the operation of a robot 2 that inserts a connector 6 into a socket 7, as shown in FIG. 1 . In the example of FIG. 1 , the socket 7 is fixed in an environment 10 in which the robot 2 operates, for example, by being positioned using a jig, and the orientation of the socket 7 in a first coordinate system is known. On the other hand, the orientation of the connector 6 may change, for example, each time the connector 6 is grasped by an end effector 2 c. The system 1 calculates a coordinate transformation between the first coordinate system and a second coordinate system corresponding to the connector 6, thereby moving the connector 6 so that the orientation of the connector 6 in the first coordinate system matches the orientation of the socket 7.

[0040] In this system 1, for example, the control device 4 calculates the orientation of the connector 6 in the first coordinate system from the following orientations Pa, Pb, and Pc: ​​Pa: orientation of the tip of the manipulator 2b in the first coordinate system, Pb: orientation of the tactile sensor 8 relative to the tip of the manipulator 2b, and Pc: ​​orientation of the connector 6 relative to the tactile sensor 8.

[0041] The control device 4 calculates the orientation Pa by performing forward kinematics calculations based on the rotation angles obtained by encoders or the like attached to the motors of the joints of the manipulator 2b. The control device 4 also calculates the orientation Pc by image recognition based on sensor data D1 obtained from the tactile sensor 8 as image data of the connector 6.

[0042] The above-mentioned orientation Pb can be calculated using, for example, CAD data of the robot 2. However, in the actual robot 2, an attachment error may occur when attaching the tactile sensor 8 to the end effector 2c. Furthermore, in the end effector 2c, the gripper opening width may change depending on, for example, the size and / or shape of the object to be grasped. Therefore, the orientation Pb of the tactile sensor 8 attached to the fingertips of the gripper during grasping may also change. In particular, when an underactuated gripper in which multiple joints move in conjunction with the actuation of some of the joints is used, such a change in the orientation of the tactile sensor 8 may be significant.

[0043] Therefore, the robot system 1 of this embodiment performs an operation to estimate the orientation Pb from data obtained when the end effector 2c actually grasps an object such as the connector 6. The operation of this system 1 will be described below with reference to FIGS.

[0044] 2-1. Overview of Operation The overview of the operation of the robot system 1 in this embodiment will be described with reference to FIG.

[0045] 3 is a diagram illustrating the operation of the robot system 1. In this embodiment, the robot system 1 grasps the connector 6 to be grasped multiple times while changing the orientation of the tactile sensor 8, with the connector 6 being fixed in the environment 10, thereby collecting data indicating the orientations Pa and Pc for each grasp. The system 1 changes at least the angle of the tactile sensor 8 as the orientation of the tactile sensor 8 in the first coordinate system according to the environment 10. Figures 3A and 3B illustrate two exemplary orientations in which the angle of the tactile sensor 8 is changed.

[0046] 3A and 3B, the connector 6 is fixed in the environment 10 by a vice 11, and the connector 6 is thereby fixed with respect to the first coordinate system. In this example, with the relative orientation between the second coordinate system and the first coordinate system corresponding to the connector 6 fixed in this way, the angle of the tactile sensor 8 is determined by the angle between the first coordinate system and the tactile sensor coordinate system. 1 θ S 3A shows the angle of the tactile sensor 8 in the first position. 1 θ S (1) 3B illustrates the y-direction component of the angle of the tactile sensor 8 in the second orientation. 1 θ S (2) Illustrates the y-direction component of

[0047] The system 1 performs processing to estimate the orientation Pb, i.e., the orientation of the tactile sensor 8 when grasping relative to the tip of the manipulator 2b (i.e., the rear end of the end effector 2c), based on data indicating the orientations Pa and Pc collected each time the orientation of the tactile sensor 8 is changed. The system 1 can calculate coordinate transformation using the orientation Pb estimated in this way, and can move the connector 6 in the environment 10 with high precision according to the orientation of the connector 6 in the first coordinate system.

[0048] 2-2. Overall Operation The overall operation of the robot system 1 of this embodiment will be described with reference to FIG.

[0049] 4 is a sequence diagram illustrating the operation of the robot system 1. In this system 1, for example, the operation unit 53 of the terminal device 5 accepts a user operation to change the attitude of the robot 2 by operating the manipulator 2b using a joystick or the like (S1). Then, the terminal device 5 transmits an operation signal corresponding to the accepted user operation for changing the attitude to the control device 4 via the communication I / F 52 (S1). In response to this user operation, the system 1 changes the attitude of the tip of the manipulator 2b in the first coordinate system, thereby changing the attitude of the tactile sensor 8 attached to the end effector 2c connected to the tip of the manipulator 2b.

[0050] The control unit 40 of the control device 4 transmits a control command to the robot 2 to change its posture via the communication I / F 42 in response to the operation signal received from the terminal device 5 (S2). The control command in step S2 is, for example, a command to the motor of each joint for the joint angle to move the robot 2 from the current posture to the instructed posture.

[0051] In the robot 2, for example, the control unit 20 of the manipulator 2b operates the manipulator 2b by driving each joint based on the control command received via the communication I / F 22b (S3). During such operation, the control unit 20 transmits encoder information such as the rotation angle of each joint to the control device 4.

[0052] The control unit 40 of the control device 4 calculates the posture of the tip of the manipulator 2b in the first coordinate system, for example, based on encoder information from the robot 2, thereby obtaining hand posture data D2 indicating such hand posture and storing it in the memory unit 41 (S4).

[0053] Furthermore, the control unit 40 of the control device 4 transmits a gripping command to the robot 2 via the communication I / F 42 to cause the end effector 2c to perform a gripping operation (S5). When the robot 2 receives the gripping command via, for example, the communication I / F 22c of the end effector 2c, it performs a gripping operation by driving the gripper of the end effector 2c (S6). The robot 2 transmits sensor data D1 of the tactile sensor 8 when gripping the connector 6 or the like to the control device 4 via, for example, the communication I / F 82.

[0054] The control device 4 acquires the sensor data D1 of the tactile sensor 8 transmitted from the robot 2 and stores it in the storage unit 41 (S7). For example, in step S7, the control unit 40 of the control device 4 may calculate the orientation in the sensor coordinate system of the object that has come into contact with the tactile sensor 8 based on the sensor data D1. In this way, the control unit 40 detects the orientation of the connector 6 with respect to the tactile sensor 8 by bringing the tactile sensor 8 into contact with the connector 6 (S5, S7).

[0055] As described above, the system 1 acquires hand posture data and sensor data during grasping while changing the posture of the tactile sensor 8 in accordance with the posture of the tip of the manipulator 2b (S1 to S7). Similar to the operation of collecting each data (S31), the system 1 repeats the collection of each data while further changing the posture in accordance with, for example, user operation on the terminal device 5 (S32).

[0056] The control unit 40 of the control device 4 performs a calculation to estimate the orientation of the tactile sensor 8 at the time of grasping relative to the tip of the manipulator 2b, based on the hand orientation data D2 collected while changing the orientation of the tactile sensor 8 and the sensor data D1 at the time of grasping (S41). In the present system 1, the orientation of the tactile sensor 8 estimated in this manner is calculated as the relative orientation of the second coordinate system as viewed from the first coordinate system, i.e., as an estimation result of a calibration parameter used to accurately calculate the orientation of the connector 6 in the first coordinate system. The control unit 40 stores the estimation result in the memory unit 41, for example.

[0057] For example, if the operation of collecting each data D1 and D2 is repeated N times (N is a natural number) as in steps S31 and S32, if the grasped connector 6 is fixed with respect to the first coordinate system, the relative orientation of the second coordinate system as seen from the first coordinate system will be the same, and the following relational equation will be obtained.

[0058] Each variable in the above equation ( 1 T2, 1 T E , E T S , S T2) represents the transformation between the two coordinate systems, e.g., a homogeneous transformation matrix. Each variable corresponds to the following pose: 1 T2: Attitude of the connector 6 in the first coordinate system (i.e., the relative attitude of the second coordinate system as viewed from the first coordinate system) 1 T E (n) : Orientation of the tip of the manipulator 2b in the first coordinate system during the nth grasping (n is a natural number equal to or less than N) E T S : Attitude of the tactile sensor 8 relative to the tip of the manipulator 2b S T2(n) : the orientation of the connector 6 with respect to the tactile sensor 8 at the nth gripping. In this way, the above-mentioned orientations Pa, Pb, and Pc are respectively 1 T E , E T S , S It can be expressed as T2.

[0059] From the above relational expressions, N simultaneous equations are obtained. In step S41, the control unit 40 calculates the postures at the first to Nth gripping times from the relationships expressed by these simultaneous equations, for example, by the least squares method. 1 T E (n) , S T2 (n) The posture of the connector 6 in the first coordinate system for N times is calculated from 1 The posture that minimizes the error of T2 E T S Calculate.

[0060] If, during multiple data collections as described above, the angle of the tactile sensor 8 in the first coordinate system is not changed and only the position is changed, the image of the connector 6 in the image represented by the sensor data D1 will only move in parallel, resulting in an infinite number of solutions to the simultaneous equations. For example, consider a case where the angle between the first coordinate system and the sensor coordinate system is θ and only the position of the tactile sensor 8 is moved by (ΔX, ΔZ) in the first coordinate system. In this case, the image of the connector 6 in the image of the sensor data D1 will move in parallel by (-ΔX cos θ + ΔZ sin θ, -ΔX sin θ - ΔZ cos θ), but this relationship is related to the posture E T S Since this holds true regardless of the collected data D1 and D2, E T S This creates a problem where it is not possible to determine the

[0061] In contrast, in the robot system 1 of this embodiment, data D1 and D2 are collected during grasping while changing the angle of the tactile sensor 8 (S31, S32), and the orientation of the tactile sensor 8 during grasping relative to the tip of the manipulator 2b is calculated. E T SIn addition, by using the data D1 and D2 obtained when actually grasping the connector 6, the attitude of the tactile sensor 8 relative to the tip of the manipulator 2b can be estimated with high accuracy (S41). E T S can be estimated including the mounting error, and the posture, which may change depending on the object being grasped, E T S can be estimated with high accuracy.

[0062] 2-3. Operation of the Control Device The operation of the control device 4 in the above-described operation of the robot system 1 will be described in detail with reference to FIG.

[0063] 5 is a flowchart illustrating the operation of the control device 4 in this embodiment. The flowchart in FIG. 5 starts in a state where a user operation to change the attitude is accepted by the terminal device 5, for example, in step S1 in FIG. 4. Each process in this flowchart is executed by the control unit 40 of the control device 4.

[0064] First, the control unit 40 determines whether a user operation to change the posture has been input (S11) based on whether an operation signal has been received from the terminal device 5 via the communication I / F 42. If the user operation has not been input (NO in S11), the control unit 40 repeats the determination in step S11, for example, at a predetermined cycle.

[0065] If a user operation to change the attitude is input (YES in S11), the control unit 40 determines a target attitude to move the tip of the manipulator 2b in the environment 10 based on the input user operation (S12).

[0066] The control unit 40 determines whether or not the existing hand posture data D2, which has been previously acquired in the same process as step S4 in Fig. 4 and stored in the storage unit 41, contains a hand posture with the same angle as the target posture determined in step S12 (S13). For example, the control unit 40 makes the determination in step S13 by comparing each hand posture in the existing hand posture data D2 with the determined target posture.

[0067] If the existing hand posture data D2 includes a hand posture with the same angle as the target posture determined in step S12 (YES in S13), the control unit 40 causes the display unit 54 of the terminal device 5 to display a warning message to alert the user of the overlapping angles (S14). In step S14, the control unit 40 outputs a control signal to the terminal device 5 via the communication I / F 42, for example, to cause the terminal device 5 to display a warning message.

[0068] The control unit 40 changes the target posture so that the angle differs from that of each hand posture in the existing hand posture data D2 (S15). The warning display in step S14 may be a display that notifies the user of such a change in angle. Also, there may be cases where the target posture is changed without displaying the warning in step S14.

[0069] After changing the target posture (S15), the control unit 40 transmits a control command to the robot 2 via the communication I / F 42 to instruct the robot 2 to change its posture to the target posture (S16). Step S16 corresponds to step S2 in Fig. 4. In response to this control command to instruct the robot 2 to change the posture of the tip of the manipulator 2b to the target posture, the manipulator 2b is driven so as to move the tip of the manipulator 2b to the target posture (S3 in Fig. 4).

[0070] Even if the existing hand posture data D2 does not contain a hand posture with the same angle as the target posture determined in step S12 (NO in S13), the control unit 40 proceeds to step S16. In this case, the control unit 40 transmits a control command to change the posture in accordance with the determined target posture (S16).

[0071] The control unit 40 acquires hand posture data D2 in the state in which the manipulator 2b is operated based on the control command from the robot 2 via the communication I / F 42 (S17). Step S17 corresponds to step S4 in FIG. 4.

[0072] The control unit 40 transmits a grip command to the robot 2 in the state after the above operation via the communication I / F 42 (S18). Step S18 corresponds to step S5 in FIG. 4. In response to this grip command, the robot 2 performs a gripping operation on the connector 6 with the end effector 2c (S6 in FIG. 4).

[0073] The control unit 40 acquires sensor data D1 generated by the tactile sensor 8 from the robot 2 gripping the connector 6 (S19). Step S19 corresponds to step S7 in Fig. 4. For example, in step S19, the control unit 40 of this embodiment calculates the orientation of the connector 6 in the image indicated by the sensor data D1 by image recognition, and further converts the calculated orientation into the orientation of the connector 6 relative to the tactile sensor 8.

[0074] For example, before executing the processing of this flowchart, the control unit 40 generates an image recognition model that detects the posture of an object in an image through deep learning, based on annotation data D3 previously stored in the storage unit 41. The annotation data D3 is data in which annotation information indicating the posture of an object in the image is added to an image captured in advance by, for example, the tactile sensor 8 through a user operation or the like. In the robot system 1 of this embodiment, for example, the terminal device 5 displays the image on the display unit 54, and then accepts a user operation on the operation unit 53 to add annotation information to the image.

[0075] The control unit 40 of the control device 4 associates, for example, multiple images with each piece of annotation information assigned in response to user operations from the terminal device 5, acquires the images as annotation data D3 via the communication I / F 42, and stores the data in the storage unit 41. The control unit 40 calculates the orientation of the connector 6 in the image in step S19 using an image recognition model generated using the annotation data D3. The control unit 40 further calculates the orientation of the connector 6 relative to the tactile sensor 8 from the orientation of the connector 6 in the image using internal parameters of the camera built into the tactile sensor 8.

[0076] For example, after acquiring the sensor data D1 (S19), the control unit 40 determines whether the sensor data D1 and the hand posture data D2 have been acquired for a predetermined number (for example, "10") of postures (S20). The control unit 40 may manage the acquisition of each data D1, D2 for one posture using a counter or the like, for example, each time the process of this flowchart is executed. The predetermined number may be, for example, the posture of the tactile sensor 8 when grasping the tip of the manipulator 2b. E T S In the estimation of the hand posture (S41 in FIG. 4), 1 T E , and the orientation of the connector 6 relative to the tactile sensor 8 S In order to reduce the influence of errors in T2, the number is set to correspond to the number of times of grasping.

[0077] According to the above process, a target posture of the tip of the manipulator 2b is determined in response to a user's operation to change the posture (S11, S12), and a control command is sent to the robot 2 to change the posture to the target posture (S16). When the robot 2 operates in response to the control command, hand posture data D2 is acquired, and further, in response to a grip command sent to the robot 2, sensor data D1 of the state in which the connector 6 is being gripped is acquired (S18, S19). The above process is repeated until the data D1 and D2 are acquired for a predetermined number of postures (S20). In this way, the data D1 and D2 can be collected for a predetermined number of postures while changing the posture of the tactile sensor 8 in response to the posture of the tip of the manipulator 2b in the environment 10.

[0078] Furthermore, according to the above process, if the collected existing hand posture data D2 includes a hand posture with the same angle as the determined target posture (YES in S13), the target posture is changed so that its angle differs from the collected hand posture (S15). This makes it possible to collect each data D1 and D2 for a predetermined number of postures while reliably changing the angle of the tactile sensor 8 in the environment 10.

[0079] In the above, an example has been described in which the target posture of the tip of the manipulator 2b is determined in response to a user operation received by the terminal device 5 in step S1 of Fig. 4 (S11, S12). The control unit 40 may determine the target posture not only in response to such a user operation but also in response to, for example, direct teaching in which the user directly operates and moves the manipulator 2b. Furthermore, the control unit 40 may automatically determine the target posture without being limited to a remote or direct operation by the user.

[0080] In the above, an example has been described in which the predetermined number in step 20 is set to a number corresponding to the number of times of grasping. For example, when each posture is determined by a movement with six degrees of freedom (three degrees of freedom for translation and three degrees of freedom for rotation), six simultaneous equations as described above are obtained by two graspings, so the predetermined number may be a number equal to or greater than "2." For example, when the predetermined number is "2," the posture to be estimated is determined by operating the manipulator 2b and acquiring the data D1 and D2 at the time of grasping while changing the posture for each degree of freedom. E T S The components corresponding to each degree of freedom can be calculated.

[0081] 2-4. Actions to move objects

[0082] In the robot system 1, the attitude of the tactile sensor 8 when grasping the tip of the manipulator 2b estimated by the above operation is E T S The operation of moving an object to a predetermined pose in the environment 10 using the above as a calibration parameter will now be described.

[0083] 6 is a sequence diagram illustrating an example of an operation of moving an object in the robot system 1. In the following, an example of moving a connector 6 as an object, similar to the example of FIG. 1, will be described. The operation shown in FIG. 6 is performed, for example, based on the posture calculated as the estimation result of the calibration parameters in step S41 of FIG. E T S is stored in the storage unit 41.

[0084] First, the control unit 40 of the control device 4 reads out the calibration parameters stored in the storage unit 41 (S51).

[0085] The control unit 40 transmits a gripping command to the end effector 2c of the robot 2 to grip the connector 6 (S52), similar to, for example, step S5 in Fig. 4. For example, the connector 6 is placed within the movable range of the end effector 2c in the environment 10. Based on the received gripping command, the robot 2 performs a gripping operation on the connector 6 with the end effector 2c (S53), similar to, for example, step S6 in Fig. 4.

[0086] The control unit 40 of the control device 4 acquires, from the robot 2 via the communication I / F 42, the sensor data D1 generated by the tactile sensor 8 when the connector 6 is gripped (S54).

[0087] The control unit 40 determines the attitude of the connector 6 with respect to the tactile sensor 8 based on the acquired sensor data D1. S In step S55, the control unit 40 calculates the attitude of the connector 6 in the sensor coordinate system based on the sensor data D1, for example, in the same manner as in step S7 of FIG.

[0088] The control unit 50 of the terminal device 5 calculates a target posture of the connector 6 in the first coordinate system. 1 The attitude to which the connector 6 is to be moved in the environment 10 is transmitted to the control device 4 via the communication I / F 52 so as to instruct the control device 4 to attain T2 (S56). 1 T2 is stored in advance in the storage unit 51 as the orientation of the connector 6 that fits into the recess of the socket 7 fixed to the environment 10 as shown in Fig. 1. The control unit 40 of the control device 4 receives the target orientation of the connector 6 from the terminal device 5 via the communication I / F 42. 1 Get T2.

[0089] The control unit 40 of the control device 4 determines the acquired target posture of the connector 6. 1 The posture to which the tip of the manipulator 2b is moved in the environment 10 in accordance with T2 is set as a target posture of the tip of the manipulator 2b in the first coordinate system. 1 T E The control unit 40 calculates the attitude of the tactile sensor 8 with respect to the tip of the manipulator 2b, which was read out as a calibration parameter in step S51, as follows (S57). ET S and the attitude of the connector 6 with respect to the tactile sensor 8 estimated in step S55. S The control unit 40 performs the calculation in step S57 based on T2. ​​The control unit 40 calculates the target posture of the tip of the manipulator 2b using the following formula: 1 T E Calculate.

[0090] The control unit 40 calculates the target attitude calculated in step S57. 1 T E In order to change the attitude of the tip of the manipulator 2b, a control command for changing the attitude is sent to the robot 2, for example, in the same manner as in step S2 of FIG. 4 (S58).

[0091] The robot 2 operates the manipulator 2b based on a control command from the control device 4, for example, in the same manner as in step S3 of FIG. 4 (S59).

[0092] For example, an object such as the same product as the connector 6 can be grasped by the robot 2 in a substantially similar manner. According to the above operation, the calibration parameters ( E T S ) to determine the target posture of the connector 6. 1 Target posture of the tip of manipulator 2b according to T2 1 T E As a result, the posture estimated by taking into account the attachment error of the tactile sensor 8 relative to the tip of the manipulator 2b and the change in posture of the tactile sensor 8 during grasping, etc., is calculated (S51 to S57). E T S are used as calibration parameters to obtain the target posture of the tip of the manipulator 2b. 1 T E can be calculated with high accuracy.

[0093] The robot system 1 of this embodiment calculates the posture of the tip of the manipulator 2b as the target posture calculated as above. 1 T E (S58, S59). As a result, the orientation of the connector 6 is changed to the target orientation. 1The connector 6 can be moved with precision to change the voltage to T2.

[0094] In the above, an example has been described in which the target posture of the connector 6 stored in advance in the storage unit 51 of the terminal device 5 is instructed to the control device 4 (S56). The control unit 50 of the terminal device 5 may receive a user input specifying the posture of the connector 6 in the first coordinate system via the operation unit 53, and instruct the control device 4 to the target posture of the connector 6 according to the user input. Also, for example, the operation of the manipulator 2b may be performed in stages by repeating the processes of steps S56 to S59. For example, the control unit 40 of the control device 4 may receive the current posture of the connector 6 in the environment 10. 1 As a posture for changing T2 (time t=0) to the final target posture, the target posture of the connector 6 is 1 The control unit 40 may calculate the target posture of the connector 6 calculated for each time, for example, T2 (t=1, 2, ...). 1 The posture of the manipulator 2b gripping the connector 6 may be controlled based on T2 (t=1, 2, . . . ).

[0095] 3. Summary As described above, the robot system 1 according to the present embodiment acquires the first posture indicating the position and angle of the connector 6 as an example of an object in the environment 10. 1 The robot system 1 is an example of a robot system that changes T2 to a target posture, which is an example of a predetermined posture. The robot system 1 includes a manipulator 2b, a tactile sensor 8, a control device 4 (an example of a sensor system), and a control unit 40 (an example of a control unit). One end of the manipulator 2b is fixed to an environment 10, and an end effector 2c that grasps a connector 6 is connected to the other end, which is different from the one end, as an example. The sensor system uses a tactile sensor 8 attached to the end effector 2c, as an example of a sensor mechanically connected to the manipulator 2b, to detect a second posture of the connector 6 relative to the tactile sensor 8. S The control unit 40 detects the second attitude T2 of the connector 6 detected by the sensor system (S7, S19, S54, S55, etc.). S Based on T2, the first position of the connector 6 1T2 to a predetermined posture, the end-effect posture of the manipulator 2b in the environment 10 is 1 T E (S57). The control unit 40 controls the connector 6 to have the first attitude. 1 Before changing T2, the angle of the attitude of the tactile sensor 8 in the environment 10 is changed by driving the manipulator 2b (S2), and each time the angle of the tactile sensor 8 is changed, the tactile sensor 8 is brought into contact with the connector 6 fixed to the environment 10 to obtain a detection result by the sensor system (S5, S7). The control unit 40 determines the first attitude of the connector 6 based on the multiple detection results obtained by the sensor system while changing the angle of the tactile sensor 8. 1 Posture to change T2 (e.g., hand posture) 1 T E (see S58 to S59) is calculated (S51, S57).

[0096] According to the robot system 1 as described above, the connector 6 is in the first posture in the environment 10. 1 Before changing T2, the tactile sensor 8 is brought into contact with the connector 6 fixed to the environment 10 each time the angle of the attitude of the tactile sensor 8 is changed, and a detection result by the sensor system is acquired (S5, S7). For example, the sensor system acquires a second attitude of the connector 6 with respect to the tactile sensor 8 when the connector 6 is held by the end effector 2c. S The robot system 1 detects the hand posture T2 based on a plurality of detection results obtained by changing the angle of the tactile sensor 8. 1 T E The first orientation of the connector 6, such as the target orientation of 1 The posture for changing T2 is calculated (S51, S57). As a result, the first posture is calculated according to the detection results of a plurality of postures in which the tactile sensor 8 is actually brought into contact with the connector 6 at different angles so as to change the relative posture with respect to the connector 6. 1 For example, even if there is an error in attaching the tactile sensor 8 to the tip of the manipulator 2b, the orientation for moving the connector 6 with high precision can be calculated.

[0097] In this embodiment, the control unit 40 determines the hand posture calculated in step S57, for example. 1 T E The first orientation of the connector 6, such as the target orientation of 1 The attitude of the manipulator 2b in the environment 10 is controlled based on the attitude for changing T2 (S58), and the first attitude of the connector 6 is 1 T2 is changed to the target posture (see S59). By controlling the posture of the manipulator 2b in this way, the connector 6 can be moved with high precision.

[0098] In this embodiment, the control unit 40 determines the attitude of the tactile sensor 8 relative to the tip of the manipulator 2b when the connector 6 is gripped by the end effector 2c based on a plurality of detection results from the sensor system. E T S (S41). This makes it possible to calculate the posture of the tactile sensor 8 that reflects, for example, an attachment error of the tactile sensor 8 relative to the tip of the manipulator 2b. For example, the posture of the tactile sensor 8 calculated in this way can be calculated as follows: E T S can be used as a calibration parameter in the calculation of coordinate transformation between the first coordinate system corresponding to the environment 10 and the second coordinate system corresponding to the connector 6. E T S In this embodiment, the first position of the connector 6 1 10 is an example of a posture calculated to change T2 to a target posture.

[0099] In this embodiment, the robot system 1 calculates the attitude of the tactile sensor 8 relative to the tip of the manipulator 2b based on a plurality of detection results from the sensor system. E T S In this way, for example, in the robot system 1, when the connector 6 is moved, the stored orientation of the tactile sensor 8 is E T S can be used as a calibration parameter (see S51).

[0100] In this embodiment, the robot system 1 further includes a control unit 40 as an example of an acquisition unit that acquires a target posture (an example of a predetermined posture) of the connector 6 in the environment 10 (see S56). E T S and a detection result by the sensor system when the connector 6 is gripped, the first posture of the connector 6 in the environment 10 is determined. 1 The posture of the tip of the manipulator 2b (i.e., the hand posture) for changing T2 to a predetermined posture 1 T E The control unit 40 calculates a target posture indicating the first posture of the connector 6 (S57). The control unit 40 controls the posture of the manipulator 2b based on the calculated target posture of the tip of the manipulator 2b (S58). 1 T2 is changed to the target posture (see S59). As a result, the stored posture of the tactile sensor 8 relative to the tip of the manipulator 2b is E T S As a calibration parameter, the orientation of the tip of the manipulator 2b can be calculated with high accuracy according to the acquired target orientation of the connector 6. By moving the manipulator 2b in this way, the first orientation of the connector 6 is 1 The connector 6 can be moved with high precision so as to change T2 to the target posture.

[0101] In this embodiment, the robot system 1 further includes an end effector 2c. The end effector 2c is provided on the tip side of the manipulator 2b, connecting the manipulator 2b to a tactile sensor 8, so as to grasp the connector 6. The tactile sensor 8 is attached to the end effector 2c. For example, in addition to an attachment error of the tactile sensor 8 relative to the tip of the manipulator 2b, the orientation of the tactile sensor 8 relative to the tip of the manipulator 2b may change depending on the object grasped by the end effector 2c. Even in such cases, the system 1 can move the connector 6 with high precision.

[0102] In this embodiment, the tactile sensor 8 generates image data by capturing an image of the connector 6. The sensor system determines the second attitude of the connector 6 based on the image represented by the image data. S T2 is detected (S7, S19, S54, S55, etc.). As a result, the second attitude of the connector 6 is detected by image recognition processing from image data captured by the reflected light according to the surface shape of the connector 6 that comes into contact with the tactile sensor 8, for example. S T2 can be detected with high accuracy.

[0103] In this embodiment, the robot system 1 further includes an operation unit 53 of the terminal device 5 as an example of an input unit that inputs annotation information to be added to the image captured by the tactile sensor 8 through a user operation. The annotation information includes the orientation of the connector 6 in the image. For example, an image recognition model that detects the orientation of the connector 6 from the image captured by the tactile sensor 8 can be obtained by deep learning or the like using, as training data, annotation data D3 in which such annotation information is associated with image data.

[0104] In this embodiment, the robot system 1 further includes an operation unit 53 of the terminal device 5 as an example of an operation unit that accepts user operations. The control unit 40 determines multiple postures by changing the angle of the tactile sensor 8 in the environment 10 in response to remote operation of the manipulator 2b by the operation unit 53 (S1, S11, S12). This allows the user to specify a grasping posture without directly touching the robot 2. The control unit 50 may also determine multiple postures by changing the angle of the postures of the tactile sensor in the environment 10 in response to direct teaching. This allows the user to specify a grasping posture through intuitive operation. The grasping posture may be specified by using a combination of the above-described remote operation and direct teaching.

[0105] In this embodiment, the control unit 40 compares each hand posture of the existing hand posture data D2 with the determined target posture as an example of two postures among the plurality of postures, and if the angle between each hand posture and the target posture is the same as an example of the angle of the tactile sensor 8 in the two postures (YES in S13), the control unit 40 changes the angle of the target posture as an example of changing the angle of the tactile sensor 8 in one of the two postures (S15). As a result, for example, if the angles in two hand postures determined in response to user operation are the same, the angles can be corrected to be different.

[0106] In this embodiment, a first attitude indicating the position and angle of a connector 6 as an example of an object in the environment 10 is 1 A control method for the robot system 1 is provided for changing T2 to a target posture, which is an example of a predetermined posture (S51 to S59, see FIGS. 4 and 5). The robot system 1 includes a manipulator 2b, a tactile sensor 8 as an example of a sensor system, and a control device 4. The manipulator 2b has one end fixed to the environment 10, and an end effector 2c that grips the connector 6 is connected to the other end different from the one end. The sensor system detects a second posture of the connector 6 relative to the tactile sensor 8 by using the tactile sensor 8 (an example of a sensor) mechanically connected to the manipulator 2b. S The control method is as follows: the control unit 40 of the control device 4, which is an example of a computer, detects the first attitude of the connector 6. 1 Before changing T2, the angle of the attitude of the tactile sensor 8 in the environment 10 is changed by driving the manipulator 2b (S2), and each time the angle of the tactile sensor 8 is changed, the tactile sensor 8 is brought into contact with the connector 6 fixed to the environment 10 to obtain the detection result of the sensor system (S5, S7). S Based on T2, the first position of the connector 6 1 T2 to a predetermined posture, the end-effect posture of the manipulator 2b in the environment 10 is 1 T EThe control unit 40 is configured to control the first attitude of the connector 6 based on a plurality of detection results obtained by the sensor system while changing the angle of the tactile sensor 8. 1 Posture to change T2 (e.g., hand posture) 1 T E (see S58 to S59) is calculated (S51, S57).

[0107] In this embodiment, a program is provided to cause the control unit 40 of the control device 4, which is an example of a computer, to execute the above-described control method. According to the above-described control method and program, the robot 2 can move an object with high precision.

[0108] (Other Embodiments) As described above, embodiment 1 has been described as an example of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which modifications, substitutions, additions, omissions, etc. are made as appropriate. Furthermore, it is also possible to combine the components described in each of the above embodiments to create a new embodiment. Therefore, other embodiments will be described below as examples.

[0109] In the above-described first embodiment, an example has been described in which, when collecting hand posture data D2 and the like, the control device 4 automatically changes the target posture (S15) if the existing hand posture data D2 contains a hand posture with the same angle as the target posture determined in response to a user operation (YES in S13 of FIG. 5). In such a case, the control device 4 may change the target posture again in response to a user operation. Such a modification of the first embodiment will be described with reference to FIG. 7.

[0110] 7 is a flowchart illustrating the operation of the control device 4 in a modification of the first embodiment. Instead of the process of automatically changing the target posture after displaying a warning to the user about angle overlap (S14, S15) in the first embodiment, the control device 4 in this modification displays a warning prompting the user to change the target posture (S14A). In this modification, the control unit 40 of the control device 4 displays the warning on the display unit 54 of the terminal device 5 (S14A), and then returns to step S11 to again determine whether a user operation to change the posture has been input from the terminal device 5. This also makes it possible to collect sensor data D1 and hand posture data D2 during grasping for multiple postures in which the angle of the tactile sensor 8 in the environment 10 is changed.

[0111] As described above, in this modified example, the robot system 1 further includes the display unit 54 of the terminal device 5 as an example of an output unit that outputs information to the user. The control unit 40 compares each hand posture of the existing hand posture data D2 with the determined target posture as an example of two postures among the plurality of postures, and if the angle between each hand posture and the target posture is the same as an example of the angle of the tactile sensor 8 in the two postures (YES in S13), the control unit 40 outputs a predetermined warning on the display unit 54 (S14A). As a result, for example, if the angles in two hand postures determined in response to user operation are the same, the warning display can prompt the user to change the angle.

[0112] In the above embodiments, an example has been described in which the posture of the connector 6 in an image of the connector 6 in contact with the tactile sensor 8 is calculated based on the sensor data D1 using an image recognition model based on deep learning (S7, S19, S55, etc.). Calculation of the posture of the connector 6 can be performed by various image recognition processes, not limited to image recognition models using deep learning. For example, image recognition processes using local features such as SIFT features may be used in the image, or template matching may be used.

[0113] In the above embodiments, an example has been described in which the orientation of the connector 6 in the image represented by the sensor data D1 is calculated and converted into the orientation of the connector 6 relative to the tactile sensor 8 (S7, S19, S55, etc.), but this is not limiting. For example, the orientation of the connector 6 relative to the tactile sensor 8 may be estimated by estimating the three-dimensional shape of the connector 6 in contact with the tactile sensor 8 using photometric stereo for the image of the sensor data D1 and comparing it with CAD data of the connector 6.

[0114] In the above embodiments, an example has been described in which the tactile sensor 8 is used in the robot system 1. The sensor in the robot system 1 is not limited to the tactile sensor 8, and any sensor capable of detecting the orientation of an object grasped by the end effector 2c relative to the sensor may be used. For example, a sensor in which various MEMS sensors or pressure sensors are arranged in an array may be used. Also, a sensor that does not come into contact with the object may be used, such as a camera that captures an image of the object using natural light or a ToF sensor. Also, although an example has been described in which the sensor is attached to the end effector 2c, the sensor may be attached to the manipulator 2b without the end effector 2c.

[0115] In each of the above embodiments, an example has been described in which the control method for the robot system 1 is executed by the control unit 40 in the control device 4 of the robot system 1. The control method of the present disclosure may be executed not only by the control unit 40 but also by the control unit 20 of the robot 2 or the control unit 50 of the terminal device 5, or may be executed by a combination of two or more of the control units 20, 40, and 50. The control method of the present disclosure may also be executed by an external computer that communicates data with the robot system 1.

[0116] In the above embodiments, an example has been described in which the robot 2 in the robot system 1 performs an operation of inserting the connector 6 into the socket 7, as shown in Fig. 1 . The operation of the robot 2 in the robot system 1 is not limited to the above example, and may be various operations in which the robot 2 moves various objects, such as assembling precision parts or attaching parts to a circuit board. Furthermore, the robot system 1 is not limited to the robot 2 as shown in Fig. 1 as an example, and can be applied to control various robots that perform various operations.

[0117] As described above, the embodiments have been described as examples of the technology in the present disclosure, and for that purpose, the accompanying drawings and detailed description have been provided.

[0118] Therefore, the components shown in the accompanying drawings and detailed description may include not only essential components for solving the problem, but also components that are not essential for solving the problem in order to illustrate the above technology. Therefore, the fact that these non-essential components are shown in the accompanying drawings or detailed description should not be interpreted as immediately indicating that these non-essential components are essential.

[0119] Furthermore, since the above-described embodiments are intended to illustrate the technology of the present disclosure, various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

[0120] (Summary of Aspects) Various aspects of the present disclosure are listed below.

[0121] A first aspect of the present disclosure is a robot system for changing a first attitude indicating the position and angle of an object in an environment to a predetermined attitude. The robot system includes a manipulator fixed at one end to the environment and connected to an end effector for grasping an object, a sensor system that detects a second attitude of the object relative to the sensor using a sensor mechanically connected to the manipulator, and a controller that controls the attitude of the manipulator in the environment to change the first attitude of the object to the predetermined attitude based on the second attitude of the object detected by the sensor system. Before changing the first attitude of the object, the controller drives the manipulator to change the angle of the attitude of the sensor in the environment, and each time the sensor angle is changed, the controller acquires a detection result by the sensor system by bringing the sensor into contact with an object fixed to the environment. The controller calculates an attitude for changing the first attitude of the object based on multiple detection results of the sensor system acquired by changing the sensor angle.

[0122] In a second aspect, in the robot system of the first aspect, the control unit controls the attitude of the manipulator in the environment based on the calculated attitude for changing the first attitude of the object, thereby changing the first attitude of the object to a predetermined attitude.

[0123] In a third aspect, in the robot system of the first or second aspect, the control unit calculates the orientation of the sensor relative to the manipulator based on a plurality of detection results by the sensor system.

[0124] In a fourth aspect, the robot system of the third aspect further includes a storage unit that stores the orientation of the sensor relative to the manipulator, calculated based on a plurality of detection results by the sensor system.

[0125] In a fifth aspect, the robot system of the fourth aspect further includes an acquisition unit that acquires a predetermined orientation of an object in an environment. The control unit calculates a target orientation indicating an orientation of the manipulator for changing the first orientation of the object in the environment to the predetermined orientation based on the orientation of the sensor with respect to the manipulator stored in the storage unit and the detection result by the sensor system. The control unit controls the orientation of the manipulator based on the calculated target orientation of the manipulator to change the first orientation of the object to the predetermined orientation.

[0126] In a sixth aspect, the robot system of any one of the first to fifth aspects further includes an end effector. The end effector is provided to the manipulator by connecting the manipulator and a sensor to grasp an object. The sensor is attached to the end effector.

[0127] In a seventh aspect, in the robot system of any of the first to sixth aspects, the sensor generates image data of an object, and the sensor system detects a second posture of the object based on an image represented by the image data.

[0128] In an eighth aspect, the robot system of the seventh aspect further includes an input unit for inputting annotation information to be added to the image by a user's operation, the annotation information including a posture of an object in the image.

[0129] In a ninth aspect, in the robot system according to any one of the first to eighth aspects, the control unit determines a plurality of postures by changing the angle of the sensor in the environment in response to direct teaching.

[0130] In a tenth aspect, the robot system according to any one of the first to ninth aspects further includes an operation unit configured to receive an operation from a user, and the control unit determines a plurality of postures by changing the angle of the sensor in the environment in response to remote operation of the manipulator by the operation unit.

[0131] In an eleventh aspect, in the robot system of the ninth or tenth aspect, the control unit compares two of the multiple postures, and if the sensor angles in the two postures are the same, changes the sensor angle in one of the two postures.

[0132] In a twelfth aspect, the robot system of the ninth or tenth aspect further includes an output unit that outputs information to be presented to a user. The control unit compares two of the plurality of postures, and if the sensor angles in the two postures are the same, outputs a predetermined warning from the output unit.

[0133] A thirteenth aspect of the present disclosure provides a control method for a robot system for changing a first attitude indicating the position and angle of an object in an environment to a predetermined attitude. The robot system includes a manipulator fixed at one end to the environment and connected to an end effector for grasping an object, and a sensor system for detecting a second attitude of the object relative to the sensor using a sensor mechanically connected to the manipulator. The control method includes, by a computer control unit, changing the angle of the attitude of the sensor in the environment by driving the manipulator before changing the first attitude of the object, and acquiring a detection result by the sensor system by bringing the sensor into contact with an object fixed to the environment each time the sensor angle is changed. The control unit is configured to control the attitude of the manipulator in the environment based on the second attitude of the object detected by the sensor system so as to change the first attitude of the object to the predetermined attitude. The control unit calculates an attitude for changing the first attitude of the object based on multiple detection results by the sensor system acquired by changing the angle of the sensor.

[0134] In a fourteenth aspect, in the control method of the thirteenth aspect, the control unit controls the attitude of the manipulator in the environment based on the calculated attitude for changing the first attitude of the object, thereby changing the first attitude of the object to a predetermined attitude.

[0135] A fifteenth aspect of the present disclosure is a program for causing a control unit of a computer to execute the control method of the thirteenth or fourteenth aspect.

[0136] The present disclosure is applicable to a variety of robotic systems for operating objects using information from sensors.

Claims

1. A robot system for changing a first attitude, which indicates the position and angle of an object in an environment, to a predetermined attitude, comprising: a manipulator having one end fixed to the environment and connected to an end effector that grasps the object; a sensor system that detects a second attitude of the object relative to the sensor using a sensor mechanically connected to the manipulator; and a control unit that controls the attitude of the manipulator in the environment so as to change the first attitude of the object to the predetermined attitude based on the second attitude of the object detected by the sensor system, wherein before changing the first attitude of the object, the control unit changes an angle of the attitude of the sensor in the environment by driving the manipulator, and each time the angle of the sensor is changed, the sensor is brought into contact with the object fixed to the environment to obtain a detection result by the sensor system, and the control unit calculates a attitude for changing the first attitude of the object based on multiple detection results by the sensor system obtained by changing the angle of the sensor.

2. The robot system described in claim 1, wherein the control unit controls the attitude of the manipulator in the environment based on the calculated attitude for changing the first attitude of the object, thereby changing the first attitude of the object to the specified attitude.

3. The robot system according to claim 1, wherein the control unit calculates the orientation of the sensor relative to the manipulator based on a plurality of detection results by the sensor system.

4. The robot system according to claim 3, further comprising a storage unit that stores the attitude of the sensor relative to the manipulator, calculated based on a plurality of detection results by the sensor system.

5. A robot system as described in claim 4, further comprising an acquisition unit that acquires the predetermined posture of the object in the environment, wherein the control unit calculates a target posture indicating the posture of the manipulator for changing the first posture of the object in the environment to the predetermined posture based on the posture of the sensor with respect to the manipulator stored in the memory unit and the detection result by the sensor system, and controls the posture of the manipulator based on the calculated target posture of the manipulator to change the first posture of the object to the predetermined posture.

6. The robot system according to claim 1, further comprising an end effector, the end effector being configured to grasp the object by connecting the manipulator to the sensor, and the sensor being attached to the end effector.

7. The robot system according to claim 1, wherein the sensor generates image data by capturing an image of the object, and the sensor system detects a second posture of the object based on an image represented by the image data.

8. The robot system according to claim 7, further comprising an input unit for inputting annotation information to be added to the image by a user's operation, the annotation information including the posture of the object in the image.

9. The robot system according to claim 1, wherein the control unit determines a plurality of postures by changing the angle of the sensor in the environment in response to direct teaching.

10. The robot system according to claim 1, further comprising an operation unit that accepts operation by a user, wherein the control unit determines a plurality of postures by changing the angle of the sensor in the environment in response to remote operation of the manipulator in the operation unit.

11. A robot system as described in claim 9 or 10, wherein the control unit compares two of the multiple postures, and if the angles of the sensor in the two postures are the same, changes the angle of the sensor in one of the two postures.

12. A robot system as described in claim 9 or 10, further comprising an output unit that outputs information to be presented to a user, wherein the control unit compares two of the multiple postures, and if the angle of the sensor in the two postures is the same, outputs a predetermined warning in the output unit.

13. A control method for a robot system for changing a first attitude indicating a position and angle of an object in an environment to a predetermined attitude, the robot system comprising: a manipulator having one end fixed to the environment and connected to an end effector for grasping the object; and a sensor system that detects a second attitude of the object relative to the sensor by a sensor mechanically connected to the manipulator, the control method including the steps of: before changing the first attitude of the object, a control unit of a computer controls an attitude of the manipulator in the environment so as to change the first attitude of the object to the predetermined attitude based on the second attitude of the object detected by the sensor system, changing an angle of the attitude of the sensor in the environment by driving the manipulator, and each time the angle of the sensor is changed, acquiring a detection result by the sensor system by bringing the sensor into contact with the object fixed to the environment, and the control unit calculates an attitude for changing the first attitude of the object based on multiple detection results by the sensor system acquired by changing the angle of the sensor.

14. The control method according to claim 13, wherein the control unit controls the attitude of the manipulator in the environment based on the calculated attitude for changing the first attitude of the object, thereby changing the first attitude of the object to the specified attitude.

15. A program for causing the control unit to execute the control method according to claim 13 or 14.

Images