CONTROL DEVICE, ROBOT SYSTEM, CONTROL METHOD, AND PROGRAM
The control device enhances robot operability by determining target positions based on object shape and environment, addressing motion discrepancies and improving interaction efficiency.
Patent Information
- Application Number
- JP2021047360
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2021-03-22
- Publication Date
- 2024-11-01
- Estimated Expiration
- 2041-03-22
AI Technical Summary
Existing robot systems fail to adequately consider the position of the object to be operated and often experience delays or discrepancies between operator and robot motions, hindering operability.
A control device that detects operator motions, determines target positions based on the object's shape and environment, estimates trajectories, and selects optimal positions using similarity analysis to guide the robot's actuators effectively.
Improves robot operability by guiding actuators to target positions that align with the operator's intentions, ensuring accurate and efficient object interaction regardless of task type.
Smart Images

Figure 0007577003000001 
Figure 0007577003000002 
Figure 0007577003000003
Abstract
Description
[Technical field]
[0001] The present invention relates to a control device, a robot system, a control method, and a program. [Background technology]
[0002] Conventionally, robot systems have been proposed that accept an operator's action and operate a robot according to the accepted action. For example, a robot system described in Patent Document 1 acquires first user attitude information indicating the attitude of a first user operating the robot, and acquires pre-change attitude information indicating a pre-change attitude before changing the attitude of the robot based on the first user attitude information. The robot system determines a target attitude of the robot that is different from the attitude of the first user based on the first user attitude information at a point in time when the robot is in the pre-change attitude indicated by the pre-change attitude information and the pre-change attitude information. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] Patent No. 6476358 Summary of the Invention [Problem to be solved by the invention]
[0004] When an operator operates a robot to perform a task, the robot needs to approach an object that is the operation target. However, in the robot system described in Patent Document 1, the position of the object that is the operation target is not fully considered. In addition, there may be a delay between the operator's action and the robot's action, and a difference between the amount of the operator's action and the amount of the robot's action. These factors hinder operability.
[0005] An aspect of the present invention has been made in consideration of the above points, and one of its objectives is to provide a control device, a robot system, a control method, and a program that can improve the operability of a robot. [Means for solving the problem]
[0006] (1) The present invention has been made to solve the above problems, and one aspect of the present invention is A control device that detects an operator's movement and makes a robot perform an action corresponding to the detected movement, a target position setting unit that determines a target position of an effector of the robot based on a shape of an object present in an operating environment of the robot; Calculating a position of the effector based on a driving state of the robot; The current status of the above-mentioned agent On the actual trajectory showing the position at each time during a specified period a trajectory estimation unit that estimates a predicted trajectory of the effector based on a predetermined function and estimates a trajectory from the current position of the effector to the target position as an approach trajectory using a predetermined function; and a target position selection unit that selects one target position based on the similarity between the predicted trajectory and each of the approach trajectories.
[0007] (2) Another aspect of the present invention is the control device of (1), wherein the target position setting unit may determine a shape of the object from an image representing the operating environment, and set the position of a feature point corresponding to the determined shape as the target position based on feature point data indicating an arrangement of feature points for each shape.
[0008] (3) Another aspect of the present invention is the control device according to (2), in which the target position setting unit may reject a target position that interferes with another object.
[0009] (4) Another aspect of the present invention is a control device of any of (1) to (3), wherein the target position setting unit may determine, for an object having a dimension larger than a predetermined dimension, an intersection between a surface of the object and the line of sight direction of an operator based on line of sight information indicating the line of sight direction of an operator, and determine the target position based on a distribution of the intersections.
[0010] (5) Another aspect of the present invention is the control device of (4), wherein the target position setting unit may classify the intersections into a plurality of subsets and determine a representative point of the distribution of intersections for each subset as the target position.
[0011] (6) Another aspect of the present invention may be a program for causing a computer to function as any one of the control devices (1) to (5).
[0012] (7) Another aspect of the present invention may be a robot system including the control device according to any one of (1) to (5) and the robot.
[0013] (8) Another aspect of the present invention is Detects the operator's movements and makes the robot perform the corresponding movement A control method in a control device, comprising: The above A first step of determining a target position of an effector of the robot based on a shape of an object present in an operating environment of the robot; Calculating a position of the effector based on a driving state of the robot; The current status of the above-mentioned agent On the actual trajectory showing the position at each time during a specified period a second step of estimating a predicted trajectory of the effector based on a predetermined function, and estimating a trajectory from the current position of the effector to the target position as an approach trajectory using a predetermined function; and a third step of selecting one target position based on the similarity between the predicted trajectory and each of the approach trajectories. Effect of the Invention
[0014] According to the above-mentioned configurations (1), (6), (7), and (8), an approach trajectory to a target position is set based on the shape of an object in the operating environment of the robot, and the target position is selected based on the similarity between a predicted trajectory estimated from the operation of the effector and the approach trajectory. By guiding the effector of the robot to a target position that is likely to meet the operator's intention, the operability of the robot can be improved. This process is based on the target position determined based on the current operation of the effector and the shape of the object in the operating environment, and is therefore independent of the type of task. Therefore, it can be applied to scenarios in which an object is approached, regardless of the type of task.
[0015] According to the above-mentioned configuration (2), the position of the feature point corresponding to the shape of the object determined from the image representing the operating environment is determined as the target position. The position of the typical feature point according to the shape of the object is set as the target position, and the effector of the robot can be guided.
[0016] According to the above-mentioned configuration (3), invalid target positions to which the effector cannot be approached are eliminated, so that the effector can be moved more reliably.
[0017] According to the above-mentioned configuration (4), the position at which the operator gazes is determined as the target position, so that even in the case of a large object whose feature points cannot be uniquely determined, it is possible to guide the operation of the effector to the object and position in which the operator is interested.
[0018] According to the above-mentioned configuration (5), even if the positions where the operator gazes are distributed discretely, the target positions corresponding to each distribution are determined. By guiding the effector of the robot to the position of the operator's interest, the operability can be improved. [Brief description of the drawings]
[0019] [Figure 1] 1 is a schematic block diagram illustrating an example of the configuration of a robot system according to a first embodiment. [Diagram 2] 2 is a schematic block diagram illustrating an example of a hardware configuration of a control device according to the first embodiment. FIG. [Diagram 3] 3A to 3C are explanatory diagrams showing examples of a target position, a predicted trajectory, and an approach trajectory according to the first embodiment. [Figure 4] 5 is a flowchart showing an example of an operation control process according to the first embodiment. [Diagram 5] FIG. 11 is an explanatory diagram showing an example of setting a target position according to the second embodiment. [Figure 6] FIG. 11 is an explanatory diagram showing another example of setting the target position according to the second embodiment. [Figure 7] 10 is a flowchart showing an example of an operation control process according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0020] <First embodiment> Hereinafter, a first embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a schematic block diagram showing an example of the configuration of a robot system S1 according to this embodiment. The robot system S1 is a control system that can control the operation of the robot 20 in accordance with the operation of an operator. The robot system S1 is a control system that controls the movement of the robot 20 in response to an operation. The robot system S1 detects the movement of the operator and causes the robot 20 to perform an action corresponding to the detected movement. The robot 20 is equipped with an effector (also called an end-effector, robot hand, end effector, effector, etc.). The effector is a member that mechanically acts on and influences other objects. The effector has multiple finger parts and is equipped with a mechanism that enables it to grasp or release other objects. Each finger part is operable in response to the movement of the corresponding finger of the operator. This allows various tasks to be realized.
[0021] The robot system S1 determines multiple target positions for the effector based on environmental information indicating the status of the operating environment of the robot 20. The target position is a target position when the effector is moved, and is also called an approach point. The robot system S1 estimates a predicted trajectory of the effector based on the current operation of the effector. On the other hand, the robot system S1 estimates a trajectory from the current position of the effector to each target position as an approach trajectory using a predetermined function. The robot system S1 selects a target position corresponding to one approach trajectory based on the similarity between the predicted trajectory and each approach trajectory. The robot system S1 moves the robot 20 toward the selected target position.
[0022] The robot system S1 includes a robot 20, a display device 50, an operating device 70, and an environmental information acquisition unit 80. The robot 20 includes one or more manipulators. The manipulator is also called a robot arm. Each manipulator includes a plurality of segments, which are connected to one another. Each of the plurality of segments includes one effector. The effector is a member that is connected to one end of the manipulator and that comes into contact with an object to exert an action. The robot 20 includes a joint for each pair of segments including two segments, and an actuator for each joint. The position of the effector can be moved by changing the angle between each pair of segments using the actuator. In the following description, the case where the number of manipulators is one (single-arm type) is mainly described, but the number of manipulators may be two or more. The robot 20 may be a stationary type that is fixed at a predetermined position, or a movable type that can move its own position.
[0023] The control device 30 of the robot 20 is connected to the display device 50, the operation device 70, and the environmental information acquisition unit 80 wirelessly or via wires so as to be able to input and output various types of information. The display device 50 and the operation device 70 may be located in a space physically separated from the robot 20 and the environmental information acquisition unit 80. In that case, the display device 50 and the operation device 70 may be connected to the control device 30 via a communication network (remote operation system).
[0024] The robot 20 includes a drive control unit 22 , a drive unit 24 , a drive sensor 26 , a power source 28 , and a control device 30 . The drive control unit 22 determines a drive command based on a control command input from the control device 30 and drive state information acquired from the drive sensor 26. More specifically, the drive control unit 22 determines a control command indicating a drive force (torque) for each actuator as an operation amount so as to reduce the difference between a target position as a control target indicated by the control command and the position of the effector. The drive control unit 22 can determine the position of the effector as an output value by the drive unit 24 based on the angle between two segments in each joint indicated in the drive state information and the size of each segment. The drive control unit 22 outputs the determined control command to the drive unit 24. The drive control unit 22 is configured to include, for example, a Proportional-Differential (PD) controller, a Proportional-Integral-Differential (PID) controller, etc. as a controller that determines an operation amount so as to reduce the difference between an output value indicating the state of the controlled object and a control target.
[0025] The drive unit 24 has an actuator for each joint, and operates according to a control command input from the drive control unit 22. Each actuator corresponds to a so-called motor, and changes the angle between the two segments connected to the drive unit 24 according to the drive amount specified by the control command. The drive sensor 26 detects the drive state of the robot 20 by the drive unit 24, and outputs drive state information indicating the detected drive state to the drive control unit 22 and the control device 30. The drive sensor 26 includes, for example, a rotary encoder that detects the angle between two segments for each joint.
[0026] The power source 28 supplies power to each component of the robot 20. The power source 28 includes, for example, a power terminal, a secondary battery, and a voltage converter. The power source terminal allows a power line to be connected, receives power from an external source, and supplies the supplied power to the secondary battery or the voltage converter. The secondary battery stores power supplied via a power supply terminal. The secondary battery is capable of supplying power to each component via a voltage converter. The voltage converter converts the voltage of the power supply terminal or the power supplied from the power supply terminal to a predetermined voltage required by each component, and supplies each component with the converted voltage.
[0027] The control device 30 includes an information acquisition unit 32, a target position setting unit 34, a trajectory estimation unit 36, a target position selection unit 38, an action command generation unit 40, a communication unit 42, and a memory unit 44. The information acquisition unit 32 acquires various information related to the status of the operator and the status of the robot 20. For example, the information acquisition unit 32 acquires gaze information and first operator movement information from the display device 50. The gaze information is information indicating the gaze direction of at least one eye of the operator at a certain time. The first operator movement information is information indicating the movement of the operator's head. The head movement is represented by the direction and position of the head at each time.
[0028] The information acquisition unit 32 acquires second operator motion information from the operation device 70. The second operator motion information is information indicating body motion related to operation by the operator. The second operator motion information mainly includes information indicating hand motion as a body part of the operator. The motion of the body part is represented using information on the position of the body part at each time. The information acquiring unit 32 acquires environmental information from the environmental information acquiring unit 80. The environmental information indicates the operating environment of the robot 20. The environmental information includes image data indicating an image showing the operating environment of the robot 20, and distance data indicating distances to objects distributed in the operating environment of the robot 20. The information acquisition unit 32 acquires driving state information from the driving sensor 26. The information acquisition unit 32 stores the acquired various pieces of information in the storage unit 44.
[0029] The target position setting unit 34 reads the latest environmental information at that time from the storage unit 44, and determines a target position based on the read environmental information. The target position is a position to which the effector can move. The target position is a candidate for the position where the effector acts on the object. A typical position where the effector acts on the object according to its shape characteristics can be adopted as the target position. The target position can be, for example, a position where the effector can easily approach the object and can stably grasp it. The target position setting unit 34 identifies objects placed in the operating environment of the robot 20 based on the image data and distance data indicated by the environmental information, and recognizes the area occupied by each object. The target position setting unit 34 can use a known image recognition process when recognizing the area of each object. The target position setting unit 34, for example, refers to the object model data and extracts the spatially continuous area occupied by each object from the surface shape in the three-dimensional space indicated by the image data and distance data. In the example shown in Fig. 3, the target position setting unit 34 specifies the regions of two objects ob10 and ob20 using image data and distance data. The object model data is configured by setting information indicating the three-dimensional region of each object in advance. The object model data is stored in the storage unit 44.
[0030] The target position setting unit 34 judges the shape of each recognized object region. When judging the shape of the object region, the target position setting unit 34 can use a known image recognition process. For example, the target position setting unit 34 refers to shape model data to judge the shape that the judged region has the most similar shape characteristics. The shape model data is configured by setting information indicating shape characteristics for each shape in advance. In the example shown in FIG. 3, the target position setting unit 34 judges that the objects ob10 and ob20 each have axial symmetry with respect to a line segment having a finite length as an axis, have a cross section perpendicular to the axis that is approximately circular, and have a cross section whose area is approximately constant regardless of the position on the axis. Therefore, the target position setting unit 34 can judge the shapes of ob10 and ob20 to be cylinders. The target position setting unit 34 can determine that the shape of a region is a triangular pyramid when the region has one side longer than the other two sides, the cross section perpendicular to the direction of the side is substantially triangular, and the cross section becomes smaller as it approaches the top surface rather than the bottom surface. The storage unit 44 stores shape model data indicating the shape characteristics for each shape.
[0031] The target position setting unit 34 determines a feature point corresponding to the shape of the determined region. The target position setting unit 34 determines the position of the feature point corresponding to the determined shape by referring to, for example, feature point data, and determines the determined position as the target position. The feature point data is formed by setting information indicating the arrangement of typical feature points for each shape in advance. Positions where work by the effector may be performed empirically are set in advance as feature points. The feature points may be set at the vertices or edges of the object. The storage unit 44 stores the feature point data. For example, in the case of a vertically elongated cylinder whose height is greater than its diameter, the target position setting unit 34 can determine the position closest to the effector among the center of the bottom surface, the center of the surface, and the cross section of the midpoint in the height direction as the target position. In the example shown in FIG. 3, the target position setting unit 34 can determine the center of the surface of the object ob01, the closest point of the cross section of the object ob01 to the effector ee01, and the center of the bottom surface of the object ob01 as the target positions ap11, ap12, and ap13, respectively. For the object ob02, the center of its surface, the closest point to the effector ee01 in the cross section, and the center of its bottom surface are determined as the target positions ap21, ap22, and ap23, respectively. The number of target positions determined in this manner can usually be two or more. Note that feature points may be set similarly for shapes other than cylinders, such as truncated cones, prisms, and truncated pyramids.
[0032] However, the target position setting unit 34 may reject a target position that interferes with another object. Interference with another object means that the effector cannot approach without contacting the other object. For example, there are cases where the target position is blocked by another object, and cases where there is not enough space around the target position to place the effector. For example, when the objects ob10 and ob20 are placed on a base and their bottom surfaces are not exposed and are in contact with the base, the effector ee01 cannot approach each bottom surface. In that case, the target position setting unit 34 excludes the target positions ap13 and ap23 set on the bottom surfaces of the objects ob10 and ob20 from the target positions. The target position setting unit 34 stores target position information indicating the determined target position in the storage unit 44.
[0033] The trajectory estimation unit 36 reads out the drive state information stored in the storage unit 44, and calculates the position of the effector at that time based on the angle of each joint indicated by the drive state information and the dimension of each segment. The trajectory estimation unit 36 generates a trajectory indicating the position at each time within a predetermined period up to the present as an actual trajectory. The trajectory estimation unit 36 performs a known regression analysis to fit the position at each time forming the actual trajectory to a predetermined continuous function that depends on time. The trajectory estimation unit 36 can use, for example, the least squares method in fitting to the continuous function. As the predetermined continuous function, for example, a quadratic function or a higher-order function that depends on time of third order or higher can be used. The trajectory estimation unit 36 extrapolates the actual trajectory using the determined continuous function, and determines a predicted trajectory indicating the position at each time from the present until a predetermined time later. In the example shown in FIG. 3, the predicted trajectory pt01 is estimated by extrapolating the actual trajectory rt01 up to the present of the effector ee01.
[0034] The trajectory estimation unit 36 may extrapolate the angle of each joint at each time indicated by the drive state information for each joint using a predetermined second continuous function that depends on time to determine the angle for each time from the present to a predetermined time later. The trajectory estimation unit 36 may determine the position at each time based on the determined angle and the size of the segment. The second continuous function is, for example, a quadratic function or a higher-order function of third order or higher that depends on time. The trajectory estimation unit 36 stores predicted trajectory information indicating the determined predicted trajectory in the storage unit 44.
[0035] On the other hand, the trajectory estimation unit 36 reads out the target position information from the storage unit 44, and determines a trajectory from the current position of the effector (sometimes referred to as the "current position" in this application) to the target position for each target position indicated by the read out target position information as an approach trajectory using a predetermined interpolation method. The trajectory estimation unit 36 can use an interpolation method such as spline interpolation. In the spline interpolation, for example, a spline curve such as a cubic or higher order spline curve or a non-uniform rational B-spline (NURBS) curve can be used.
[0036] When a high-order spline curve is used, the trajectory estimation unit 36 determines the parameters of the curve so that the angle and angular velocity are continuous for each joint of the robot for each segment point that divides the section from the current position to the target position into a plurality of sub-sections. The number of sub-sections is preset in the trajectory estimation unit 36. The position of the effector is determined using the angle of each joint and the dimension of the segment. When a NURBS curve is used, the trajectory estimation unit 36 determines the parameters of the curve so that the angle and angular velocity are continuous for each joint of the robot 20 for each of a plurality of control points. The control points do not necessarily pass through the trajectory. The number of control points is preset. In the example shown in FIG. 3, approach trajectories pp11, pp12, pp13, pp21, pp22, and pp23 are determined as approach trajectories corresponding to the target positions ap11, ap12, ap13, ap21, ap22, and ap23, respectively.
[0037] The trajectory estimation unit 36 may use a transition curve instead of a spline curve. A transition curve is a curve whose curvature at the current position is set to 0 and whose curvature changes continuously. For example, a clothoid curve may be used as the transition curve. A clothoid curve is a trajectory obtained by changing the curvature at a constant rate. In a clothoid curve, the product of the radius of curvature and the length of the trajectory is constant. In addition, the trajectory estimation unit 36 may impose a constraint condition in trajectory estimation that the velocity at the target position is zero or the angular velocity of each joint is zero. Since the velocity when the effector reaches the target position can be approximated to zero, it is possible to avoid or mitigate the impact that may occur when the effector contacts an object.
[0038] The trajectory estimation unit 36 stores approach trajectory information indicating the approach trajectory determined for each target position in the storage unit 44. The trajectory may be represented by the parameters of a curve indicating the position of the effector at each time, may be represented by the position of the effector at each time until the target position is reached, or may be represented by the angle of each joint of the robot 20 at each time.
[0039] The target position selection unit 38 reads out the predicted orbit information and the approach orbit information from the storage unit 44. When the approach orbit information indicates two or more approach orbits, the target position selection unit 38 calculates the similarity between the predicted orbit indicated by the predicted orbit information and each approach orbit indicated by the approach orbit information. Here, the degree of approximation between a first distribution of positions at each time after the present as a set of positions forming the predicted orbit and a second distribution of positions at each corresponding time as a set of positions forming one approach orbit is evaluated as the similarity. An index value that quantitatively indicates the two distributions can be used as the index value of the similarity. For example, the Bhattacharyya distance corresponds to such an index value. The Bhattacharyya distance is an index value that indicates that the similarity between the two distributions is lower as its value increases. However, such an index value is not limited to this, and may be the Wasserstein distance or the like. The target position selection unit 38 identifies the approach orbit that gives the highest similarity among the similarities calculated for each approach orbit. The target position selection unit 38 selects a target position corresponding to the identified approach trajectory, and stores target position information indicating the selected target position in the storage unit 44. In the example shown in Fig. 3, the approach trajectory pp12 is selected as the approach trajectory that is closest to the predicted trajectory pt01. Therefore, the target position ap12 corresponding to the approach trajectory pp12 is identified. When the approach trajectory information indicates one approach trajectory, the target position selection unit 38 may directly adopt the target position corresponding to the one approach trajectory indicated by the approach trajectory information. The target position selection unit 38 stores the target position information indicating the adopted target position in the storage unit 44.
[0040] Furthermore, if no approach trajectory is identified, the target position selection unit 38 may determine, by coordinate conversion, a position corresponding to the hand position indicated by the latest second operator operation information stored in the storage unit 44 as the target position. In the coordinate conversion, coordinate values in the operator coordinate system indicating the position indicated by the second operator operation information are converted into coordinate values in the robot coordinate system. The operator coordinate system is a coordinate system based on the position and orientation of the operator. The robot coordinate system is a coordinate system based on the position and orientation of the robot 20.
[0041] The motion command generating unit 40 reads out the target position information from the storage unit 44, and generates a motion command for moving the effector from the current position to the target position indicated by the read out target position information. The motion command generating unit 40 reads out the drive state information stored in the storage unit 44, for example, and performs known inverse kinematics analysis and inverse dynamics analysis based on the current motion situation (angle of each joint) indicated by the drive state information and the moment of inertia of each segment to determine the drive force (torque) at each joint for moving the effector to the target position. The motion command generating unit 40 outputs a motion command indicating the determined drive force of each joint to the drive control unit 22.
[0042] The communication unit 42 transmits and receives various information wirelessly or via wires between each component of the robot 20 and other components (i.e., the display device 50, the operation device 70, and the environmental information acquisition unit 80). The communication unit 42 includes, for example, an input / output interface, a communication interface, and the like. The storage unit 44 temporarily or permanently stores data used for various processes in the control device 30, various data acquired by the control device 30, etc. The storage unit 44 includes a storage medium such as a RAM (Random Access Memory) or a ROM (Read Only Memory).
[0043] The display device 50 includes a display unit 52, a gaze detection unit 54, a motion detection unit 56, a control unit 58, and a communication unit 60. The display device 50 includes a support member that can be worn on the head of an operator who is a user, and may be configured as a head mounted display (HMD).
[0044] The display unit 52 receives image data from the environmental information acquisition unit 80 by using the communication unit 60 via the control device 30. Based on the received image data, the display unit 52 displays an image showing the operating environment of the robot 20. When the display unit 52 is worn on the head of the operator, the display unit 52 may be disposed in a position where the screen faces directly in front of both eyes. The gaze detection unit 54 includes a gaze sensor that detects the gaze direction of both eyes of the operator. The gaze detection unit 54 transmits gaze information indicating the gaze direction detected at each time to the control device 30 using the communication unit 60. The gaze detection unit 54 may be disposed at a position exposed to at least one of the eyes when worn on the operator's head. The motion detection unit 56 detects the motion of the head of the operator, and transmits first operator motion information indicating the detected motion to the control device 30 using the communication unit 60. The motion detection unit 56 is configured to include, for example, an acceleration sensor for detecting the motion of the operator.
[0045] The control unit 58 controls the functions of the display device 50. The control unit 58 includes a processor such as a CPU (Central Processing Unit). The communication unit 60 transmits and receives various information to and from the control device 30. The communication unit 60 includes a communication interface.
[0046] The operation device 70 includes a motion detection unit 72, a control unit 74, and a communication unit 76. The operation device 70 includes a support member that can be worn on the hand of an operator who is a user, and may be configured as a data glove. The motion detection unit 72 detects the motion of the operator's hand, and transmits second operator motion information indicating the detected motion to the control device 30 using the communication unit 76. The motion detection unit 72 is configured to include, for example, an acceleration sensor for detecting the motion of the operator's hand.
[0047] The control unit 74 controls the functions of the operation device 70. The control unit 74 includes a processor such as a CPU. The communication unit 76 transmits and receives various information to and from the control device 30. The communication unit 76 includes a communication interface. The motion detection unit 72 may be connectable to the communication unit 60 of the display device 50 and transmit the second operator motion information to the control device via the communication unit 60. In that case, the control unit 74 and the communication unit 76 may be omitted.
[0048] The environmental information acquisition unit 80 includes an image capture unit 82, a distance measurement unit 84, and a communication unit 86. The environmental information acquisition unit 80 may be installed in the housing of the robot 20, or may be installed in a location separate from the robot 20. The photographing unit 82 photographs an image of the operating environment within a predetermined range from the robot 20. The operating environment includes the range that the effector of the robot 20 can reach. The photographed image does not necessarily include the entire image of the robot 20. The photographing unit 82 is a digital video camera that photographs images at predetermined time intervals. The photographing unit 82 transmits image data showing the photographed image to the control device 30 via the communication unit 86.
[0049] The distance measuring unit 84 measures the distance from the robot 20 to the surface of an object in the operating environment within a predetermined range. The distance measuring unit 84 includes, for example, a wave transmitting unit, a wave receiving unit, and a distance detecting unit. The wave transmitting unit transmits waves such as infrared rays. The wave transmitting unit includes, for example, a light emitting diode. The wave receiving unit receives reflected waves generated by reflection on the surface of the object. The wave receiving unit includes, for example, a photodiode. The reflected waves are generated in response to the wave transmitted from the wave transmitting unit being incident on the surface of the object as an incident wave. The distance detecting unit detects the phase difference between the transmitted wave and the reflected wave, and can determine the distance to the surface of the object based on the detected phase difference. The distance detecting unit transmits distance data indicating the distance determined for each direction corresponding to each pixel of the image to the control device 30 via the communication unit 86. Note that the wave used to measure the distance is not limited to infrared rays, and may be millimeter waves, ultrasonic waves, or the like. The communication unit 86 transmits and receives various information to and from the control device 30. The communication unit 86 includes a communication interface. In addition, when the environmental information acquisition unit 80 is installed in the housing of the robot 20 and can transmit and receive various information to and from other functional units of the robot 20, the communication unit 86 may be omitted.
[0050] (Hardware configuration) Next, an example of the hardware configuration of the control device 30 according to the present embodiment will be described. 2 is a schematic block diagram showing an example of a hardware configuration of the control device 30 according to this embodiment. The control device 30 functions as a computer including a processor 102, a ROM 104, a RAM 106, an auxiliary storage unit 108, and an input / output unit 110. The processor 102, the ROM 104, the RAM 106, the auxiliary storage unit 108, and the input / output unit 110 are connected to each other so as to be able to input and output various data.
[0051] The processor 102, for example, reads out programs and various data stored in the ROM 104, and executes the programs to execute processes for realizing the functions of each part of the control device 30 and processes for controlling the functions. The processor 102 is, for example, a CPU. In this application, the execution of processes instructed by various commands written in a program may be referred to as "executing a program" or "running a program."
[0052] The ROM 104 stores, for example, a program for execution by the processor 102 .
[0053] The RAM 106 functions as a working area for temporarily storing various data and programs used by the processor 102, for example. The auxiliary storage unit 108 permanently stores various types of data. The auxiliary storage unit 108 stores data acquired by the control device 30. The auxiliary storage unit 108 is, for example, a storage medium such as a hard disk drive (HDD) or a solid state drive (SSD).
[0054] The input / output unit 110 is capable of wirelessly or wired input / output of various data with other devices. The input / output unit 110 may be connected to other devices via a network. The input / output unit includes, for example, one or both of an input / output interface and a communication interface. The display device 50, the operation device 70, and the environmental information acquisition section 80 may also have the same hardware configuration as that shown in FIG. 2 and be configured as a computer that realizes the functions of each device.
[0055] (Operation control processing) Next, an example of the operation control process according to the present embodiment will be described with reference to a flowchart shown in FIG 4. (Step S102) The information acquisition unit 32 acquires driving state information from the driving sensor 26. (Step S104) The trajectory estimation unit 36 calculates the position of the effector based on the angle of each joint indicated by the actuation state information. The trajectory estimation unit 36 fits the actual trajectory, which is the trajectory of the effector's position up to the present, to a predetermined continuous function. The target position setting unit 34 uses the fitted continuous function to estimate a predicted trajectory, which is the trajectory of the position at each time point in the future up to a predetermined time.
[0056] (Step S106) The information acquiring unit 32 acquires image data and distance data as environmental information indicating the operating environment of the robot 20 from the environmental information acquiring unit 80. (Step S108) The target position setting unit 34 performs a known image recognition process to identify the area of each object from the surface shape in the operating environment indicated by the image data and distance data. (Step S110) The target position setting unit 34 performs a known image recognition process to determine the shape of each object. The target position setting unit 34 refers to preset object model data and sets the position of a feature point corresponding to the determined shape as a target position.
[0057] (Step S112) The trajectory estimation unit 36 estimates, for each estimated target position, a trajectory from the current position of the effector to the target position as an approach trajectory using a predetermined interpolation method. (Step S114) The target position selection unit 38 calculates an index value indicating the similarity between each estimated approach trajectory and the predicted trajectory. The target position selection unit 38 identifies the approach trajectory that gives the index value indicating the highest similarity, and selects the target position corresponding to the identified approach trajectory. (Step S116) The action command generator 40 generates an action command for moving the effector to the selected target position, and outputs the generated action command to the drive controller 22. This causes the effector to move toward the target position. After that, the process shown in FIG. 4 ends.
[0058] <Second embodiment> Next, a second embodiment of the present invention will be described. In the following description, the differences from the first embodiment will be mainly described, and the functions and configurations common to the first embodiment will be described in the first embodiment unless otherwise specified. The robot system S1 according to this embodiment may have a configuration similar to that shown in FIG.
[0059] In general, there may be objects in the operating environment of the robot 20 that have dimensions larger than a predetermined dimension. The object has a surface that has dimensions sufficiently larger than the diameter of a typical effector, such as a desk, table, or base. On such a surface, the operator arbitrarily determines the position at which the effector is to be applied. Therefore, the position at which the effector is to be applied is not necessarily constant. Even if a feature point is set in advance, it may not be a valid target position. Therefore, in this embodiment, the target position setting unit 34 may determine whether the area of the detected object has dimensions larger than a threshold value of a predetermined dimension. The size that the effector can grasp may be set as the threshold value of the predetermined dimension in the target position setting unit 34. The target position setting unit 34 may determine whether the area of the detected object has dimensions larger than a threshold value of a predetermined dimension based on whether the area of the recognized object does not have dimensions that the effector can grasp (for example, at least one of width, thickness, height, etc.).
[0060] The target position setting unit 34 determines a target position for an object determined to have a dimension larger than a threshold value of a predetermined dimension based on the line of sight information. The target position setting unit 34 reads out the line of sight information from the storage unit 44, and determines the intersection of the line of sight direction from the center of the eyeball (or head) and the object at each time within a predetermined period (for example, 1 to 60 seconds) up to the present time indicated by the line of sight information as the operator's attention point. The target position setting unit 34 determines a representative point of the coordinates of the attention point determined for each time as the target position. The representative point may be any of the center of gravity, the most frequent point, etc., as long as it is a position that statistically represents the attention points within the predetermined period. In the example shown in FIG. 5, unlike the objects ob32 and ob34, the work table wst30 is determined to be an object having a dimension larger than the threshold value of a predetermined dimension. Then, an intersection is with the surface of the work table wst30 is determined for each line of sight direction. A representative point of the distribution of these intersections is is determined as the target position ap30. The target position setting unit 34 may determine the position indicated by the coordinate values obtained by weighting the coordinate values of the individual attention points as the target position. The weighting coefficient used in the weighted average may be set to a larger real value for the line of sight direction related to a time closer to the present. The target position is determined with more importance given to newer line of sight directions.
[0061] However, when the observed gaze direction is a direction based on the direction of the operator's head (hereinafter referred to as "head direction"), the gaze direction is corrected using the position and direction of the head. The target position setting unit 34 reads out the first operator motion information from the storage unit 44, and converts the position and direction of the operator's head at each time indicated by the read out first operator motion information into the robot coordinate system. The target position setting unit 34 then corrects the gaze direction using the converted head position and direction, and determines the point of attention using the corrected gaze direction.
[0062] In the example shown in Fig. 5, the intersections is distributed around the representative point, but this is not limited to this. As shown in Fig. 6, the intersections is may be distributed in the form of multiple subsets. In such a case, the operator may not be interested in the representative point of the entire distribution of the intersections is. The target position setting unit 34 performs a known clustering process on the distribution of the attention points determined for each time and classifies the distribution into a plurality of subsets (clusters). The target position setting unit 34 can use, for example, the k-means method. The k-means method is a type of non-hierarchical clustering.
[0063] The k-means method is a method having the following steps (1a)-(1d): (1a) initialization, (1b) cluster center determination, (1c) reallocation, and (1d) convergence determination. In (1a) initialization, the target position setting unit 34 randomly assigns each attention point to one of a predetermined k (k is an integer equal to or greater than 2) clusters. In (1b) cluster center determination, the target position setting unit 34 determines the calculated average value of the coordinates of each attention point for each cluster as the center value of the cluster. In (1c) reallocation, the target position setting unit 34 calculates the distance between the coordinate value of each attention point and the center value of each cluster, and assigns the attention point to the cluster with the shortest distance to the center value. In (1d) convergence determination, the target position setting unit 34 determines whether convergence has occurred based on whether the allocation to the cluster remains unchanged or the number of cluster changes is equal to or less than a predetermined number. In (1d) convergence determination, the target position setting unit 34 repeats the processes of (1b) and (1c) until it is determined that convergence has not occurred.
[0064] In the example shown in FIG. 6, an intersection point is with the surface of the work table wst30 is determined for each line of sight direction. A clustering process is performed on the distribution of these intersection points is, and they are classified into two subsets (clusters). Representative points of each subset are determined as target positions ap42 and ap44. Even in this case, the target position setting unit 34 may determine the position indicated by the coordinate value obtained by weighting the coordinate values of each attention point as the target position. The weighting coefficient used for the weighted average may be set to a larger real value for the line of sight direction related to a time closer to the present.
[0065] The target position setting unit 34 may use other methods in the clustering process. The target position setting unit 34 may classify the points of interest by, for example, performing hierarchical clustering. Hierarchical clustering is a method having the following steps (2a)-(2d): (2a) The target position setting unit 34 sets clusters each having one point of interest as a member. (2b) The target position setting unit 34 integrates the clusters having the highest similarity among the similarities between the clusters to form one cluster. (2c) The target position setting unit 34 ends the process when the number of clusters becomes one, and otherwise proceeds to the process of (2d). (2d) The target position setting unit 34 calculates the similarity between the cluster formed in the process of (2b) and each of the other clusters, and returns to the process of (2b). Hierarchical clustering is applicable even when the number of clusters is not known in advance. Therefore, the similarity between clusters can be calculated regardless of the number of points of interest belonging to each cluster.
[0066] An example of a hierarchical clustering method is the group average method. The group average method is a method in which the similarity between attention points selected from each of two clusters is calculated for each set of attention points in this hierarchical clustering, and the average of the calculated similarities is determined as the similarity between the two clusters. According to the group average method, a representative (e.g., a center of gravity) of each cluster to be finally obtained can be obtained as a representative of attention points. Also, since a clear cluster where attention points are concentrated is often formed in a specific area that is of high interest to the operator, a representative point determined for that area can be used as a target position. The target position setting unit 34 may discard any cluster that has fewer attention points than a predetermined threshold value for the number of attention points among the classified clusters. This eliminates target positions based on attention points even if the attention points are sparsely distributed in an area of low interest to the operator. Therefore, target positions that are likely to be positions of interest to the operator can be left.
[0067] (Operation control processing) Next, an example of the operation control process according to this embodiment will be described. Fig. 7 is a flowchart showing an example of the operation control process according to this embodiment. The process shown in Fig. 7 includes steps S102-S108, steps S112-S116, and steps S122-S130. The process shown in FIG. 7 proceeds to step S122 after step S108 is completed. (Step S122) The target position setting unit 34 determines whether any of the detected objects is a target object to be processed, and determines whether the target object has a dimension larger than a predetermined dimension. If it is determined that the target object has a dimension larger than a predetermined dimension (YES in step S122), the process proceeds to step S124. If it is determined that the target object does not have a dimension larger than a predetermined dimension (NO in step S122), the process proceeds to step S126. (Step S124) The target position setting unit 34 determines the intersection of the line of sight direction and the target object at each time within a predetermined period up to the present time indicated by the line of sight information as the operator's attention point, and determines the representative point of the coordinates of the determined attention point as the target position. Then, the process proceeds to step S128.
[0068] (Step S126) The target position setting unit 34 determines the shape of the target object using the same method as in step S110, and sets the position of the feature point corresponding to the determined shape as the target position with reference to the object model data. (Step S128) The target position setting unit 34 judges whether or not there is an unprocessed object among the detected objects. If there is an unprocessed object (step S128 YES), the process proceeds to step S130. If there is no unprocessed object (step S128 NO), the target position setting unit 34 proceeds to step S112. (Step S130) The target position setting unit 34 changes the target object to any one of the unprocessed objects, and returns to the process of step S122.
[0069] As described above, the control device 30 includes a target position setting unit 34 that determines a target position of the effector of the robot 20 based on the form of an object in the operating environment of the robot 20. The control device 30 includes a trajectory estimation unit 36 that estimates a predicted trajectory of the effector based on the operation of the effector up to the present, and estimates a trajectory from the current position of the effector to the target position as an approach trajectory using a predetermined function. The control device 30 also includes a target position selection unit 38 that selects one target position based on the similarity between the predicted trajectory and each approach trajectory. According to this configuration, an approach trajectory to a target position is set based on the shape of an object in the operating environment of the robot 20, and the target position is selected based on the similarity between a predicted trajectory estimated from the operation of the effector and the approach trajectory. By guiding the effector of the robot 20 to a target position that is likely to meet the operator's intention, the operability of the robot 20 can be improved. This processing is based on the target position determined based on the current operation of the effector and the shape of the object in the operating environment, and is therefore independent of the type of task. Therefore, it can be applied to scenarios in which an object is approached, regardless of the type of task.
[0070] The target position setting unit 34 may determine the shape of an object from an image representing the operating environment, and based on feature point data indicating the arrangement of feature points for each shape, set the position of the feature point corresponding to the determined shape as the target position. According to this configuration, the position of the feature point corresponding to the shape of the object determined from the image representing the operating environment is determined as the target position. The action mechanism of the robot 20 can be guided to the target position of a typical feature point according to the shape of the object.
[0071] The target position setting unit 34 may reject a target position that interferes with another object. According to this configuration, invalid target positions to which the effector cannot be approached are eliminated, so that the effector can be moved more reliably.
[0072] For an object having dimensions larger than a predetermined dimension, the target position setting unit 34 may determine an intersection between the surface of the object and the line of sight direction of the operator based on line of sight information indicating the line of sight direction, and determine a target position based on the distribution of the intersections. With this configuration, the position at which the operator gazes is determined as the target position, so even in the case of a large object whose feature points cannot be uniquely determined, the operation of the effector can be guided to the object and position in which the operator is interested.
[0073] The target position setting unit 34 may classify the intersections into a plurality of subsets, and set a representative point of the distribution of the intersections for each subset as the target position. According to this configuration, even if the positions where the operator gazes are distributed discretely, the target positions corresponding to each distribution are determined. By guiding the effector of the robot 20 to the position of the operator's interest, the operability can be improved.
[0074] Although the embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to the above, and various design changes, etc. are possible within the scope that does not deviate from the gist of the present invention.
[0075] For example, in the environmental information acquisition unit 80, the distance measurement unit 84 may be integrated with the photographing unit 82 and configured as a three-dimensional image photographing unit. Moreover, the environmental information acquisition unit 80 may include a multi-viewpoint photographing unit instead of the photographing unit 82 and the distance measurement unit 84. The multi-viewpoint photographing unit is a photographing unit that has a plurality of viewpoints and can photograph an image for each viewpoint. The multi-viewpoint photographing unit includes a so-called stereo camera. The operating environment of the robot 20 is three-dimensionally expressed by the images (multi-viewpoint images) for each viewpoint photographed by the multi-viewpoint photographing unit. The target position setting unit 34 can identify the area occupied by an object from the operating environment of the robot 20 by analyzing the multi-viewpoint images.
[0076] In the first embodiment, a part or the whole of the display device 50 may be omitted. In the second embodiment, the gaze information mainly indicates the gaze direction of one eye (for example, the left eye) of the operator, but the present invention is not limited to this. The gaze information may indicate the gaze direction of each of the left eye and the right eye. When the gaze direction of one eye is not available at a certain time (for example, blinking), the target position setting unit 34 may use the gaze direction of the other eye when determining the attention point. When the gaze directions of both eyes are available, the target position setting unit 34 may determine the intersection of the line segments of the gaze directions of each of the two eyes, or the midpoint of the closest point of each of these line segments, as the gaze point. Then, the target position setting unit 34 may determine the direction from the head to the gaze point as the gaze direction representing the gaze directions of both eyes, and determine the intersection of the determined gaze direction and the surface of the object. The determined intersection is used to set the target position.
[0077] The robot 20 may include multiple manipulators. The robot 20 may be a dual-arm robot that includes two manipulators. Each manipulator can be operated in response to the movement of one of the forearms and hands of an operator. For a dual-arm robot, for example, one and the other manipulator may correspond to the left and right hands of an operator, respectively. Also, multiple manipulators may correspond to one of the forearms and hands of multiple operators. The control device 30 may determine one target position for each manipulator using the above method and operate the manipulators toward the determined target position.
[0078] The target position setting unit 34 may use a machine learning model when determining the area of each object from the operating environment of the robot 20. The target position setting unit 34 may use a machine learning model when determining the shape of the determined object area and determining a target position according to the determined shape. The target position setting unit 34 may use a neural network such as a convolutional neural network or a recurrent neural network. The target position setting unit 34 is not limited to a neural network, and may use other machine learning models such as a random forest.
[0079] In the above description, when estimating the target position, the target position setting unit 34 mainly identifies objects placed in the operating environment of the robot 20 using image data and distance data sequentially acquired from the environment information acquisition unit 80 as environmental information indicating the shape of the object, but this is not limited to this. For example, the target position setting unit 34 may identify an object by using object identification information indicating each object instead of or together with the image data and distance data. For example, RFID (Radio Frequency Identification) or the like can be used as the object identification information. The RFID may indicate the shape of each object, or may indicate information indicating the type of the object, such as its product name and model number. The target position setting unit 34 can identify the shape corresponding to the type of object indicated by the RDIF by referring to an object type table indicating the shape of each type of object in advance.
[0080] However, one or more RD tags are attached to the surface of each object placed in the operating environment of the robot 20 as a transmitter that transmits radio waves or electromagnetic waves (hereinafter collectively referred to as "electromagnetic waves, etc.") that carry RFID. The robot system S1 has one or more RF readers as detectors that detect the electromagnetic waves, etc. transmitted from the RF tags. The RF readers transmit the RFID carried by the detected electromagnetic waves, etc. to the control device 30 as object identification information. The information acquisition unit 32 stores the object identification information received from the RF tags in the memory unit 44, making the object identification information available to the target position setting unit 34.
[0081] The number of RF readers provided in the robot system S1 may be three or more. In that case, the target position setting unit 34 can detect the position of the RD tag that emits the RFID as the position of the object, for each of two RF readers of electromagnetic waves or the like that carry a common RFID from three or more RF readers, based on the phase difference generated between the RF readers and the positions of the individual RF readers that are set in advance. Also, the number of RD tags installed on the surface of one object may be two or more, and the target position setting unit 34 may estimate the positions of each RD tag. The target position setting unit 34 can detect the orientation of the object from the estimated positions of the multiple RD tags by setting in advance attachment position information indicating the positions where the individual RD tags are attached on the object in advance. The target position setting unit 34 can determine the target position as described above based on the shape, position, and orientation of the detected object without necessarily using image data and distance data.
[0082] The robot system S1 may also include a simulator that synthesizes three-dimensional data representing the shapes, positions, and orientations of various objects in the space of the operating environment of the robot 20 as environmental information, instead of or together with the image data and distance data. The simulator may synthesize a three-dimensional image as environmental information using a known computer graphics technique, or may acquire object information that expresses the shapes, positions, and orientations of individual objects in the three-dimensional image by numerical values or codes. The simulator transmits the synthesized or acquired environmental information to the control device 30. The information acquisition unit 32 stores the object identification information received from the simulator in the storage unit 44, thereby making the object identification information available to the target position setting unit 34. This allows the robot system S1 to control the operation of the robot 20 without necessarily using images captured in reality or distances measured in reality. In other words, the robot system S1 can be applied to operation control in a virtual space that simulates an operating environment, or to operation control in an augmented reality space that combines the virtual space with the operating environment of the real world. [Explanation of symbols]
[0083] S1...robot system, 20...robot, 22...drive control unit, 24...drive unit, 26...drive sensor, 28...power supply, 30...control unit, 32...information acquisition unit, 34...target position setting unit, 36...trajectory estimation unit, 38...target position selection unit, 40...motion command generation unit, 42...communication unit, 44...storage unit, 50...display unit, 52...display unit, 54...gaze detection unit, 56...motion detection unit, 58...control unit, 60...communication unit, 70...operation device, 72...motion detection unit, 74...control unit, 76...communication unit, 80...environmental information acquisition unit, 82...image capture unit, 84...distance measurement unit, 86...communication unit
Claims
1. A control device that detects an operator's movement and causes a robot to perform an action corresponding to the movement, a target position setting unit that determines a target position of an action mechanism of the robot based on a shape of an object present in an operating environment of the robot; Calculating a position of the effector based on a driving state of the robot; estimating a predicted trajectory of the effector based on an actual trajectory indicating a position of the effector at each time within a predetermined period up to the present; a trajectory estimation unit that estimates a trajectory from a current position of the effector to the target position as an approach trajectory using a predetermined function; a target position selection unit that selects one target position based on a similarity between the predicted trajectory and each of the approach trajectories; A control device comprising:
2. The target position setting unit determining a shape of the object from an image representative of the operating environment; Based on feature point data indicating the arrangement of feature points for each shape, the position of the feature point corresponding to the determined shape is determined as the target position. The control device according to claim 1 .
3. The target position setting unit Reject target positions that collide with other objects The control device according to claim 2.
4. The target position setting unit For objects having a dimension greater than a given dimension, determining an intersection between a surface of the object and the line of sight direction based on line of sight information indicating the line of sight direction of an operator; The target position is determined based on the distribution of the intersection points. The control device according to any one of claims 1 to 3.
5. The target position setting unit The intersections are classified into a plurality of subsets, and a representative point of the distribution of intersections for each subset is determined as a target position. The control device according to claim 4.
6. A program for causing a computer to function as the control device according to any one of claims 1 to 5.
7. A control device according to any one of claims 1 to 5 and the robot Robot system.
8. A control method for a control device that detects an operator's motion and causes a robot to perform a motion corresponding to the motion, comprising: A first step of determining a target position of an effector of the robot based on a shape of an object present in an operating environment of the robot; Calculating a position of the effector based on a driving state of the robot; estimating a predicted trajectory of the effector based on an actual trajectory indicating a position of the effector at each time within a predetermined period up to the present; a second step of estimating a trajectory from the current position of the effector to the target position as an approach trajectory using a predetermined function; a third step of selecting one target position based on a similarity between the predicted trajectory and each of the approach trajectories; The control method includes:
Citation Information
Patent Citations
Qualitative inference processing method
JP1989076358A
Robot motion control device
JP2017196678A
Systems and methods for operating a robotic system and executing robotic interactions
US20190291277A1
Control device, robot control method, and robot control system
WO2018212226A1
