Information processing device, control method and program

Abstract

The object of the present invention is to allow an object to be smoothly transferred. An information processing device comprises a control unit (12, 11) which controls a manipulator so that the object held by the manipulator maintains a continuous moving speed when transferring the object to a recipient.

Description

Technical Field

[0001] The present disclosure relates to an information processing device, a control method, and a program. Background Art

[0002] In recent years, robots have been actively developed for interacting with humans in homes, care institutions, stores, etc. The interaction between humans and robots includes, for example, conversation, transfer of objects, and the like.

[0003] Reference List

[0004] Patent Literature

[0005] Patent Document 1: JP 2013-111737 A Summary of the invention

[0006] Technical issues

[0007] Here, in the delivery of the object, fine control is required according to the behavior, condition, etc. of the object as the target to be delivered or the person as the delivery target in order to achieve smooth delivery of the object.

[0008] Therefore, the present disclosure proposes an information processing device, a control method, and a program that enable smooth transfer of an object.

[0009] Solution to the problem

[0010] In order to solve the above problem, an information processing apparatus according to one aspect of the present disclosure includes: a control unit that controls a manipulator so that when an object gripped by the manipulator is transferred to a transfer target person, a moving speed of the object maintains continuity.

[0011] (Action) According to the information processing device according to the embodiment of the present disclosure, the manipulator is controlled so that when the object is delivered to the delivery target person, the continuity of the moving speed of the object in the delivery direction is maintained. Therefore, the sudden change of the displacement of the object in the delivery direction can be reduced, and thus it becomes possible to smoothly deliver the object. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram for describing an operation of handing an object from an autonomous robot to a delivery target person according to an embodiment.

[0013] Figure 2 is a graph showing an example of the optimal transfer operation in the embodiment.

[0014] Figure 3 is a graph showing an example of a non-optimal transfer operation in the embodiment.

[0015] Figure 4is a block diagram showing a schematic configuration example of an autonomous robot according to an embodiment.

[0016] Figure 5 is an external view showing a configuration example of a hand according to an embodiment.

[0017] Figure 6 It is shown from Figure 5 This is an external view of a configuration example of a case where the hand is viewed from the direction A2.

[0018] Figure 7 is a block diagram illustrating an example of functional components for performing a transfer operation of an autonomous robot according to an embodiment.

[0019] Figure 8 is a diagram illustrating an example of input / output information in a machine learning process according to an embodiment.

[0020] Fig. 9 is a flowchart illustrating an exemplary operation example according to an embodiment.

[0021] Fig.10 is a flowchart illustrating a transfer operation according to a first example of an embodiment.

[0022] Fig.11 is a flowchart illustrating a transfer operation according to a second example of the embodiment.

[0023] Fig.12 is a flowchart illustrating a transfer operation according to a third example of the embodiment.

[0024] Fig.13 is a flowchart illustrating a transfer operation according to a fourth example of the embodiment.

[0025] Fig.14 is a diagram for describing a blend between a clamping operation and a releasing operation according to an embodiment.

[0026] Fig.15 : is a diagram for describing a mixture between a clamping operation and a releasing operation according to a modification example of the embodiment.

[0027] Fig.16 is a schematic diagram showing an example of a slip sensor configured using a vision sensor according to an embodiment.

[0028] Fig.17 It is shown Fig.16 Figure 3 shows an example of a deformed portion.

[0029] Fig.18 is a diagram for describing an initial slide detected by a vision sensor according to an embodiment (part 1).

[0030] Fig.19 is a diagram for describing an initial slide detected by a vision sensor according to an embodiment (part 2).

[0031] Fig. 20 is a diagram for describing an initial slide detected by a vision sensor according to an embodiment (Part 3).

[0032] Fig.21 is a schematic diagram showing an example of a slide sensor configured using a pressure distribution sensor according to an embodiment.

[0033] Fig. 22 is a diagram for describing initial slip detected by a pressure distribution sensor according to an embodiment (Part 1).

[0034] Fig.23 is a diagram for describing initial slip detected by the pressure distribution sensor according to an embodiment (Part 2).

[0035] Fig.24 is a block diagram showing an example of a system configuration according to an embodiment. DETAILED DESCRIPTION

[0036] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Note that in each of the following embodiments, the same parts will be denoted by the same reference numerals, and repeated descriptions thereof will be omitted.

[0037] Furthermore, the present disclosure will be described in the following order of items.

[0038] 1. Implementation Method

[0039] 1.1 Overview of Autonomous Robots

[0040] 1.2 Object transfer operations

[0041] 1.3 Schematic configuration of the autonomous robot

[0042] 1.4 Hand configuration

[0043] 1.5 Functional components of autonomous robots

[0044] 1.6 Machine Learning on Transfer Actions

[0045] 1.7 Operation Examples

[0046] 1.8 Specific examples of transfer operations

[0047] 1.8.1 First Example

[0048] 1.8.2 Second Example

[0049] 1.8.3 Third Example

[0050] 1.8.4 Fourth Example

[0051] 1.9 Mixing of Clamping and Release Operations

[0052] 1.9.1 Modification Example

[0053] 1.10 Measurement of initial slip

[0054] 1.10.1 Vision Sensor

[0055] 1.10.2 Pressure distribution sensor

[0056] 1.11. System Configuration

[0057] 1.12 Effects

[0058] 1. Implementation Method

[0059] Hereinafter, an information processing device, a control method, and a program according to an embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. In this embodiment, an information processing device, a control method, and a program will be described by way of example, which enables physical interaction of exchanging unknown objects between an autonomous robot with an arm (also referred to as a "manipulator") such as a robot hand, a humanoid robot, a pet robot, etc. and a human to be smoothly performed.

[0060] 1.1 Overview of Autonomous Robots

[0061] Figure 1 2 is a schematic diagram for describing an operation of handing an object from an autonomous robot to a person (hereinafter, referred to as a “transfer target person”) according to the present embodiment. Figure 1 As shown, the autonomous robot 1 is, for example, a humanoid robot including a head 41, a body 42, a carriage portion 43, and manipulators 44L and 44R. The body 42 and the carriage portion 43 constitute a moving body that can move on, for example, a floor P1 or the like.

[0062] In addition to the travel mechanism such as wheels, tracks, etc., the bracket portion 43 also accommodates, for example, a travel motor, a battery, a control unit, etc. However, the travel mechanism is not limited to wheels, tracks, etc., and may be a walking mechanism including two or more legs, etc. In addition, the autonomous robot 1 is not limited to a humanoid robot, and various robots having at least one arm such as only a manipulator, an autonomous mobile body equipped with a manipulator, etc. may be employed as the autonomous robot 1.

[0063] Each of the manipulators 44L and 44R (hereinafter, in the case where the manipulators 44L and 44R are not distinguished from each other, their reference numerals are simply 44) includes: an upper arm portion 441, which is attached to a position corresponding to the shoulder of the body portion 42; a forearm portion 442, which is attached to the upper arm portion 441 at a position corresponding to the elbow of the manipulator 44; and a hand portion 443, which is attached to the forearm portion 442 at a position corresponding to the wrist of the manipulator 44. The upper arm portion 441 and the forearm portion 442 constitute, for example, an arm portion in the manipulator 44. Each joint corresponding to the shoulder, elbow, and wrist of the manipulator 44 is provided with, for example, a drive unit, a joint mechanism, and the like for moving the manipulator 44 like a human arm. As the drive unit, for example, an electromagnetic motor, a hydraulic actuator, a pneumatic actuator, and the like can be used.

[0064] In addition, the hand 443 is also provided with a finger portion as a mechanism for gripping an object. Note that in the following description, a case where the hand 443 has two fingers will be exemplified, but the number of fingers of the hand 443 is not limited thereto, and various modifications may be made, for example, three fingers, four fingers, five fingers, etc. In addition, a hand using a jamming transition, a hand using suction performed by air pressure control, etc. may also be used as the hand 443.

[0065] 1.2 Object transfer operations

[0066] In such an autonomous robot 1, in the case of handing the object B1 gripped by the hand 443 to the hand H1 of the transfer target person, it is preferable that during the execution period of the transfer operation, the displacement of the object B1 in the transfer direction A1 is continuous, for example, linear, such as Figure 2 On the other hand, Figure 3 As shown, when the displacement of object B1 in the transfer direction A1 includes sudden changes during the execution period of the transfer operation, it is very likely that object B1 cannot be successfully transferred from hand 443 to hand H1 of the transfer target person, causing an event in which object B1 falls or the contents of object B1 overflows.

[0067] The object transfer operation between the person and the autonomous robot includes a clamping operation in which the autonomous robot clamps the object, a releasing operation in which the autonomous robot releases the object, and a receiving operation in which the person receives the object, and the autonomous robot 1 releases the object clamped by the clamping operation through a releasing operation when transferring the object to the person. That is, when transferring the clamped object to the person, the autonomous robot 1 changes the blend ratio between the clamping operation and the releasing operation to gradually increase the ratio of the releasing operation, and finally releases the object.

[0068] Here, for example, the release operation performed by the autonomous robot does not have a predetermined pattern, and the optimal release operation is different for each object or each delivery target person. For example, the optimal release operation is different between the case of delivering a teacup filled with hot water and the case of delivering a tennis ball.

[0069] This is because the amount of movement of object B1 per unit time in the transfer direction A1 changes according to the magnitude of the external force applied to object B1 by the transfer target person during the receiving operation, but this change in the amount of movement varies depending on the differences in the unique characteristics of object B1 (for example, static friction coefficient, dynamic friction coefficient, mass, shape and size, stiffness, strength, temperature, humidity, etc.).

[0070] For this reason, it is difficult for the designer of the autonomous robot to plan the best release operation for all objects with different characteristics in advance. Similarly, it is difficult for the designer to plan the best release operation for all situations where the behaviors, conditions, etc. of the delivery target person are different from each other in advance.

[0071] Here, in this embodiment, specific examples will be used to describe the configuration and operation of achieving high-quality physical interaction (specifically, the transfer of objects) between humans and autonomous robots in an environment where the object to be transferred is unknown or in an environment where the behavior, condition, etc. of the transfer target person are unknown (for example, a home, a nursing facility, a store, etc. where humans and autonomous robots coexist), that is, an environment where it is impossible to create a model of the optimal release operation in advance.

[0072] For example, in the present embodiment, by utilizing information from various sensors mounted on the autonomous robot 1, an optimal delivery operation can be performed according to the behavior, condition, etc. of the object B1 to be delivered or the delivery target person.

[0073] Note that in this specification, the optimal transfer operation may be, for example, that the rate of change of the sliding amount in the transfer direction when the object is transferred is maintained continuously (see, for example, Figure 2 ). However, the optimal transfer operation is not limited thereto, and various transfer operations in which a transfer target person can receive the object from the autonomous robot 1 without stress may be defined as the optimal transfer operation.

[0074] On the other hand, an inappropriate transfer operation may, for example, be an operation that results in events such as: an event in which a force greater than the necessary degree is generated when the transfer target person receives the object from the autonomous robot 1; an event in which a force is generated that causes deformation, damage, or ejection of the contents of the object when the transfer target person receives the object from the autonomous robot 1; and the like.

[0075] Furthermore, an inappropriate delivery operation may include an operation in which the delivery target person stops receiving the object by taking more time than the time required for the release operation, or thereby causes the object to fall or the contents of the object to jump out.

[0076] In the case where such an inappropriate transfer operation is performed, an external force or disturbance greater than necessary is generated in the autonomous robot 1, and the rate of change of the sliding amount of the object in the transfer direction becomes discontinuous (see, for example, Figure 3 ). In addition, in the case where an inappropriate transfer operation is performed, the rate of change of the sliding amount of the object in the transfer direction is intermittent, which also makes the rate of change of the sliding amount of the object in the transfer direction discontinuous.

[0077] Therefore, in the present embodiment, for example, machine learning is performed using the rate of change of the gripping force of the hand 443, information detected by various sensors mounted on the hand 443, etc. as inputs, and the continuity of the moving speed of the object B1 in the transfer direction A1 as output. Therefore, it is possible to achieve smooth transfer of the object B1 based on the characteristics of the object B1 (static friction coefficient, dynamic friction coefficient, mass, shape and size, rigidity, strength, temperature, humidity, etc.) and the behavior, condition, etc. of the transfer target person.

[0078] In addition, in the present embodiment, for example, a change in the sliding amount or initial sliding amount of the object B1 clamped by the hand 443 is measured, and when the change includes a component in a direction different from the gravity direction, such as a direction opposite to gravity, a direction opposite to the torque due to gravity, or a component in the transfer direction A1 (for example, the position direction of the hand H1 of the transfer target person), the release operation is started. Therefore, the release operation can be started at the initial stage of the transfer operation, and thus the fluctuation of the load applied to the object B1 or the clamping force of the hand 443 can be suppressed to a minimum, so that it becomes possible to transfer the object B1 more smoothly.

[0079] In addition, in this embodiment, in order to further improve the quality of physical interaction, in addition to the above, the presence, emotional changes, etc. of the target person obtained from image input information, voice input information, etc. can be added to the input or output of machine learning.

[0080] For example, the release operation can be started after confirming the presence of the transfer target person or the presence of the hand H1 using the result of image processing or the information from the distance measuring sensor. Therefore, it is possible to avoid starting the release operation by mistake, and thus it becomes possible to perform the transfer of the object B1 more safely and reliably. Note that the presence of the transfer target person can be confirmed using, for example, voice input / output information, etc., as well as the result of image processing or the information from the distance measuring sensor. In addition, after recognizing the intention of the transfer target person to clamp the object or the emotional change of the transfer target person during the release operation using voice input / output information, etc., it is possible to perform a determination on the start of the release operation or the continuation of the operation.

[0081] In addition, for example, by continuously measuring the change in the sliding amount of the object B1 in the transfer direction A1 during the release operation, the release operation can be controlled so that the continuity of the moving speed of the object B1 in the transfer direction A1 is maintained. Therefore, the sudden change in the displacement of the object B1 in the transfer direction A1 can be reduced, and thus it becomes possible to transfer the object B1 more smoothly.

[0082] Furthermore, for example, during the release operation, the sliding amount or initial sliding amount of the object B1 in the gravity direction can be continuously measured. Therefore, it becomes possible to reduce erroneous dropping of the object B1 or unnatural vertical displacement of the object B1 within the hand 443.

[0083] Note that in the present specification, the transmission direction A1 is defined, for example, as the direction of the carpal bone on a line connecting the upper end of the metacarpal bone of the ring finger and the carpal bone to each other after identifying the position, posture, and shape of the hand H1 of the transmission target person. In addition, the transmission direction A1 may also be defined as the sternal direction on a line connecting the center of the carpal bone and the center of the sternal bone to each other after identifying the position and shape of the hand H1 and torso of the transmission target person. Alternatively, the transmission direction A1 may also be defined as the humeral head direction on a line connecting the center of the carpal bone and the center of the humeral head to each other after identifying the position and shape of the hand H1 and torso of the transmission target person. However, the definition of the transmission direction A1 is not limited to the above, and various modifications may be made.

[0084] In addition, the release operation according to the present embodiment may include the gripping force control of the hand 443, the operation control of the arm in the manipulator 44 (hereinafter referred to as "arm operation control"), the whole body operation control of the autonomous robot 1, etc. The gripping force control of the hand 443, the arm operation control of the arm in the manipulator 44, the whole body operation control of the autonomous robot 1, etc. are mixed with each other in a certain mixing ratio, but in the present embodiment, an evaluation function for evaluating the mixing ratio is defined, and in order to obtain the best solution for the evaluation function, the above-mentioned machine learning is utilized which uses the rate of change of the gripping force applied to the hand 443, the information detected by various sensors installed on the hand 443, etc. as inputs and uses the continuity of the moving speed of the object B1 in the transfer direction A1 as output.

[0085] Note that in this specification, the gripping force control may be to control the amount of change in force per unit time when the gripping force generated in the hand 443 to grip the object B1 is reduced to release.

[0086] In addition, the arm operation control may be to change the posture of the arm that clamps the object B1 and arranges the object B1 in the target coordinate space to move the position of the object B1 in the transfer direction A1, or may be to control the amount of change per unit time at this time. Through such arm operation control, it becomes possible to reduce the inertial force in the transfer direction A1 generated in the object B1 when the object B1 is immediately released from the hand 443 at the time point when the start of the release operation has been detected.

[0087] In addition, the whole-body operation control of the robot may be to change the position and posture of the autonomous robot 1 that grips the object B1 and arranges the object B1 in the target coordinate space to move the position of the object B1 in the transfer direction A1, or may be to control the amount of change in the position per unit time at this time.

[0088] 1.3 Schematic configuration of the autonomous robot

[0089] Next, a schematic configuration of the autonomous robot 1 according to the present embodiment will be described in detail with reference to the drawings. Figure 4 is a block diagram showing a schematic configuration example of an autonomous robot according to the present embodiment. Figure 4 As shown, the autonomous robot 1 includes, for example, a control unit 10 and a battery 18 as a power source of the autonomous robot 1, and the control unit 10 is formed by connecting a central processing unit (CPU) 12, a dynamic random access memory (DRAM) 13, a flash read-only memory (ROM) 14, a personal computer (PC) card interface (I / F) 15, a wireless communication unit 16 and a signal processing circuit 11 to each other via an internal bus 17.

[0090] In addition, the autonomous robot 1 includes a movable unit 26 and an actuator 27 for driving the movable unit 26. The movable unit 26 is, for example, a joint part of a manipulator 44, a joint part of a body part 42 (neck joint, waist joint, etc.), a wheel, a track, etc. The actuator 27 serves as a movable mechanism for realizing operations such as movement, limb movements, etc.

[0091] In addition, the autonomous robot 1 includes an inertial measurement unit (IMU) 20 for detecting the direction or acceleration of movement of the autonomous robot 1 and an encoder (or potentiometer) 28 for detecting the driving amount of the actuator 27 as sensors (hereinafter referred to as "internal sensors") for acquiring information such as moving distance, moving speed, moving direction, posture, etc. Note that in addition to the IMU 20 and the encoder 28, an acceleration sensor, an angular velocity sensor, etc. can also be used as the internal sensor.

[0092] In addition, the autonomous robot 1 includes a camera 19 that captures an image of an external situation and a time-of-flight (ToF) sensor 21 that measures a distance to an object existing in a specific direction relative to the autonomous robot 1, as sensors (hereinafter referred to as "external sensors") that acquire information such as the terrain around the autonomous robot 1, the distance or direction to an object existing around the autonomous robot 1, and the like. Note that, in addition to the camera 19 and the ToF sensor 21, a light detection and ranging or laser imaging detection and ranging (LIDAR) sensor, a global positioning system (GPS) sensor, a magnetic sensor, a radio wave intensity measuring unit in the wireless communication unit 16 such as Bluetooth (registered trademark) or Wi-Fi (registered trademark) (hereinafter referred to as a "radio wave intensity sensor"), and the like can also be used as the external sensor.

[0093] In addition, the autonomous robot 1 may be provided with a touch sensor 22 for detecting physical pressure received from the outside, a microphone 23 for collecting external sound, a speaker 24 for outputting voice etc. to the surroundings, a display unit 25 for displaying various information to a user etc., and the like.

[0094] In addition, the movable unit 26 of the autonomous robot 1 includes a 6-axis force sensor 501, a 3-axis force sensor 502, a sliding sensor 503, a distance measuring sensor 504, a 3-axis sensor 261, a 1-axis sensor 262, a 3-axis sensor 263, a 1-axis sensor 264, a 2-axis sensor 265 and a 3-axis sensor 266 as sensors for controlling the transfer of the object B1.

[0095] The 6-axis force sensor 501 is attached to, for example, the wrist portion of the manipulator 44 , and detects the magnitude and direction of force and torque applied to the wrist portion.

[0096] The 3-axis force sensor 502 is attached to, for example, each finger joint in the hand 443 , and detects the magnitude and direction of a force or torque applied to the finger joint.

[0097] The sliding sensor 503 is attached to, for example, a portion of the hand 443 that contacts the object B1 to be clamped, such as a palm, a fingertip, etc., and detects the size (sliding amount) and direction of shear slip between the object B1 and the portion in contact with the object B1. In addition, the sliding sensor 503 can detect the size (initial sliding amount) and direction of initial slip generated between the object B1 and the portion in contact with the object B1. As the sliding sensor 503, for example, a visual sensor that observes the deformation of a viscoelastic body having a predetermined shape attached to the portion of the hand 443 that contacts the object B1, a pressure distribution sensor that measures the two-dimensional distribution of pressure, etc. can be used.

[0098] The distance measuring sensor 504 is attached to, for example, a part of the manipulator 44 that can observe the object B1 held by the hand 443 , such as the wrist, palm, back of hand, fingertips, etc., and measures the distance between the hand 443 and the object B1 .

[0099] The 3-axis sensor 261 is attached to, for example, a shoulder portion, and detects a roll angle, a pitch angle, and a yaw angle of the upper arm portion 441 relative to the body portion 42 .

[0100] The 1-axis sensor 262 is attached to the elbow portion, for example, and detects the pitch angle of the forearm portion 442 relative to the upper arm portion 441 .

[0101] The 3-axis sensor 263 is attached to, for example, the wrist portion, and detects the roll angle, pitch angle, and yaw angle of the hand 443 relative to the forearm portion 442 .

[0102] The 1-axis sensor 264 is attached to, for example, each finger joint of the hand 443 , and detects the pitch angle of each joint.

[0103] The 2-axis sensor 265 is attached to, for example, a joint portion between the bracket portion 43 and the body portion 42 , and detects a roll angle and a pitch angle of the body portion 42 relative to the bracket portion 43 .

[0104] The 3-axis sensor 266 is attached to, for example, the neck portion, and detects a roll angle, a pitch angle, and a yaw angle of the head portion 41 relative to the body portion 42 .

[0105] In the configuration described above, various sensors such as IMU 20, touch sensor 22, ToF sensor 21, microphone 23, speaker 24, encoder (or potentiometer) 28, display unit 25, actuator 27, camera 19 and battery 18 are connected to the signal processing circuit 11 of the control unit 10.

[0106] The signal processing circuit 11 sequentially acquires sensor data or image data and voice data supplied from the above-mentioned various sensors, and sequentially stores these data at predetermined locations in the DRAM 13 via the internal bus 17. In addition, the signal processing circuit 11 sequentially acquires battery remaining amount data indicating the remaining amount of battery supplied from the battery 18 together with this, and stores the battery remaining amount data at predetermined locations in the DRAM 13.

[0107] Each sensor data, image data, voice data, and battery remaining amount data stored in the DRAM 13 in this manner is used when the CPU 12 performs operation control of the autonomous robot 1, and when necessary, is transmitted to an external server or the like via the wireless communication unit 16. Note that the wireless communication unit 16 may be a communication unit for communicating with an external server or the like via a predetermined network such as a wireless local area network (LAN), a mobile communication network, or the like in addition to Bluetooth (registered trademark), Wi-Fi (registered trademark), or the like.

[0108] The CPU 12 reads a control program stored in a memory card 30 loaded in a PC card slot (not shown) or a flash ROM 14 via the PC card I / F 15 or directly, and stores the control program in the DRAM 13, for example, at the initial stage of supplying power to the autonomous robot 1.

[0109] In addition, the CPU 12 determines the condition of the autonomous robot 1 and its surroundings, whether there are instructions or actions from the user, etc. based on each sensor data, image data, voice data, and battery remaining data sequentially stored in the DRAM 13 from the signal processing circuit 11 as described above.

[0110] Furthermore, the CPU 12 can perform self-position estimation or various operations using map data stored in the DRAM 13 or the like or map data acquired from an external server or the like via the wireless communication unit 16 and various information.

[0111] Then, the CPU 12 determines the subsequent action based on the above-mentioned determination results, the estimated own position, the control program stored in the DRAM 13, etc., and drives the required actuator 27 based on the determination results of the subsequent action to perform various actions in addition to the transfer operation of the object B1, such as movement, limb movements, etc.

[0112] At this time, the CPU 12 generates voice data, if necessary, and supplies the voice data as a voice signal to the speaker 24 via the signal processing circuit 11 to output voice based on the voice signal to the outside or cause the display unit 25 to display various information.

[0113] In this way, the autonomous robot 1 is configured to be able to autonomously act according to the conditions of the autonomous robot 1 and its surroundings, instructions and actions from the user.

[0114] 1.4 Hand configuration

[0115] Next, the configuration of the hand 443 of the autonomous robot 1 according to the present embodiment will be described in detail with reference to the drawings. Note that in this specification, for simplicity, a case where the hand 443 has two fingers is exemplified. Figure 5 : is an external view showing an example of the configuration of the hand according to the present embodiment. Figure 6 It is shown from Figure 5 This is an external view of a configuration example of a case where the hand is viewed from direction A2.

[0116] like Figure 5 and Figure 6 As shown, the hand 443 includes, for example, a base 4431 corresponding to the palm and the back of the hand, and two finger portions 4432a and 4432b (hereinafter, in the case where the finger portions 4432a and 4432b are not distinguished from each other, their reference numerals are both 4432). The base 4431 is attached to the forearm portion 442 via a joint mechanism 444 corresponding to the wrist, for example.

[0117] As described above, the joint mechanism 444 corresponding to the wrist is provided with a 6-axis force sensor 501 for detecting the magnitude and direction of the force and torque applied to the wrist part and a 3-axis sensor 263 for detecting the roll angle, pitch angle and yaw angle of the hand 443 relative to the forearm 442.

[0118] Each finger portion 4432 includes: a base joint portion 4434 attached to the base portion 4431 via a joint mechanism 4433 corresponding to a finger joint (third joint); and a distal joint portion 4436 attached to the base joint portion 4434 via a joint mechanism 4435 corresponding to a finger joint (first joint). For example, the two finger portions 4432a and 4432b are attached to the base portion 4431 so that the surfaces of the two finger portions corresponding to the finger pulps face each other.

[0119] The joint mechanisms 4433 and 4435 corresponding to each finger joint of each finger portion 4432 are provided with a 3-axis force sensor 502 for detecting the magnitude and direction of a force or moment applied to each finger joint and a 1-axis sensor 264 for detecting the pitch angle of each joint.

[0120] In addition, the portion of the finger portion 4432 that contacts the object B1 when the object B1 is clamped (for example, the portion of the distal joint portion 4436 corresponding to the finger pulp) is provided with a slide sensor 503. In addition, the surface of the base portion 4431 corresponding to the palm, that is, the surface facing the slide sensor 503 attached to the distal joint portion 4436 when the finger portion 4432 is folded, is provided with a ToF sensor 504a as a distance measurement sensor 504. In addition, the portion of the base portion 4431 corresponding to the back of the hand is provided with a camera device 504b as the distance measurement sensor 504, the roll angle, pitch angle, and yaw angle of which can be adjusted.

[0121] 1.5 Functional components of autonomous robots

[0122] Next, functional components for performing a transfer operation of the autonomous robot 1 according to the present embodiment will be described in detail with reference to the drawings. Figure 7 is a block diagram showing an example of functional components for performing a transfer operation of an autonomous robot according to the present embodiment. Figure 7 The components other than the physical interaction execution unit 55 and the learning information storage unit 54 in the functional components shown can be, for example, Figure 4 The CPU 12 shown is implemented by executing a predetermined program stored in the flash ROM 14 or the memory card or a program downloaded via the wireless communication unit 16. In addition, the physical interaction execution unit 55 may be implemented by, for example Figure 4 The CPU 12, signal processing circuit 11, movable unit 26 and actuator 27 shown are implemented. In addition, the learning information storage unit 54 may be implemented by, for example Figure 4 This is accomplished by the flash ROM 14 or the memory card 30 shown.

[0123] like Figure 7 As shown, the autonomous robot 1 includes a transfer target person recognition unit 51, an object recognition unit 52, a transfer action planning unit 53, a learning information storage unit 54, a physical interaction execution unit 55, a gripping information acquisition unit 56, a response time measurement unit 57, an emotion map generation unit 58, and a transfer action evaluation unit 59 as functional components for performing a transfer operation of the object B1 with the transfer target person. In addition, for example, the camera 19, the microphone 23, the 6-axis force sensor 501, the 3-axis force sensor 502, the slide sensor 503, and the distance measurement sensor 504 among the sensors included in the autonomous robot 1 constitute a sensor group 50 for acquiring various information used in performing the transfer operation.

[0124] The delivery target person recognition unit 51 recognizes the existence of the delivery target person and / or the delivery target person's hand H1, the delivery target person's intention to receive the object B1, or the delivery target person's behavior, condition, etc. (hereinafter referred to as "the delivery target person's posture") by analyzing, for example, image data acquired by the camera 19 or voice data input from the microphone 23, and inputs the recognition result to the delivery action planning unit 53. In addition, the delivery target person recognition unit 51 detects emotional changes of the delivery target person during the release operation, such as emotional changes such as interruption or abandonment of the receiving operation due to enthusiasm, indifference, timing mismatch, etc., by analyzing the image data acquired by the camera 19 or voice data input from the microphone 23 during the release operation performed by the physical interaction execution unit 55 described later, and inputs the detection result to the emotion map generation unit 58.

[0125] The object recognition unit 52 recognizes or estimates the position of the object B1 or the characteristics of the object B1, such as the static friction coefficient, the dynamic friction coefficient, the mass, the shape and size, the stiffness, the strength, the temperature, the humidity, etc., based on, for example, the analysis results of the image data obtained by the camera device 19, the sensor data input from the 6-axis force sensor 501, the 3-axis force sensor 502 and the sliding sensor 503, and inputs the recognition or estimation results to the transfer action planning unit 53.

[0126] The learning information storage unit 54 stores, for example, a learned model constructed by performing machine learning on the delivery operations performed in the past, a delivery action plan planned by the delivery action planning unit 53 and its evaluation results, etc. Note that the component that constructs the learned model by performing machine learning on the delivery operations performed in the past may be arranged in the autonomous robot 1, for example, or may be arranged on a server connected to the autonomous robot 1 via a predetermined network.

[0127] The delivery action planning unit 53 creates a delivery action plan for delivering the object B1 to the delivery target person based on the recognition result of the delivery target person input from the delivery target person recognition unit 51 and the recognition or estimation result of the object B1 input from the object recognition unit 52, based on the learned model stored in the learning information storage unit 54, the delivery action plan planned in the past and its evaluation result, etc. For example, the created delivery action plan may include an operation of lifting the object B1 from a table, a floor, etc. or receiving the object B1 from a person, etc., and an operation of delivering the object B1 to the delivery target person.

[0128] The physical interaction executing unit 55 executes a physical interaction (delivery action) of delivering the object B1 to the delivery target person by executing the delivery action plan created by the delivery action planning unit 53 .

[0129] The clamping information acquisition unit 56 detects the timing at which the target person has started to receive the operation based on, for example, the sliding amount or the initial sliding amount detected by the sliding sensor 503 or the component of the sliding amount detected by the distance measuring sensor 504 in the direction opposite to gravity, the direction opposite to the torque caused by gravity, or the position direction of the hand H1 of the target person, so as to determine the start timing of the release operation performed by the physical interaction execution unit 55, and inputs the detected timing to the physical interaction execution unit 55.

[0130] In addition, during the period when the physical interaction execution unit 55 is performing a release operation, for example, the clamping information acquisition unit 56 continuously inputs the sliding amount of the object B1 in the transfer direction A1 detected by the sliding sensor 503 and / or the distance measurement sensor 504 to the physical interaction execution unit 55 to ensure the continuity of the moving speed of the object B1 in the transfer direction A1 during the period when the physical interaction execution unit 55 is performing a release operation.

[0131] In addition, during the period when the physical interaction execution unit 55 performs the release operation, for example, the gripping information acquisition unit 56 continuously inputs the sliding amount of the object B1 in the gravity direction detected by the sliding sensor 503 and / or the distance measurement sensor 504 to the physical interaction execution unit 55, so as to reduce the falling or unnatural vertical movement of the object B1 during the period when the physical interaction execution unit 55 performs the release operation.

[0132] In addition, the gripping information acquisition unit 56 inputs information such as the gripping force detected by the 3-axis force sensor 502, the sliding amount and / or the initial sliding amount detected by the sliding sensor 503, etc., detected by various sensors provided on the hand 443 during the period of performing the release operation by the physical interaction execution unit 55, to the machine learning unit (not shown) as an input in the machine learning of the transfer operation. In addition, the gripping information acquisition unit 56 inputs, for example, the distance to the object B1 detected by the distance measurement sensor 504 during the period of performing the release operation by the physical interaction execution unit 55 to the machine learning unit (not shown) as an output in the machine learning of the transfer operation.

[0133] The response time measurement unit 57 measures, for example, the time (response time) from when the autonomous robot 1 starts a release operation to reduce the gripping force of the hand 443 until the moving speed of the object B1 in the transfer direction A1 increases, and inputs the measured response time to the physical interaction execution unit 55.

[0134] The emotion map generation unit 58 generates an emotion map in which emotion changes of the delivery target person along the time axis during the execution of the release operation are mapped, based on information on emotion changes of the delivery target person input from the delivery target person identification unit 51 during the period of the physical interaction execution unit 55 executing the release operation, for example, and inputs the generated emotion map to the delivery action evaluation unit 59. Note that the mapped emotions may include not only negative emotions such as enthusiasm, indifference, timing mismatch, etc., but also positive emotions such as emotions indicating that the delivery target person can comfortably receive the object.

[0135] The delivery action evaluation unit 59 evaluates the delivery action plan planned by the delivery action planning unit 53 based on the emotion map input from the emotion map generation unit 58 , and inputs the evaluation result together with the delivery action plan into the learning information storage unit 54 .

[0136] 1.6 Machine Learning on Transfer Actions

[0137] Here, machine learning of transfer actions will be described. Figure 8 is a schematic diagram showing an example of input / output information in a machine learning process. Figure 8 As shown, in the machine learning process of the transfer action in the present embodiment, information detected by various sensors provided on the hand 443 or information obtained from such information, for example, information such as the clamping force or the rate of change of the clamping force detected by the 3-axis force sensor 502, the sliding amount and / or the initial sliding amount or the rate of change of the sliding amount and / or the initial sliding amount detected by the sliding sensor 503, etc., is given to the input layer, information about the continuity of the moving speed of the object B1 in the transfer direction A1 is given to the output layer, and the weight of each edge 61 connecting the nodes (also referred to as "neurons") 60 of each layer from the input layer via the hidden layer to the output layer is obtained. Thus, a learned model that is optimal for the transfer operation is created. Note that, as described above, the emotions of the transfer target person can also be used as the input or output of the machine learning.

[0138] Note that in machine learning, for example, as the continuity of the moving speed of the object B1 in the transfer direction A1 becomes higher, a positive reward is set. On the other hand, as the continuity of the moving speed of the object B1 in the transfer direction A1 becomes lower, that is, as the moving amount per unit time of the object B1 in the transfer direction A1 becomes more discontinuous, a negative reward is set.

[0139] Furthermore, in machine learning, a reward may be set based on the emotion and / or emotion change of the transfer target person during the period when the physical interaction execution unit 55 performs the release operation. For example, in the case where the physical interaction execution unit 55 recognizes a negative emotion from the transfer target person during the period when the release operation is performed, a negative reward may be set, and in the case where the physical interaction execution unit 55 recognizes a positive emotion from the transfer target person during the period when the release operation is performed, a positive reward may be set. Whether the emotion of the transfer target person is a negative emotion or a positive emotion may be determined based on, for example, an emotion map generated by the emotion map generation unit 58.

[0140] 1.7 Operation Examples

[0141] Next, an operation example of the autonomous robot 1 according to the present embodiment will be described in detail with reference to the drawings. Fig. 9 1 is a flowchart showing an exemplary operation example according to the present embodiment. Fig. 9 As shown, in this operation, first, the object recognition unit 52 recognizes the position, shape, size, etc. of the object B1, which is a target to be transferred, by analyzing the image data acquired by the camera 19 (step S101). Subsequently, the physical interaction execution unit 55 grips the object B1 based on the position, shape, size, etc. of the object B1 recognized by the object recognition unit 52 (step S102).

[0142] Next, when the physical interaction execution unit 55 lifts the object B1, the object identification unit 52 calculates the load caused by lifting the object B1 based on the sensor data detected by the various sensors of the manipulator 44 (6-axis force sensor 501, 3-axis force sensor 502, sliding sensor 503, etc.) (step S103), and identifies or estimates the characteristics of the object B1 based on the calculated load, such as the static friction coefficient, dynamic friction coefficient, mass, stiffness, strength, temperature, humidity, etc. (step S104).

[0143] Next, the delivery target person recognition unit 51 recognizes the position of the delivery target person by analyzing, for example, image data acquired by the camera 19 or voice data input from the microphone 23 (step S105). Subsequently, the delivery action planning unit 53 creates a movement plan from the position of the delivery target person recognized by the delivery target person recognition unit 51 to the delivery place of the determined object B1 (step S106). Then, the physical interaction execution unit 55 moves the autonomous robot 1 to the delivery place according to the movement plan created by the delivery action planning unit 53 (step S107).

[0144] Next, the transfer target person identification unit 51 identifies the position of the hand H1 of the transfer target person by analyzing, for example, image data acquired by the camera device 19 (step S108). Subsequently, the transfer action planning unit 53 creates a posture control plan for the transfer posture of the autonomous robot 1 from the position of the hand H1 of the transfer target person identified by the transfer target person identification unit 51 to the determined autonomous robot 1 (step S109). Then, the physical interaction execution unit 55 controls the posture of the autonomous robot 1 according to the posture control plan created by the transfer action planning unit 53 (step S110). Note that the posture control of the autonomous robot 1 may include, for example, style control of the manipulator 44, tilt control of the body part 42 and the head 41, and the like. Furthermore, in step S108, in addition to the position of the hand H1 of the transfer target person, the behavior, condition, and the like of the hand H1 of the transfer target person may also be identified.

[0145] Next, the transfer action planning unit 53 creates a transfer operation plan for transferring the object B1 to the transfer target person based on the learned model stored in the learning information storage unit 54, the transfer action plan planned in the past and its evaluation results, etc., according to the characteristics of the object B1 identified or estimated by the object recognition unit 52 and the position, behavior, condition, etc. of the hand H1 of the transfer target person identified by the transfer target person recognition unit 51 (step S111). Subsequently, the physical interaction execution unit 55 executes the transfer action of transferring the object B1 to the transfer target person according to the transfer operation plan created by the transfer action planning unit 53 (step S112). Thereafter, this operation ends.

[0146] 1.8 Specific examples of transfer operations

[0147] Next, we will use some examples to describe Fig. 9 The transfer operation is shown in step S112.

[0148] 1.8.1 First Example

[0149] First, the transfer operation according to the first example will be described. Fig.10 : is a flowchart showing the transfer operation according to the first example of the present embodiment. Note that in the first example, the case where the physical interaction execution unit 55 performs the release operation using only the gripping force control is illustrated.

[0150] like Fig.10As shown, in the release operation according to the first example, first, the clamping information acquisition unit 56 determines whether a component in the direction opposite to gravity, the direction opposite to the torque due to gravity, or the position direction of the hand H1 of the transfer target person (hereinafter, these directions are referred to as "specific directions"), for example, of the sliding amount U or the initial sliding amount u detected by the sliding sensor 503 or the sliding amount U detected by the distance measuring sensor 504 has become greater than zero (step S121). That is, it is determined whether the receiving operation by the transfer target person has started. In the case where the sliding amount U or the initial sliding amount u in the specific direction is zero or less ("No" in step S121), that is, in the case where the receiving operation by the transfer target person has not yet started, the present operation returns to step S121, and waits for the start of the receiving operation by the transfer target person.

[0151] On the other hand, in the case where the sliding amount U or the initial sliding amount u in the specific direction has become greater than zero ("Yes" in step S121), that is, in the case where the receiving operation by the transfer target person has started, the gripping information acquisition unit 56 specifies the transfer direction A1 by analyzing, for example, image data acquired by the camera device 504b in the distance measurement sensor 504 (step S122). Based on the fact that the sliding amount U or the initial sliding amount u in the specific direction has become greater than zero, the specified transfer direction A1 is input from the gripping information acquisition unit 56 to the physical interaction execution unit 55 together with the start trigger of the release operation. In response to this, the physical interaction execution unit 55 starts the release operation of the object B1 in the transfer direction A1 (step S123).

[0152] When the release operation is started in this way, the physical interaction execution unit 55 then performs the gripping force control of the hand 443 (step S124). Specifically, the physical interaction execution unit 55 controls the force change amount per unit time when the gripping force F generated in the hand 443 to grip the object B1 is reduced to release.

[0153] In addition, after the physical interaction execution unit 55 starts the release operation, the clamping information acquisition unit 56 measures the moving speed of the object B1 in the transfer direction A1 by continuously measuring the change in the sliding amount of the object B1 in the transfer direction A1 (step S125). The measured moving speed of the object B1 in the transfer direction A1 is input to the physical interaction execution unit 55 and is used for the clamping force control in the physical interaction execution unit 55. That is, the physical interaction execution unit 55 controls the amount of reduction per unit time of the clamping force generated in the hand 443 so that the moving speed of the object B1 in the transfer direction A1 maintains continuity (feedback control).

[0154] Thereafter, the physical interaction execution unit 55 determines whether the clamping force F generated in the hand 443 has reached zero (step S126), completes the release operation (step S127) when the clamping force F has reached zero ("yes" in step S126), and ends the transfer operation.

[0155] On the other hand, when the clamping force F has not reached zero (No in step S126), the operation returns to step S124, and the subsequent operations are repeatedly performed until the clamping force F reaches zero.

[0156] In this way, by adopting a configuration in which the physical interaction execution unit 55 starts the release operation (step S123) based on the fact that the sliding amount U or the initial sliding amount u detected by the sliding sensor 503 or the component in a specific direction in the sliding amount U detected by the distance measurement sensor 504 has become greater than zero ("Yes" in step S121), it becomes possible to start the release operation at the initial stage of the transfer operation. Therefore, it becomes possible to suppress the fluctuation of the load applied to the object B1 or the gripping force of the hand 443 to the minimum, and thus it becomes possible to transfer the object B1 more smoothly.

[0157] Furthermore, by continuously measuring the change in the sliding amount of the object B1 in the transfer direction A1 after the physical interaction execution unit 55 starts the release operation (step S125), and controlling the amount of reduction per unit time of the gripping force F generated in the hand 443 based on the measured change in the sliding amount of the object B in the transfer direction A1, it is possible to reduce a sudden change in the displacement of the object B1 in the transfer direction A1. Therefore, it becomes possible to transfer the object B1 more smoothly, and it becomes possible to reduce an erroneous drop of the object B1 or an unnatural vertical displacement of the object B1 in the hand 443.

[0158] Note that after the physical interaction execution unit 55 starts the release operation in step S123, the physical interaction execution unit 55 can stop or end the release operation, for example based on the response time measured by the response time measurement unit 57, if the moving speed of the object B1 in the transfer direction A1 does not increase for a certain time or longer.

[0159] Furthermore, in the present embodiment, the case where the hand 443 has two fingers has been exemplified, but for example, in the case where the hand 443 has three or more fingers, in the gripping force control, the gripping force may be gradually reduced from the finger portions other than the two finger portions between which the object B1 is gripped. Thus, the risk that the object B1 may fall is reduced, making it possible to perform the transfer operation more stably.

[0160] 1.8.2 Second Example

[0161] Next, the transfer operation according to the second example will be described. Fig.11 : is a flowchart showing a transfer operation according to a second example of the present embodiment. In the second example, a case is illustrated where the physical interaction execution unit 55 performs a release operation by mixing the gripping force control and the arm operation control with each other at a mixing ratio specified based on a learned model.

[0162] like Fig.11 As shown, in the transfer operation according to the second example, for example, in the case of Fig. 9 In a process similar to the process of the transfer operation described in the first example, Fig. 9 Step S124 is replaced by step S224.

[0163] In step S224, the physical interaction execution unit 55 performs the clamping force control of the hand 443 and the arm operation control of the arm, which are mixed with each other at a mixing ratio specified based on the learned model. Specifically, the physical interaction execution unit 55 controls (clamping force control) the change in force per unit time when the clamping force F generated in the hand 443 for clamping the object B1 is reduced to release, and controls the position change per unit time when the position of the object B1 moves in the transfer direction A1 by changing the posture of the arm that clamps the object B1 and arranges the object B1 in the target coordinate space. At this time, the reduction in the clamping force F per unit time and the change in the posture of the arm are mixed with each other at the mixing ratio mentioned above, so that the movement speed of the object B1 in the transfer direction A1 maintains continuity.

[0164] 1.8.3 Third Example

[0165] Next, the transfer operation according to the third example will be described. Fig.12 3 is a flowchart showing a transfer operation according to a third example of the present embodiment. In the third example, a case is illustrated where the physical interaction execution unit 55 performs a release operation by mixing the gripping force control, the arm operation control, and the whole body operation control with each other at a mixing ratio specified based on a learned model.

[0166] like Fig.12 As shown, in the transfer operation according to the third example, for example, in the case of Fig. 9 In a process similar to the process of the transfer operation described in the first example, Fig. 9 Step S124 is replaced by step S324.

[0167] In step S324, the physical interaction execution unit 55 performs the gripping force control of the hand 443, the arm operation control of the arm, and the whole body operation control of the autonomous robot 1, which are mixed with each other at a mixing ratio specified based on the learned model. Specifically, the physical interaction execution unit 55 controls (gripping force control) the amount of change in force per unit time when the gripping force F generated in the hand 443 for gripping the object B1 is reduced to release, controls the amount of position change per unit time when the position of the object B1 moves in the transfer direction A1 by changing the posture of the arm that grips the object B1 and arranges the object B1 in the target coordinate space, and moves the position of the object B1 in the transfer direction A1 by changing the position and posture of the autonomous robot 1 that grips the object B1 and arranges the object B1 in the target coordinate space. At this time, the amount of reduction in the gripping force F per unit time, the amount of change in the posture of the arm, and the amount of change in the position and posture of the autonomous robot 1 are mixed with each other at the above-mentioned mixing ratio, so that the moving speed of the object B1 in the transfer direction A1 maintains continuity.

[0168] 1.8.4 Fourth Example

[0169] Next, the transfer operation according to the fourth example will be described. Fig.13 is a flowchart showing a transfer operation according to a fourth example of the present embodiment. In the fourth example, for example, in addition to the transfer operation shown in the third example, a case where the physical interaction execution unit 55 stops or ends the release operation according to the change in the emotion of the transfer target person is illustrated. Note that in this specification, the third example is used as a basis, but the fourth example is not limited thereto, and the first example or the second example may also be used as a basis.

[0170] like Fig.13 As shown, in the transfer operation according to the fourth example, for example, in the case of Fig.12 In a flow similar to the flow of the transfer operation according to the third example described, steps S401 to S403 are added between step S324 and step S125.

[0171] In step S401, the transfer target person recognition unit 51 detects emotional changes of the transfer target person by analyzing, for example, image data acquired by the camera 19 or voice data input from the microphone 23. Information on the detected emotional changes of the transfer target person can be input to the emotion map generation unit 58 as input or output of machine learning, for example.

[0172] Next, in step S402, the transfer target person identification unit 51 determines whether the detected changed emotion is a specific emotion. The specific emotion may be, for example, a negative emotion, which leads to the interruption or abandonment of the transfer operation by the transfer target person. In the event that the specific emotion is not detected ("No" in step S402), the operation proceeds to step S125. On the other hand, in the event that the specific emotion is detected ("Yes" in step S402), the physical interaction execution unit 55 stops or ends the release operation (step S403), and then returns to step S121. Note that the instruction to stop the release operation can be directly input from the transfer target person identification unit 51 to the physical interaction execution unit 55, or can be input from the transfer target person identification unit 51 to the physical interaction execution unit 55 via the emotion map generation unit 58.

[0173] As described above, after the physical interaction execution unit 55 starts the release operation in step S123, the delivery target person recognition unit 51 can detect the change in the emotion of the delivery target person by analyzing the image data from the camera 19 or the voice data from the microphone 23, and can be configured so that in the case where the detected emotion is a specific emotion that causes the interruption or abandonment of the delivery operation, the release operation performed by the physical interaction execution unit 55 is interrupted. Therefore, when the delivery target person interrupts or abandons the receiving operation, it becomes possible to quickly stop or end the release operation, and thus it becomes possible to reduce the falling of the object B1, the jumping out of the content, etc.

[0174] 1.9 Mixing of Clamping and Release Operations

[0175] Next, the mixing between the clamping operation and the releasing operation will be described in detail with reference to the drawings. Fig.14 is a diagram for describing the mixing between the clamping operation and the releasing operation according to the present embodiment. Fig.14 , there is illustrated a case where the physical interaction execution unit 55 executes a release operation by mixing the gripping force control, the arm operation control, and the whole body operation control with each other at a mixing ratio specified based on a learned model (third example).

[0176] like Fig.14 As shown, when the autonomous robot 1 delivers the gripped object to the delivery target person, the gripping operation 71 and the release operation 72 are mixed (added) with each other at a mixing ratio 76, and the gripping force control 74 is performed based on the mixing result (73). In the process of the autonomous robot 1 releasing the object B1, the mixing ratio 76 is reduced, so that the autonomous robot 1 finally releases the object B1.

[0177] The mixing ratio 76 can be obtained based on the result of calculating (75) the load based on sensor data (clamping force F, rotational torque M, etc.) detected by various sensors of the manipulator 44 (6-axis force sensor 501, 3-axis force sensor 502, sliding sensor 503, etc.), for example, after the physical interaction execution unit 55 performs a release operation.

[0178] The mixing ratio obtained in this way is also used as a parameter in the arm operation control 77. The control amount of the arm operation control 77 is determined based on the sensor data (gripping force F, rotational torque M, etc.) detected by the various sensors (6-axis force sensor 501, 3-axis force sensor 502, slide sensor 503, etc.) of the manipulator 44 after the physical interaction execution unit 55 performs the release operation and the mixing ratio 76, and then the control amount of the whole body operation control 78 is determined to support the arm operation control 77.

[0179] 1.9.1 Modification Example

[0180] Fig.15 2 is a diagram for describing a mixture between a clamping operation and a releasing operation according to a modification example of the present embodiment. Fig.15 As shown, in addition to the sensor data (gripping force F, rotational torque M, etc.), the mixing ratio 76 can be obtained based on the response time T measured by the response time measurement unit 57 or the emotional change of the delivery target person detected by the delivery target person recognition unit 51. Therefore, it becomes possible to adjust the mixing ratio 76 to release the object B1 more politely or more quickly, for example, in a case where the delivery target person feels uncomfortable.

[0181] 1.10 Measurement of initial slip

[0182] Next, the measurement of the initial slip will be described using a specific example.

[0183] 1.10.1 Vision Sensor

[0184] First, a case where a visual sensor is used as a sensor for measuring an initial slide (corresponding to the slide sensor 503 ) will be described. Fig.16 is a schematic diagram showing an example of a slide sensor configured using a vision sensor.

[0185] like Fig.16 As shown, the slide sensor 503A configured using the vision sensor 83 includes a deformation portion 82 provided in a part of a housing 81 of the autonomous robot 1 and the vision sensor 83 observing the deformation of the deformation portion 82 from the inside of the housing 81 .

[0186] like Fig.17As shown, the deformation part 82 is formed of a viscoelastic body 821 such as silicone rubber. The viscoelastic body 821 is provided with a plurality of marks 822 arranged in a two-dimensional grid pattern, for example. Fig.17 When pressure is applied to the region R1 shown, the mark 822 in the region R1 causes deformation or positional deviation. Therefore, the slide sensor 503A can specify which region the pressure is applied to, that is, which region is in contact with the object B1, by observing the region of the mark 822 causing deformation or positional deviation using the visual sensor 83.

[0187] Furthermore, for example, when external force is applied to the object B1 in contact with the deformation portion 82, initial sliding occurs as a stage before the object B1 actually starts to slide relative to the deformation portion 82. The initial sliding is a phenomenon in which sliding occurs in the peripheral portion while sliding does not occur in the central portion of the contact area.

[0188] For example, Figures 18 to 20 As shown, when an external force in the right direction in the figure is applied to the object B1 in contact with the entire deformation portion 82, an initial sliding 823 occurs from the left side of the deformation portion 82 (see Fig.18 ), the sliding area gradually expands (see Figures 18 to 20 ), and finally sliding is detected in the entire deformation portion 82.

[0189] Therefore, when using the sliding sensor 503A configured using the visual sensor 83, it becomes possible to set the state of the hand 443 clamping and lifting the object B1 as a reference, detect the initial slide in the direction opposite to gravity, the direction opposite to the torque caused by gravity, or the direction of the position of the hand H1 of the transfer target person based on the state through the sliding sensor 503A, and set the timing of detecting the initial slide as the timing for the transfer target person to start receiving the operation.

[0190] 1.10.2 Pressure distribution sensor

[0191] Furthermore, the pressure distribution sensor may also be used as a sensor for measuring the initial slip (corresponding to the slip sensor 503). Fig.21 is a schematic diagram showing an example of a slide sensor configured using a pressure distribution sensor.

[0192] like Fig.21 As shown in FIG. 1 , in the slide sensor 503B configured using the pressure distribution sensor 91, the surface in contact with the object B1 is deformed. Fig. 22 As shown in FIG. 1 , when the object B1 is brought into contact with the slide sensor 503B, the pressure detected in the central portion of the contact region R2 becomes the highest. Fig. 22 and Fig.23, it is assumed that the density of the color in the contact area R2 indicates the height of the pressure.

[0193] When an external force in the right direction in the figure is applied in this state, the area with the highest pressure in the contact area R2 does not move, but its surrounding area is offset in the right direction in the figure, as shown in FIG. Fig.23 As shown.

[0194] Therefore, when using the sliding sensor 503B configured using the pressure distribution sensor 91, it becomes possible to set the state of the hand 443 clamping and lifting the object B1 as a reference, detect the initial slide in the direction opposite to gravity, the direction opposite to the torque caused by gravity, or the direction of the position of the hand H1 of the transfer target person based on the state through the sliding sensor 503B, and set the timing of detecting the initial slide as the timing for the transfer target person to start receiving the operation.

[0195] 1.11 System Configuration

[0196] like Fig.24 As shown, for example, the autonomous robot 1 according to the present embodiment can be connected to the server 2 via a predetermined network 3. For example, the server 2 can be a server group including a plurality of servers, such as a cloud server, etc. As the network 3, for example, various networks such as the Internet, a LAN, a mobile communication network, etc. can be adopted.

[0197] In such a system configuration, the above-mentioned machine learning unit may be arranged in each autonomous robot 1 or may be arranged in the server 2. Note that it is preferred that the result of machine learning performed by the machine learning unit (e.g., a learned model) can be shared by a plurality of autonomous robots 1 connected to each other via a predetermined network 3, regardless of whether the machine learning unit is arranged in each autonomous robot 1 or in the server 2.

[0198] As a device for implementing the machine learning unit, a general-purpose computer or processor (CPU) may be used, or a general-purpose computing graphics processing unit (GPGPU), a large personal computer (PC), a field programmable gate array (FPGA), etc. may be used.

[0199] 1.12 Effects

[0200] As described above, according to the present embodiment, by continuously measuring the change in the sliding amount of the object B1 in the transfer direction A1 during the release operation, the release operation is controlled so as to maintain the continuity of the moving speed of the object B1 in the transfer direction A1. Therefore, the sudden change in the displacement of the object B1 in the transfer direction A1 can be reduced, and thus it becomes possible to smoothly transfer the object B1.

[0201] Furthermore, in the present embodiment, machine learning is performed using the rate of change of the gripping force of the hand 443, information detected by various sensors mounted on the hand 443, etc. as inputs, and the continuity of the moving speed of the object B1 in the transfer direction A1 as output. Therefore, a smoother transfer of the object B1 can be achieved based on the characteristics of the object B1 (static friction coefficient, dynamic friction coefficient, mass, shape and size, rigidity, strength, temperature, humidity, etc.) and the behavior, condition, etc. of the transfer target person.

[0202] Furthermore, in the present embodiment, for example, the change in the sliding amount or the initial sliding amount of the object B1 clamped by the hand 443 is measured, and when the direction of the change has a component in the direction opposite to the gravity, the direction opposite to the torque due to the gravity, or the position direction of the hand H1 of the transfer target person, the release operation is started. Therefore, the release operation can be started at the initial stage of the transfer operation, and thus the fluctuation of the load applied to the object B1 or the clamping force of the hand 443 can be suppressed to a minimum, so that it becomes possible to transfer the object B1 more smoothly.

[0203] Furthermore, in the present embodiment, the sliding amount or initial sliding amount of the object B1 in the gravity direction is continuously measured during the release operation. Therefore, it becomes possible to reduce erroneous dropping of the object B1 or unnatural vertical displacement of the object B1 in the hand 443.

[0204] In addition, in the present embodiment, the presence, emotional changes, etc. of the delivery target person obtained from the image input information, voice input information, etc. are added to the input and / or output of machine learning. Therefore, further improvement in the quality of physical interaction can be achieved.

[0205] In addition, in the present embodiment, the release operation can be started after confirming the presence of the transfer target person or the presence of the hand H1 using the result of image processing or the information from the distance measuring sensor. Therefore, it is possible to avoid starting the release operation by mistake, and thus it becomes possible to perform the transfer of the object B1 more safely and reliably. Note that the presence of the transfer target person can be confirmed using, for example, voice input / output information, etc., as well as the result of image processing or the information from the distance measuring sensor. In addition, after recognizing the intention of the transfer target person to clamp the object or the emotional change of the transfer target person during the release operation using voice input / output information, etc., it becomes possible to perform a determination on the start of the release operation or the continuation of the operation.

[0206] Note that the above embodiment has been exemplified in which the object B1 is gripped by one manipulator 44 and is delivered to the delivery target person, but the present disclosure is not limited thereto. For example, the above embodiment can be applied to a case where both manipulators 44R and 44L are used to grip the object B1 and deliver the object B1 to the delivery target person, etc.

[0207] Furthermore, the above-described embodiment can also be applied to a case where a plurality of autonomous robots 1 and one autonomous robot 1 lift the object B1 in cooperation with each other and deliver the object B1 to one or more delivery target persons.

[0208] Furthermore, the above-described embodiment can also be applied to a case where the object B1 is received from a target person.

[0209] The embodiments of the present disclosure have been described above, but the technical scope of the present disclosure is not limited to each of the above embodiments, and various modifications can be made without departing from the gist of the present disclosure. In addition, components across different embodiments and modifications can be appropriately combined with each other.

[0210] Furthermore, the effects in each embodiment described in this specification are merely examples and are not limited, and other effects may be provided.

[0211] Note that the present technology may also have the following configurations. (1)

[0213] An information processing device includes a control unit that controls a manipulator so that when an object gripped by the manipulator is transferred to a transfer target person, the moving speed of the object maintains continuity. (2)

[0215] The information processing device according to (1), wherein the control unit controls the amount of change in the gripping force of the manipulator per unit time so that the moving speed of the object in the transfer direction maintains continuity. (3)

[0217] The information processing device according to (1),

[0218] wherein the manipulator comprises an arm and a hand for holding the object, the hand being attached to one end of the arm, and

[0219] The control unit controls the amount of change in the gripping force of the hand per unit time and controls the amount of change in the posture of the arm per unit time so that the moving speed of the object in the transfer direction maintains continuity. (4)

[0221] The information processing device according to any one of (1) to (3),

[0222] wherein the manipulator is attached to a movable moving body, and

[0223] The control unit controls the manipulator and controls the movement of the moving body so that the movement speed of the object maintains continuity. (5)

[0225] The information processing device according to any one of (1) to (4), further comprising:

[0226] a first detection unit that detects a sliding amount of the object at a portion where the object is in contact with the manipulator,

[0227] The control unit controls the manipulator based on the sliding amount detected by the first detection unit so that the moving speed of the object maintains continuity. (6)

[0229] The information processing device according to any one of (1) to (5), further comprising:

[0230] a second detection unit that detects a sliding amount or an initial sliding amount of the object at a portion where the object is in contact with the manipulator,

[0231] Here, when the change in the sliding amount or the initial sliding amount detected by the second detection unit includes a component in a direction different from the gravity direction, the control unit causes the manipulator to start an operation of transferring the object to the transfer target person. (7)

[0233] The information processing device according to (6), wherein the direction different from the gravity direction is a direction opposite to gravity, a direction opposite to torque due to gravity, or a delivery direction to the delivery target person. (8)

[0235] The information processing device according to any one of (1) to (7), further comprising:

[0236] a third detection unit that detects a sliding amount or an initial sliding amount of the object in a gravity direction at a portion where the object is in contact with the manipulator,

[0237] The control unit controls the manipulator based on the sliding amount or the initial sliding amount in the gravity direction detected by the third detection unit. (9)

[0239] The information processing device according to any one of (1) to (8), further comprising:

[0240] a target person recognition unit that recognizes an emotion of the transfer target person during an operation of transferring the object to the transfer target person,

[0241] The control unit stops or ends the operation of transferring the object to the transfer target person based on the change of the emotion of the transfer target person detected by the target person recognition unit. (10)

[0243] An information processing device according to (9), wherein the target person recognition unit recognizes the emotion of the delivery target person based on at least one of image data obtained by capturing an image of the delivery target person and voice data obtained by collecting voice uttered by the delivery target person. (11)

[0245] The information processing device according to any one of (1) to (10), further comprising:

[0246] a planning unit that plans a transfer operation to be performed by the manipulator to transfer the object to the transfer target person,

[0247] The control unit controls the manipulator according to the transfer operation planned by the planning unit so that the moving speed of the object maintains continuity. (12)

[0249] The information processing device according to (11), further comprising:

[0250] an object recognition unit that recognizes or estimates characteristics of said object,

[0251] wherein the planning unit plans the transfer operation based on characteristics of the object identified or estimated by the object identification unit. (13)

[0253] The information processing device according to (12), wherein the characteristics of the object include at least one of a static friction coefficient, a dynamic friction coefficient, a mass, a shape and size, a rigidity, a strength, a temperature, and a humidity. (14)

[0255] The information processing device according to any one of (11) to (13), wherein the planning unit plans the delivery operation according to a learned model. (15)

[0257] An information processing device according to (14), wherein the learned model is a model created by machine learning using the rate of change of the clamping force of the manipulator clamping the object as input and the continuity of the moving speed of the object in the transfer direction as output. (16)

[0259] An information processing device according to (15), wherein the machine learning is machine learning in which a positive reward is set as the continuity of the moving speed of the object in the transfer direction becomes higher, and a negative reward is set as the continuity of the moving speed of the object in the transfer direction becomes lower. (17)

[0261] The information processing device according to (5), wherein the first detection unit includes at least one of a viscoelastic body, a visual sensor, a pressure distribution sensor, and a distance measurement sensor. (18)

[0263] The information processing device according to (6) or (7), wherein the second detection unit includes a viscoelastic body and a visual sensor or a pressure distribution sensor. (19)

[0265] A control method controls a manipulator so that when an object gripped by the manipulator is transferred to a transfer target person, the moving speed of the object maintains continuity. (20)

[0267] A program for causing a computer to operate a manipulator,

[0268] The program causes the computer to perform the following operations: controlling the manipulator so that when an object held by the manipulator is delivered to a delivery target person, the moving speed of the object maintains continuity.

[0269] Reference numerals list

[0270] 1 Autonomous Robots

[0271] 2 Server

[0272] 3 Network

[0273] 10 Control unit

[0274] 11 Signal Processing Circuit

[0275] 12 CPU

[0276] 13 DRAM

[0277] 14 Flash ROM

[0278] 15 PC card I / F

[0279] 16 Wireless communication unit

[0280] 17 Internal bus

[0281] 18 Batteries

[0282] 19 Camera

[0283] 20 IMU

[0284] 21 ToF sensor

[0285] 22 Touch Sensor

[0286] 23 Microphone

[0287] 24 Speakers

[0288] 25 Display unit

[0289] 26 removable units

[0290] 261 3-axis sensor (shoulder)

[0291] 262 1-axis sensor (elbow)

[0292] 263 3-axis sensor (wrist)

[0293] 264 1-axis sensor (knuckle)

[0294] 265 2-axis sensor (waist)

[0295] 266 3-axis sensor (neck)

[0296] 27 Actuator

[0297] 28 Encoder (potentiometer)

[0298] 30 Memory Card

[0299] 41 Head

[0300] 42 Body

[0301] 43 Bracket

[0302] 44, 44L, 44R manipulator

[0303] 441 Upper arm

[0304] 442 Forearm

[0305] 443 Hands

[0306] 444 Joint mechanism

[0307] 4431 Base

[0308] 4432, 4432a, 4432b Finger

[0309] 4433, 4435 joint mechanism

[0310] 4434 Basic joints

[0311] 4436 Distal joint

[0312] 50 Sensor Group

[0313] 501 6-axis force sensor

[0314] 502 3-axis force sensor

[0315] 503, 503A, 503B Slide Sensor

[0316] 504 Distance Measurement Sensor

[0317] 504a ToF sensor

[0318] 504b Camera device

[0319] 51 Transfer target person identification unit

[0320] 52 Object Recognition Unit

[0321] 53 Transfer action plan unit

[0322] 54 Learning Information Storage Unit

[0323] 55 Physical Interaction Execution Unit

[0324] 56 Clamping information acquisition unit

[0325] 57 Response time measurement unit

[0326] 58 Emotion Map Generation Unit

[0327] 59 Transfer Action Evaluation Unit

[0328] 60 nodes (neurons)

[0329] 61 Edge

[0330] 71 Clamping operation

[0331] 72 Release Operation

[0332] 73 α×clamping operation+(1-α)×releasing operation

[0333] 74 Clamping force control

[0334] 75 Load Calculation

[0335] 76 Mixing ratio

[0336] 77 Arm Operation Control

[0337] 78 Whole body operation control

[0338] 81 Housing

[0339] 82 Deformation

[0340] 83 Vision Sensor

[0341] 821 Viscoelastic

[0342] 822 Mark

[0343] 823 Initial Slide

[0344] 91 Pressure distribution sensor

[0345] A1 Transfer direction

[0346] B1 object

[0347] H1 Lot

[0348] P1 Floor

[0349] R1 Region

[0350] R2 contact area

Claims

1. An information processing device, comprising: a control unit that controls the manipulator so that when an object held by the manipulator is transferred to a transfer target person, a moving speed of the object maintains continuity; as well as a target person recognition unit that recognizes an emotion of the transfer target person during an operation of transferring the object to the transfer target person, wherein the manipulator is attached to a movable body, The control unit controls the manipulator and controls the movement of the moving body so that the moving speed of the object maintains continuity, and The control unit stops or ends the operation of delivering the object to the delivery target person based on a change in the emotion of the delivery target person detected by the target person recognition unit.

2. The information processing device according to claim 1, wherein: The control unit controls the amount of change in the gripping force of the manipulator per unit time so that the moving speed of the object in the transfer direction maintains continuity.

3. The information processing device according to claim 1, in, The manipulator includes an arm and a hand for holding the object, the hand being attached to one end of the arm, and The control unit controls the amount of change in the gripping force of the hand per unit time and controls the amount of change in the posture of the arm per unit time so that the moving speed of the object in the transfer direction maintains continuity.

4. The information processing device according to claim 1, further comprising: a first detection unit that detects a sliding amount of the object at a portion where the object is in contact with the manipulator, The control unit controls the manipulator based on the sliding amount detected by the first detection unit so that the moving speed of the object maintains continuity.

5. The information processing device according to claim 1, further comprising: a second detection unit that detects a sliding amount or an initial sliding amount of the object at a portion where the object is in contact with the manipulator, Here, when the change in the sliding amount or the initial sliding amount detected by the second detection unit includes a component in a direction different from the gravity direction, the control unit causes the manipulator to start an operation of transferring the object to the transfer target person.

6. The information processing device according to claim 5, wherein: The direction different from the direction of gravity is a direction opposite to gravity, a direction opposite to the torque due to gravity, or a delivery direction to the delivery target person.

7. The information processing device according to claim 1, further comprising: a third detection unit that detects a sliding amount or an initial sliding amount of the object in a gravity direction at a portion where the object is in contact with the manipulator, The control unit controls the manipulator based on the sliding amount or the initial sliding amount in the gravity direction detected by the third detection unit.

8. The information processing device according to claim 1, wherein: The target person recognition unit recognizes an emotion of the transfer target person based on at least one of image data obtained by capturing an image of the transfer target person and voice data obtained by collecting voice uttered by the transfer target person.

9. The information processing device according to claim 1, further comprising: a planning unit that plans a transfer operation to be performed by the manipulator to transfer the object to the transfer target person, The control unit controls the manipulator according to the transfer operation planned by the planning unit so that the moving speed of the object maintains continuity.

10. The information processing device according to claim 9, further comprising: an object recognition unit, which recognizes or estimates characteristics of the object, wherein the planning unit plans the transfer operation based on characteristics of the object identified or estimated by the object identification unit.

11. The information processing device according to claim 10, wherein: The property of the object includes at least one of a static friction coefficient, a dynamic friction coefficient, a mass, a shape and size, a rigidity, a strength, a temperature and a humidity.

12. The information processing device according to claim 9, wherein: The planning unit plans the delivery operation according to the learned model.

13. The information processing device according to claim 12, wherein: The learned model is a model created by machine learning using a rate of change of a gripping force of the manipulator gripping the object as an input and continuity of a moving speed of the object in a transfer direction as an output.

14. The information processing device according to claim 13, wherein: The machine learning is machine learning in which a positive reward is set as the continuity of the moving speed of the object in the transfer direction becomes higher, and a negative reward is set as the continuity of the moving speed of the object in the transfer direction becomes lower.

15. The information processing device according to claim 4, wherein: The first detection unit includes at least one of a viscoelastic body, a visual sensor, a pressure distribution sensor, and a distance measurement sensor.

16. The information processing device according to claim 5, wherein: The second detection unit includes a viscoelastic body and a visual sensor or a pressure distribution sensor.

17. A control method, the control method comprising: controlling the movement of a manipulator and a movable body to which the manipulator is attached so that when an object gripped by the manipulator is transferred to a transfer target person, the movement speed of the object maintains continuity; recognizing an emotion of the transfer target person during the operation of transferring the object to the transfer target person; as well as Based on the detected change in the emotion of the transfer target person, the operation of transferring the object to the transfer target person is stopped or ended.

18. A program product for causing a computer to operate a manipulator, The program product causes the computer to perform the following operations: controlling the movement of the manipulator and a movable body to which the manipulator is attached so that when an object gripped by the manipulator is delivered to a delivery target person, the moving speed of the object maintains continuity; recognizing an emotion of the transfer target person during the operation of transferring the object to the transfer target person; as well as Based on the detected change in the emotion of the transfer target person, the operation of transferring the object to the transfer target person is stopped or ended.

19. A removable medium storing a program for causing a computer controlling a manipulator to operate, The program causes the computer to perform the following operations: controlling the movement of the manipulator and a movable body to which the manipulator is attached so that when an object gripped by the manipulator is delivered to a delivery target person, the moving speed of the object maintains continuity; recognizing an emotion of the transfer target person during the operation of transferring the object to the transfer target person; as well as Based on the detected change in the emotion of the transfer target person, the operation of transferring the object to the transfer target person is stopped or ended.

Images