Robotic language controller

Abstract

Systems and methods of controlling a robot are described. In some embodiments, a method includes obtaining an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task. A control signal is provided to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot. Feedback information about the end effector resulting from the control signal is obtained, and output speech is generated by modulating the input speech based on the feedback information about the end effector.

Description

ROBOTIC LANGUAGE CONTROLLERGOVERNMENT FUNDING

[0001] This invention was made with government support under IIS2133072 awarded by the National Science Foundation. The government has certain rights in the invention.RELATED APPLICATIONS

[0002] This application claims priority under 35 U.S.C. § 119(e) to U.S. Provisional Patent Application No. 63 / 578,951, filed August 25, 2023, entitled “ROBOTIC LANGUAGE CONTROLLER,” which is incorporated herein by reference in its entirety.FIELD

[0003] Disclosed embodiments are related to a robotic language controller.BACKGROUND

[0004] Robots are deployed in a large variety of applications. Some uses of robots, such as in automated factories and in delivery tasks, have become fairly common. Other applications are in areas, such as in care settings, that require more interaction between robots and humans. These areas involve both verbal and physical interaction.SUMMARY

[0005] In some embodiments, a method of controlling a robot includes obtaining an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task. The method also includes providing a control signal to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot. By obtaining feedback information about the end effector resulting from the control signal, output speech is generated by modulating the input speech based on the feedback information about the end effector.

[0006] In some embodiments, a system to control a robot includes one or more controllers. The one or more controllers may obtain an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task, and provide a control signal to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot. The one or more controllers may also obtain feedback information about the endeffector resulting from the control signal, and generate output speech by modulating the input speech based on the feedback information about the end effector.

[0007] It should be appreciated that the foregoing concepts, and additional concepts discussed below, may be arranged in any suitable combination, as the present disclosure is not limited in this respect. Further, other advantages and novel features of the present disclosure will become apparent from the following detailed description of various non-limiting embodiments when considered in conjunction with the accompanying figures.BRIEF DESCRIPTION OF DRAWINGS

[0008] The accompanying drawings are not intended to be drawn to scale. In the drawings, each identical or nearly identical component that is illustrated in various figures may be represented by a like numeral. For purposes of clarity, not every component may be labeled in every drawing. In the drawings:

[0009] FIG. 1 is a block diagram of an exemplary system to synchronize robotic language and motion according to one embodiment;

[0010] FIG. 2 is a flow diagram of an exemplary embodiment of a method 200 of synchronizing robotic language and motion according to one embodiment;

[0011] FIG. 3 shows a series of images from an exemplary physical therapy session;

[0012] FIG. 4 is a block diagram depicting an exemplary general framework of a language controller according to one embodiment; and

[0013] FIG. 5 is an exemplary block diagram of a robotic control system according to one embodiment.DETAILED DESCRIPTION OF THE INVENTION

[0014] As noted above, robots may be used in applications that requires verbal and physical interaction with humans. For example, a robot may perform physical therapy and guide a person through exercises using speech, as well as physical touch. As another example, a robot caregiver may physically assist a person with a task while verbally communicating to ensure that the person understands what the robot is doing. A person performing similar tasks naturally synchronizes physical movements with speech. For example, when assisting a person with walking by helping to move one foot at a time, if the person slows, a human caregiver would naturally adjust tone and duration of speech so that they are not talking about assistingmovement of the left foot before the right foot is in the appropriate position to support the person’s weight. If the caregiver did not synchronize speech and movement, the person may become confused and uncomfortable.

[0015] In view of the above, the inventors have recognized and appreciated the benefits associated with synchronizing speech and movement of a robot intended to interact with a subject. A language controller of the robot that considers feedback resulting from movement of the robot interacting with the surrounding environment, such as interactions with the subject, may be a potential solution to the need to synchronize robotic speech and movement. In some embodiments, an admittance control scheme may be used to control the physical interaction of the robot and, more specifically, an end effector of the robot, with a subject (e.g., physical therapy patient). The movements of the end effector, or other portion of a robot, generally, may follow an input trajectory. For example, the specific movements involved in helping a person stand from a seated position may be provided as an input trajectory. In carrying out each of the movements, admittance control considers external contact forces (e.g., a resistance force from the subject opposing motion of one or more portions of the robot). By measuring interaction forces with the subject (e.g., resistance force of the subject), the set point of a reference following velocity controller may be changed.

[0016] In some embodiments, in addition to using feedback information resulting from imposing a force with the end effector on the subject to control subsequently applied forces and / or motions of one or more portions of a robot, some of the feedback information may also be used for language control. Input speech may be modulated according to the feedback information to generate output speech. Specifically, the actual speed of the end effector may be obtained as feedback and compared with an initially commanded speed according to the input trajectory. The difference may be used to adjust a time scale (i.e., duration) and / or other appropriate parameter of the output speech. For example, input speech may be slowed or sped up to match slower or faster motions of the robot. According to some embodiments, when the time scale indicates that output speech should be longer than input speech, filler speech (e.g., “uhm”) may be inserted. Based on feedback indicating high resistance force from the subject, the filler speech may include terms such as “relax.” However, any appropriate method for synchronizing the duration and timing of the output and / or other appropriate parameter of the output speech with the modified motion of the robot may be used as the disclosure is not so limited.

[0017] According to some embodiments, pitch of the input speech may be maintained during adjustments to duration in the output speech. Pitch may be maintained by performing a Fourier transform on the input speech, performing phase unwrapping to determine the instantaneous frequency, and using the frequency for duration-adjusted output speech obtained by implementing an inverse Fourier transform. In some embodiments, a phase vocoder may be used for time-scaling without pitch shifting of the input speech to produce the output speech.

[0018] In some embodiments, the input speech may be generated based on an input description. That is, a description of the task to be performed by the robot may be provided along with an input trajectory. A generative language-based model may generate speech in real time in some embodiments. In some embodiments, a language model may generate text from the description that is converted to speech by a text-to-speech algorithm. However, the use of predefined input speech, directly recorded input speech, and / or any other appropriate method of generating the desired input speech may be used as the disclosure is not so limited.

[0019] In some embodiments, to synchronize the output speech with the movements of the robot or end effector of the robot, the input speech (or text) may be provided to the motion controller, as well as to the language controller. In this way, the parameters such as pose, commanded trajectory, and / or force output by the motion controller match the speech to be output by the language controller.

[0020] Turning to the figures, specific non-limiting embodiments are described in further detail. It should be understood that the various systems, components, features, and methods described relative to these embodiments may be used either individually and / or in any desired combination as the disclosure is not limited to only the specific embodiments described herein.

[0021] FIG. 1 is a block diagram of an exemplary system 100 to synchronize robotic language and motion according to one embodiment. A task description 105 may be provided to a language model 110. The task description 105 may describe robotic actions associated with physical therapy, assistance, or another task performed by a robot 101 that involves physical and auditory interaction with a subject. The language model 110 may provide input text 115 based on the task description 105. This input text 115 may be provided to a motion controller 140 to facilitate synchronization of robotic movement and speech. The language model 110 may be optional and, as shown in FIGs. 4 and 5, input language or speech may be provided to a language controller and a motion controller.

[0022] In the exemplary embodiment shown in FIG 1, the motion controller 140 also obtains an input motion trajectory 130 associated with the task described by the task description 105. The input motion trajectory 130 may indicate an ideal speed 135 for an end effector 102 of the robot 101 that makes contact with the subject. This ideal speed 135 may be provided to a language controller 120.

[0023] An exemplary embodiment of the motion controller 140 is discussed with reference to FIG. 5. The motion controller 140 may provide a control signal in the form of a torque 145 applied to the end effector 102. The movement performed by the robot 101 or, more particularly, by the end effector 102 may result in feedback obtained by a feedback controller 150. The feedback may include variables 160 such as end effector position and velocity, as well as resistance force applied by the subject. The feedback may also include actual speed 155 of the end effector 102, which is provided to the language controller 120.

[0024] The input text 115 is also provided to a language controller 120 in the exemplary embodiment of FIG. 1. An exemplary embodiment of the language controller 120 is discussed with reference to FIG. 5. The language controller 120 may use the input text 115 and a difference between the ideal speed 135 and actual speed 155 of the end effector 102 to modulate audio and provide modulated audio 125 as an output. This modulated audio 125 may be output from the robot 101 in synchronization with movements performed, for example, by the end effector 102.

[0025] FIG. 2 is a flow diagram of an exemplary method 200 of synchronizing robotic language and motion according to one embodiment. At 210, obtaining information for a task may include obtaining a task description 105 and input motion trajectory 130, as discussed with reference to FIG. 1. In this case, implementing a language model at 220 may result in input text 115 that is obtained at 230 and is also provided for generating motion commands at 270. In alternate embodiments, such as one discussed with reference to FIG. 5, the information may include input speech. In that case, a generative language-based artificial intelligence (Al) model may extract linguistic features from the speech (e.g., words, pace, tone, pauses).

[0026] Motion commands generated at 270, with the language input, facilitate synchronization between the movement and speech outputs of the robot 101. Generating motion commands (at 270), like generating output speech (at 250), also is based on feedback information from the environment and subject. Motion commands generated at 270 may include poses (e.g., position of the end effector 102) and speed. At 280, generating a control signal mayrefer to providing torque to be applied to the joint of the end effector 102 to trigger end effector action. At 290, obtaining and processing feedback information from the robot, environment, and the subject of the end effector action results in different feedback provided to different aspects of the system 100. The feedback may include speech by the human subject. Resistance from the subject and actual position and speed of the robot 101 (end effector 102) may be among the information 160 provided for generating subsequent motion commands and a subsequent control signal. Actual speed of the end effector 102 may be provided for output speech generation at 250.

[0027] With respect to language generation, at 240, text to input speech generation may be performed based on input text 115 obtained at 230. As noted, the input, provided with the input motion trajectory, may itself be speech, and an artificial intelligence model may be implemented to extract features (e.g., words, pace, tone, pauses), in real time, to be used as input speech at 250.

[0028] At 250, modulating input speech to generate output speech may include obtaining ideal speed 135 of the end effector based on the input motion trajectory 130, as well as actual speed 155 obtained as feedback. Specifically, the difference between ideal speed 135 and actual speed 155 of the end effector 102 may be used to modify duration of speech. This time scaling ensures correspondence between movement of the end effector 102 and speech output by the robot 101. When it is determined that duration of speech is to be increased, filler sounds (e.g., uhm, okay) may be added and when duration of speech is to be decreased, words may be eliminated. For example, path through input speech may be changed from “now would you please lean forward” to “please lean forward.” In addition, pitch may be maintained by processing the input speech in the frequency domain. In some embodiments, a phase vocoder may be implemented. The resulting output speech may be provided as audio that is synchronized with the end effector movement.

[0029] Example: General Framework

[0030] Humans naturally adjust the pace and tone of their speech to match their motion. FIG. 3 shows a series of images from an exemplary physical therapy session involving a therapist performing a “shoulder external rotation” therapy. Here, the therapist adapts speech with the length of pauses during the intended motion. Similarly, a robot interacting with a human subject can adapt the pace of the speech as it guides the user along a given trajectory according to various embodiments.

[0031] A language controller may augment a robot controller. A typical robot controller may perform feedback control such that the end effector’s motion follows a given trajectory. In the exemplary framework, a language controller also takes in a piece of text or speech audio along with the trajectory. The speech may closely related to the trajectory in that it explains the purpose of the motion and / or asks a user to comply or give directions on the next steps. For simple tasks, the trajectory may be hard-coded and the speech pre-recorded. For complex tasks, the speech-trajectory pair may be the output of a high- level multi-modal planner, possibly chained to text-to- speech systems. A language controller outputs modulated audio, which may be played directly by a speaker on the robot.

[0032] The language controller may modulate the input audio in real-time so that the speech is uttered in synchronization with the motion based on the state of the robot, observations of the environment, and the state of the motion controller. For example, a language controller may consider the force measured by sensors on the end effectors and user speech or action (e.g., interruption). Speech may be manipulated by:1) adjusting the parameters to a text-to-speech model, if the reference speech is provided in the form of text,2) in general, changing the pitch and scaling the duration of the audio through frequency domain techniques (e.g., phase- vocoder),3) changing the length of pauses, phonemes based on language- specific knowledge,4) cropping out part of audio, or injecting fillers (e.g., “. . . uhm. . . ” or “. . . relax”).

[0033] The language controller 120 may provide feedback to the motion controller, such as the speech progress and estimated completion time. Such feedback, for example, allows the motion controller to slow down when necessary to prevent finishing the trajectory before finishing the speech.

[0034] FIG. 4 is a block diagram depicting an exemplary general framework of a language controller 120. In the figure, r represents applied joint torque, q and q pertain to joint position and velocity, and x and x pertain to robot position and velocity. Robot state and observations are among the collected robot and environment states 410 and may include speech by the human subject.

[0035] Example: Language Control

[0036] Generally, motion of a robot may be controlled with an admittance controller 510 and speech modulation may be achieved with a phase vocoder 520 as shown in the exemplary block diagram in FIG. 5.

[0037] A. Admittance Control

[0038] Motion of the robot may be controlled by an admittance controller 510 that converts external force feedback Fext (from 410) to a desired velocity through a virtual dynamics model 511 of the robot. Because humans have active control over their limb motion and generate resistive force in case their limb movement deviates from their expectations, human users bear characteristics of an impedance. This, in turn, may call for treating the robot as an admittance. The core of admittance control is the virtual dynamics model 511.

[0039] The robot’s end effector 102 (whose position is denoted as x) may be conceptualized as a virtual mass M, and its movement through a desired trajectory in free space may be conceptualized as a being driven by a virtual force Fdrive(x) with damping D, such that:Fdriveix) is designed such that it is tangent to the trajectory for every position x on the trajectory, and pointing toward the trajectory for every off-trajectory point so as to guide the end effector back on course. When the robot is in contact with the user, the user exerts a force Fext which is sensed by a force sensor mounted on the end effector 102. In the context of rehabilitation, the users may typically be passive and exerted forces may be resistance. Hence, the complete virtual dynamics model may be given by:

[0040] The model according to (2) has two characteristics:1) As the resistance Fext increases, the end effector 102 slows down. When Fext = F 'drive, the end effector comes to a stop due to the damping. When Fexl> F drive, the end effector 102 backs off and breaks the contact. Together, Udrive|| represents the maximum persistent force that the end effector 102 may theoretically exert on the user. In other words, the compliance or safety of the system may be intuitively internalized by a parameter of the model.2) The presence of damping implies a maximum velocity xmax= D ~1Fdrivein the absence of resistance. Having a maximum end effector velocity, set according to velocity controller 512, may be helpful in controlling the end effector. In practice, the maximum safe force and the maximum safe velocity may be a starting point, and D may then be chosen suchthat xmax= D ~^Fdriveholds. The inertia M may be low enough to make the end effector 102 appear compliant but also high enough to stabilize the system.

[0041] In practice, the admittance controller 510 may continuously compute x from the dynamics equation and integrate it to give the desired velocity. The computed end-effector velocity may then be passed to the low-level velocity controller 512 that actuates the robot.

[0042] B. Time Stretching

[0043] The speech audio may be modulated by time- stretching it with a phase vocoder (PV) 520. A phase vocoder520 may first transform the audio into the frequency domain by performing Short-time Fourier Transform (STFT) 521 along short windows centered around a series of sampling instants based on windowing 524. By making the interval between the sampling points larger and smaller, correcting phases, re- synthesizing the time-domain audio with spectra around each sample point, and composing the results, the phase vocoder 520 scales the duration of an audio without pitch shifts (which would occur in time-domain-based methods) at 522. As indicated in FIG. 5, an inverse STFT (ISTFT) 523 may be used to obtain the speech in the time domain prior to output at the composing process 526. PV 520 does not require specific prior knowledge of the language of the speech, and human speech may remain highly intelligible after PV time stretching.

[0044] C. Determine Time-Scaling Ratio

[0045] Generally, humans speak slower when they move slower, and faster when they move faster. Following this intuition, the time-scaling ratio c of the phase vocoder may be set at 525 as the ratio of the current end-effector speed along the trajectory, vCUrrent, over an ideal speed Videai- The raw time-scaling ration crawmay be given by:The time-scaling ratio may be clamped to prevent excessive and unnatural scaling, which humans do not generally do. The ideal speed Videai can be computed from (2) assuming an ideal / typical resistance that a typical cooperative user exhibits. For example:Current end-effector speed Vcurrent and ideal speed Videai are provided by the admittance controller 510 at 513.

[0046] The time-scaling ratio may go through a low-pass filter to avoid abrupt changes in the speed of speech due to fluctuating end effector speed or measurement noises. Thus, the time-scaling ratio may be given by:

[0047] The above-described embodiments of the technology described herein can be implemented in any of numerous ways. For example, the embodiments may be implemented using hardware, software or a combination thereof. When implemented in software, the software code can be executed on any suitable processor or collection of processors, whether provided in a single computing device or distributed among multiple computing devices. Such processors may be implemented as integrated circuits, with one or more processors in an integrated circuit component, including commercially available integrated circuit components known in the art by names such as CPU chips, GPU chips, microprocessor, microcontroller, or co-processor. Alternatively, a processor may be implemented in custom circuitry, such as an ASIC, or semicustom circuitry resulting from configuring a programmable logic device. As yet a further alternative, a processor may be a portion of a larger circuit or semiconductor device, whether commercially available, semi-custom or custom. As a specific example, some commercially available microprocessors have multiple cores such that one or a subset of those cores may constitute a processor. Though, a processor may be implemented using circuitry in any suitable format.

[0048] Further, it should be appreciated that a computing device may be embodied in any of a number of forms, such as a rack-mounted computer, a desktop computer, a laptop computer, or a tablet computer. Additionally, a computing device may be embedded in a device not generally regarded as a computing device but with suitable processing capabilities, including a Personal Digital Assistant (PDA), a smart phone, tablet, or any other suitable portable or fixed electronic device.

[0049] Also, a computing device may have one or more input and output devices. These devices can be used, among other things, to present a user interface. Examples of output devices that can be used to provide a user interface include display screens for visual presentation of output and speakers or other sound generating devices for audible presentation of output. Examples of input devices that can be used for a user interface include keyboards, individual buttons, and pointing devices, such as mice, touch pads, and digitizing tablets. As anotherexample, a computing device may receive input information through speech recognition or in other audible format.

[0050] Such computing devices may be interconnected by one or more networks in any suitable form, including as a local area network or a wide area network, such as an enterprise network or the Internet. Such networks may be based on any suitable technology and may operate according to any suitable protocol and may include wireless networks, wired networks or fiber optic networks.

[0051] Also, the various methods or processes outlined herein may be coded as software that is executable on one or more processors that employ any one of a variety of operating systems or platforms. Additionally, such software may be written using any of a number of suitable programming languages and / or programming or scripting tools, and also may be compiled as executable machine language code or intermediate code that is executed on a framework or virtual machine.

[0052] In this respect, the embodiments described herein may be embodied as a computer readable storage medium (or multiple computer readable media) (e.g., a computer memory, one or more floppy discs, compact discs (CD), optical discs, digital video disks (DVD), magnetic tapes, flash memories, RAM, ROM, EEPROM, circuit configurations in Field Programmable Gate Arrays or other semiconductor devices, or other tangible computer storage medium) encoded with one or more programs that, when executed on one or more computers or other processors, perform methods that implement the various embodiments discussed above. As is apparent from the foregoing examples, a computer readable storage medium may retain information for a sufficient time to provide computer-executable instructions in a non-transitory form. Such a computer readable storage medium or media can be transportable, such that the program or programs stored thereon can be loaded onto one or more different computing devices or other processors to implement various aspects of the present disclosure as discussed above. As used herein, the term "computer-readable storage medium" encompasses only a non- transitory computer-readable medium that can be considered to be a manufacture (i.e., article of manufacture) or a machine. Alternatively or additionally, the disclosure may be embodied as a computer readable medium other than a computer-readable storage medium, such as a propagating signal.

[0053] The terms “program” or “software” are used herein in a generic sense to refer to any type of computer code or set of computer-executable instructions that can be employed toprogram a computing device or other processor to implement various aspects of the present disclosure as discussed above. Additionally, it should be appreciated that according to one aspect of this embodiment, one or more computer programs that when executed perform methods of the present disclosure need not reside on a single computing device or processor, but may be distributed in a modular fashion amongst a number of different computers or processors to implement various aspects of the present disclosure.

[0054] Computer-executable instructions may be in many forms, such as program modules, executed by one or more computers or other devices. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. Typically the functionality of the program modules may be combined or distributed as desired in various embodiments.

[0055] The embodiments described herein may be embodied as a method, of which an example has been provided. The acts performed as part of the method may be ordered in any suitable way. Accordingly, embodiments may be constructed in which acts are performed in an order different than illustrated, which may include performing some acts simultaneously, even though shown as sequential acts in illustrative embodiments.

[0056] Further, some actions are described as taken by a “user.” It should be appreciated that a “user” need not be a single individual, and that in some embodiments, actions attributable to a “user” may be performed by a team of individuals and / or an individual in combination with computer-assisted tools or other mechanisms.

[0057] While the present teachings have been described in conjunction with various embodiments and examples, it is not intended that the present teachings be limited to such embodiments or examples. On the contrary, the present teachings encompass various alternatives, modifications, and equivalents, as will be appreciated by those of skill in the art. Accordingly, the foregoing description and drawings are by way of example only.

Claims

CLAIMSWhat is claimed is:

1. A method of controlling a robot, the method comprising: obtaining an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task; providing a control signal to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot; obtaining feedback information about the end effector resulting from the control signal; and generating output speech by modulating the input speech based on the feedback information about the end effector.

2. The method according to claim 1, wherein the feedback information includes position and velocity of the robot.

3. The method according to any of claims 1-2, wherein interacting with the environment with the end effector includes interacting with a subject with the end effector.

4. The method according to any of claims 1-3, further comprising generating motion commands based on the input motion trajectory and generating the control signal based on the motion commands, wherein the motion commands specify a speed of the end effector.

5. The method according to any of claims 1-4, further comprising synchronizing the output speech with movement of the end effector of the robot.

6. The method according to any of claims 1-5, wherein the modulating the input speech includes adjusting a duration while maintaining pitch of the input speech based on the feedback information.

7. The method according to any of claims 1-6, further comprising implementing one or more of a motion controller and an audio controller.

8. A system to control a robot, the system comprising one or more controllers configured to: obtain an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task; provide a control signal to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot; obtain feedback information about the end effector resulting from the control signal; and generate output speech by modulating the input speech based on the feedback information about the end effector.

9. The system according to claim 8, wherein the one or controllers implements an artificial intelligence language model to generate the input speech from a description of the task.

10. The system according to any of claims 8-9, wherein the one or more controllers includes a motion controller configured to perform admittance control.

11. The system according to claim 10, wherein the motion controller is configured to generate motion commands based on the input motion trajectory and generate the control signal based on the motion commands, wherein the motion commands specify a speed of the end effector.

12. The system according to claim 11, wherein the motion controller is configured to use the input speech to generate the motion commands with correspondence between the movement of the end effector and the output speech.

13. The system according to any of claim 8-12, wherein the one or more controllers includes an audio controller that implements phase vocoding.

14. The system according to claim 13, wherein the audio controller adjusts a duration of the input speech while maintaining pitch to generate the output speech.

15. A non-transitory computer readable storage medium encoded with one or more programs that, when executed by one or more processors, perform a method of controlling a robot, the method comprising: obtaining an input motion trajectory defining movements of the robot to perform a task and input speech corresponding with descriptions of the task; providing a control signal to control movement of the robot based on the input motion trajectory to interact with an environment surrounding the robot with an end effector of the robot; obtaining feedback information about the end effector resulting from the control signal; and generating output speech by modulating the input speech based on the feedback information about the end effector.16 The non-transitory computer readable storage medium according to claim 15, wherein the feedback information includes position and velocity of the robot.

17. The non-transitory computer readable storage medium according to any of claims 15-16, wherein interacting with the environment with the end effector includes interacting with a subject with the end effector.

18. The non-transitory computer readable storage medium according to any of claims 15-17, further comprising generating motion commands based on the input motion trajectory and generating the control signal based on the motion commands, wherein the motion commands specify a speed of the end effector.

19. The non-transitory computer readable storage medium according to any of claims 15-18, further comprising synchronizing the output speech with movement of the end effector of the robot.

20. The non-transitory computer readable storage medium according to any of claims 15-19, wherein the modulating the input speech includes adjusting a duration while maintaining pitch of the input speech based on the feedback information.

Images