Remote operation system, program, and robot

Abstract

A remote operation system 1 comprises: a robot 2 that moves through space in accordance with user's instructions from an operating terminal 4; the operating terminal 4, which inputs instructions for movement into the robot 2; and a processing device 3 connected to the robot 2 and the operating terminal 4. The processing device 3 comprises: an acquiring unit 32 for acquiring movement information relating to the robot 2; and a transmitting unit 33 for transmitting, to the operating terminal 4, movement sound data capable of specifying a timing at which a sound is generated, using a pre-specified movement method, in accordance with the movement information relating to the robot 2. The operating terminal 4 is provided with a reproducing unit 42 which, in accordance with the movement sound data, reproduces sound data generated using the movement method in accordance with a movement amount.

Description

Remote control systems, programs and robots

[0001] The present invention relates to a remote control system, a program, and a robot.

[0002] In recent years, with the advancement of information processing devices and communication network technologies, opportunities for people in remote locations to communicate with each other have increased. Research and development is also progressing in a system in which a user operates a robot in a remote location as a robot avatar on their behalf, and the user acts in the remote location via the robot.

[0003] Various studies have been conducted to improve the sense of realism in remote control. For example, there is a technology that provides feedback to the user by providing sound when the feet of a bipedal robot land on the floor (Non-Patent Documents 1 and 2).

[0004] Emmanuele Tidoni, Pierre Gergondet, Abderrahmane Kheddar and Salvatore M. Aglioti, "Audio-visual feedback improves the BCI performance in navigational control of a humanoid robot", 2014 Jun 17, ORIGINAL RESEARCH ARTICLEEmmanuele Tidoni, Pierre Gergondet, Gabriele Fusco, Abderrahmane Kheddar, "The Role of Audio-Visual Feedback in a Thought-Based Control of a Humanoid Robot: A BCI Study in Healthy and Spinal Cord Injured People", 2017 Jun 17, IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING, VOL. 25, NO. 6

[0005] However, in Non-Patent Documents 1 and 2, sound feedback is given to the user when the robot's feet land on the floor, and the frequency of the sound feedback depends on the specifications of the robot. Therefore, the sense of realism that the user can experience through the robot is limited.

[0006] The present disclosure has been made in consideration of the above circumstances, and an object of the present disclosure is to provide a technology that can improve the sense of realism that a user can experience through a robot located in a remote location.

[0007] A remote control system according to one aspect of the present disclosure includes a robot that moves through space in accordance with user instructions from an operation terminal, an operation terminal that inputs instructions to the robot to move, and a processing device connected to the robot and the operation terminal, the processing device including an acquisition unit that acquires movement information of the robot, and a transmission unit that transmits to the operation terminal movement sound data that can identify the timing at which sound is generated in a movement method that is pre-specified in accordance with the movement information of the robot, and the operation terminal including a playback unit that plays back sound data generated in accordance with the movement information in accordance with the movement sound data.

[0008] A program according to one aspect of the present disclosure causes a processing device used in a remote control system including a robot that moves through space in accordance with instructions from the user via an operation terminal, an operation terminal that inputs instructions to the robot to move, and a processing device connected to the robot and the operation terminal to function as an acquisition unit that acquires movement information of the robot and a transmission unit that transmits movement sound data to the operation terminal that can identify the timing at which sound is generated in a movement method that is pre-specified in accordance with the movement information of the robot.

[0009] A remote control system according to one aspect of the present disclosure includes a robot that moves through space in accordance with user instructions from an operation terminal, an operation terminal that inputs instructions to the robot to move, and a processing device connected to the robot and the operation terminal, the robot having artificial feet at positions corresponding to human feet and artificial ears at positions corresponding to human ears, the artificial feet having a movement sound generating device that generates sounds generated in a pre-specified movement method corresponding to the movement of the robot, the artificial ears having a microphone that collects sounds in the space in which the robot is located and generates spatial sound data, the processing device having an acquisition unit that acquires the spatial sound data and a transmission unit that transmits the spatial sound data to the operation terminal, and the operation terminal having a playback unit that plays back the spatial sound data.

[0010] A robot according to one aspect of the present disclosure is a robot that moves through space in accordance with instructions from a user via an operating terminal, and has artificial feet at positions corresponding to human feet and artificial ears at positions corresponding to human ears, the artificial feet having a movement sound generating device that generates sounds generated in a pre-specified movement method in response to the movement of the robot, and the artificial ears having a microphone that collects sounds in the space in which the robot is located and generates spatial sound data.

[0011] According to the present disclosure, it is possible to provide a technology that can improve the sense of realism that a user experiences through a robot located in a remote location.

[0012] FIG. 1 is a diagram illustrating the system configuration of a remote control system according to a first embodiment and functional blocks of a processing device. FIG. 2 is a sequence diagram illustrating the processing of the remote control system according to the first embodiment. FIG. 3 is a diagram illustrating the system configuration of a remote control system according to a second embodiment and functional blocks of a processing device. FIG. 4 is a sequence diagram illustrating the processing of the remote control system according to the second embodiment. FIG. 5 is a diagram illustrating an overview of a footstep generating device. FIG. 6 is a diagram illustrating the mechanism (part 1) of the footstep generating device. FIG. 7 is a diagram illustrating the mechanism (part 2) of the footstep generating device. FIG. 8 is a diagram illustrating the hardware configuration of a computer used in the processing device.

[0013] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. In the description of the drawings, the same parts are designated by the same reference numerals and the description thereof will be omitted.

[0014] The remote control system 1 according to the present disclosure allows a user to operate a robot 2 located in a remote location and provides the user with feedback on the situation in the space where the robot 2 is located. This allows the user to experience a sense of realism as if they were actually in the robot's location. Furthermore, because the user can synchronize the operation of the robot 2 with the images and sounds fed back to the user in a more realistic manner, it is expected that the sense of discomfort felt when remotely controlling the robot 2 or so-called virtual reality (VR) sickness will be reduced.

[0015] First Embodiment In a remote control system 1 according to a first embodiment, a user is provided with feedback on sounds generated in a pre-specified movement method in response to the movement of the robot 2. The user can set his / her own stride length in advance. The user can view video data captured from the robot 2's eye level in real time, and can also hear in real time the footsteps generated at a pre-specified stride length as the robot 2 moves.

[0016] Here, "pre-specified" means that the movement method is specified prior to the timing of feedback. Here, the movement method may be specified by the user or may be set in accordance with the attributes of the space P in which the robot 2 is located. Furthermore, the sound generated by the movement method may also be specified by the user or may be set in accordance with the attributes of the space P in which the robot 2 is located. Alternatively, the movement method or the sound generated by the movement method may be specified by the user or may be set by computer processing based on the user's attributes. In the first embodiment, a case will be described in which the movement method is walking, and the sound generated by the movement method is footsteps generated in accordance with the amount of movement and stride length specified by the user. The movement method and footstep attributes such as stride length and sound data will be specified in advance by the user. Furthermore, "corresponding to the movement of the robot 2" means that the amount of movement, movement direction, etc. are taken into consideration.

[0017] In the first embodiment, the user can experience a sense of realism as if he or she were moving in the space P where the robot 2 is present.

[0018] In the present disclosure, "real time" refers to the robot 2 being driven instantly in accordance with instructions input to the operation terminal 4, and environmental data such as video data acquired by the robot 2 being instantly played back on the operation terminal 4. "Real time" may also include various delays such as the processing time of each computer and the time required for communication.

[0019] 1, the remote operation system 1 includes a robot 2, a processing device 3, and an operation terminal 4. The robot 2 moves in a space P according to instructions from a user via the operation terminal 4. The operation terminal 4 inputs instructions to move to the robot 2. The processing device 3 is connected to the robot 2 and the operation terminal 4.

[0020] The robot 2 is located in a space P. The space P is different from the space in which the operation terminal 4 is located. The user operating the operation terminal 4 cannot directly grasp the situation in the space P, but is in a position where he or she can grasp the situation through the operation terminal 4.

[0021] The robot 2 has a shape that resembles a human being, and includes at least artificial feet 22, artificial ears 23, and artificial eyes (not shown).

[0022] The artificial foot 22 is provided at a position corresponding to a human foot and includes wheels for movement and a motor for driving the wheels. In the example shown in Fig. 1, the wheels are specifically circular in side view relative to the direction of travel and are covered with tires, but this is not limited to this. The wheels may also be caterpillar-shaped, having an elliptical shape in side view relative to the direction of travel.

[0023] The artificial ear unit 23 is provided at a position corresponding to a human ear and has a microphone capable of collecting surrounding sounds. The artificial ear unit 23 has a structure similar to the inside of a human ear, and the microphone may be a binaural microphone.

[0024] The pseudo-eye unit corresponds to a human eye and has a camera. Video data captured by the camera facing the direction of the human eye is transmitted to the operation terminal 4. The pseudo-eye unit may have cameras provided at positions corresponding to the left and right eyes of a human. The operation terminal 4 can display three-dimensional video data using the video data captured by the two cameras.

[0025] The robot 2 has a control unit 21 that controls each piece of equipment that the robot 2 has. The pieces of equipment that the robot 2 has include, for example, wheels, motors, microphones, and cameras. In the remote control system 1 shown in FIG. 1 , the control unit 21 is a computer that controls each piece of equipment of the robot 2 in accordance with operation instructions input from the processing device 3. The control unit 21 transmits data acquired from each piece of equipment of the robot 2 to the processing device 3.

[0026] Specifically, the control unit 21 drives the motor in accordance with the input operation instructions to move the robot 2 in the direction and at the speed specified by the operation instructions. The control unit 21 also transmits sound data collected by the microphone and video data captured by the camera to the processing device 3 in real time.

[0027] The processing device 3 is a computer that relays in real time the transmission and reception of data between the operation terminal 4 and the robot 2. Each function of the processing device 3 may be implemented in a server on a communication network, or may be implemented in the control unit 21 in the operation terminal 4 or the robot 2.

[0028] The operation terminal 4 is a computer used by a user who operates the robot 2. The operation terminal 4 includes an instruction unit 41 and a playback unit 42.

[0029] The instruction unit 41 transmits operation instructions regarding the speed and direction of movement of the robot 2, which are input from an input device (not shown) by a user's operation, to the robot 2 via the processing device 3. The input device may be a general mouse and keyboard, or a controller equipped with a joystick.

[0030] The playback unit 42 plays back on an output device the video data and footstep data transmitted from the processing device 3. The output device may be a general mouse and speaker, or a head-mounted display and headphones.

[0031] (Processing Device) The processing device 3 will be described. The processing device 3 includes an operation unit 31, an acquisition unit 32, and a transmission unit 33.

[0032] The operation unit 31 receives operation instructions from the operation terminal 4 and transmits them to the robot 2 .

[0033] The acquisition unit 32 acquires video data and robot movement information from the robot 2. The movement information includes whether the robot 2 is moving in the space P, and if it is moving, the amount of movement, speed, direction, etc. The amount of movement is the distance traveled by the robot 2. The acquisition unit 32 sequentially acquires the video data and the amount of movement of the robot 2 as the robot 2 moves.

[0034] The acquisition unit 32 includes a generation unit 35 that generates movement sound data capable of identifying the timing at which sound will be generated by a movement method designated in advance in accordance with the movement information of the robot 2. In this embodiment, a case will be described in which the movement sound data is data that identifies the timing at which footsteps generated by walking will be generated when the movement method is "walking." The operation terminal 4 transmits a stride value to the processing device 3 in accordance with an input from the user, and the processing device 3 holds the stride value input from the operation terminal 4. The generation unit 35 calculates the timing at which footsteps will be generated from the amount of movement acquired from the robot 2, assuming that the robot 2 is moving with the stride designated by the user, and generates footstep data capable of identifying the calculated timing.

[0035] The transmission unit 33 transmits the video data acquired by the acquisition unit 32 and the footstep data generated by the generation unit 35 to the operation terminal 4. The transmission unit 33 transmits the video data and the footstep data to the operation terminal 4 in real time.

[0036] The playback unit 42 of the operation terminal 4 plays back the video data and footstep data. Specifically, the playback unit 42 plays back the video data on the display, and also plays back sound data such as footsteps generated by a pre-specified movement method in accordance with the footstep data and in response to the movement information.

[0037] The footstep data is, for example, data that notifies the timing of footstep generation. The generation unit 35 successively monitors the amount of movement of the robot 2, and when the timing of footstep generation arrives, notifies the playback unit 42 of the arrival of the timing of footstep generation via the transmission unit 33. The playback unit 42 plays the footstep sound data through the speaker at the notified timing.

[0038] The footstep data may be data that is superimposed on the sound data of the video data. The generation unit 35 successively monitors the amount of movement of the robot 2, and when the timing for footstep generation arrives, superimposes the footstep sound data on the sound data of the video data. The video data with the footstep sound data superimposed thereon is transmitted by the transmission unit 33 and played back by the playback unit 42.

[0039] At this time, the footstep sound data may also be specified by the user. For example, footsteps are considered to vary depending on the sound of the user's shoes, the user's weight, or the floor material of the space P in which the robot 2 is located. Therefore, the user may notify the robot 2 or the processing device 3 of the type of shoes, weight, etc., in addition to the stride length. The system administrator may also specify the type of floor material of the space P in which the robot 2 is located. Alternatively, the generation unit 35 may estimate the type of floor material by referencing photographed image data of the floor and a model that associates floor images with floor material types. The footstep sound data may also be data of the footsteps of a virtual character. For example, the footstep sound data may be a "hopping" sound when a character jumps around, a "thud thud thud" sound when a heavy-looking character moves, or any other arbitrary sound. The user may specify the footstep sound data as appropriate, or the generation unit 35 may set the sound data as appropriate according to the attributes of the space P in which the robot 2 is located.

[0040] The generation unit 35 may acquire the stride length set by the user for each movement direction of the robot 2, and generate sound data of footsteps for each stride length set by the user according to the movement direction of the robot 2. The movement direction may be, for example, forward, backward, straight, or curved.

[0041] The stride length specified in advance may be the stride length of the user of the operation terminal 4, or may be a stride length set based on the height assumed from the viewpoint of the camera, i.e., the viewpoint of the video data obtained from the robot 2. Alternatively, under the setting that the user is moving as a virtual character in the space P where the robot 2 is located, a stride length assumed for that character, specifically a stride length unrelated to the user's stride length, may be set.

[0042] As long as the user can hear footsteps generated in accordance with the amount of movement of the robot 2 and the stride length specified by the user on the operation terminal 4, the footstep data may be generated in any manner.

[0043] The footstep data specifies the timing at which footsteps are to be generated when the cumulative amount of movement of the robot 2 since it started moving reaches a natural number multiple of the stride length specified by the user. Since the timing at which footsteps are to be generated is specified from the amount of movement of the robot and the stride length of the user, the user can get a sense of realism as if they were actually moving in space P, from the changes in the scenery that can be confirmed in the video data and the footsteps that are generated at the user's own stride length as they move.

[0044] The footstep data also identifies the timing of footsteps when the robot 2 stops, as the timing of footsteps. When a human stops moving, they land with both feet, so the sense of realism that the user gets is improved by feeding back the footsteps generated at that time to the operation terminal 4. Furthermore, when the robot 2 stops, the generation unit 35 initializes the cumulative movement amount from when the robot 2 started moving.

[0045] In the first embodiment, the amount of movement transmitted from the robot 2 to the processing device 3 is the distance traveled by the robot 2 in the space P. The robot 2 measures the amount of movement in the space P and transmits it to the processing device 3.

[0046] In another embodiment, the movement amount may be calculated by the generation unit 35 of the processing device 3 in accordance with an operation instruction input from the operation terminal 4. For example, the processing device 3 stores in advance correspondence data that associates an operation instruction input from the operation terminal 4 with the movement amount of the robot 2 in response to that operation instruction. When an operation instruction is input from the operation terminal 4, the generation unit 35 may calculate the movement amount by referring to the correspondence data, and generate footstep data that can identify the timing at which footsteps occur with a stride length specified by the user corresponding to the calculated movement amount.

[0047] (Remote Control Method) The processing in the remote control system 1 according to the first embodiment will be described with reference to Fig. 2. The drive processing and feedback processing are each executed sequentially in real time.

[0048] First, the drive process will be described.

[0049] In step S101, the operation terminal 4 transmits an operation instruction to the processing device 3. In step S102, the processing device 3 transmits the operation instruction received from the operation terminal 4 to the robot 2. In step S103, the robot 2 drives its wheels and the like in accordance with the operation instruction received from the processing device 3.

[0050] In step S104, the robot 2 transmits the video data captured by the camera to the processing device 3. In step S105, the processing device 3 transmits the video data received from the robot 2 to the operation terminal 4. In step S106, the operation terminal 4 plays back the video data received from the processing device 3. Note that the robot 2 transmits the video data to the operation terminal 4 at any time, regardless of whether or not an operation instruction has been received from the operation terminal 4, and the operation terminal 4 plays back the video data at any time.

[0051] Next, the feedback process will be described.

[0052] In step S151, the robot 2 transmits the amount of movement of the robot 2 to the processing device 3. In step S152, the processing device 3 calculates the cumulative amount of movement of the robot 2 since the start of movement. In step S153, the processing device 3 determines whether the cumulative amount of movement calculated in step S152 has reached a natural number multiple of the stride length set in advance from the operation terminal 4. If it has reached a natural number multiple, the process proceeds to step S155. If it has not reached a natural number multiple, the process proceeds to step S154.

[0053] In step S154, the processing device 3 determines whether or not the robot 2 has stopped moving. If the robot 2 has not stopped moving, the processing proceeds to step S152 and waits to receive the amount of movement from the robot 2. If the robot 2 has stopped moving, the processing device 3 initializes the cumulative amount of movement and proceeds to step S155.

[0054] In step S155, the processing device 3 determines the timing of footstep generation. Specifically, the processing device 3 determines the timing of footstep generation when the accumulated movement amount reaches a natural number multiple of the stride length or when the robot 2 has finished moving.

[0055] In step S155, the processing device 3 transmits information to the operation terminal 4 indicating that it is time for footsteps to be heard, and the operation terminal 4 plays the footstep sound data at the timing when the footstep data is received. Alternatively, the processing device 3 superimposes the footstep sound data on the sound data of the video data at the timing when the footsteps are heard. The operation terminal 4 plays the video data on which the footstep sound data has been superimposed.

[0056] 2, the case where the accumulated movement amount of the robot 2 is calculated and initialized when the robot 2 stops has been described, but this is not limited to this. When the movement amount of the robot 2 reaches the stride length, the processing device 3 may identify this as the timing of footstep generation and initialize the movement amount. Furthermore, if the robot 2 moves on wheels, the robot 2 may notify the processing device 3 of the timing when the wheels have rotated a predetermined number of times, thereby notifying the processing device 3 of the movement amount of the robot 2.

[0057] In the first embodiment, every time the amount of movement of the robot 2 reaches a natural number multiple of the stride length, sound data of footsteps is reproduced on the operation terminal 4. The user can perceive the movement of the robot 2 in the space P as the user's own movement, thereby improving the sense of realism.

[0058] Second Embodiment In a remote control system 1a according to a second embodiment, a user is provided with feedback on sounds generated in a pre-specified movement method in response to the movement of a robot 2a. Footsteps are generated in a space P where the robot 2a is located, the footsteps are collected by a microphone, and are played on the operation terminal 4. The user can hear the footsteps generated in the space P where the robot 2a is located, specifically, footsteps that reflect reverberations that depend on the environment, such as the materials of the floor, walls, or ceiling in the space P and the size of the space P. The user can experience a sense of realism as if they were actually moving in the space P where the robot 2a is located.

[0059] Here, "pre-specified" means that it is specified prior to the timing of feedback. Here, the movement method may be specified by the user or may be set in accordance with the attributes of the space P in which the robot 2 is located. The sound generated by the movement method may also be specified by the user or may be set in accordance with the attributes of the space P in which the robot 2a is located. Alternatively, the movement method or the sound generated by the movement method may be specified by the user or may be set by computer processing based on the user's attributes. In the second embodiment, a case will be described in which the movement method is walking and the sound generated by the movement method is footsteps generated in accordance with the movement amount and the user's stride. A case will be described in which the movement method and footstep attributes such as stride length and sound data are specified in advance by the user. "In response to the movement of the robot 2a" means that the movement amount or movement direction, etc. are taken into consideration.

[0060] Furthermore, the user can preset his / her own stride length for the robot 2 a. The user can view video data captured from the robot 2 a's line of sight, and can also hear in real time the footsteps of the robot 2 a, which are generated at the preset stride length as the robot 2 a moves.

[0061] In the second embodiment, the user can experience a sense of realism as if he or she were moving in the space P where the robot 2a is present.

[0062] 3, the remote operation system 1a includes a robot 2a, a processing device 3a, and an operation terminal 4. As in the first embodiment, the robot 2a moves in a space P according to instructions from a user via the operation terminal 4. The operation terminal 4 inputs instructions to move to the robot 2a. The processing device 3a is connected to the robot 2a and the operation terminal 4.

[0063] The artificial feet 22 of the robot 2a include wheels for movement, motors for driving the wheels, and a footstep generator 24 for generating footsteps at a stride length specified by the user in accordance with the amount of movement of the robot 2a. For example, the footstep generator 24 generates footsteps when the cumulative amount of movement since the start of movement reaches a natural number multiple of the stride length preset by the user, or when movement stops. The user can perceive the movement of the robot 2a in the space P as the user's own movement, thereby improving the sense of realism.

[0064] The artificial ear unit 23 of the robot 2a collects sounds in the space P in which the robot 2a is located and generates spatial sound data. The microphones provided in the artificial ear unit 23 are preferably binaural microphones, with one binaural microphone provided for each of the left and right ears. The binaural microphones can reverberate the footsteps emitted by the footstep generator 24 in the space P and capture them as three-dimensional sounds. The artificial ear unit 23 is preferably made of silicone artificial skin, urethane, or the like, which imitates the texture of a human ear, and has a structure similar to the structure of a human ear. By collecting sound inside the artificial ear with the binaural microphone, reverberation sounds within a structure that mimics the structure and texture of a human ear can be collected and made audible to the user.

[0065] The robot 2 a may acquire a stride length set by the user for each moving direction of the robot 2 a, and generate sound data of footsteps for each stride length set by the user according to the moving direction of the robot 2 a. The moving direction may be, for example, forward, backward, straight, or curved.

[0066] The control unit 21a of the robot 2a drives the motor in accordance with the input operation instructions to move the robot 2a in the direction and at the speed specified by the operation instructions. The control unit 21a also transmits spatial sound data collected by the microphone and video data captured by the camera to the processing device 3a in real time.

[0067] The spatial sound data is data obtained by collecting footsteps generated by the pseudo feet 22 corresponding to human feet in the human-shaped robot 2a using microphones provided in the pseudo ears 23 corresponding to human ears. By playing the spatial sound data on the operation terminal 4, the user can experience a sense of realism as if they were hearing their own footsteps generated in the space P where the robot 2a is located with their own ears. The spatial sound data may be data separate from the video data, or may be sound data within the video data.

[0068] The processing device 3a includes an operation unit 31, an acquisition unit 32a, and a transmission unit 33. In the second embodiment, the processing device 3a relays the transmission and reception of data between the robot 2a and the operation terminal 4, but data may be transmitted and received directly between the robot 2a and the operation terminal 4 without the processing device 3a.

[0069] The operation unit 31 receives operation instructions from the operation terminal 4 and transmits them to the robot 2a.

[0070] The acquisition unit 32a acquires the video data and spatial sound data from the robot 2a. The acquisition unit 32a sequentially acquires the video data and spatial sound data as the robot 2a moves.

[0071] The transmitting unit 33 transmits the video data and spatial sound data acquired by the acquiring unit 32a to the operation terminal 4. The transmitting unit 33 transmits the video data and spatial sound data to the operation terminal 4 in real time.

[0072] The operation terminal 4 is a computer used by a user who operates the robot 2 a. The operation terminal 4 includes an instruction unit 41 and a playback unit 42.

[0073] The instruction unit 41 transmits operation instructions regarding the speed and direction of movement of the robot 2a, which are input from an input device (not shown) by a user's operation, to the robot 2a via the processing device 3a.

[0074] The reproduction unit 42 reproduces the video data and spatial sound data transmitted from the processing device 3a on the output device.

[0075] (Remote Control Method) The process in the remote control system 1a according to the second embodiment will be described with reference to Fig. 4. The drive process and feedback process are each executed sequentially in real time. The drive process shown in steps S101 to S105 in Fig. 4 is the same as the drive process described with reference to Fig. 2.

[0076] Next, the feedback process will be described.

[0077] In step S201, the robot 2a determines whether the cumulative movement amount since the start of movement has reached a natural number multiple of the stride length preset on the operation terminal 4. If it has reached a natural number multiple, the process proceeds to step S203. If it has not reached a natural number multiple, the process proceeds to step S202.

[0078] In step S202, the robot 2a determines whether the robot 2a has stopped moving. If the robot 2a has not stopped moving, the process proceeds to step S201 and waits for the amount of movement of the robot 2a to be acquired. If the robot 2a has stopped moving, the robot 2a initializes the accumulated amount of movement and proceeds to step S203.

[0079] In step S203, the footstep generating device 24 of the robot 2a generates footsteps. In step S204, the robot 2a collects sounds in the space P with a microphone and generates spatial sound data. In step S205, the robot 2a transmits the spatial sound data to the processing device 3a. In step S206, the processing device 3a transmits the spatial sound data received from the robot 2a to the operation terminal 4. The operation terminal 4 plays back the spatial sound data received from the processing device 3a.

[0080] In the second embodiment, every time the amount of movement of the robot 2a reaches a natural number multiple of the stride length, sound data of footsteps is reproduced on the operation terminal 4. The user can perceive the movement of the robot 2a in the space P as the user's own movement, thereby improving the sense of realism.

[0081] (Footstep Generating Device) The footstep generating device 24 is an example of a movement sound generating device that generates sounds generated by a movement method designated in advance by a user or the like, in accordance with the movement of the robot 2. The footstep generating device 24 is provided on the pseudo foot 22 of the robot 2a. The footstep generating device 24 generates footsteps with a stride length designated by the user in accordance with the amount of movement of the robot. An example of the footstep generating device 24 will be described.

[0082] (First Example) The footstep generating device 24 according to the first example is a speaker that outputs footstep sound data. Because the robot 2a has a shape that resembles a human, a speaker is provided on the robot 2a close to the floor. The footstep generating device 24 of the robot 2a monitors the amount of movement of the robot 2a, and reproduces footstep sound data each time the amount of movement of the robot 2a from the start of movement reaches a natural number multiple of the stride length specified by the user. The footstep generating device 24 also reproduces footstep sound data when the robot 2a stops.

[0083] As another example, the control unit 21a monitors the amount of movement of the robot 2a, and when the amount of movement of the robot 2a from the start of movement reaches a natural number multiple of the stride length specified by the user, notifies the footstep generating device 24. The control unit 21a notifies the footstep generating device 24 when the robot 2a stops. Upon receiving the notification from the control unit 21a, the footstep generating device 24 outputs footstep sound data from a speaker.

[0084] At this time, the user may also specify the sound data of the footsteps. For example, it is considered that the sound of footsteps changes depending on the sound of the user's shoes, the user's weight, or the floor material of the space P where the robot 2a is located. Therefore, the user may notify the robot 2a of the type of shoes or weight, in addition to the stride length. Furthermore, a system administrator may specify the type of floor material of the space P to the robot 2a. Alternatively, the footstep generating device 24 may estimate the type of floor material by referring to photographed image data of the floor and a model that associates floor images with floor material types. For example, the sound data of the footsteps may be appropriately specified by the user, such as a "hopping" sound when a character jumps around, a "thud thud" sound when a solid character moves, or any other sound.

[0085] The footstep sound generating device 24 has data that associates conditions such as shoe type, weight, or floor material type with sound data of footsteps to be generated under those conditions. The footstep sound generating device 24 refers to that data and selects sound data of footsteps based on the shoe type, weight, or floor material type specified by the user. The footstep sound generating device 24 outputs the selected sound data from a speaker at the timing when footsteps are to be generated.

[0086] The footstep generating device 24 according to the first example outputs footstep sound data from the feet of the robot 2a, and spatial sound data obtained by collecting footsteps in the space P with a microphone provided in the artificial ear unit 23 of the robot 2a is reproduced on the operation terminal 4. The user can hear the footsteps generated by reflecting the environment, such as reverberation, in the space P, collected by the microphone provided at a position corresponding to the ear. The user can experience a sense of realism as if they were actually moving in the space P where the robot 2a is located.

[0087] 5, a footstep generating device 24 according to the second example has a contact part 51 that can come into contact with the floor, and generates footsteps by striking the contact part 51 against the floor. The footstep generating device 24 controls the contact part 51 so that it strikes the floor of the space P at desired timing and moves it away from the floor at other timings. The footstep generating device 24 of the robot 2a monitors the amount of movement of the robot 2a, and strikes the contact part 51 against the floor every time the amount of movement of the robot 2a from the start of movement reaches a natural number multiple of the stride length specified by the user. The footstep generating device 24 also strikes the contact part 51 against the floor when the robot 2a stops.

[0088] As another example, the control unit 21a monitors the amount of movement of the robot 2a, and when the amount of movement of the robot 2a from the start of movement reaches a natural number multiple of the stride length specified by the user, notifies the footstep generating device 24. The control unit 21a notifies the footstep generating device 24 when the robot 2a stops. Upon receiving the notification from the control unit 21a, the footstep generating device 24 brings the contact unit 51 into contact with the floor.

[0089] In the example shown in FIG. 5, the contact portion 51 has a shape resembling a foot, but it may have any shape as long as it can be struck against the floor.

[0090] In the second example, there are several possible methods for controlling the footstep sound generating device 24 so that the contact part 51 hits the floor of the space P at a desired timing and lifts it off the floor at other timings.

[0091] 6, the footstep generating device 24 includes a contact portion 51, a rotation shaft 52, and a pin 53. The rotation shaft 52 is fixed to the robot 2a so as not to move with the rotation of the wheels. The rotation shaft 52 includes a support rod 54 and the contact portion 51. The rotation shaft 52 is formed so as to maintain a predetermined angle between the support rod 54 and the contact portion 51.

[0092] 6A shows a state in which the support rod 54 and the pin 53 are separated. The support rod 54 is disposed substantially parallel to the floor. In this state, the contact portion 51 is adjusted so as not to contact the floor.

[0093] 6(b), when the support rod 54 and the pin 53 come into contact, the support rod 54 is pushed by the pin 53. As the support rod 54 tilts, the contact portion 51 rises. When the wheel rotates further, the contact between the support rod 54 and the pin 53 is released, and the reaction force hits the contact portion 51 against the floor.

[0094] 6, the footstep generating device 24 can generate footsteps by striking the contact portion 51 against the floor at a predetermined cycle. By matching the circumference of the wheels to the length of the stride specified by the user, the footstep generating device 24 can generate footsteps with the stride specified by the user in accordance with the amount of movement of the robot 2a.

[0095] While Fig. 6 describes a case where the circumference of the wheel is made to match the stride length, the footstep generator 24 shown in Fig. 7 drives the motor 55 to strike the contact part 51 against the floor. The footstep generator 24 of the robot 2a monitors the amount of movement of the robot 2a, and drives the motor 55 to strike the contact part 51 against the floor every time the amount of movement of the robot 2a from the start of movement reaches a natural number multiple of the stride length specified by the user. Furthermore, when the robot 2a stops, the footstep generator 24 drives the motor 55 to strike the contact part 51 against the floor.

[0096] 7, footsteps can be generated by striking the contact portion 51 against the floor by driving the motor of the footstep generating device 24. The footstep generating device 24 can generate footsteps with a stride length specified by the user in accordance with the amount of movement of the robot 2a.

[0097] Other Embodiments In the first and second embodiments, a method of feeding back footsteps in the space P to the user has been described, but the present invention is not limited to this. For example, sounds made by a movement method designated in advance by the user or the like in the space P may be fed back to the user of the operation terminal 4 in accordance with the amount of movement of the robot 2. Furthermore, the air flow or temperature felt by the user when the user moves in the space P may be fed back to the user of the operation terminal 4 in accordance with the amount of movement of the robot 2. In this case, a fan or heater controllable by the operation terminal 4 may be installed, and the operation terminal 4 controls the fan or heater to present the air flow or temperature to the user.

[0098] For example, the sound emitted in the movement method specified by the user may be the sound of clothes rustling as the arms are swung, and the sound of clothes rustling as the arms are swung while moving may be fed back to the user at the operation terminal 4. For example, the user sets the time interval of the user's arm swing while moving in the processing device 3 or the robot 2a. In this case, too, as described in the first embodiment, the timing for playing the sound of clothes rustling may be specified by the processing device 3, or as described in the second embodiment, the movement sound generating device may generate the sound of clothes rustling in the robot 2a, and the operation terminal 4 may play back the spatial sound data collected by the microphone. The time interval of the user's arm swing while moving and the sound may be set based on the user's attributes such as age and weight, and the attributes of the space P in which the robot 2 is installed may also be taken into consideration.

[0099] The processing device 3 performs a process of specifying the timing for playing the clothes rustling sound. The generator 35 of the processing device 3 notifies the operation terminal 4 that the clothes rustling sound data will be played at a timing when a natural number multiple of a set time interval has elapsed since the robot 2 started moving. Alternatively, the generator 35 of the processing device 3 superimposes the clothes rustling sound data onto the video data at a timing when a natural number multiple of a set time interval has elapsed since the robot 2 started moving.

[0100] The process of generating clothing rustling sounds and collecting the sounds with a microphone in the robot 2a will be described. The right and left sides of the robot 2a are each provided with a rustling sound generator, which is a type of movement sound generator. After the robot 2a starts moving, the left and right side rustling sound generators alternately output the clothing rustling sound to a speaker at a timing when a natural number multiple of a set time interval has elapsed. Alternatively, after the robot 2a starts moving, the left and right side rustling sound generators each use a motor or the like to put on clothing at a timing when a natural number multiple of a set time interval has elapsed. At this time, the robot 2a may be made to wear clothing specified by the user or clothing that is expected to be worn based on the user's attributes.

[0101] The sound of clothes rustling when the user swings their arms is fed back to the operation terminal 4 at time intervals when the user swings their arms while moving, thereby further improving the user's sense of realism.

[0102] Furthermore, the sound emitted in the movement method designated by the user may be a breathing sound or a heartbeat sound, and the breathing sound or heartbeat sound accompanying the movement may be fed back to the user of the operation terminal 4. For example, the user sets the time interval of the breathing sound or heartbeat sound during the user's movement in the processing device 3 or the robot 2a. In this case, too, as described in the first embodiment, the timing for reproducing the breathing sound or heartbeat sound may be specified by the processing device 3, or as described in the second embodiment, a movement sound generating device may generate the breathing sound or heartbeat sound near the mouth or heart of the robot 2a, and the operation terminal 4 may reproduce the spatial sound data collected by a microphone. The breathing sound or heartbeat sound during the user's movement and its time interval may be designated by the user, or may be set based on the user's attributes such as age and weight, and may further take into consideration the attributes of the space P in which the robot 2 is installed.

[0103] The breathing sound or heartbeat sound is fed back to the operation terminal 4 at the time intervals when the breathing sound or heartbeat sound occurs while the user is moving, thereby further improving the sense of realism for the user.

[0104] Furthermore, the sound generated by the movement method specified by the user may be wind noise, and the wind noise generated by the movement may be fed back to the user of the operation terminal 4. For example, the user may set the wind noise to be generated when the user moves in the processing device 3 or the robot 2a. This wind noise is the sound of wind blowing on the clothes worn by the user. The wind noise may also be set for each movement condition, such as the movement speed or movement direction. In this case, as described in the first embodiment, the processing device 3 may specify the timing for generating the wind noise as when the robot 2 is moving. As described in the second embodiment, the movement sound generating device may generate wind noise in the robot 2a, and the operation terminal 4 may play back the spatial sound data collected by the microphone. The timing for generating the wind noise may be specified by the user, or the attributes of the space P in which the robot 2 is installed may be taken into consideration.

[0105] The wind noise that occurs when the user moves around in the space P where the robot 2 is installed is fed back to the operation terminal 4, further enhancing the user's sense of presence.

[0106] The processing device 3 of the present embodiment described above is, for example, a general-purpose computer system including a CPU (Central Processing Unit, processor) 901, a memory 902, a storage 903 (HDD: Hard Disk Drive, SSD: Solid State Drive), a communication device 904, an input device 905, and an output device 906. In this computer system, the CPU 901 executes a program loaded on the memory 902, thereby realizing each function of the processing device 3.

[0107] The processing device 3 may be implemented by one computer or by multiple computers, or may be a virtual machine implemented in a computer.

[0108] The program of the processing device 3 can be stored in a computer-readable recording medium such as a HDD, SSD, USB (Universal Serial Bus) memory, CD (Compact Disc), DVD (Digital Versatile Disc), or can be distributed via a network. The computer-readable recording medium is, for example, a non-transitory recording medium.

[0109] The present disclosure is not limited to the above-described embodiments, and various modifications are possible within the scope of the present disclosure.

[0110] REFERENCE SIGNS LIST 1 Remote operation system 2 Robot 3 Processing device 4 Operation terminal 21 Control unit 22 Artificial foot unit 23 Artificial ear unit 24 Footstep generation device 25 Microphone 31 Operation unit 32 Acquisition unit 33 Transmission unit 35 Generation unit 41 Instruction unit 42 Playback unit 51 Contact unit 52 Rotation axis 53 Pin 54 Support rod 55 Motor 901 CPU 902 Memory 903 Storage 904 Communication device 905 Input device 906 Output device P Space

Claims

1. (delete)

2. (delete)

3. (delete)

4. (delete)

5. A robot that moves through space according to instructions from a user via an operating terminal; an operation terminal for inputting a movement instruction to the robot; a processing device connected to the robot and the operation terminal, The robot The device has a pseudo foot portion at a position corresponding to a human foot and a pseudo ear portion at a position corresponding to a human ear, the artificial foot unit has a moving unit for moving the robot, and a moving sound generating device that generates a sound that is generated when the robot moves in a predetermined moving manner in response to the movement of the robot, The artificial ear unit has a microphone that collects sounds in a space where the robot is located and generates spatial sound data, The processing device includes: an acquisition unit that acquires the spatial sound data; a transmitting unit that transmits the spatial sound data to the operation terminal, The operation terminal a playback unit that plays back the spatial sound data Remote control system.

6. The movement sound generating device generates footstep sounds by walking, and reproduces sound data of footsteps every time the amount of movement of the robot from the start of movement reaches a natural number multiple of a pre-specified stride length. The remote control system according to claim 5 .

7. The moving sound generating device reproduces footstep sound data when the robot stops. The remote control system according to claim 6.

8. The moving sound generating device has a contact part that can come into contact with a floor, and the contact part comes into contact with the floor every time the amount of movement of the robot from the start of movement reaches a predetermined stride length. The remote control system according to claim 5 .

9. The moving sound generating device causes the contact portion to contact the floor when the robot stops. The remote control system according to claim 8.

10. A robot that moves through space according to instructions from a user via an operation terminal, The device has a pseudo foot portion at a position corresponding to a human foot and a pseudo ear portion at a position corresponding to a human ear, the artificial foot unit has a moving unit for moving the robot, and a moving sound generating device that generates a sound that is generated when the robot moves in a predetermined moving manner in response to the movement of the robot, The artificial ear unit has a microphone that collects sounds in the space where the robot is located and generates spatial sound data. robot.

11. the pre-specified movement method includes a stride length of the user, The movement sound generating device generates footsteps according to the movement amount and the stride length of the robot. The remote control system according to claim 5 .

12. The sound generated by the movement method is the sound of clothes rubbing against each other when swinging arms, The moving sound generating devices are provided at positions corresponding to the right and left sides of a person. The remote control system according to claim 5 .

13. The sound generated by the movement method is a breathing sound, The moving sound generating device is provided at a position corresponding to a human mouth. The remote control system according to claim 5 .

14. The sound generated by the movement method is a heartbeat sound, The moving sound generating device is provided at a position corresponding to the human heart. The remote control system according to claim 5 .

15. The sound generated by the movement method is wind noise, The movement sound generating device generates wind noise on the robot. The remote control system according to claim 5 .