Movable apparatus and information processing method

Abstract

A movable apparatus that is provided in a reality space includes: one or more sensors; an acquisition section that acquires parameter information of a virtual representation relating to at least one of a corresponding virtual object that corresponds to the movable apparatus and is provided in a virtual space or a corresponding virtual space that corresponds to the reality space, and sound-related data corresponding to the parameter information; and an output controller that controls output of a sound corresponding to the virtual representation by using the acquired sound-related data on the basis of the acquired parameter information and sensing data from the one or more sensors.

Description

MOVABLE APPARATUS AND INFORMATION PROCESSING METHODCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of Japanese Priority Patent Application JP 2024-151409 filed September 3, 2024, the entire contents of which are incorporated herein by reference.

[0002] The present technology relates to a movable apparatus such as an autonomous mobile robot, and to an information processing method.

[0003] Patent Literature 1 discloses the technology that allows capturing of images suitable for image processing for, for example, recognizing a target object existing in the vicinity using an example of a dog-like robot capable of quadrupedal walking.

[0004] Patent Literature 2 discloses an entertainment system capable of providing a new environment in which humans interact with robots.

[0005] Japanese Patent Application Laid-open No. 2021-177582

[0006] WO Publication No. 2019 / 116521Summary

[0007] The development of movable apparatuses such as autonomous mobile robots has been advanced, and the movable apparatuses have been used as entertainment robots, for example. There is a demand for the technology capable of providing a high-quality user experience to a user who uses such movable apparatuses.

[0008] In view of the circumstances as described above, it is desirable to provide a movable apparatus and an information processing method that are capable of providing a high-quality user experience to a user.

[0009] According to an embodiment of the present technology, there is provided an apparatus capable of motion, the apparatus comprising: one or more sensor; processing circuitry configured to: acquire parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to the apparatus and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquire sound-related data corresponding to the parameter information; and control output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from the one or more sensor.

[0010] According to an embodiment of the present technology, there is provided a method, comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus.

[0011] According to an embodiment of the present technology, there is provided a non-transitory computer readable medium storing instructions which when executed by a computer cause the computer to perform a method, the method comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus.

[0012] Fig. 1 is a schematic diagram showing a configuration example of an entertainment system according to an embodiment of the present technology.

[0013] Fig. 2 is a block diagram showing a configuration example of a real robot.

[0014] Fig. 3 is a flowchart showing a basic flow of output of a sound corresponding to a virtual representation.

[0015] Fig. 4 is a flowchart showing a processing example in a user terminal (application), for outputting a sound corresponding to a virtual item.

[0016] Fig. 5 is a schematic view showing a configuration example of a selection screen for shoes.

[0017] Fig. 6A are schematic diagrams each showing an example of information that shoes have been attached, which is recorded in a DB.Fig. 6B are schematic diagrams each showing an example of information that shoes have been attached, which is recorded in a DB.

[0018] Fig. 7 is a flowchart showing a processing example in the real robot, for outputting a sound corresponding to a virtual item.

[0019] Fig. 8 is a schematic view showing an example of a map corresponding to a reality space.

[0020] Fig. 9 is a flowchart showing a processing example in the user terminal (application), for outputting a sound corresponding to a virtual object.

[0021] Fig. 10 is a flowchart showing a processing example in the real robot, for outputting a sound corresponding to a virtual object.

[0022] Figs. 11A are schematic views each showing an example of outputting a sound corresponding to a proficiency level.Figs. 11B are schematic views each showing an example of outputting a sound corresponding to a proficiency level.

[0023] Fig. 12 is a block diagram showing a hardware configuration example of a computer that can be used for establishing an entertainment system according to an embodiment of the present technology.

[0024] Hereinafter, embodiments according to the present technology will be described with reference to the drawings.Outline of Entertainment System

[0025] The outline of an entertainment system according to an embodiment of the present technology will be described. An entertainment system to be described below corresponds to an embodiment of an information processing system according to the present technology.

[0026] Fig. 1 is a schematic diagram showing a configuration example of an entertainment system 1. As shown in Fig. 1, the entertainment system 1 includes a robot 2, a user terminal 3, and a database (DB) 4.

[0027] The robot 2, the user terminal 3, and the DB 4 each include hardware necessary for a computer, for example, a processor such as a CPU, a GPU, or a DSP, a memory such as a ROM or a RAM, and a storage device such as a hard disk drive (HDD), as a basic configuration (see Fig. 12). The processor loads a program according to one embodiment of the present technology, which is stored in the storage or the memory, to the RAM and executes the program, so that an information processing method according to one embodiment of the present technology (a method of controlling a movable apparatus) is executed.

[0028] As shown in Fig. 1, the robot 2, the user terminal 3, and the DB 4 are communicably connected to each other via a network 5. The network 5 is established by, for example, the Internet or a wide area communication network. In addition thereto, any wide area network (WAN), local area network (LAN), or the like may be used, and a protocol for establishing the network 5 is not limited.

[0029] Further, in order to achieve the connection capable of communication between the devices shown in Fig. 1, any wireless communication technology, for example, wireless LAN communication such as Wi-Fi, short-distance wireless communication such as Bluetooth (registered trademark), and mobile communication such as Long Term Evolution (LTE), can be used.

[0030] The robot 2 is configured as an entertainment robot capable of automated moving. As shown in Fig. 1, in this embodiment, a dog-like robot 2 capable of quadrupedal walking is used.

[0031] A user 6 can communicate with the robot 2 in various manners in a reality space RS and can have a highly enjoyable user experience. For example, the user 6 touches the robot 2 or talks to the robot 2. In response to this, the robot 2 can behave in a manner such as feeling delight or respond by barking.

[0032] Further, the robot 2 can also walk or run freely in the reality space RS. For example, the robot 2 can also follow the user 6 or play chase with the user 6.

[0033] The user terminal 3 is a terminal used by the user 6 and has, for example, an imaging function, an image displaying function, and an input receiving function. The imaging function can be implemented, for example, by mounting an imaging device such as a digital camera. The displaying function can be implemented, for example, by mounting a display device including liquid crystal, EL, or the like. The input receiving function can be implemented, for example, by mounting an operation device such as a keyboard, a pointing device, or a touch panel.

[0034] In the example shown in Fig. 1, a smartphone is used as the user terminal 3. The imaging function is implemented by a digital camera of the smartphone (illustration is omitted). Further, the image displaying function and the input receiving function are implemented by a touch panel 7 of the smartphone. Other computers such as a tablet terminal and a laptop PC can also be used as the user terminal 3.

[0035] The DB 4 is provided as a cloud-type DB 4. In order to provide the cloud-type DB 4, a computer such as a DB server may be appropriately used.

[0036] In the DB 4, various types of information and data regarding this entertainment system 1 are stored. For example, information regarding the robot 2, information regarding the user 6, information regarding the user terminal 3, and the like are stored in the DB 4. Further, parameter information of a virtual representation, sound-related data corresponding to parameter information, which will be described later, and the like are stored in the DB 4.

[0037] (Cooperation of Reality Space RS and Virtual Space VS) In this entertainment system 1, the user 6 can also communicate with a virtual robot 8 in a virtual space VS via the user terminal 3.

[0038] For example, the user 6 downloads an application (application program) to use this entertainment system 1 to the user terminal 3. For example, the user 6 inputs information such as an ID or a password to create an account for using the entertainment system 1. The account information is stored in the DB 4 as user information of the user 6.

[0039] The user 6 activates a registering function in the application to register the robot 2 that spends time together in the reality space RS. For example, registration of the robot 2 is performed by, for example, registering device information such as a model number of the robot 2 or reading coat information attached to the robot 2. The information of the registered robot 2 is stored in the DB 4.

[0040] Upon completion of the registration of the robot 2, as shown in Fig. 1, a virtual robot 8 corresponding to the robot 2 provided in the reality space RS (hereinafter, described as real robot 2) is displayed on the touch panel 7 of the user terminal 3. The user 6 can communicate in various manners with the virtual robot 8 displayed on the user terminal 3.

[0041] For example, it is assumed that the user 6 is at home with the real robot 2. When the user 6 goes outside, the user 6 activates the application of this entertainment system 1 by using the user terminal 3.

[0042] This allows the user 6 to stroke or talk to the virtual robot 8 displayed on the user terminal 3. Further, it is also possible to give treats or food to the virtual robot 8 in the virtual space VS.

[0043] The information regarding various communications performed by the user 6 to the virtual robot 8 is stored in the DB 4. This information regarding various communications is appropriately downloaded to and shared with the real robot 2.

[0044] For example, the real robot 2 may request the DB 4 to transmit the information regarding communications with the virtual robot 8 at a predetermined timing or on the basis of a predetermined trigger. Alternatively, the information regarding communications may be transmitted from the DB 4 to the real robot 2 at a predetermined timing or on the basis of a predetermined trigger.

[0045] This makes it possible to share the results of various communications, performed on the virtual robot 8 by the user 6, with the real robot 2 in this entertainment system 1. For example, it is assumed that the user 6 strokes the virtual robot 8 while on the go, and the closeness of the virtual robot 8 to the user 6 increases. In this case, it is also feasible that the increase in closeness is reflected in the real robot 2, and the real robot 2 performs a behavior corresponding to the improved closeness when the user 6 comes back home.

[0046] Further, it is assumed that the user 6 gives food to the virtual robot 8 while on the go, and the satiety level of the virtual robot 8 increases (the hunger level decreases). In this case, it is also possible that the increase in satiety level is reflected in the real robot 2, and the real robot 2 performs a behavior as being full up when the user 6 comes back home.

[0047] In such a manner, this entertainment system 1 makes it possible to cause the communication with the real robot 2 in the reality space RS and the communication with the virtual robot 8 in the virtual space VS to cooperate with each other, and to provide a very high-quality user experience.

[0048] In this embodiment, the real robot 2 corresponds to one embodiment of a movable apparatus provided in a reality space according to the present technology, and to one embodiment of a mobile apparatus capable of automated moving according to the present technology. Further, the virtual robot 8 corresponds to one embodiment of a corresponding virtual object that corresponds to the movable apparatus and is provided in a virtual space (a corresponding virtual object provided in a virtual space to correspond to the movable apparatus provided in the reality space).

[0049] Fig. 2 is a block diagram showing a configuration example of the real robot 2. As shown in Fig. 2, the real robot 2 includes a sensor section 9, a speaker 10, a light source section 11, a display 12, and a switch 13. The real robot 2 also includes a controller 14, a drive section 15, storage 16, a communication section 17, and a power supply section 18.

[0050] The sensor section 9 includes one or more sensors and includes any sensor capable of detecting information (data) regarding an environment around the real robot 2 and information (data) regarding the state of the real robot 2. For example, any sensor such as an image sensor (camera), a ranging sensor, a touch sensor, a microphone, a GPS sensor, a GNSS sensor, an inertial measurement unit (IMU) sensor, a compass, a human detection sensor, an illuminance sensor, a temperature sensor, or a humidity sensor may be mounted.

[0051] As the image sensor (camera), for example, a complementary metal-oxide semiconductor (CMOS) sensor or a charge coupled device (CCD) sensor is used. Further, an infrared ray camera or the like may be mounted.

[0052] As the ranging sensor, for example, various types of ranging sensors such as an optical-laser ranging sensor, an ultrasonic ranging sensor, a stereo camera, a time-of-flight (ToF) sensor, LiDAR(Light Detection and Ranging, Laser Imaging Detection and Ranging), and a structured-light ranging sensor can be used.

[0053] As the touch sensor, for example, various types of touch sensors such as a pressure-sensitive touch sensor and a capacitive touch sensor can be used.

[0054] The specific configuration of the microphone, the GPS sensor, the GNSS sensor, the IMU sensor, the compass, the human detection sensor, the illuminance sensor, the temperature sensor, the humidity sensor, or the like is not limited, and any configuration may be adopted. As a matter of course, other sensors may be mounted.

[0055] In this embodiment, a position sensor section, a motion sensor section, and a landing sensor section are provided as the sensor section 9. The position sensor section is configured to detect data regarding the position of the real robot 2 in the reality space RS. For example, the position sensor section can be implemented by an image sensor, a ranging sensor, a GPS sensor, a GNSS sensor, or the like. As a matter of course, other devices may be used.

[0056] The motion sensor section is configured to detect data regarding the motion of the real robot 2. For example, the motion sensor section can be implemented by an IMU sensor, a compass, or the like. As a matter of course, other devices may be used.

[0057] The landing sensor section is configured to detect the landing of four paws 19 (19a to 19d) of the real robot 2 onto the ground. For example, switches are arranged at the leading edges of the four paws 19 of the real robot 2 (e.g., the soles of the feet), so that the landing of each paw 19 can be detected. Further, it is also possible to arrange ranging sensors at the leading edges of the four paws 19 and detect the landing of each paw 19 on the basis of the detected distance to the ground. Furthermore, it is also possible to detect the landing of each paw 19 on the basis of a detection result of the IMU sensor.

[0058] The speaker 10 can output sounds, and the specific configuration thereof is not limited. The speaker 10 corresponds to one embodiment of a sound output section according to the present technology.

[0059] The light source section 11 is configured by, for example, an LED and provided to each part of the real robot 2. Light emission of the light source section 11 allows presentation of various information to the user 6. Further, the light emission of the light source section 11 also allows expression of feelings such as delight, anger, sorrow, and pleasure. Furthermore, the light emission of the light source section 11 also allows indication of the remaining amount of power supply (battery) life, a connection state of the network, or the like.

[0060] The display 12 is configured by, for example, liquid crystal or EL and can display various images. In this embodiment, a small display is disposed at an eye position of the real robot 2. For example, various eye images are displayed by the display 12, so that various facial expressions can be expressed.

[0061] For example, a self-light-emitting display device (OLED) can also be disposed as the display 12 in each of the right and left eyes of the real robot 2. The OLED is disposed, so that various representations regarding eyes, such as eye blink, the white part of the eye, the iris and pupil, and the motion of the iris and pupil, can be achieved in each of the right and left eyes. Further, arranging lenses or cover glass to cover the OLEDs makes it possible to three-dimensionally configure the right and left eyes.

[0062] For example, arranging lenses to cover the OLEDs makes it possible to deflect the light so as to widen display by the OLEDs. Further, the lenses can also be used to express the spherical shape of eyeballs. Furthermore, arranging cover glasses to cover the lenses makes it possible to form curved surfaces that are continuous with the surface of the head of the real robot 2 and to present a smooth feel when the user 6 touches the eyes and their surrounding area of the real robot 2.

[0063] As the switch 13, for example, a power supply button, a sound volume button, and a switch for wireless communication connection are disposed. As a matter of course, the switch 13 is not limited to those switches.

[0064] The drive section 15 is a mechanism for implementing the motion of the real robot 2, such as quadrupedal walking. The drive section 15 includes, for example, drive units disposed at the respective joints of the real robot 2. The drive unit includes a motor that performs a rotation operation around the axis, an encoder that detects a rotation position of the motor, and a driver that adaptively controls the rotation position and rotation speed of the motor on the basis of the output of the encoder.

[0065] For example, the drive units are disposed such that each of the head, mouth, neck, front paws, back paws, tail, waist, ears, etc. of the real robot 2 can move in one or more axes. When each drive unit operates appropriately, the real robot 2 can perform various motions including quadrupedal walking.

[0066] The storage 16 is, for example, a storage device such as a nonvolatile memory. For example, an HDD, a solid-state drive (SSD), or the like is used. Any other non-transitory computer-readable storage medium may be used.

[0067] The communication section 17 is a module for executing network communication, short-distance wireless communication, mobile communication, or the like with other devices. For example, a wireless LAN module for Wi-Fi or the like, a communication module for Bluetooth (registered trademark) or the like, or a mobile communication module for LTE or the like is provided.

[0068] The power supply section 18 supplies power to the sections of the real robot 2. The power supply section 18 includes, for example, a rechargeable battery and a charge / discharge controller that manages the state of charge / discharge of the rechargeable battery.

[0069] The controller 14 controls the operation of each block of the real robot 2. The controller 14 includes hardware necessary for the computer, for example, a processor, a memory, and a storage device.

[0070] As the controller 14, for example, a programmable logic device (PLD) such as a field programmable gate array (FPGA) and other devices such as an application specific integrated circuit (ASIC) may be used.

[0071] In this embodiment, the controller 14 executes a program according to an embodiment of the present technology, so that a recognition section 20, a recognized information management section 21, an action controller 22, a drive controller 23, a sound output controller 24, a parameter information acquisition section 25, and a sound-related data acquisition section 26 are implemented as functional blocks. In order to implement the functional blocks, dedicated hardware such as an integrated circuit (IC) may be appropriately used.

[0072] The program is installed on the real robot 2, for example, via various recording media. Alternatively, the program may be installed via the Internet or the like. The types of recording media on which the program is recorded, and the like are not limited, and any computer-readable recording media may be used. For example, any non-transitory computer-readable storage media may be used.

[0073] The recognition section 20 performs recognition processing on the basis of a detection result of the sensor section 9 and recognizes the environment around the real robot 2. For example, the recognition section 20 performs human recognition processing, speech recognition processing, spatial recognition processing, or the like.

[0074] The human recognition processing makes it possible to recognize the presence or absence of humans in a surrounding space on the basis of, for example, image information or speech information detected by the sensor section 9. Further, the human recognition processing also makes it possible to identify a person located in a surrounding space and to recognize the line of sight, a facial expression, or the like of a person whose presence has been recognized.

[0075] The speech recognition processing makes it possible to recognize the contents (context) of speech or conversation of the user 6 located in a surrounding space on the basis of, for example, the speech information detected by the sensor section 9.

[0076] The spatial recognition processing makes it possible to perform obstacle recognition, shape recognition (e.g., wall recognition or floor recognition), or object recognition and recognize objects or shapes in a surrounding space on the basis of, for example, the image information, distance information, or the like detected by the position sensor section of the sensor section 9. Further, the spatial recognition processing also makes it possible to recognize characteristics of a route, such as a width, a distance, and surface roughness of a passable route to a destination.

[0077] The specific algorithm of the recognition processing by the recognition section 20 is not limited, and any algorithm may be used. For example, any machine learning algorithm using deep neural network (DNN), recurrent neural network (RNN), convolutional neural network (CNN), or the like may be used. For example, recognition processing of a surrounding environment can be performed with high accuracy using artificial intelligence (AI) or the like that performs deep learning.

[0078] Note that the application of the machine learning algorithm may be performed on any processing in the present disclosure. In other words, processing using machine learning may be performed on any processing described in the present disclosure. As a matter of course, any algorithm based on rules can also be used without being limited to the machine learning algorithm.

[0079] The recognized information management section 21 manages each piece of information recognized by the recognition section 20 to be converted into information suitable to be used in other blocks such as the action controller 22.

[0080] For example, the recognized information management section 21 can determine the presence or absence of detection of the user 6 in the surrounding space or can calculate the degree of attention of the user 6 to the real robot 2. The degree of attention of the user 6 to the real robot 2 can be used to determine whether or not an affection expression that lets the user 6 have affection to the real robot 2 is performed, for example.

[0081] Further, in this embodiment, the recognized information management section 21 can generate a map of the surrounding space on the basis of a recognition result of the surrounding space by the recognition section 20. For example, the recognized information management section 21 can generate an obstacle map or movement region map indicating regions that can be passed through. Further, the recognized information management section 21 can also generate an object map showing the locations of various objects in the surrounding space, and a topological map showing the name, relevance, or significance of each region.

[0082] The specific algorithm for generating the map of the surrounding space is not limited, and any algorithm such as simultaneous localization and mapping (SLAM) may be used. Any other self-location estimation algorithms may be used.

[0083] In such a manner, in this embodiment, the recognition section 20 and the recognized information management section 21 can generate a map corresponding to the reality space RS on the basis of the sensing data by the sensor section 9 (one or more sensors). The generated map is stored in the DB 4 and downloaded to the user terminal 3. The generated map is then displayed on the touch panel 7 of the user terminal 3 as a corresponding virtual space that corresponds to the reality space RS.

[0084] The action controller 22 determines the action to be performed by the real robot 2 on the basis of the recognized information generated by the recognized information management section 21 or various types of parameter information showing the states of the real robot 2 acquired from the DB 4. In other words, the action controller 22 determines what motion the real robot 2 makes and what sound the real robot 2 outputs. Further, the action controller 22 controls light emission by the light source section 11 or display of various eye images by the display 12 disposed on the eye position.

[0085] Note that the motion of the real robot 2 to the user 6 and the output of a sound can also be an expression of the real robot to the user 6. For example, the motion of the real robot 2 to the user 6 can also be represented as a motion expression. Further, the output of a sound can also be represented as a sound expression.

[0086] The action controller 22 can also be an expression controller that controls the state of the real robot 2 to perform expression control of a sound or a motion.

[0087] For example, the action controller 22 can cause the real robot 2 to walk or run to a destination. In such a case, the action controller 22 can determine a route for the real robot 2 to move to the destination on the basis of the map generated by the recognized information management section 21.

[0088] Further, the action controller 22 can determine whether to perform various emotional expressions that let the user 6 feel an affection, and can cause the real robot 2 to perform various motions for achieving emotional expressions. For example, the action controller 22 can cause the real robot 2 to perform various motions such as jumping on the moment, wagging its tail, opening and closing its mouth, moving the ears, making a posture of giving its paw, making a posture of lying down, looking away, making a motion of eating, dancing, barking, singing, and growling. In addition, the action controller 22 can determine various actions of the real robot 2.

[0089] The drive controller 23 outputs a control command to perform the action determined by the action controller 22 to the drive section 15. For example, the drive unit disposed at each joint of the real robot 2 is driven on the basis of the control signal outputs from the drive controller 23 to achieve various motions such as quadrupedal walking.

[0090] The sound output controller 24 controls the output of various sounds by the speaker 10. For example, the sound output controller 24 acquires sound data (e.g., audio files such as wave) from the DB 4 or the like and causes the speaker 10 to output it.

[0091] Further, in this embodiment, the sound output controller 24 can dynamically generate sound data (audio files). For example, the sound output controller 24 can dynamically generate sound data by synthesizing a plurality of pieces of sound data or by increasing or decreasing a predetermined frequency component in predetermined sound data.

[0092] For example, the sound output controller 24 downloads sound setting information for dynamically generating sound data from the DB 4 or the like. Sound data may be generated on the basis of the downloaded sound setting information. Further, for example, sound data may be generated by processing or synthesizing the sound data stored in advance in the storage 16 to output a sound from the speaker 10.

[0093] In this embodiment, the sound data or sound setting information is contained in the sound-related data according to one embodiment of the present technology. In other words, the sound-related data according to one embodiment of the present technology may be a sound waveform itself or may be parameters of a synthesizer for generating sounds or data written for filtering applied to a sound waveform. The parameters for generating sounds correspond to, for example, the degree of application of filter, algorithms, changes in pitch, and the like.

[0094] The parameter information acquisition section 25 and the sound-related data acquisition section 26 will be described later.

[0095] (Output of Sounds Corresponding to Virtual Representations) As described above, in this entertainment system 1, it is possible to coordinate the communication with the real robot 2 in the reality space RS and the communication with the virtual robot 8 in the virtual space VS.

[0096] Regarding the coordination between the reality space RS and the virtual space VS, this entertainment system 1 makes it possible to achieve the output of sounds corresponding to virtual representations and provide a very high-quality user experience. The output of sounds corresponding to virtual representations will be described below.

[0097] Fig. 3 is a flowchart showing a basic flow of the output of sounds corresponding to virtual representations. The parameter information acquisition section 25 acquires parameter information of a virtual representation relating to at least one of the virtual robot 8 that corresponds to the real robot 2 and is provided in the virtual space VS, or a corresponding virtual space (map) that corresponds to the reality space RS (Step 101).

[0098] The parameter information of a virtual representation relating to the virtual robot 8 includes, for example, any parameter information for achieving various virtual representations relating to the virtual robot 8 displayed on the touch panel 7 of the user terminal 3. For example, information regarding the state of the virtual robot 8, information regarding a virtual item attached to the virtual robot 8, a proficiency level of the virtual robot 8 with respect to a predetermined target to be learned, and the like are acquired as parameter information of virtual representations.

[0099] The parameter information of a virtual representation relating to a corresponding virtual space (map) includes, for example, any parameter information for achieving various virtual representations relating to the map displayed on the touch panel 7 of the user terminal 3. For example, information regarding a virtual object arranged on the map, information regarding a region with a predetermined attribute set for the map, and the like are acquired as parameter information of virtual representations.

[0100] Note that the acquisition of data (information) in the present disclosure includes all of receiving the data (information) from another apparatus or another functional block, downloading the data (information) from the DB or the like, generating the data (information) by executing a predetermined algorithm by itself, and reading the data (information) stored in advance in the storage or the like of its own apparatus.

[0101] The sound-related data acquisition section 26 acquires sound-related data corresponding to the parameter information acquired in Step 101 (Step 102).

[0102] The action controller 22 and the sound output controller 24 control the output of a sound corresponding to a virtual representation by using the sound-related data acquired in Step 102 on the basis of the parameter information acquired in Step 101 and the sensing data by the sensor section 9 (one or more sensors).

[0103] In this embodiment, the action controller 22 and the sound output controller 24 achieve an output controller according to one embodiment of the present technology. Note that the recognition section 20 and the recognized information management section 21 can also be regarded as part of the output controller according to one embodiment of the present technology.

[0104] In other words, the action controller 22 determines the action of the real robot 2, including the output of a sound corresponding to a virtual representation, on the basis of a recognition result by the recognition section 20 and the recognized information generated by the recognized information management section 21. The sound output controller 24 then outputs a sound corresponding to a virtual representation from the speaker 10. In this case, it is also possible to say that the recognition section 20, the recognized information management section 21, the action controller 22, and the sound output controller 24 control the output of a sound corresponding to a virtual representation from the speaker 10.

[0105] Hereinafter, detailed embodiments of the output of a sound corresponding to a virtual representation will be described. (First Embodiment: Output of Sound Corresponding to Virtual Item)

[0106] In this entertainment system 1, it is possible to output a sound corresponding to a virtual item attached to the virtual robot 8 as the output of a sound corresponding to a virtual representation.

[0107] Fig. 4 is a flowchart showing a processing example in the user terminal 3 (application), for outputting a sound corresponding to a virtual item.

[0108] The user 6 selects shoes to be attached to the virtual robot 8 in the application (Step 201). For example, the user 6 activates a virtual item attaching function in the application to display a selection screen (GUI) of the shoes to be attached to the virtual robot 8.

[0109] Fig. 5 is a schematic view showing a configuration example of a selection screen 27 of shoes. In the example shown in Fig. 5, the virtual robot 8 is displayed at substantially the center of the touch panel 7 of the user terminal 3. On the lower side of the virtual robot 8, shoes with bells 28a, wooden shoes 28b, and boa slippers 28c are displayed as options of shoes to be attached.

[0110] The user 6 can appropriately select a left front paw 29a, a right front paw 29b, a left back paw 29c, and a right back paw 29d of the virtual robot 8, and select shoes 28 to be attached to the respective paws 29. It is possible to put the same type of shoes 28 on the four paws 29 of the virtual robot 8, and it is also possible to put different types of shoes 28 on the respective paws 29.

[0111] The shoes 28 that can be attached are not limited to the shoes with bells 28a, the wooden shoes 28b, and the boa slippers 28c, and any shoes such as high heels and Japanese sandals can be attached. Further, the mode in which the user 6 purchases shoes 28 can also be adopted.

[0112] The information that the shoes have been attached to the virtual robot 8 is recorded in the cloud DB 4 (Step 202).

[0113] Figs. 6A and 6B are schematic diagrams each showing an example of information that the shoes have been attached, which is recorded in the DB 4. In this embodiment, attachment data 31 shown in Fig. 6A and item data 32 shown in Fig. 6B are recorded in the DB 4 as the information that the shoes have been attached.

[0114] As shown in Fig. 6A, the attachment data 31 includes the following information.

[0115] "Individual" c Information for identifying the real robot 2. In Fig. 6A, the characters of "Robot A" are shown, but any information with which the real robot 2 can be identified, such as a model number, an ID, or the like of the real robot 2, may be used.

[0116] "Site" c A site of the virtual robot 8 to which a virtual item is attached. In this embodiment, information of the four paws 29 of the virtual robot 8 are stored.

[0117] "Type of item" c Information for identifying a virtual item. In this embodiment, information of the three types of shoes 28 are stored as "Shoe A", "Shoe B", and "Shoe C". "Shoe A" corresponds to the shoes with bells 28a shown in Fig. 5, and "Shoe B" corresponds to the wooden shoes 28b. Further, "Shoe C" corresponds to the boa slippers 28c. As the information of the shoes 28, any information with which the shoes 28 can be identified may be used. Further, for the "Site" to which no virtual item is attached, "Type of item" is left blank.

[0118] In the attachment data 31 shown in Fig. 6A, for the same virtual robot 8 (= "Robot A"), "Shoe A" is attached to the left front paw 29a, and "Shoe B" is attached to the right front paw 29b. Further, "Shoe C" is attached to the left back paw 29c, and no shoe is attached to the right back paw 29d.

[0119] As shown in Fig. 6B, the item data 32 includes the following information.

[0120] "Type of item" c Information for identifying a virtual item (typically, the same data as "Type of item" of the attachment data 31).

[0121] "Type of sound" c Type of sound corresponding to an attached item. In "Type of sound", information of the type of sound that is output when a virtual item is attached is stored. In this embodiment, "sound of bells" is stored for "Shoe A", and "sound of wooden shoes" is stored for "Shoe B". Further, "sound of boa slippers" is stored for "Shoe C".

[0122] As described above, in this embodiment, sounds to be reproduced are associated with the respective shoes 28 in advance, and the information thereof is stored in the DB 4.

[0123] The information that the shoes have been attached (the attachment data 31 and the item data 32), shown in Figs. 6A and 6B, corresponds to parameter information of virtual representations relating to the virtual robot 8.

[0124] Referring back to Fig. 4, the virtual robot 8 in the application is displayed with the shoes selected by the user 6 being attached thereto (Step 203). In the example shown in Figs. 6A and 6B, the shoe with bells 28a is attached to the left front paw 29a of the virtual robot 8, and the wooden shoe 28b is attached to the right front paw 29b. Further, the boa slipper 28c is attached to the left back paw 29c of the virtual robot 8, and no shoe is attached to the right back paw 29d. The virtual robot 8 is displayed in such a state on the touch panel 7 of the user terminal 3.

[0125] Fig. 5 shows the state in which the shoes 28 are not yet selected by the user 6. When the user 6 selects shoes 28 for the respective paws 29, the virtual robot 8 is displayed with the selected shoes 28 being attached to the respective paws 29.

[0126] In the application, sounds corresponding to the attached shoes 28 are reproduced according to walking of the virtual robot 8. For example, the sound of bells (e.g., "ting-a-ling"), the sound of wooden shoes (e.g., "clack, clack, clack"), and the sound of boa slippers (e.g., "pitter-patter") are reproduced according to the landing of each paw 29 of the virtual robot 8 in the application.

[0127] Fig. 7 is a flowchart showing a processing example in the real robot 2, for outputting a sound corresponding to a virtual item.

[0128] The parameter information acquisition section 25 of the real robot 2 downloads the information that the shoes have been attached, which has been recorded in the DB 4, from the DB 4 (Step 301). Downloading of the information that the shoes have been attached in Step 301 corresponds to one embodiment of the acquisition of parameter information of a virtual representation in Step 101 shown in Fig. 3.

[0129] The sound-related data acquisition section 26 of the real robot 2 determines whether or not there is sound-related data corresponding to the attached shoe 28 (Step 302). In this embodiment, the sound-related data acquisition section 26 refers to the information that the shoes have been attached shown in Figs. 6A and 6B. The sound-related data acquisition section 26 then determines whether or not there is sound-related data for outputting the "sound of bells" corresponding to the shoe with bells 28a, which has been attached to the left front paw 29a of the virtual robot 8.

[0130] Further, the sound-related data acquisition section 26 determines whether or not there is sound-related data for outputting the "sound of wooden shoes" corresponding to the wooden shoe 28b, which has been attached to the right front paw 29b of the virtual robot 8. Furthermore, the sound-related data acquisition section 26 determines whether or not there is sound-related data for outputting the "sound of boa slippers" corresponding to the boa slipper 28c, which has been attached to the left back paw 29c of the virtual robot 8.

[0131] In this embodiment, the sound-related data acquisition section 26 determines whether or not the sound-related data for outputting the "sound of bells", the sound-related data for outputting the "sound of wooden shoes", and the sound-related data for outputting the "sound of boa slippers" are stored in the DB 4.

[0132] If none of the sound-related data of the "sound of bells", the "sound of wooden shoes", and the "sound of boa slippers" is stored, it is determined No in Step 302, and the output of the sound corresponding to the virtual item is terminated.

[0133] If any of the sound-related data of the "sound of bells", the "sound of wooden shoes", and the "sound of boa slippers" is stored, it is determined Yes in Step 302, and the sound-related data stored in the DB 4 is downloaded (Step 303). In this embodiment, it is assumed that the sound-related data of each of the "sound of bells", the "sound of wooden shoes", and the "sound of boa slippers" is stored in the DB 4 and is downloaded by the sound-related data acquisition section 26.

[0134] Steps 302 and 303 correspond to one embodiment of the acquisition of the sound-related data in Step 102 shown in Fig. 3.

[0135] Note that the sound-related data may be stored in advance in the storage 16 of the real robot 2. In this case, the sound-related data of each of the "sound of bells", the "sound of wooden shoes", and the "sound of boa slippers" is read from the storage 16 by the sound-related data acquisition section 26.

[0136] The action controller 22 determines whether or not output conditions for outputting sounds corresponding to virtual representations in the real robot 2 are satisfied (Step 304). For example, if sounds corresponding to virtual representations are constantly being output, the user 6 may feel annoyed. In this embodiment, appropriately setting the output conditions makes it possible to eliminate such a problem.

[0137] The output conditions include the following conditions, for example: the sounds corresponding to the virtual representations are output for a certain period of time after the shoes 28 have been attached by the application, and are stopped after the lapse of the certain period of time. Further, the following conditions can also be set: the sounds corresponding to the virtual representations are output for a certain period of time after the activation of the application; and the sounds corresponding to the virtual representations are output for a limited period of time of a day (a desired time zone or a predetermined time zone).

[0138] Alternatively, it is also possible to set a condition where the sounds corresponding to the virtual representations are output in a predetermined area. For example, it is also possible to set a condition where sounds are output in the living room but sounds are not output in the bed room. Further, it is also possible to set a condition where sounds are not output when there is a particular person (e.g., a baby) nearby.

[0139] Further, the output conditions can also be set for the motion or state of the real robot 2. For example, it is also possible to set a condition where the sounds corresponding to the virtual representations are not output when the remaining amount of battery life is small (hungry state). Further, it is also possible to determine whether to output the sounds at random using a random number or the like.

[0140] If the output conditions for the sounds corresponding to the virtual representations are not satisfied, it is determined No in Step 304, and the output of the sound corresponding to the virtual item is terminated.

[0141] If the output conditions for the sounds corresponding to the virtual representations are satisfied, the output of the sound corresponding to the virtual representation is performed. In this embodiment, the processing proceeds from Step 304 to Step 305, and the action controller 22 and the sound output controller 24 output footsteps corresponding to the shoes 28 attached to the respective paws 29 of the virtual robot 8 in coordination with the landing of the paws 19 of the real robot 2 shown in Fig. 1.

[0142] In other words, the action controller 22 determines the action including the output of the footsteps corresponding to the shoes 28 attached to the respective paws 29 of the virtual robot 8 in coordination with the landing of the paws 19 of the real robot 2.

[0143] The sound output controller 24 generates the sound data of the footsteps corresponding to the shoes 28 attached to the respective paws 29 of the virtual robot 8 on the basis of the sound-related data downloaded from the DB 4. If the sound-related data is the sound data of footsteps, the sound data is used as it is. If the sound-related data is sound setting information, the sound data of footsteps is generated on the basis of the sound setting information.

[0144] The sound output controller 24 then causes the speaker 10 to output the footsteps corresponding to the shoes 28 attached to the respective paws 29 of the virtual robot 8 in coordination with the landing of the paws 19 of the real robot 2 on the basis of a detection result of a landing sensor section provided in the sensor section 9.

[0145] In this embodiment, the sound of bells is output in coordination with the landing of the left front paw 19a of the real robot 2 shown in Fig. 1, and the sound of wooden shoes is output in coordination with the landing of the right front paw 19b. Further, the sound of boa slippers is output in coordination with the landing of the left back paw 19c of the real robot 2.

[0146] Note that no shoe is attached to the right back paw 29d of the virtual robot 8, and thus no footsteps are output when the right back paw 19d of the real robot 2 has landed. As a matter of course, it may also be possible to set footsteps when no shoes are attached, and to output such footsteps in coordination with the landing of the paws 19 on which no shoes are attached.

[0147] In this embodiment, the walking of the real robot 2 and the landing of the paws 19 correspond to one embodiment of a predetermined motion of the movable apparatus according to the present technology and one embodiment of a moving motion of the movable apparatus according to the present technology. Note that in the present disclosure the walking also includes a motion of the real robot 2 that causes each paw 19 to land on the moment, and the motion is one embodiment of a moving motion of the movable apparatus.

[0148] As described above, in the first embodiment, the sound expression (operation sound) of the real robot 2 is changed depending on the virtual item attached to the virtual robot 8 in the virtual space SV. This makes it possible to attach items such as shoes, which are difficult to attach to the real robot 2, to the virtual robot 8 and possible for user 6 to feel a strong association between the real robot 2 and the virtual robot 8.

[0149] Note that it is also possible to change the sounds corresponding to the virtual representations to be reproduced, depending on the intensiveness of the landing, the frequency of the landing, moving speed, acceleration, and the like of the paws 19 of the real robot 2. For example, it is also possible to change the sounds corresponding to the virtual representations depending on the cases such as landing on the floor swiftly, walking slowly and dashing.

[0150] Further, it is possible to change the sounds corresponding to the virtual representations depending on the actual floor state in the reality space RV. Furthermore, it is also possible to change the sounds corresponding to the virtual representations depending on the situation of walking such as going up stairs.

[0151] Note that the sounds corresponding to the virtual representations may be changed in accordance with the elapse of time after the shoes 28 have been attached to the virtual robot 8 or other parameter information. For example, it is also possible to perform the processing of setting a predetermined period of time corresponding to the wear of the shoes 28 to stop reproducing the footsteps when the predetermined period of time has passed. Further, it is also possible to set parameter information such as the degree of wear and to perform processing of gradually diminishing the footsteps in accordance with the degree of wear.

[0152] As a matter of course, the virtual item is not limited to the shoes 28, and the present technology can also be applied to any virtual items including clothing, barrettes, sunglasses, musical instruments such as bells, and the like. For example, in the example shown in Fig. 5, the virtual robot 8 wears clothing 30 like a Santa Claus. A bell is attached as a virtual item to the clothing 30, the neck of the virtual robot 8, or the like. The sound of bell is then output in coordination with the walking of the real robot 2 (the landing of all the four paws 19). Such processing can also be performed.

[0153] If a bell or the like is attached to the neck of the virtual robot 8, the sound of bell may be output in coordination with the motion of the neck of the real robot 2.

[0154] Further, the sound of bells that is heard generally on Christmas Day, or the like may be output in coordination with the walking of the real robot 2 in response to the attachment of the clothing 30 like a Santa Claus. Furthermore, a Christmas song may be reproduced in coordination with the landing of each paw 19.

[0155] Further, bells whose generated sounds have different pitches may be attached as virtual items to the respective paws 29 of the virtual robot 8. The sound of bells with different pitches may be output in coordination with the walking of the real robot 2. Furthermore, the chords of the sound of bells may be output in coordination with the walking of the real robot 2.

[0156] In addition, the motion, behavior, mood, and personality may be controlled by the virtual items attached to the virtual robot 8. For example, the following examples can be provided.

[0157] When socks are put on the paws 29 of the virtual robot 8, the real robot 2 walks slowly and quietly.

[0158] When a dress is put on the virtual robot 8, the real robot 2 behaves gracefully and walks slowly without stomping its feet or running.

[0159] When cat ears are put on the virtual robot 8, the real robot 2 meows like a cat and moves like a cat.

[0160] When a hat with the shape of sheep is put on the virtual robot 8, the real robot 2 cries like a sheep.

[0161] When a helmet for a motorcycle is put on the virtual robot 8, the real robot 2 imitates sounds of a motorcycle or outputs sound effects regarding a motorcycle.

[0162] When a collar with a bell is put on the virtual robot 8, the sound of bell is output in coordination with the walking of the real robot 2.

[0163] When a soccer uniform is put on the virtual robot 8, the real robot 2 imitates kicking a ball.

[0164] When the virtual robot 8 is provided with a musical instrument, the real robot 2 sometimes reproduces a phrase as if it were practicing the instrument.

[0165] When the virtual robot 8 is provided with a whistle, the real robot 2 sometimes outputs the sound of the whistle.

[0166] When high heels are put on the paws 29 of the virtual robot 8, the walking style of the real robot 2 changes.

[0167] In addition, the present technology can be applied in various modes, which makes it possible to provide a high-quality user experience. (Second Embodiment: Output of Sound Corresponding to Virtual Object)

[0168] In this entertainment system 1, it is possible to output a sound corresponding to a virtual object arranged in a corresponding virtual space, as the output of a sound corresponding to a virtual representation. For example, it is possible to output a sound corresponding to a virtual object arranged in a map corresponding to the reality space RS generated on the basis of the sensing data detected by the sensor section 9.

[0169] Note that the virtual object is a concept including virtual items, and for example, the shoes 28 put on the virtual robot 8 are also included in the concept of the virtual object.

[0170] Fig. 8 is a schematic view showing an example of a map corresponding to the reality space RS. In the example shown in Fig. 8, a two-dimensional map 37 is displayed on the touch panel 7 of the user terminal 3. The map 37 is a map corresponding to the space in which the user 6 stays with the real robot 2 in the reality space RS (e.g., home).

[0171] In the map 37 shown in Fig. 8, gray blocks represent regions 38 impassable by the real robot 2. The impassable regions 38 correspond to regions in which obstacles such as a desk and a wall exist in the reality space RS.

[0172] Further, in the map 37 shown in Fig. 8, white blocks represent regions 39 passable by the real robot 2. The passable regions 39 correspond to regions in which obstacles such as a desk and a wall do not exist in the reality space RS. Note that the passable regions 39 may be classified into finer types, for example, regions easy to pass and regions difficult to pass.

[0173] Further, in the map 37 shown in Fig. 8, a virtual object of dog (icon) 40 represents the current position of the real robot 2. A triangle mark displayed in the vicinity of the virtual object of dog 40 represents the current orientation of the real robot 2.

[0174] Further, in the map 37 shown in the Fig. 8, a virtual object of house 41 is displayed. The virtual object of house 41 represents the position of a charging station where the charging of the real robot 2 is performed.

[0175] Fig. 9 is a flowchart showing a processing example in the user terminal 3 (application), for outputting the sound corresponding to the virtual object.

[0176] The user 6 arranges a puddle as a virtual object in a predetermined area of the map 37 corresponding to the reality space RS in the application (Step 401). For example, the user 6 arranges a virtual object of puddle 42 as exemplified in Fig. 8. Note that a method or GUI for the user 6 to select a virtual object desired to be arranged on the map 37, a method or GUI for arranging the selected virtual object on the map 37, and the like are not limited, and any methods or GUIs may be adopted.

[0177] The information that the virtual object of puddle 42 has been arranged is recorded in the cloud DB 4 (Step 402). For example, the type of the virtual object (= puddle) and the position information of the region where the virtual object has been arranged are associated with each other to be recorded in the DB 4.

[0178] As the position information of the region where the virtual object has been arranged, for example, the position information corresponding to self-location information in the reality space RS used to generate the map 37 is recorded. In other words, the position information of a region in the reality space RS (hereinafter, referred to as a first corresponding region), which corresponds to the region on the map 37 in which the virtual object of puddle 42 has been arranged, is recorded.

[0179] The information that the virtual object of puddle 42 has been arranged (the type of virtual object and the position information of the region where the virtual object has been arranged) corresponds to parameter information of a virtual representation relating to a corresponding virtual space.

[0180] As exemplified in Fig. 8, the virtual object of puddle 42 is displayed on the map 37 in the application (Step 403).

[0181] Fig. 10 is a flowchart showing a processing example in the real robot 2, for outputting a sound corresponding to a virtual object.

[0182] The parameter information acquisition section 25 of the real robot 2 downloads the information that the virtual object of puddle 42 has been arranged, which has been recorded in the DB 4, from the DB 4 (Step 501). Downloading of the information that the virtual object of puddle 42 has been arranged in Step 501 corresponds to one embodiment of the acquisition of parameter information of a virtual representation in Step 101 shown in Fig. 3.

[0183] The sound-related data acquisition section 26 of the real robot 2 determines whether or not there is sound-related data corresponding to the virtual object of puddle 42 (Step 502). In this embodiment, the sound-related data acquisition section 26 refers to the information that the virtual object of puddle 42 has been arranged, which has been downloaded from the DB 4. The sound-related data acquisition section 26 then determines whether or not there is sound-related data for outputting a sound corresponding to the "puddle" arranged on the map 37.

[0184] In this embodiment, the sound-related data acquisition section 26 determines whether or not sound-related data for outputting a sound corresponding to the "puddle" is stored in the DB 4. If the sound-related data corresponding to the "puddle" is not stored, it is determined No in Step 502, and the output of the sound corresponding to the virtual object is terminated.

[0185] If the sound-related data corresponding to the "puddle" is stored, it is determined Yes in Step 502, and the sound-related data corresponding to the "puddle", which has been stored in the DB 4, is downloaded (Step 503). The sound-related data corresponding to the "puddle" is sound-related data for outputting footsteps (e.g.,"splish-splash"), for example, in walking in a puddle in the reality space RS.

[0186] Note that the sound-related data may be stored in advance in the storage 16 of the real robot 2. In this case, the sound-related data acquisition section 26 reads the sound-related data corresponding to the "puddle" from the storage 16.

[0187] Steps 502 and 503 correspond to one embodiment of the acquisition of the sound-related data in Step 102 shown in Fig. 3.

[0188] The action controller 22 determines whether or not the real robot 2 has entered the region where the "puddle" has been arranged, that is, the first corresponding region, in the reality space RS (Step 504).

[0189] If the real robot 2 has not entered the first corresponding region, it is determined No in Step 504, and the output of the sound corresponding to the virtual object is terminated.

[0190] If the real robot 2 has entered the first corresponding region, the output of the sound corresponding to the virtual object is performed. In this embodiment, the processing proceeds from Step 504 to Step 505, and the action controller 22 and the sound output controller 24 output the footsteps in walking in a puddle in coordination with the landing of the paws 19 of the real robot 2 shown in Fig. 1.

[0191] In other words, the action controller 22 determines the action including the output of the footsteps in walking in a puddle in coordination with the landing of the paws 19 of the real robot 2.

[0192] The sound output controller 24 generates the sound data of the footsteps in walking in a puddle on the basis of the sound-related data downloaded from the DB 4. If the sound-related data is the sound data of footsteps, the sound data is used as it is. If the sound-related data is sound setting information, the sound data of footsteps is generated on the basis of the sound setting information.

[0193] The sound output controller 24 then causes the speaker 10 to output the footsteps in walking in a puddle in coordination with the landing of the paws 19 of the real robot 2 on the basis of a detection result of a landing sensor section provided in the sensor section 9.

[0194] In such a manner, in this embodiment, when the real robot 2 has entered the first corresponding region in the reality space RS, which corresponds to the arrangement region for the virtual object of puddle 42 that has been arranged by the user 6 on the map 37 serving as a corresponding virtual space, the footsteps in walking in a puddle, such as "splish-splash", are output in accordance with the walking of the real robot 2 (the landing of the paws 19).

[0195] In this embodiment, the walking of the real robot 2 (the landing of the paws 19) in the first corresponding region corresponds to one embodiment of a predetermined motion of the movable apparatus in the first corresponding region according to the present technology.

[0196] In the example shown in Fig. 8, a virtual object of drum 43 is also arranged on the map 37.

[0197] For example, in Step 401 shown in Fig. 9, the user 6 arranges a virtual object of drum 43 on the map 37. In Step 402, the information of the arrangement of the virtual object of drum 43 is recorded in the cloud DB 4. In Step 403, the virtual object of drum 43 is displayed on the map 37 in the application.

[0198] In Step 501 shown in Fig. 10, the parameter information acquisition section 25 of the real robot 2 downloads the information of the arrangement of the virtual object of drum 43, which has been recorded in the DB 4, from the DB 4.

[0199] In Step 502, the sound-related data acquisition section 26 of the real robot 2 determines whether or not there is sound-related data corresponding to the virtual object of drum 43. If sound-related data for outputting a sound corresponding to "drum" is stored in the DB 4, the sound-related data is downloaded. The sound-related data corresponding to "drum" is, for example, sound-related data for outputting a musical performance sound of a drum (e.g., "dub-a-dub").

[0200] In Step 504, it is determined whether or not the real robot 2 has entered the region where the "drum" is arranged, that is, the first corresponding region, in the reality space RS. If the real robot 2 has entered the first corresponding region, the musical performance sound of the drum is output in coordination with the landing of the paws 19 of the real robot 2 in Step 505. Note that the musical performance sound of the drum may be output when the real robot 2 has entered the first corresponding region without coordinating with the landing of the paws 19.

[0201] In such a manner, it is possible to change the sounds corresponding to the virtual representations depending on the virtual objects displayed on the map 37. As a matter of course, when the virtual object of drum 43 moves, the location from which the sound of the drum is generated is also changed accordingly.

[0202] Further, the motion, behavior, mood, and personality may be controlled depending on the virtual objects arranged on the map 37. For example, the following processing can also be possible: the real robot 2, which has entered the first corresponding region, behaves in a manner such as playing a drum, so that the sound of the drum is output. Furthermore, for example, the following setting can also be possible: the real robot 2 plays a drum whenever it feels like it in response to the arrangement of the virtual object of drum 43. Furthermore, it is also possible to perform processing such as changing the types of drum from a toy drum to a Japanese drum and changing a reproduced sound depending on a proficiency level (the degree of skill) regarding playing a drum.

[0203] In such a manner, in this entertainment system 1, it is possible to arrange various virtual objects on the map 37. When the real robot 2 enters the first corresponding region corresponding to the region where a virtual object is arranged, a sound corresponding to the virtual object can be output from the real robot 2.

[0204] For example, it is also possible to set that music is played by footsteps when the real robot 2 walks in a corresponding first corresponding region in accordance with the arrangement of a virtual object of "score". For example, it is possible to set that the score advances in accordance with the landing of each paw 19 of the real robot 2. For example, when the "score" of "Twinkle, Twinkle, Little Star" is arranged, the sound "Do-Do-So-So-La-La-So" is reproduced in order for each landing. For example, if the real robot 2 walks in the order of its right front paw 19b, left back paw 19c, left front paw 19a, and right back paw 19d, the sound like "Do, Do, So, So" is reproduced for each landing. Note that no sound is reproduced at the rest timing even if the foot lands. Different music is also played depending on the type of "score". Such settings are also possible.

[0205] For example, footsteps of walking on snow or on tap-dancing flooring are output to correspond to the virtual objects of "snowy road" and "tap-dancing flooring". The sound of fireworks is output to correspond to a virtual object of "fireworks". Such various settings are also possible.

[0206] In such a manner, in the second embodiment, the sound expression (operation sound) of the real robot 2 changes depending on the virtual objects arranged on the map 37 corresponding to the reality space RV. For example, it is difficult to actually arrange an item that specifies a puddle in the reality space RS, but applying the present technology makes it possible to achieve the same effect. (Third Embodiment: Output of Sound Corresponding to Region with Predetermined Attribute)

[0207] It is also possible to set part of the region of the map 37 as a region with a predetermined attribute, instead of arranging the virtual object on the map 37 that has been described in the second embodiment.

[0208] For example, part of the region of the map 37 is set as a region with an attribute of "puddle". This makes it possible to provide a user experience similar to that of the case where a virtual object of puddle 42 is arranged on the map 37.

[0209] For example, the user 6 sets part of the region of the map 37 as a region with an attribute of "puddle" in the application. Information regarding the region of "puddle" is recorded in the cloud DB 4. For example, the attribute (= puddle) and the position information of the set region are associated with each other to be recorded in the DB 4. The fact that the region of "puddle" has been set is displayed on the map 37.

[0210] The information regarding the region of "puddle" (the attribute and the position information of the set region) corresponds to the parameter information of the virtual representation associated with the corresponding virtual space.

[0211] The parameter information acquisition section 25 of the real robot 2 downloads the information regarding the region of "puddle", which has been recorded in the DB 4, from the DB 4. Downloading of the information regarding the region of "puddle" corresponds to one embodiment of the acquisition of parameter information of a virtual representation in Step 101 shown in Fig. 3.

[0212] The sound-related data acquisition section 26 of the real robot 2 determines whether or not there is sound-related data corresponding to the region with the attribute of "puddle". If the sound-related data corresponding to the region with the attribute of "puddle" is stored in the DB 4, such sound-related data is downloaded. The sound-related data corresponding to the region with the attribute of "puddle" is, for example, sound-related data for outputting footsteps (e.g.,"splish-splash") in walking in a puddle in the reality space RS.

[0213] Note that the sound-related data may be stored in advance in the storage 16 of the real robot 2. In this case, the sound-related data acquisition section 26 reads the sound-related data corresponding to the region with the attribute of "puddle" from the storage 16.

[0214] Determination of the presence / absence of the sound-related data corresponding to the region with the attribute of "puddle", and download of the sound-related data correspond to one embodiment of the acquisition of sound-related data in Step 102 shown in Fig. 3.

[0215] The action controller 22 determines whether or not the real robot 2 has entered the region of the reality space RS (hereinafter, referred to as a second corresponding region), corresponding to the region with the attribute of "puddle" in the map 37, in the reality space RS.

[0216] If the real robot 2 has entered the second corresponding region, a sound corresponding to the region with the attribute of "puddle" is output. In this embodiment, the footsteps in walking in a puddle are output in coordination with the walking of the real robot 2 (the landing of the paws 19).

[0217] The predetermined attribute is not limited, and attributes of "snowy road" and "tap-dancing flooring" can be set for regions on the map 37. Further, an attribute such as "fireworks" can also be set for a region on the map 37.

[0218] In this embodiment, the walking of the real robot 2 (the landing of the paws 19) in the second corresponding region corresponds to one embodiment of a predetermined motion of the movable apparatus in the second corresponding region according to the present technology.

[0219] In such a manner, in the third embodiment, the sound expression (operation sound) of the real robot 2 changes depending on the region with a predetermined attribute set on the map 37 corresponding to the reality space RV. This makes it possible to achieve the same effect as the second embodiment in which a virtual object is arranged on the map 37.

[0220] In the second and third embodiments, a three-dimensional map (3D map) may be generated as a map corresponding to the reality space RS. Further, the map is not limited to the case where it is generated by the real robot 2. The map may be explicitly created by the user 6 or may be created by another means. Further, it is possible to use map data such as a floor map corresponding to the reality space RS by downloading it from the outside via the network 5. Furthermore, it is also possible to acquire the map generated by another real robot 2 via the network 5.

[0221] Further, when the real robot 2 has approached a corresponding region (first corresponding region and second corresponding region), a sound corresponding to a virtual representation may be output. For example, in the second embodiment, when the real robot 2 has approached the first corresponding region, a sound corresponding to a virtual object may be output. Further, in the third embodiment, when the real robot 2 has approached the second corresponding region, a sound corresponding to a region with a predetermined attribute may be output.

[0222] It is possible to determine whether or not the real robot 2 has approached a corresponding region (first corresponding region and second corresponding region) by, for example, setting a threshold for a distance between the real robot 2 and the region (first corresponding region and second corresponding region). A specific value of the threshold is not limited and may be discretionally set. For example, the threshold can be discretionally set in the range from several centimeters to several meters. As a matter of course, the threshold is not limited to this range. Further, another method may be adopted as the method of determining whether or not the real robot 2 has approached a corresponding region (first corresponding region and second corresponding region).

[0223] The real robot 2 may behave in a manner such as enjoying playing in the water in a corresponding region (first corresponding region and second corresponding region) in accordance with the arrangement of the virtual object of "puddle" 42 or the setting of the region with the attribute of "puddle". Further, the sound of footsteps may change from the sound of water to the sound of walking on dry ground under the assumption that the puddle is drying up, in accordance with the elapse of time.

[0224] The following setting can also be made: the real robot 2 walks slowly and quietly in a corresponding region (first corresponding region and second corresponding region) in accordance with the arrangement of a virtual object of "carpet" or the setting of a region with an attribute of "carpet". Further, the sound of footsteps may change under the assumption that a fluffy carpet becomes hard in accordance with the elapse of time. (Fourth Embodiment: Output of Sound Corresponding to Proficiency Level)

[0225] In this entertainment system 1, it is possible to output a sound corresponding to a proficiency level of a corresponding virtual object with respect to a predetermined target to be learned, as the output of a sound corresponding to a virtual representation. For example, it is possible to cause the real robot 2 to output a sound corresponding to a proficiency level of the virtual robot 8 with respect to a predetermined target to be learned.

[0226] As the predetermined target to be learned, for example, various musical performances are assumed. The musical performances include musical performances using musical instruments and musical performances performed without using musical instruments, such as singing, whistling, and handclapping. Further, tap dancing is also included as a predetermined target to be learned. In addition, any other techniques (skills) to perform expression by using generated sounds can be adopted as a predetermined target to be learned according to one embodiment of the present technology.

[0227] It is possible to cause the real robot 2 to output a sound corresponding to a proficiency level regarding a musical performance or dancing.

[0228] Figs. 11A and 11B are schematic views each showing the output of a sound corresponding to a proficiency level. In the examples shown in Figs. 11A and 11B, singing of a predetermined melody is adopted as a predetermined target to be learned.

[0229] Fig. 11A shows a case where a proficiency level of singing of the virtual robot 8 displayed on the user terminal 3 is relatively high. In this case, in the reality space RS, a sound corresponding to a relatively high proficiency level is output from the real robot 2. For example, a sound of singing a predetermined melody beautifully is output from the real robot 2.

[0230] Fig. 11B shows a case where a proficiency level of singing of the virtual robot 8 displayed on the user terminal 3 is relatively low. In this case, in the reality space RS, a sound corresponding to a relatively low proficiency level is output from the real robot 2. For example, a sound of singing a predetermined melody badly, which is out of tune in terms of sound pitch (scale), rhythm, timing, etc., is output from the real robot 2.

[0231] For example, the user 6 improves the proficiency level of singing of the virtual robot 8 in the application. Then, in the reality space RS, the output of the sound of poor singing shown in Fig. 11B is improved to the output of the sound of good singing shown in Fig. 11A. Note that the virtual robot 8 performs singing at the same proficiency level as the real robot 2 also in the virtual space VS.

[0232] The proficiency level regarding singing corresponds to the parameter information of the virtual representation relating to the virtual robot 8.

[0233] As a method of specifying a music piece or the like that is a target to be learned, for example, the user 6 can adopt a method of selecting a music piece or the like that the user 6 wants the real robot 2 and the virtual robot 8 to learn in the application. Further, it is also possible to cause the real robot 2 and the virtual robot 8 to recognize a music piece or the like to be learned by reproducing the music piece with a music player in the virtual space VS.

[0234] Further, music is played to the virtual robot 8 in the virtual space VS by linking with streaming services, etc. The virtual robot 8 then analyzes the music piece or downloads meta-information stored in a separate server to obtain information of the music piece. Such a method is also possible.

[0235] Furthermore, it is also possible for the user 6 to play a musical instrument in the virtual space VS and let the virtual robot 8 listen to it to recognize the music to be learned. For example, in the case of causing the virtual robot 8 to learn a predetermined phrase of a snare drum or learn a predetermined melody of a piano, it is effective to cause the virtual robot 8 to listen to the musical performance of the musical instrument in the virtual space VS.

[0236] Further, it is also possible to cause the real robot 2 to recognize a music piece to be learned by reproducing an acquired music piece or the like through a speaker and causing the real robot 2 to listen to it with a microphone in the reality space RS. Further, it is also possible to cause the real robot 2 to recognize a music piece to be learned by the user 6 playing an actual musical instrument in the reality space RS.

[0237] As the method of improving the proficiency level of the virtual robot 8, for example, it is possible to adopt a method for the user 6 to instruct the virtual robot 8 to practice a music piece or the like in the application. For example, pressing down a particular button in the application to cause the virtual robot 8 to make a practice makes it possible to improve a proficiency level. For example, the number of times that the particular button is pressed down is counted, and as the number of times becomes higher, the proficiency level becomes higher. Further, it is also possible to adopt a calculation method in which as the practice time of the virtual robot 8 becomes longer in the virtual space VS, the proficiency level becomes higher. The calculated proficiency level is recorded in the cloud DB 4.

[0238] Further, another method can also be adopted, in which the virtual robot 8 is caused to listen to a model music piece (e.g., the sound source of a CD or the musical performance sound played by the user 6) many times in the virtual space VS, to improve its proficiency level. As the virtual robot 8 is caused to listen to the music piece or the like more times and a longer time, the proficiency level becomes higher.

[0239] The proficiency level may be improved by meeting other virtual robots already having a high proficiency level in the virtual space VS. As the proficiency level of the other virtual robots becomes higher, the proficiency level of the virtual robot 8 becomes higher. Further, as the virtual robot 8 meets other virtual robots more times, the proficiency level of the virtual robot 8 becomes higher.

[0240] The proficiency level may be improved by giving a particular virtual item to the virtual robot 8 in the virtual space VS. For example, a musical note-shaped cookie is given to the virtual robot 8. When the virtual robot 8 eats the musical note-shaped cookie, its proficiency level is improved. Such a setting is also possible.

[0241] While the virtual robot 8 is practicing a music piece or the like in the virtual space VS, the virtual robot 8 may behave in a way corresponding to the practice. In addition, the fact that the proficiency level has been improved may be displayed in the application.

[0242] The user 6 can improve the proficiency level of the real robot 2 in the reality space RS. For example, the real robot 2 is caused to listen to a model music piece (e.g., the sound source of a CD or the musical performance sound played by the user 6) in the reality space RS. When the real robot 2 recognizes that a music piece or the like specified as a target to be learned is being reproduced via a microphone, the real robot 2 enters a practice mode and behaves in a way corresponding to the practice. As the real robot 2 is caused to listen to the music piece or the like more times and a longer time, the proficiency level becomes higher.

[0243] An example of the method of calculating the proficiency level of the virtual robot 8 will be described. For example, the virtual robot 8 enters a practice mode for practicing a music piece or the like. The user 6 may instruct the virtual robot 8 to transfer to the practice mode, or the virtual robot 8 may automatically transfer to the practice mode.

[0244] In the system, a practice state and a non-practice state are defined for the virtual robot 8.

[0245] The proficiency level varies depending on the practice by the virtual robot 8, other actions (encounters with other virtual robots, acquisition of specific items, etc.), the passage of time, and the like. For example, the proficiency level is calculated on the basis of the number of times the user 6 instructs the robot to make a practice, the total time the user 6 operates for practice, the number of times the user 6 shows the robot an example, and so on. The proficiency level may directly change on the basis of the number of encounters with other virtual robots, the number of times a particular item is obtained, and so on, in addition to the practice. The proficiency level can also be set to decrease on the basis of the elapsed time from the practice.

[0246] As for the method of outputting the sound corresponding to the proficiency level, any method that can represent the progress of the real robot 2 in music with sound may be adopted.

[0247] In this embodiment, for example, complete musical performance sound data (sound data of a musical performance when the musical performance has succeeded) is recorded in advance in the storage 16 of the real robot 2. For example, the pitch of the sound, the length of the sound, the timing of generating the sound, the tone, etc. are recorded as complete musical performance sound data.

[0248] The sound corresponding to the proficiency level is output by modifying the complete musical performance sound data in accordance with the proficiency level. For example, the sound data corresponding to the proficiency level is output by changing the pitch of the sound and the timing of generating the sound, and adding or deleting sounds from the musical performance sound data.

[0249] If the proficiency level is low, modification is performed many times to approach the complete musical performance sound data as the proficiency level becomes higher. As the proficiency level becomes higher, the difference from the original complete musical performance sound data decreases. When the proficiency level reaches the highest value (or a predetermined threshold or more), the sound is output as the original musical performance sound data.

[0250] Specific examples of the modification of the complete musical performance sound data will be described. It is assumed that the pitch of the sound, the length of the sound, the timing of generating the sound, and the like are recorded as the complete musical performance sound data. The modification processing including the following five modification items is performed when the sound corresponding to the proficiency level is reproduced. (1) Addition of errors: For a specific sound, a sound at a pitch different from the original one is reproduced. (2) Delay of reproduction: A pause of approximately one second is inserted during the reproduction of the sound after (1) or after the correct sound. (3) Fluctuation of reproduction speed: The reproduction speed is not constant, but slowed down at a part where a series of short sounds are continuously output. (4) Termination of reproduction: The music piece is not reproduced to the end, but is terminated in the middle. (5) Repeat: The middle part of a music piece is reproduced over and over.

[0251] For example, as the proficiency level becomes higher, the number of modification items to be performed is reduced among the five modification items. Further, as the proficiency level becomes higher, the degree of modification of each modification item is reduced. Alternatively, as the proficiency level becomes higher, the probability of performing the modification using each modification item is decreased.

[0252] For example, the proficiency level is classified into six levels (lowest proficiency level 1 to highest proficiency level 6). If the proficiency level corresponds to the lowest proficiency level 1, the modification using all of the above five modification items is performed. Each time the proficiency level increases by one level, the number of modification items to be performed is reduced by one (a modification item to be performed may be selected randomly). When the proficiency level corresponds to the highest proficiency level 6, none of the modification items is performed, and the sound is output on the basis of the original complete musical performance sound data. Such processing is also possible.

[0253] Further, in the case of the proficiency levels 1 to 5, a parameter of (1 / proficiency level) may be set as the probability of performing each modification item. For example, in the case of the lowest proficiency level 1, all modification items are performed with a probability of 100%. In the case of the proficiency level 5, all modification items are performed with a probability of 20%. For the highest proficiency level 6, no modification is performed. Such processing is also possible.

[0254] Thus, in the fourth embodiment, the virtual robot 8 practices a music piece or the like in the virtual space VS, and the musical performance sound of the real robot 2 changes according to the proficiency level of the virtual robot 8. This makes it possible to provide a very high user experience.

[0255] Further, in this embodiment, successful musical performance sound data that is musical performance sound data obtained when a musical performance has succeeded, or sound setting information for generating successful musical performance sound data is adopted as the sound-related data. Subsequently, the modification processing is performed on the successful musical performance sound data on the basis of the proficiency level, so that the sound corresponding to the proficiency level is output.

[0256] Since it is possible to dynamically perform the modification processing corresponding to the proficiency level as described above, it is possible to perform a variety of expressions at low costs, for example, as compared to the case of preparing a plurality of pieces of musical performance sound data corresponding to the proficiency level in advance.

[0257] For example, in the case of preparing a plurality of pieces of musical performance sound data corresponding to the proficiency level in advance and appropriately selecting and outputting a musical performance sound in accordance with the proficiency level, the musical performance sounds to be output will be the same every time, from a poor musical performance to a good musical performance. In this embodiment, selecting the five modification items described above or setting the probability makes it possible to output different musical performance sounds each time, even in the musical performance sounds of the same proficiency level. This can achieve a variety of expressions and an enjoyable user experience.

[0258] Note that in applying the present technology, it is also possible to adopt a method of preparing a plurality of pieces of musical performance sound data corresponding to the proficiency level in advance and appropriately selecting and outputting a musical performance sound in accordance with the proficiency level. In other words, it is also possible to adopt, as sound-related data, a plurality of pieces of corresponding musical performance sound data corresponding to the proficiency level or sound setting information for generating a plurality of pieces of corresponding musical performance sound data corresponding to the proficiency level. Subsequently, by selecting one of the plurality of pieces of corresponding musical performance sound data on the basis of the proficiency level, the sound corresponding to the proficiency level may be output. Adopting this method makes it possible to simplify the processing.

[0259] The motion of the real robot 2 may be changed in accordance with the modification of the complete musical performance sound data. For example, the expressions of the eyes and the expressions of the ears of the real robot 2, the expressions when the real robot 2 recognizes failures, and the like may be performed in coordination with the output of the musical performance sound.

[0260] Further, the modification processing may be controlled in accordance with the fatigue level or mood of the real robot 2 (virtual robot 8). For example, it is possible to set that, when the real robot 2 (virtual robot 8) is tired, many modifications are performed thereon to make more mistakes. For example, it is also possible to set that the real robot 2 (virtual robot 8) makes more mistakes when the remaining amount of battery life is low.

[0261] It is also possible to change the proficiency level on the basis of the unique parameters of the real robot 2 (virtual robot 8). The unique parameters of the real robot 2 (virtual robot 8) include, for example, the parameters randomly set at the time of purchasing the real robot 2 or the parameters set at the beginning.

[0262] For example, the unique parameters include a parameter set by the user 6, such as gender. The unique parameters also include parameters set on the basis of the relationship between the user 6 and the real robot 2, such as a startup time, the frequency of voice recognition, and the frequency of stroking.

[0263] Those unique parameters may change the mode in which the proficiency level changes. For example, the unique parameters may change the time required for learning music or dance. By an increase in the interaction with the real robot 2 (lots of stroking, lots of praise, etc.), the proficiency level of the real robot 2 is improved with less practice. The unique parameters determine which songs the real robot 2 is good at or bad at learning. Such a variety of settings can be achieved, so that a variety of expressions can be achieved.

[0264] When the sound corresponding to the proficiency level is output, not only the pitch of the sound, the length of the sound, the timing of generating the sound, but also the parameters relating to musical performance expressions may be controlled to correspond to the proficiency level.

[0265] For example, a high pitch sound is set to be lower in the case of singing voices, or the envelope (time variation) of tones is changed in the case of wind instruments, so that the lack of proficiency in musical performance can be expressed.

[0266] As the proficiency level increases and the skill is improved, expressions such as adding vibrato (a broad expression with periodic changes in pitch) to a guitar sound or a singing voice, and changing the timing of a part to be heard (creating a pause) may be adopted. It is also possible to set that, when the proficiency level is high, a musical performance can be achieved very quickly.

[0267] Any other various musical performance expressions that are difficult to express by the sheet music may be achieved in accordance with the proficiency level. For example, performing machine learning on the basis of musical performance sound data of a world-famous musician makes it possible to acquire musical performance sound data in the case of a very high proficiency level, and also makes it possible to output a very high-level, beautiful musical performance sound from the real robot 2.

[0268] Further, it is also possible to link the change in pause or timing with the body motion of the real robot 2.

[0269] As described above, in the entertainment system 1 and the real robot 2 according to each embodiment described above, the parameter information of the virtual representation relating to at least one of the virtual robot 8 that corresponds to the real robot 2 or the map 37 serving as a corresponding virtual space that corresponds to the reality space RS, and the sound-related data corresponding to the parameter information are acquired. Subsequently, the output of the sound corresponding to the virtual representation is controlled using the sound-related data on the basis of the parameter information and the sensing data of the sensor section 9. This makes it possible to provide a high-quality user experience to the user 6 who uses the real robot 2.

[0270] In the entertainment robot, use of the virtual robot 8 that has the same appearance as the real robot 2 in the virtual space VS makes it possible to achieve expressions that are physically difficult to achieve or are expensive in the case of the real robot 2.

[0271] For example, it is possible to achieve the expressions that are difficult for the real robot 2 to perform in the reality space RS, such as an expression of the virtual robot 8 entering a swimming pool or being located in outer space. It is also possible to put virtual clothing on the virtual robot 8, give virtual foods, and cause the virtual robot 8 to play a virtual musical instrument. This makes it possible for the user 6 to have a wider range of satisfying experiences than when spending time only with the real robot 2.

[0272] However, there arises a problem that, as the expressions of the virtual robot 8 in the virtual space VS become richer, this makes it more difficult for the user 6 to feel the relationship with the real robot 2 in the reality space RS.

[0273] In this entertainment system 1, the parameter changes in the virtual space VS can be reflected in the expressions of the real robot 2 in the reality space RS, and the user 6 can feel the relationship between the real robot 2 and the virtual robot 8 more strongly. <Other Embodiments>

[0274] The present technology is not limited to the embodiments described above and can achieve various other embodiments.

[0275] In each of the above embodiments, augmented reality (AR) display may be possible in the user terminal 3. For example, it is possible to superimpose the display of a virtual object on the image acquired by the camera of a smartphone in real time. It is also possible to use a head-mounted display (HMD) such as AR glasses as the user terminal 3 to superimpose the display of a virtual object on the field of view of the user 6.

[0276] For example, in the case of attaching the shoes 28 described in the first embodiment, when the real robot 2 is imaged through a camera of a smartphone, AR display is performed as if the real robot 2 were wearing the shoes 28 selected by the user 6.

[0277] Further, in the setting of the arrangement of the virtual object of "puddle" 42 described in the second embodiment or the region with the attribute of "puddle" described in the third embodiment, when the first corresponding region and the second corresponding region are seen through a camera of a smartphone, AR display is performed as if there were a puddle on the floor. For example, linking with the self-location estimation technology using a camera of a smartphone makes it possible to achieve such AR display.

[0278] Further, after the real robot 2 leaves the first corresponding region and the second corresponding region corresponding to the "puddle", wet footprints may be AR-displayed on the ground.

[0279] In the output of the sound corresponding to the proficiency level of the musical performance described in the third embodiment, it is possible to superimpose a virtual musical instrument as if the real robot 2were playing the musical instrument, using AR display via a camera of a smartphone.

[0280] Such various types of AR display can provide a higher-quality user experience.

[0281] The dog-like real robot 2 capable of automated walking has been described above as an example. Of course, the application of the present technology is not limited to this type of real robot 2.

[0282] For example, the present technology can also be applied to a movable apparatus that does not move. For example, in a real robot that rotates in a stationary state, an item such as a bell is attached to a corresponding virtual robot. Then, the sound linked to the rotation of the real robot 2 changes. If an item such as shoes with bells is set, the frequency of ringing the bells changes in accordance with the rotation speed. Such settings are also possible.

[0283] If actual physical clothing is put on the real robot 2, the clothing may be reflected on the virtual robot 8 in the virtual space VS. In this case, if clothing is put on the real robot 2, a sound linked with the motion of the real robot 2, such as walking, may be added. Further, in the case of a car, a running sound may be changed by changing the settings of the car in the application. For example, the running sound may be changed by setting virtual tires.

[0284] A flying object (flying robot) that can fly may be configured as the real robot 2. For example, the present technology is also applicable to drones. In such a flying object, for example, the user 6 changes the parameter information of virtual representations relating to the virtual robot 8 corresponding to the flying object in the application, using the user terminal 3. It is also possible to set that the sound emitted when the real robot 2 (flying object) flies changes in accordance with the change of the parameter information.

[0285] For example, in the application, a virtual item like a fairy's wing is attached to the virtual robot 8 (flying object). Then, when the real robot 2 (flying object) flies in the reality space RV, a cute sound like imagining a fairy's flight is reproduced. Such an enjoyable setting is also possible.

[0286] For example, the attachment of the shoes 28 exemplified in the first embodiment and the arrangement of the virtual object of "puddle" described in the second embodiment may be combined. Special sounds and the like may be output depending on the combination of the virtual item attached to the virtual robot 8 and the virtual object arranged on the map 37.

[0287] Regarding the map 37 corresponding to the reality space RS, if sounds can be output through the AR glasses or speakers, sounds may be output when a person enters an area that is set to output a sound, such as a virtual puddle. Further, if the real robot 2 includes a display region such as a display, such an expression that the color of the "puddle" changes may be provided when the real robot 2 approaches the puddle. Further, the motion of the real robot 2 may be changed according to the season, time, date, etc. For example, the following processing is also possible: when it rains, a puddle is naturally formed in the virtual space VS, and the sound may change in that area.

[0288] In addition, the output of the footsteps of the real robot 2 can be determined in accordance with the environment, e.g., a sound is not output to be quiet when a person is watching TV, talking, or listening to music, to determine the presence / absence of the reproduction of a sound (e.g., a sound is output not so much depending on the time zone, e.g., at night).

[0289] In the above description, the sound corresponding to the virtual representation is output by the speaker 10 mounted on the real robot 2. The present technology is not limited to the above and can also be applied to a speaker (sound output section) that is configured separately from the real robot 2.

[0290] For example, if a very small real robot is used, there may be a case where a speaker cannot be mounted thereon. In such a case, it is effective to output a sound corresponding to a virtual representation from a speaker (sound output section) that is configured separately from the real robot 2

[0291] Alternatively, it is also possible to incorporate a speaker into a floor to output a variety of sounds corresponding to virtual representations of the real robot 2 in accordance with its walking. This makes it possible to provide a high-quality user experience.

[0292] Fig. 12 is a block diagram showing a hardware configuration example of a computer 60 that can be used for establishing the entertainment system 1 according to one embodiment of the present technology.

[0293] The computer 60 includes a central processing unit (CPU) 61, a read only memory (ROM) 62, a random access memory (RAM) 63, an input / output interface 65, and a bus 64 that connects those components to each other. A display section 66, an input section 67, a storage 68, a communication section 69, a drive section 70, and the like are connected to the input / output interface 65.

[0294] The display section 66 is, for example, a display device using liquid crystal, electro-luminescence (EL), or the like. The input section 67 is, for example, a keyboard, a pointing device, a touch panel, or another operation device. If the input section 67 includes a touch panel, the touch panel may be integrated with the display section 66.

[0295] The storage 68 is a nonvolatile storage device and is, for example, an HDD, a flash memory, or another solid-state memory. The drive section 70 is, for example, a device capable of driving a removable recording medium 71 such as an optical recording medium or a magnetic recording tape.

[0296] The communication section 69 is a modem, a router, or another communication device that can be connected to a LAN, a WAN, or the like for communicating with other devices. The communication section 69 may communicate using wires or radios. The communication section 69 is often used separately from the computer 60.

[0297] The information processing by the computer 60 having the hardware configuration as described above is implemented in cooperation with the software stored in the storage 68, the ROM 62, or the like and the hardware resources of the computer 60. Specifically, the information processing method according to one embodiment of the present technology is implemented when a program configuring the software, which is stored in the ROM 62 or the like, is loaded into the RAM 63 and then executed.

[0298] The program is installed in the computer 60, for example, through the recording medium 71. Alternatively, the program may be installed in the computer 60 via a global network or the like. In addition, any non-transitory computer-readable storage medium may be used.

[0299] The information processing method (the method of controlling a movable apparatus) and the program according to one embodiment of the present technology may be executed, and the information processing system according to one embodiment of the present technology may be provided, by linking a plurality of computers communicably connected via a network or the like.

[0300] In other words, the information processing method, the information processing system, and the program according to one embodiment of the present technology can be performed not only in a computer system formed of a single computer, but also in a computer system in which a plurality of computers operates cooperatively.

[0301] Note that, in the present disclosure, the system refers to a set of components (such as apparatuses and modules (parts)) and it does not matter whether all of the components are in a single housing. Thus, a plurality of apparatuses accommodated in separate housings and connected to each other through a network, and a single apparatus in which a plurality of modules is accommodated in a single housing are both the system.

[0302] Execution of the information processing method, the information processing system, and the program according to one embodiment of the present technology by the computer system includes, for example, both a case in which the acquisition of the parameter information of the virtual representation, the acquisition of the sound-related data, the generation of the map that is a corresponding virtual space, the output of a sound corresponding to the virtual representation, and the like are performed by a single computer; and a case in which the respective processes are performed by different computers. Further, the execution of each process by a predetermined computer includes causing another computer to perform part or all of the processes and obtaining a result thereof.

[0303] In other words, the information processing method, the information processing system, and the program according to one embodiment of the present technology are also applicable to a configuration of cloud computing in which a single function is shared and cooperatively processed by a plurality of apparatuses through a network.

[0304] The configurations of the entertainment system, the real robot, the user terminal, the DB, the GUI in the application, and the like described with reference to the respective figures; and the processing flows of the acquisition of the parameter information of the virtual representation, the acquisition of the sound-related data, the generation of the map that is a corresponding virtual space, the output of a sound corresponding to the virtual representation, and the like are merely embodiments, and any modifications may be made thereto without departing from the spirit of the present technology. In other words, any other configurations or algorithms for purpose of practicing the present technology may be employed.

[0305] In the present disclosure, to easily understand the description, the words such as "nearly", "substantially", "approximately", and "about" are appropriately used. Meanwhile, it does not define a clear difference between the case where those words such as "nearly", "substantially", "approximately", and "about" are used and the case where those words are not used.

[0306] In other words, in the present disclosure, concepts defining shapes, sizes, positional relationships, states, and the like, such as "central", "middle", "uniform", and "equal", are concepts including "substantially central", "substantially middle", "substantially uniform", and "substantially equal".

[0307] For example, the states included in a predetermined range (e.g., range of }10%) with reference to "completely central", "completely middle", "completely uniform", "completely equal", and the like are also included.

[0308] Therefore, even if the words such as "substantially", "approximately", and "about" are not added, the concept that may be expressed by adding so-called "substantially", "approximately", and "about" thereto can be included. To the contrary, the complete states are not necessarily excluded from the states expressed by adding "substantially", "approximately", "about", and the like.

[0309] In the present disclosure, expressions using the term "than" such as "greater than A" and "less than A" are expressions that comprehensively include concepts that include the case of being equal to A and concepts that do not include the case of being equal to A. For example, "greater than A" is not limited to the case where it does not include "equal to A"; however, it also includes "equal to or greater than A". Further, "less than A" is not limited to "less than A"; it also includes "equal to or less than A".

[0310] Upon implementation of the present technology, specific settings and other settings may be appropriately adopted from the concepts that are included in "greater than A" and "less than A" to achieve the effects described above.

[0311] At least two of the features among the features described above according to one embodiment of the present technology can also be combined. In other words, various features described in the respective embodiments may be combined discretionarily regardless of the embodiments. Further, the various effects described above are merely illustrative and not restrictive, and other effects may be exerted.

[0312] Note that the present technology may also take the following configurations. (1) A movable apparatus that is provided in a reality space, including: one or more sensors; an acquisition section that acquires parameter information of a virtual representation relating to at least one of a corresponding virtual object that corresponds to the movable apparatus and is provided in a virtual space or a corresponding virtual space that corresponds to the reality space, and sound-related data corresponding to the parameter information; and an output controller that controls output of a sound corresponding to the virtual representation by using the acquired sound-related data on the basis of the acquired parameter information and sensing data from the one or more sensors. (2) The movable apparatus according to (1), further including a sound output section, in which the output controller controls the output of the sound corresponding to the virtual representation from the sound output section. (3) The movable apparatus according to (1) or (2), in which the parameter information includes information regarding a virtual item attached to the corresponding virtual object, and the output controller causes a sound corresponding to the virtual item to be output. (4) The movable apparatus according to (3), in which the one or more sensors acquire data relating to a motion of the movable apparatus, and the output controller causes the sound corresponding to the virtual item to be output in accordance with a predetermined motion of the movable apparatus. (5) The movable apparatus according to (4), in which the output controller causes the sound corresponding to the virtual item to be output in accordance with a moving motion of the movable apparatus. (6) The movable apparatus according to any one of (1) to (5), in which the parameter information includes at least one of information regarding a virtual object arranged in the corresponding virtual space or information regarding a region with a predetermined attribute that is set in the corresponding virtual space, and the output controller causes at least one of a sound corresponding to the virtual object or a sound corresponding to the region with the predetermined attribute to be output. (7) The movable apparatus according to (6), in which the one or more sensors acquire data regarding a motion of the movable apparatus, and the output controller performs at least one of the output of the sound corresponding to the virtual object if the movable apparatus enters or approaches a first corresponding region, in the reality space, corresponding to a region in which the virtual object is arranged in the corresponding virtual space, or the output of the sound corresponding to the region with the predetermined attribute if the movable apparatus enters or approaches a second corresponding region, in the reality space, corresponding to the region with the predetermined attribute in the corresponding virtual space. (8) The movable apparatus according to (7), in which the output controller performs at least one of the output of the sound corresponding to the virtual object in accordance with a predetermined motion of the movable apparatus in the first corresponding region, or the output of the sound corresponding to the region with the predetermined attribute in accordance with a predetermined motion of the movable apparatus in the second corresponding region. (9) The movable apparatus according to any one of (1) to (8), in which the corresponding virtual space is a map corresponding to the reality space, the map being generated on the basis of the sensing data from the one or more sensors. (10) The movable apparatus according to any one of (1) to (9), in which the parameter information includes a proficiency level of the corresponding virtual object regarding a predetermined target to be learned, and the output controller causes a sound corresponding to the proficiency level to be output. (11) The movable apparatus according to (10), in which the proficiency level includes a proficiency level of the corresponding virtual object regarding a musical performance. (12) The movable apparatus according to (11), in which the sound-related data corresponding to the parameter information includes successful musical performance sound data that is musical performance sound data when the musical performance has succeeded, or sound setting information for generating the successful musical performance sound data, and the output controller performs modification processing on the successful musical performance sound data on the basis of the proficiency level, to cause the sound corresponding to the proficiency level to be output. (13) The movable apparatus according to (11), in which the sound-related data corresponding to the parameter information includes a plurality of pieces of corresponding musical performance sound data corresponding to the proficiency level, or sound setting information for generating the plurality of pieces of corresponding musical performance sound data corresponding to the proficiency level, and the output controller selects any one of the plurality of pieces of corresponding musical performance sound data on the basis of the proficiency level, to cause the sound corresponding to the proficiency level to be output. (14) The movable apparatus according to any one of (1) to (13), in which the output controller performs the output of the sound corresponding to the virtual representation when a condition for performing the output of the sound corresponding to the virtual representation is satisfied. (15) The movable apparatus according to any one of (1) to (14), in which the movable apparatus is configured as a mobile apparatus capable of automated moving. (16) An information processing method that is executed by a computer system, including: acquiring parameter information of a virtual representation relating to at least one of a corresponding virtual object that corresponds to a movable apparatus provided in a reality space and is provided in a virtual space or a corresponding virtual space that corresponds to the reality space, and sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation by using the acquired sound-related data on the basis of the acquired parameter information and sensing data from the one or more sensors of the movable apparatus. (17) An apparatus capable of motion, the apparatus comprising: one or more sensor; processing circuitry configured to: acquire parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to the apparatus and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquire sound-related data corresponding to the parameter information; and control output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from the one or more sensor. (18) The apparatus according to (17), further comprising an audio transducer, wherein the processing circuitry is configured to cause the sound corresponding to the virtual representation to be output by the audio transducer. (19) The apparatus according to (17), wherein the parameter information includes information regarding a virtual item attached to the corresponding virtual object, and the processing circuitry is configured to cause a sound corresponding to the virtual item to be output. (20) The apparatus according to (19), wherein the one or more sensor is configured to acquire data relating to motion of the apparatus, and the processing circuitry is configured to cause the sound corresponding to the virtual item to be output in accordance with a predetermined motion of the apparatus. (21) The apparatus according to (20), wherein the predetermined motion includes a moving motion. (22) The apparatus according to (17), wherein the parameter information includes at least one of: information regarding a virtual object arranged in the corresponding virtual space, or information regarding a region with a predetermined attribute that is set in the corresponding virtual space, and the processing circuitry is configured to cause at least one of: a sound corresponding to the virtual object to be output, or a sound corresponding to the region with the predetermined attribute to be output. (23) The apparatus according to (22), wherein the one or more sensor is configured to acquire data relating to motion of the apparatus, and the processing circuitry is configured to cause at least one of: the sound corresponding to the virtual object to be output when the apparatus enters or approaches a first corresponding region, in the real space, corresponding to a region in which the virtual object is arranged in the corresponding virtual space, or the sound corresponding to the region with the predetermined attribute to be output when the apparatus enters or approaches a second corresponding region, in the real space, corresponding to the region with the predetermined attribute in the corresponding virtual space. (24) The apparatus according to (23), wherein the processing circuitry is configured to cause at least one of: the sound corresponding to the virtual object to be output in accordance with a predetermined motion of the apparatus in the first corresponding region, or the sound corresponding to the region with the predetermined attribute to be output in accordance with a predetermined motion of the apparatus in the second corresponding region. (25) The apparatus according to (17), wherein the corresponding virtual space is a map corresponding to the reality space, and the map is generated on a basis of the data from the one or more sensor. (26) The apparatus according to (17), wherein the parameter information includes a proficiency level of the corresponding virtual object regarding a predetermined target to be learned, and the processing circuitry is configured to cause a sound corresponding to the proficiency level to be output. (27) The apparatus according to (26), wherein the proficiency level includes a proficiency level of the corresponding virtual object regarding a musical performance. (28) The apparatus according to (27), wherein the sound-related data corresponding to the parameter information includes: musical performance sound data, or sound setting information for generating the musical performance sound data, and the processing circuitry is configured to perform modification processing on the musical performance sound data on a basis of the proficiency level. (29) The apparatus according to (27), wherein the sound-related data corresponding to the parameter information includes: a plurality of pieces of corresponding musical performance sound data, or sound setting information for generating the plurality of pieces of corresponding musical performance sound data, and the processing circuitry is configured to select any one of the plurality of pieces of corresponding musical performance sound data on a basis of the proficiency level. (30) The apparatus according to (17), wherein the processing circuitry is configured to cause the sound corresponding to the virtual representation to be output when a condition for causing the sound corresponding to the virtual representation to be output is satisfied. (31) The apparatus according to (17), wherein the apparatus is capable of automated moving. (32) The apparatus according to (17), wherein acquire the sound-related data corresponding to the parameter information includes generating the sound-related data corresponding to the parameter information. (33) The apparatus according to (17), wherein the processing circuitry is configured to cause the apparatus to move. (34) The apparatus according to (33), further comprising a plurality of drive units each including a motor configured to perform a rotation operation around an axis, an encoder configured to detect a rotation position of the motor, and a driver configured to control the rotation position of the motor and a rotation speed of the motor on a basis of the output of the encoder, wherein causing the apparatus to move includes controlling the plurality of drive units. (35) A method, comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus. (36) A non-transitory computer readable medium storing instructions which when executed by a computer cause the computer to perform a method, the method comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus.

[0313] It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and alterations may occur depending on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.

[0314] RS reality space VS virtual space entertainment system real robot user terminal DB network user virtual robot 19 paw of real robot 28 shoes 29 paw of virtual robot 30 clothing 31 attachment data 32 item data 37 map 40 to 43 virtual object 60 computer

Claims

1. An apparatus capable of motion, the apparatus comprising: one or more sensor; processing circuitry configured to: acquire parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to the apparatus and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquire sound-related data corresponding to the parameter information; and control output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from the one or more sensor.

2. The apparatus according to claim 1, further comprising an audio transducer, wherein the processing circuitry is configured to cause the sound corresponding to the virtual representation to be output by the audio transducer.

3. The apparatus according to claim 1, wherein the parameter information includes information regarding a virtual item attached to the corresponding virtual object, and the processing circuitry is configured to cause a sound corresponding to the virtual item to be output.

4. The apparatus according to claim 3, wherein the one or more sensor is configured to acquire data relating to motion of the apparatus, and the processing circuitry is configured to cause the sound corresponding to the virtual item to be output in accordance with a predetermined motion of the apparatus.

5. The apparatus according to claim 4, wherein the predetermined motion includes a moving motion.

6. The apparatus according to claim 1, wherein the parameter information includes at least one of: information regarding a virtual object arranged in the corresponding virtual space, or information regarding a region with a predetermined attribute that is set in the corresponding virtual space, and the processing circuitry is configured to cause at least one of: a sound corresponding to the virtual object to be output, or a sound corresponding to the region with the predetermined attribute to be output.

7. The apparatus according to claim 6, wherein the one or more sensor is configured to acquire data relating to motion of the apparatus, and the processing circuitry is configured to cause at least one of: the sound corresponding to the virtual object to be output when the apparatus enters or approaches a first corresponding region, in the real space, corresponding to a region in which the virtual object is arranged in the corresponding virtual space, or the sound corresponding to the region with the predetermined attribute to be output when the apparatus enters or approaches a second corresponding region, in the real space, corresponding to the region with the predetermined attribute in the corresponding virtual space.

8. The apparatus according to claim 7, wherein the processing circuitry is configured to cause at least one of: the sound corresponding to the virtual object to be output in accordance with a predetermined motion of the apparatus in the first corresponding region, or the sound corresponding to the region with the predetermined attribute to be output in accordance with a predetermined motion of the apparatus in the second corresponding region.

9. The apparatus according to claim 1, wherein the corresponding virtual space is a map corresponding to the reality space, and the map is generated on a basis of the data from the one or more sensor.

10. The apparatus according to claim 1, wherein the parameter information includes a proficiency level of the corresponding virtual object regarding a predetermined target to be learned, and the processing circuitry is configured to cause a sound corresponding to the proficiency level to be output.

11. The apparatus according to claim 10, wherein the proficiency level includes a proficiency level of the corresponding virtual object regarding a musical performance.

12. The apparatus according to claim 11, wherein the sound-related data corresponding to the parameter information includes: musical performance sound data, or sound setting information for generating the musical performance sound data, and the processing circuitry is configured to perform modification processing on the musical performance sound data on a basis of the proficiency level.

13. The apparatus according to claim 11, wherein the sound-related data corresponding to the parameter information includes: a plurality of pieces of corresponding musical performance sound data, or sound setting information for generating the plurality of pieces of corresponding musical performance sound data, and the processing circuitry is configured to select any one of the plurality of pieces of corresponding musical performance sound data on a basis of the proficiency level.

14. The apparatus according to claim 1, wherein the processing circuitry is configured to cause the sound corresponding to the virtual representation to be output when a condition for causing the sound corresponding to the virtual representation to be output is satisfied.

15. The apparatus according to claim 1, wherein the apparatus is capable of automated moving.

16. The apparatus according to claim 1, wherein acquire the sound-related data corresponding to the parameter information includes generating the sound-related data corresponding to the parameter information.

17. The apparatus according to claim 1, wherein the processing circuitry is configured to cause the apparatus to move.

18. The apparatus according to claim 17, further comprising a plurality of drive units each including a motor configured to perform a rotation operation around an axis, an encoder configured to detect a rotation position of the motor, and a driver configured to control the rotation position of the motor and a rotation speed of the motor on a basis of the output of the encoder, wherein causing the apparatus to move includes controlling the plurality of drive units.

19. A method, comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus.

20. A non-transitory computer readable medium storing instructions which when executed by a computer cause the computer to perform a method, the method comprising: acquiring parameter information of a virtual representation relating to at least one of: a corresponding virtual object that corresponds to an apparatus capable of motion and that is provided in a virtual space, or a corresponding virtual space that corresponds to a real space in which the apparatus is provided; acquiring sound-related data corresponding to the parameter information; and controlling output of a sound corresponding to the virtual representation based on the acquired sound-related data, the acquired parameter information, and data from one or more sensor of the apparatus.

Images