Movable device and information processing method

Abstract

To provide a movable device and an information processing method that can provide users with a high-quality user experience. [Solution] To achieve the above objective, a movable device according to one embodiment of this technology is a movable device constructed in real space and comprises one or more sensors, an acquisition unit, and an output control unit. The acquisition unit acquires parameter information of a virtual representation related to at least one of a corresponding virtual object corresponding to the movable device constructed in virtual space and a corresponding virtual space corresponding to the real space, and sound-related data corresponding to the parameter information. The output control unit controls the output of sound corresponding to the virtual representation using the acquired sound-related data, based on the acquired parameter information and sensing data from the one or more sensors.

Description

Technical Field

[0001] The present technology relates to a movable device such as an autonomous mobile robot and an information processing method.

Background Art

[0002] Patent Document 1 discloses a technology that enables taking an image suitable for image processing for recognizing an object existing in the surroundings, taking as an example a dog-shaped robot capable of quadruped walking.

[0003] Patent Document 2 discloses an entertainment system capable of providing a new environment in which a human touches a robot.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0005] The development of movable devices such as autonomous mobile robots has advanced and is used, for example, as an entertainment robot. There is a demand for a technology capable of providing a high-quality user experience to users who use such movable devices.

[0006] In view of the above circumstances, an object of the present technology is to provide a movable device and an information processing method capable of providing a high-quality user experience to users.

Means for Solving the Problems

[0007] To achieve the above objective, a movable device according to one embodiment of this technology is a movable device constructed in real space and comprises one or more sensors, an acquisition unit, and an output control unit. The acquisition unit acquires parameter information of a virtual representation related to at least one of a corresponding virtual object corresponding to the movable device constructed in the virtual space and a corresponding virtual space corresponding to the real space, and sound-related data corresponding to the parameter information. The output control unit controls the output of the sound corresponding to the virtual representation using the acquired sound-related data, based on the acquired parameter information and sensing data from the one or more sensors.

[0008] This movable device acquires parameter information for a virtual representation related to at least one of a corresponding virtual object corresponding to the movable device and a corresponding virtual space corresponding to the real space, as well as sound-related data corresponding to the parameter information. Based on the parameter information and sensing data, the sound output corresponding to the virtual representation is controlled using the sound-related data. This makes it possible to provide a high-quality user experience to users of the movable device.

[0009] The movable device may further include a sound output unit. In this case, the output control unit may control the output of sound corresponding to the virtual representation from the sound output unit.

[0010] The parameter information may include information about a virtual item attached to the corresponding virtual object. In this case, the output control unit may output a sound corresponding to the virtual item.

[0011] The one or more sensors mentioned above may acquire data relating to the movement of the movable device. In this case, the output control unit may output a sound corresponding to the virtual item in accordance with a predetermined movement of the movable device.

[0012] The output control unit may output a sound corresponding to the virtual item in accordance with the movement of the movable device.

[0013] The parameter information may include at least one of the following: information about a virtual object placed in the corresponding virtual space, and information about a region of a predetermined attribute set in the corresponding virtual space. In this case, the output control unit may output at least one of the sound corresponding to the virtual object and the sound corresponding to the region of the predetermined attribute.

[0014] The one or more sensors mentioned above may acquire data relating to the movement of the movable device. In this case, the output control unit may output a sound corresponding to the virtual object when the movable device enters or approaches a first corresponding area in the real space that corresponds to an area in the corresponding virtual space where the virtual object is located, and output a sound corresponding to an area of ​​predetermined attribute when the movable device enters or approaches a second corresponding area in the real space that corresponds to an area of ​​predetermined attribute in the corresponding virtual space.

[0015] The output control unit may output a sound corresponding to the virtual object in accordance with a predetermined movement of the movable device in the first corresponding area, and output a sound corresponding to the area of ​​the predetermined attribute in accordance with a predetermined movement of the movable device in the second corresponding area.

[0016] The corresponding virtual space may be a map corresponding to the real space, generated based on sensing data from one or more sensors.

[0017] The parameter information may include the proficiency level of the corresponding virtual object with respect to a predetermined learning target. In this case, the output control unit may output a sound corresponding to the proficiency level.

[0018] The aforementioned level of proficiency may include the level of proficiency of the corresponding virtual object related to performance.

[0019] The sound-related data corresponding to the parameter information may include successful performance sound data that is data of the performance sound when the performance is successful, or sound setting information for generating the successful performance sound data. In this case, the output control unit may execute a modification process on the data of the successful performance sound based on the proficiency level to output a sound corresponding to the proficiency level.

[0020] The sound-related data corresponding to the parameter information may include data of a plurality of corresponding performance sounds corresponding to the proficiency level, or sound setting information for generating the data of the plurality of corresponding performance sounds corresponding to the proficiency level. In this case, the output control unit may select any one of the data of the plurality of corresponding performance sounds based on the proficiency level to output a sound corresponding to the proficiency level.

[0021] The output control unit may execute the output of the sound corresponding to the virtual expression when the condition for executing the output of the sound corresponding to the virtual expression is satisfied.

[0022] The movable device may be configured as a movable device capable of autonomous movement.

[0023] An information processing method according to an aspect of the present technology is an information processing method executed by a computer system, including acquiring parameter information of a virtual expression related to at least one of a corresponding virtual object constructed in a virtual space corresponding to a movable device constructed in the real space and a corresponding virtual space corresponding to the real space, and sound-related data corresponding to the parameter information. Based on the acquired parameter information and sensing data from one or more sensors of the movable device, the output of the sound corresponding to the virtual expression is controlled using the acquired sound-related data.

Brief Description of the Drawings

[0024] [Figure 1] It is a schematic diagram showing a configuration example of an entertainment system according to an embodiment of the present technology. [Figure 2]This is a block diagram showing an example of the configuration of a real robot. [Figure 3] This flowchart shows the basic flow of sound output corresponding to virtual representation. [Figure 4] This flowchart shows an example of processing within a user terminal (application) to output sound corresponding to a virtual item. [Figure 5] This is a schematic diagram showing an example of the configuration of the shoe selection screen. [Figure 6] This is a schematic diagram showing an example of shoe-wearing information recorded in the database. [Figure 7] This flowchart shows an example of processing within a real robot to produce sound output corresponding to a virtual item. [Figure 8] This is a schematic diagram illustrating an example of a map corresponding to real space. [Figure 9] This flowchart shows an example of processing within a user terminal (application) to execute sound output corresponding to a virtual object. [Figure 10] This flowchart shows an example of processing within a real robot to produce sound output corresponding to a virtual object. [Figure 11] This is a schematic diagram showing an example of sound output corresponding to different levels of proficiency. [Figure 12] This is a block diagram showing an example of a computer hardware configuration that can be used to build an entertainment system related to this technology. [Modes for carrying out the invention]

[0025] The embodiments of this technology will be described below with reference to the drawings.

[0026] [Overview of the Entertainment System] An overview of an entertainment system according to one embodiment of this technology will be described below. The entertainment system described below corresponds to one embodiment of the information processing system related to this technology.

[0027] Figure 1 is a schematic diagram showing an example configuration of the entertainment system 1. As shown in Figure 1, the entertainment system 1 includes a robot 2, a user terminal 3, and a database (DB) 4.

[0028] Robot 2, user terminal 3, and DB4 have, as a basic configuration, the necessary computer hardware such as a processor (CPU, GPU, DSP, etc.), memory (ROM, RAM, etc.), and storage devices (HDD, etc.) (see Figure 12). The processor loads the program related to this technology, which is stored in the storage unit or memory, into RAM and executes it, thereby executing the information processing method (control method for movable devices) related to this technology.

[0029] As shown in Figure 1, the robot 2, user terminal 3, and DB 4 are connected to each other via network 5, enabling communication. Network 5 is constructed using, for example, the internet or a wide-area communication network. In addition, any WAN (Wide Area Network) or LAN (Local Area Network) may be used, and the protocol for constructing network 5 is not limited.

[0030] Furthermore, in order to achieve a communication-enabled connection for each device shown in Figure 1, it is possible to use any wireless communication technology, such as Wi-Fi or other wireless LAN communication, Bluetooth® or other short-range wireless communication, or LTE or other mobile communication.

[0031] Robot 2 is constructed as an autonomously mobile entertainment robot. As shown in Figure 1, in this embodiment, a quadrupedal, dog-shaped robot 2 is used.

[0032] User 6 can engage in various forms of communication with Robot 2 in the real-world RS environment, resulting in a highly enjoyable user experience. For example, User 6 can touch or talk to Robot 2. In response, Robot 2 can perform actions that show it delight or respond by barking.

[0033] Furthermore, Robot 2 can freely walk and run within the real-world RS. For example, Robot 2 can follow User 6 around or play chase with User 6.

[0034] User terminal 3 is a terminal used by user 6 and includes, for example, a shooting function, an image display function, and an input reception function. The shooting function can be implemented by equipping it with a shooting device such as a digital camera. The display function can be implemented by equipping it with a display device such as an LCD or EL. The input reception function can be implemented by equipping it with an operation device such as a keyboard, pointing device, or touch panel.

[0035] In the example shown in Figure 1, a smartphone is used as the user terminal 3. The smartphone's digital camera (not shown) provides the shooting function. The smartphone's touch panel 7 provides the image display and input reception functions. Other computers, such as tablet devices or notebook PCs, can also be used as the user terminal 3.

[0036] DB4 will be built as a cloud-based DB4. To build this cloud-based DB4, computers such as DB servers may be used as appropriate.

[0037] DB4 stores various information and data related to this entertainment system 1. For example, DB4 stores information about robot 2, user 6, user terminal 3, etc. DB4 also stores parameter information for virtual representations, which will be explained later, and sound-related data corresponding to that parameter information.

[0038] [Integration between the real world RS and the virtual world VS] In this entertainment system 1, user 6 can also communicate with virtual robot 8 in the virtual space VS via user terminal 3.

[0039] For example, user 6 downloads an application (application program) to use the entertainment system 1 to user terminal 3. For example, user 6 creates an account to use the entertainment system 1 by entering information such as an ID and password. This account information is stored in DB4 as user information for user 6.

[0040] User 6 activates the registration function within the application and registers Robot 2, with whom they are spending time in the real-world RS. For example, registration of Robot 2 is performed by registering equipment information such as the model number of Robot 2, or by reading information such as the coat information attached to Robot 2. The registered information of Robot 2 is stored in DB4.

[0041] Once registration of robot 2 is complete, a virtual robot 8 corresponding to the robot 2 (hereinafter referred to as "real robot 2") constructed in the real-world space RS is displayed on the touch panel 7 of the user terminal 3, as shown in Figure 1. User 6 can communicate with the virtual robot 8 displayed on the user terminal 3 in various ways.

[0042] For example, suppose user 6 is spending time at home with the actual robot 2. When user 6 goes out, user 6 uses user terminal 3 to launch the application of this entertainment system 1.

[0043] Furthermore, it becomes possible to pet and talk to the virtual robot 8 displayed on the user terminal 3. It is also possible to give snacks and food to the virtual robot 8 in the virtual space VS.

[0044] Information regarding various communications between user 6 and virtual robot 8 is stored in DB4. This communication information is then downloaded to and shared with the actual robot 2 as needed.

[0045] For example, the actual robot 2 may request information regarding communication with the virtual robot 8 from DB4 at a predetermined timing or based on a predetermined trigger. Alternatively, DB4 may send information regarding communication to the actual robot 2 at a predetermined timing or based on a predetermined trigger.

[0046] This allows the entertainment system 1 to share the results of various communications that user 6 has with virtual robot 8 with the actual robot 2. For example, suppose user 6 pets virtual robot 8 while out, increasing the virtual robot 8's affinity with user 6. In this case, this increase in affinity is reflected in the actual robot 2, and when user 6 returns home, the actual robot 2 can perform actions corresponding to the increased affinity.

[0047] Furthermore, suppose that user 6 feeds virtual robot 8 while out, causing virtual robot 8's satiety level to increase (hungry level to decrease). In this case, this increase in satiety is reflected in the real robot 2, and when user 6 returns home, the real robot 2 may behave as if it is full.

[0048] Thus, this entertainment system 1 makes it possible to link communication with a real robot 2 in the real space RS and communication with a virtual robot 8 in the virtual space VS, thereby providing a very high-quality user experience.

[0049] In this embodiment, the real robot 2 corresponds to one embodiment of a movable device constructed in real space according to this technology, and one embodiment of a mobile device capable of autonomous movement according to this technology. The virtual robot 8 corresponds to one embodiment of a corresponding virtual object (a corresponding virtual object constructed in virtual space in relation to a movable device constructed in real space) that corresponds to a movable device constructed in virtual space.

[0050] Figure 2 is a block diagram showing an example configuration of the actual robot 2. As shown in Figure 2, the actual robot 2 has a sensor unit 9, a speaker 10, a light source unit 11, a display 12, and a switch 13. The actual robot 2 also has a control unit 14, a drive unit 15, a storage unit 16, a communication unit 17, and a power supply unit 18.

[0051] The sensor unit 9 is composed of one or more sensors and includes any sensors capable of detecting information (data) about the environment surrounding the actual robot 2 and information (data) about the state of the actual robot 2. For example, any sensors such as an image sensor (camera), distance sensor, touch sensor, microphone, GPS sensor, GNSS sensor, IMU (Inertial Measurement Unit) sensor, compass, human presence sensor, illuminance sensor, temperature sensor, humidity sensor, etc. may be installed.

[0052] For the image sensor (camera), for example, a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor may be used. An infrared camera may also be incorporated.

[0053] Various types of distance measuring sensors can be used, such as optical laser distance measuring sensors, ultrasonic distance measuring sensors, stereo cameras, ToF (Time of Flight) sensors, LiDAR (Light Detection and Ranging, Laser Imaging Detection and Ranging), and Structured Light distance measuring sensors.

[0054] Various types of touch sensors can be used, such as pressure-sensitive touch sensors and capacitive touch sensors.

[0055] The specific configuration of the microphone, GPS sensor, GNSS sensor, IMU sensor, compass, motion sensor, illuminance sensor, temperature sensor, humidity sensor, etc., is not limited, and any configuration may be adopted. Of course, other sensors may also be included.

[0056] In this embodiment, the sensor unit 9 is constructed to include a position sensor unit, a motion sensor unit, and a landing sensor unit. The position sensor unit is configured to detect data related to the position of the actual robot 2 in the real space RS. For example, the position sensor unit can be realized using an image sensor, a distance measuring sensor, a GPS sensor, a GNSS sensor, etc. Of course, other devices may also be used.

[0057] The motion sensor unit is configured to detect data related to the movement of the actual robot 2. For example, the motion sensor unit can be implemented using an IMU sensor, a compass, etc. Of course, other devices may also be used.

[0058] The landing sensor unit is configured to detect when the four legs 19 (19a to 19d) of the actual robot 2 touch the ground. For example, by placing switches on the tips of the four legs 19 of the actual robot 2 (e.g., the soles of the feet), it is possible to detect the landing of each leg 19. Alternatively, distance measuring sensors can be placed on the tips of the four legs 19, and the landing of each leg 19 can be detected based on the detected distance to the ground. It is also possible to detect the landing of each leg 19 based on the detection results of the IMU sensor.

[0059] The speaker 10 is capable of outputting sound, and its specific configuration is not limited. The speaker 10 corresponds to one embodiment of the sound output unit according to this technology.

[0060] The light source unit 11 is composed of, for example, an LED and is installed in various parts of the actual robot 2. The light emitted by the light source unit 11 makes it possible to present various information to the user 6. It is also possible to express emotions such as joy, anger, sadness, and happiness through the light emitted by the light source unit 11. Furthermore, it is possible to indicate the remaining power (battery) level and the connection status of the network, etc., through the light emitted by the light source unit 11.

[0061] The display 12 is composed of, for example, liquid crystal, electroluminescent, etc., and is capable of displaying various images. In this embodiment, a small display is placed at the eye position of the actual robot 2. For example, by displaying various eye images on the display 12, it is possible to express various facial expressions.

[0062] For example, it is possible to place a self-emissive display device (OLED) as a display 12 in each of the left and right eyes of the actual robot 2. By placing OLEDs, it becomes possible to realize various expressions related to the eyes, such as blinking, the whites of the eyes, the pupils, and the movement of the pupils, in each of the left and right eyes. Furthermore, by placing lenses or cover glass to cover the OLEDs, it is also possible to construct the left and right eyes in a three-dimensional manner.

[0063] For example, by positioning a lens to cover the OLED, it is possible to refract light to create a wider display on the OLED. The lens can also be used to represent the spherical shape of an eyeball. Furthermore, by positioning a cover glass to cover the lens, it becomes possible to create a curved surface that is continuous with the surface of the head of the actual robot 2, allowing the user 6 to experience a smooth sensation when touching the eyes and surrounding area of ​​the actual robot 2.

[0064] Switch 13 could include, for example, a power button, volume buttons, or a switch for wireless communication connection. Of course, it is not limited to these switches.

[0065] The drive unit 15 is a mechanism for realizing the movements of the actual robot 2, such as quadrupedal walking. The drive unit 15 includes, for example, drive units arranged at each joint of the actual robot 2. The drive unit includes a motor that performs rotational motion around an axis, an encoder that detects the rotational position of the motor, and a driver that adaptively controls the rotational position and rotational speed of the motor based on the output of the encoder.

[0066] For example, drive units are positioned in each of the actual robot's parts, such as the head, mouth, neck, forelegs, hind legs, tail, waist, and ears, to allow for movement on one or more axes. By having each drive unit operate appropriately, the robot can perform various movements, including quadrupedal walking.

[0067] The memory unit 16 is a storage device such as non-volatile memory, for example, an HDD or SSD. Alternatively, any non-transient storage medium that is computer-readable may be used.

[0068] The communication unit 17 is a module for performing network communication, short-range wireless communication, mobile communication, etc., with other devices. For example, it may be equipped with a wireless LAN module such as WiFi, a communication module such as Bluetooth®, or a mobile communication module such as LTE.

[0069] The power supply unit 18 supplies power to various parts within the actual robot 2. The power supply unit 18 consists of, for example, a rechargeable battery and a charge / discharge control unit that manages the charging and discharging state of the rechargeable battery.

[0070] The control unit 14 controls the operation of each block of the actual robot 2. The control unit 14 has the necessary computer hardware, such as a processor, memory, and storage devices.

[0071] The control unit 14 may be a PLD (Programmable Logic Device) such as an FPGA (Field Programmable Gate Array), or other devices such as an ASIC (Application Specific Integrated Circuit).

[0072] In this embodiment, the control unit 14 executes a program related to this technology, thereby realizing the following functional blocks: a recognition unit 20, a recognition information management unit 21, an action control unit 22, a drive control unit 23, a sound output control unit 24, a parameter information acquisition unit 25, and a sound-related data acquisition unit 26. Dedicated hardware such as ICs (integrated circuits) may be used as appropriate to realize each functional block.

[0073] The program is installed on the actual robot 2, for example, via various recording media. Alternatively, the program may be installed via the internet or the like. The type of recording media on which the program is stored is not limited; any computer-readable recording media may be used. For example, any non-transient storage medium that is computer-readable may be used.

[0074] The recognition unit 20 performs recognition processing based on the detection results from the sensor unit 9 and recognizes the environment surrounding the actual robot 2. For example, the recognition unit 20 performs human recognition processing, speech recognition processing, and spatial recognition processing.

[0075] Human recognition processing makes it possible to recognize the presence or absence of people in the surrounding space based on image information, audio information, etc., detected by, for example, the sensor unit 9. Furthermore, human recognition processing makes it possible to identify individuals in the surrounding space, and to recognize the gaze and facial expressions of the people whose presence has been recognized.

[0076] Through speech recognition processing, it is possible to recognize the content (context) of user 6's speech and conversations in the surrounding space, for example, based on speech information detected by the sensor unit 9.

[0077] Spatial recognition processing enables the recognition of objects or shapes in the surrounding space by performing obstacle recognition, shape recognition (e.g., wall recognition, floor recognition, etc.), or object recognition based on image information and distance information detected by, for example, the position sensor unit 9. Furthermore, spatial recognition processing can also recognize, for example, the width, distance, or surface irregularities of a passable path to a destination.

[0078] The specific algorithm used for recognition processing by the recognition unit 20 is not limited, and any algorithm may be used. For example, any machine learning algorithm using DNN (Deep Neural Network), RNN (Recurrent Neural Network), CNN (Convolutional Neural Network), etc., may be used. For example, by using an AI (Artificial Intelligence) that performs deep learning, it is possible to perform recognition processing of the surrounding environment with high accuracy.

[0079] The application of machine learning algorithms may be performed on any process described in this disclosure. In other words, machine learning-based processing may be performed on any process described in this disclosure. Of course, it is not limited to machine learning algorithms; any rule-based algorithm can also be used.

[0080] The recognition information management unit 21 manages each piece of information recognized by the recognition unit 20 and converts it into information suitable for use by other blocks such as the action control unit 22.

[0081] For example, the recognition information management unit 21 can determine whether or not user 6 has been detected in the surrounding space, and calculate the degree to which user 6 is paying attention to the actual robot 2. The degree to which user 6 is paying attention to the actual robot 2 can be used, for example, as a factor in deciding whether or not to have the actual robot 2 perform emotional expressions that evoke affection for user 6.

[0082] In this embodiment, the recognition information management unit 21 can generate a map of the surrounding space based on the recognition results of the recognition unit 20. For example, the recognition information management unit 21 can generate an obstacle map or a movement area map showing passable areas. It can also generate an object map showing the locations of various objects in the surrounding space, or a topological map showing the names, relationships, or meanings of each area.

[0083] The specific algorithm for generating a map of the surrounding space is not limited; for example, any algorithm such as SLAM (Simultaneous Localization and Mapping) may be used. In addition, any self-localization algorithm may be used.

[0084] Thus, in this embodiment, the recognition unit 20 and the recognition information management unit 21 can generate a map corresponding to the real-world space RS based on sensing data from the sensor unit 9 (one or more sensors). The generated map is stored in DB4 and downloaded to the user terminal 3. It is then displayed on the touch panel 7 of the user terminal 3 as a corresponding virtual space that corresponds to the real-world space RS.

[0085] The action control unit 22 determines the actions to be performed by the actual robot 2 based on the recognition information generated by the recognition information management unit 21 and various parameter information indicating the state of the actual robot 2 obtained from DB4. In other words, the action control unit 22 determines what kind of movements the actual robot 2 will make and what kind of sounds it will output. The action control unit 22 also controls the illumination of the light source unit 11 and the display of various eye images on the display 12 positioned at the eyes.

[0086] Furthermore, the movements and sound outputs of the actual robot 2 to user 6 can also be considered expressions of the actual robot to user 6. For example, the movements of the actual robot 2 to user 6 can be expressed as an expression of movement. Similarly, the sound output can be expressed as an expression of sound.

[0087] The behavior control unit 22 can also be described as an expression control unit that controls the state of the actual robot 2 and controls the expression of sound and movement.

[0088] For example, the behavior control unit 22 can make the actual robot 2 walk or run to its destination. In doing so, it can determine the path for the actual robot 2 to move to the destination based on the map generated by the recognition information management unit 21.

[0089] Furthermore, the behavior control unit 22 can determine whether or not to perform various emotional expressions that evoke affection in the user 6, and can make the actual robot 2 perform various movements to realize these emotional expressions. For example, it can make the actual robot 2 perform various movements such as jumping in place, wagging its tail, opening and closing its mouth, moving its ears, making a paw-shaking gesture, lying down, turning its head away, making a gesture of eating, dancing, barking, singing, and growling. In addition, the behavior control unit 22 can determine various other actions of the actual robot 2.

[0090] The drive control unit 23 outputs control commands to the drive unit 15 to execute the actions determined by the action control unit 22. For example, based on the control signals output from the drive control unit 23, the drive units located at each joint of the actual robot 2 are driven, enabling various movements such as quadrupedal walking.

[0091] The sound output control unit 24 controls the output of various sounds from the speaker 10. For example, the sound output control unit 24 acquires sound data (such as audio files like wave files) from DB4 or the like and outputs it from the speaker 10.

[0092] Furthermore, in this embodiment, the sound output control unit 24 enables the dynamic generation of sound data (audio files). For example, it is possible to dynamically generate audio data by synthesizing multiple sound data or by processing predetermined sound data such as increasing or decreasing predetermined frequency components.

[0093] For example, the sound output control unit 24 downloads sound setting information from DB4 or the like for dynamically generating sound data. Sound data may be generated based on the downloaded sound setting information. Alternatively, sound data may be generated by processing or combining sound data previously stored in the storage unit 16, and sound may be output from the speaker 10.

[0094] In this embodiment, sound data and sound setting information are included in the sound-related data relating to this technology. That is, the sound-related data relating to this technology may be the sound waveform itself, the parameters of a synthesizer for generating sound, or data describing the filter processing applied to the sound waveform. Parameters for generating sound include, for example, the degree of filter application, the algorithm, and the pitch change.

[0095] The parameter information acquisition unit 25 and the sound-related data acquisition unit 26 will be described later.

[0096] [Sound output corresponding to virtual representation] As described above, this entertainment system 1 makes it possible to link communication with a real robot 2 in the real space RS and communication with a virtual robot 8 in the virtual space VS.

[0097] Regarding this collaboration between the real world RS and the virtual world VS, this entertainment system 1 is capable of outputting sound that corresponds to the virtual representation, enabling a very high-quality user experience. The following describes the sound output that corresponds to the virtual representation.

[0098] Figure 3 is a flowchart showing the basic flow of sound output corresponding to the virtual representation. The parameter information acquisition unit 25 acquires parameter information of the virtual representation related to at least one of the virtual robot 8 corresponding to the real robot 2 constructed in the virtual space VS, and the corresponding virtual space (map) corresponding to the real space RS (step 101).

[0099] The parameter information for the virtual representation related to the virtual robot 8 includes, for example, arbitrary parameter information for realizing various virtual representations related to the virtual robot 8 that are displayed on the touch panel 7 of the user terminal 3. For example, information about the state of the virtual robot 8, information about virtual items attached to the virtual robot 8, and the virtual robot 8's proficiency level with respect to a predetermined learning target are acquired as parameter information for the virtual representation.

[0100] The parameter information for virtual representations related to the corresponding virtual space (map) includes, for example, arbitrary parameter information for realizing various virtual representations related to the map displayed on the touch panel 7 of the user terminal 3. For example, information about virtual objects placed on the map and information about areas with predetermined attributes set on the map are acquired as parameter information for virtual representations.

[0101] In this disclosure, the acquisition of data (information) includes receiving such data (information) from other devices or other functional blocks, downloading such data (information) from a database or the like, generating such data (information) by executing a predetermined algorithm, and reading such data (information) that is pre-stored in the storage unit or the like of the machine itself.

[0102] The sound-related data acquisition unit 26 acquires sound-related data corresponding to the parameter information acquired in step 101 (step 102).

[0103] The behavior control unit 22 and the sound output control unit 24 control the output of sound corresponding to the virtual representation based on the parameter information acquired in step 101 and sensing data from the sensor unit 9 (one or more sensors), using the sound-related data acquired in step 102.

[0104] In this embodiment, the output control unit according to this technology is realized by the action control unit 22 and the sound output control unit 24. It is also possible to consider the recognition unit 20 and the recognition information management unit 21 as part of the output control unit according to this technology.

[0105] In other words, based on the recognition results from the recognition unit 20 and the recognition information management unit 21, the action control unit 22 determines the actions of the actual robot 2, including the output of sound corresponding to the virtual representation. Then, the sound output control unit 24 outputs sound corresponding to the virtual representation from the speaker 10. In this case, it can also be said that the recognition unit 20, the recognition information management unit 21, the action control unit 22, and the sound output control unit 24 control the output of sound corresponding to the virtual representation from the speaker 10.

[0106] The following describes a detailed embodiment of the sound output corresponding to the virtual representation.

[0107] (First embodiment: Sound output corresponding to a virtual item) This entertainment system 1 is capable of outputting sounds corresponding to virtual items attached to virtual robot 8 as sound outputs that correspond to virtual representations.

[0108] Figure 4 is a flowchart showing an example of processing within user terminal 3 (application) to output sound corresponding to a virtual item.

[0109] User 6 selects shoes to be worn by the virtual robot 8 within the application (step 201). For example, User 6 activates the virtual item attachment function within the application, and a screen (GUI) for selecting shoes to be worn by the virtual robot 8 is displayed.

[0110] Figure 5 is a schematic diagram showing an example configuration of the shoe selection screen 27. In the example shown in Figure 5, a virtual robot 8 is displayed near the center of the touch panel 7 of the user terminal 3. Below the virtual robot 8, the following shoes are displayed as candidates for wearing: shoes with bells 28a, wooden clogs 28b, and boa slippers 28c.

[0111] User 6 can appropriately select the left front leg 29a, right front leg 29b, left hind leg 29c, and right hind leg 29d of the virtual robot 8, and select the shoes 28 to be attached to each leg 29. It is also possible to attach the same type of shoe 28 to all four legs 29 of the virtual robot 8, or to attach different types of shoes 28 to each leg 29.

[0112] The shoes that can be worn are not limited to the shoes with bells 28a, wooden clogs 28b, and boa slippers 28c; any shoes such as high heels or sandals can be worn. It is also possible to adopt a model in which the user 6 purchases the shoes 28.

[0113] Information about shoe attachment to the virtual robot 8 is recorded in the cloud's DB4 (step 202).

[0114] Figure 6 is a schematic diagram showing an example of shoe-wearing information recorded in DB4. In this embodiment, the shoe-wearing data 31 shown in Figure 6A and the item data 32 shown in Figure 6B are recorded in DB4 as shoe-wearing information.

[0115] As shown in Figure 6A, the mounting data 31 includes the following information. "Individual"... Information used to identify the actual robot 2. In Figure 6A, the words "Robot A" are shown, but any information that can identify the actual robot 2, such as the model number or ID of the actual robot 2, may be used. "Part"... The part of the virtual robot 8 to which the virtual item is attached. In this embodiment, information for each of the four legs 29 of the virtual robot 8 is stored. "Item Type"... Information for identifying a virtual item. In this embodiment, information for three types of shoes 28 is stored as "Shoe A," "Shoe B," and "Shoe C." "Shoe A" corresponds to the shoe with a bell 28a shown in Figure 5, and "Shoe B" corresponds to the wooden shoe 28b. "Shoe C" corresponds to the boa slipper 28c. Any information that can identify the shoe 28 may be used as the information for the shoe 28. For "body parts" that do not have a virtual item attached, the "Item Type" will be blank.

[0116] In the mounting data 31 shown in Figure 6A, for the same virtual robot 8 (= "Robot A"), "Shoe A" is mounted on the left foreleg 29a, and "Shoe B" is mounted on the right foreleg 29b. Additionally, "Shoe C" is mounted on the left hind leg 29c, while no shoe is mounted on the right hind leg 29d.

[0117] As shown in Figure 6B, item data 32 includes the following information. "Item Type"... Information used to identify a virtual item (typically the same data as "Item Type" in the equipped data 31). "Sound Type"... The type of sound corresponding to the equipped item. "Sound Type" stores information about the type of sound output when a virtual item is equipped. In this embodiment, "Shoes A" stores the "bell sound," "Shoes B" stores the "wooden clogs sound," and "Shoes C" stores the "boa slipper" sound.

[0118] In this embodiment, each shoe 28 is associated with a pre-played sound, and this information is stored in DB4.

[0119] The shoe-wearing information shown in Figure 6 (wearing data 31 and item data 32) corresponds to the parameter information of the virtual representation related to the virtual robot 8.

[0120] Returning to Figure 4, the virtual robot 8 in the application is displayed with the shoes selected by user 6 attached (step 203). In the example shown in Figure 6, a shoe with a bell 28a is attached to the left foreleg 29a of the virtual robot 8, and a wooden shoe 28b is attached to the right foreleg 29b. A boa slipper 28c is attached to the left hind leg 29c of the virtual robot 8, and no shoe is attached to the right hind leg 29d. In this state, the virtual robot 8 is displayed on the touch panel 7 of the user terminal 3.

[0121] Figure 5 illustrates the state before the user 6 selects a shoe 28. Once the user 6 selects a shoe 28 for each foot 29, the virtual robot 8 is displayed with the selected shoe 28 attached to each foot 29.

[0122] Within the application, sounds corresponding to the shoes 28 worn by the virtual robot 8 are played in sync with the robot's walking. For example, sounds such as bells (e.g., "ring, ring, ring"), wooden clogs (e.g., "tap, tap, tap"), and boa slippers (e.g., "swoosh, swoosh, swoosh") are played in conjunction with each foot 29 of the virtual robot 8 landing within the application.

[0123] Figure 7 is a flowchart showing an example of processing within the actual robot 2 to produce sound output corresponding to a virtual item.

[0124] The parameter information acquisition unit 25 of the actual robot 2 downloads the shoe-wearing information recorded in DB4 from DB4 (step 301). The download of shoe-wearing information in step 301 corresponds to one embodiment of acquiring the parameter information of the virtual representation in step 101 shown in Figure 3.

[0125] The sound-related data acquisition unit 26 of the actual robot 2 determines whether or not there is sound-related data corresponding to the attached shoe 28 (step 302). In this embodiment, the sound-related data acquisition unit 26 refers to the shoe attachment information shown in Figure 6. It then determines whether or not there is sound-related data for outputting a "bell sound" corresponding to the bell-attached shoe 28a attached to the left front foot 29a of the virtual robot 8.

[0126] Furthermore, the sound-related data acquisition unit 26 determines whether or not there is sound-related data to output the "sound of wooden shoes" corresponding to the wooden shoes 28b attached to the right front leg 29b of the virtual robot 8. Also, the sound-related data acquisition unit 26 determines whether or not there is sound-related data to output the "sound of bore slippers" corresponding to the bore slippers 28c attached to the left rear leg 29c of the virtual robot 8.

[0127] In this embodiment, the sound-related data acquisition unit 26 determines whether sound-related data for outputting the "sound of a bell," sound-related data for outputting the "sound of wooden clogs," and sound-related data for outputting the "sound of boa slippers" are stored in DB4.

[0128] If none of the sound-related data for "bell sound," "wooden clogs sound," or "boa slipper sound" are stored, step 302 is rejected, and the output of the sound corresponding to the virtual item ends.

[0129] If any of the sound-related data for "bell sound," "wooden shoe sound," or "boa slipper sound" is stored, step 302 is affirmative, and the sound-related data stored in DB4 is downloaded (step 303). In this embodiment, each of the sound-related data for "bell sound," "wooden shoe sound," and "boa slipper sound" is stored in DB4 and downloaded by the sound-related data acquisition unit 26.

[0130] Steps 302 and 301 correspond to one embodiment of acquiring sound-related data in step 102, as shown in Figure 3.

[0131] In addition, sound-related data may be pre-stored in the memory unit 16 of the actual robot 2. In this case, the sound-related data acquisition unit 26 reads out the respective sound-related data for "bell sound," "wooden clogs sound," and "boa slipper sound" from the memory unit 16.

[0132] The action control unit 22 determines whether the output conditions for outputting sound corresponding to the virtual representation in the actual robot 2 are met (step 304). For example, if sound corresponding to the virtual representation is output constantly, it may be considered bothersome to the user 6. In this embodiment, such a problem can be solved by appropriately setting the output conditions.

[0133] Output conditions include, for example, outputting sound corresponding to the virtual representation for a certain period of time after the application puts on the shoes 28, and stopping the sound output after that period of time has elapsed. It is also possible to set conditions such as outputting sound corresponding to the virtual representation for a certain period of time after the application starts, or outputting sound corresponding to the virtual representation during limited times of the day (desired time period or predetermined time period).

[0134] Alternatively, it is possible to set conditions such as outputting sounds corresponding to virtual representations in designated areas. For example, it is possible to set conditions such as outputting sounds in the living room but not in the bedroom. It is also possible to set conditions such as not outputting sounds if a specific person (for example, a baby) is nearby.

[0135] Furthermore, it is possible to set output conditions for the movement and state of the actual robot 2. For example, it is possible to set a condition that the sound output corresponding to the virtual representation will not be performed when the battery level is low (when the robot is hungry). It is also possible to randomly determine whether or not to perform sound output using random numbers or other methods.

[0136] If the sound output conditions corresponding to the virtual representation are not met, step 304 is negated, and the sound output corresponding to the virtual item ends.

[0137] If the conditions for outputting a sound corresponding to the virtual representation are met, the output of a sound corresponding to the virtual representation is performed. In this embodiment, the process proceeds from step 304 to step 305, and the action control unit 22 and the sound output control unit 24 output footstep sounds corresponding to the shoes 28 attached to each foot 29 of the virtual robot 8 in conjunction with the landing of each foot 19 of the actual robot 2 shown in Figure 1.

[0138] In other words, the action control unit 22 determines an action that includes outputting footstep sounds corresponding to the shoes 28 attached to each foot 29 of the virtual robot 8 in conjunction with the landing of each foot 19 of the actual robot 2.

[0139] The sound output control unit 24 generates sound data for footsteps corresponding to the shoes 28 attached to each foot 29 of the virtual robot 8, based on the sound-related data downloaded from DB4. If the sound-related data is footstep sound data, that sound data is used as is. If the sound-related data is sound setting information, the sound output control unit 24 generates sound data for footsteps based on the sound setting information.

[0140] The sound output control unit 24 then outputs footstep sounds from the speaker 10 corresponding to the shoes 28 worn on each foot 29 of the virtual robot 8, in conjunction with the landing of each foot 19 of the actual robot 2, based on the detection results of the landing sensor unit built in the sensor unit 9.

[0141] In this embodiment, a bell sound is output in conjunction with the landing of the left front foot 19a of the actual robot 2 shown in Figure 1, and a wooden shoe sound is output in conjunction with the landing of the right front foot 19b. In addition, a boa slipper sound is output in conjunction with the landing of the left rear foot 19c of the actual robot 2.

[0142] Since the virtual robot 8's right hind leg 29d does not have a shoe attached, no footstep sound is emitted when the real robot 2's right hind leg 19d lands. Of course, it is also possible to set a footstep sound for when no shoe is attached, and have the footstep sound emitted in conjunction with the landing of the foot 19 which does not have a shoe attached.

[0143] In this embodiment, the walking of the actual robot 2 and the landing of each foot 19 correspond to one embodiment of a predetermined movement of the movable device according to this technology, and one embodiment of the movement of the movable device according to this technology. In this disclosure, the movement of the actual robot 2 landing each foot 19 in place is also included in walking and constitutes one embodiment of the movement of the movable device.

[0144] In this first embodiment, the sound output (operational sounds) of the real robot 2 changes depending on the virtual items worn by the virtual robot 8 in the virtual space SV. This makes it possible to attach items such as shoes, which would be difficult for the real robot 2, to the virtual robot 8, while also allowing the user 6 to feel a stronger connection between the real robot 2 and the virtual robot 8.

[0145] Furthermore, it is possible to change the sound corresponding to the virtual representation played by each of the 19 legs of the actual robot 2, depending on the intensity of the landing, the frequency of landings, the movement speed, acceleration, etc. For example, it is possible to change the sound corresponding to the virtual representation depending on whether the robot lands forcefully on the floor, walks slowly, or dashes.

[0146] Furthermore, it is possible to change the sound corresponding to the virtual representation based on the actual floor conditions in the real-world RV. It is also possible to change the sound corresponding to the virtual representation based on walking conditions, such as climbing stairs.

[0147] Furthermore, the sound corresponding to the virtual representation may be changed depending on the elapsed time since the shoes 28 were attached to the virtual robot 8, and other parameter information. For example, a predetermined time corresponding to the wear of the shoes 28 can be set, and when that time has elapsed, the playback of the footsteps can be stopped. Alternatively, parameter information such as the degree of wear can be set, and the footsteps can be made to gradually decrease in volume according to the degree of wear.

[0148] Of course, virtual items are not limited to shoes 28; this technology can be applied to any virtual item, such as clothes, hair accessories, sunglasses, or musical instruments like bells. For example, in the example shown in Figure 6, the virtual robot 8 is wearing clothes 30 modeled after Santa Claus. A bell is attached as a virtual item to the clothes 30 or around the neck of the virtual robot 8. Then, the sound of the bell is output in conjunction with the walking of the real robot 2 (all four feet 19 landing). Such processing is also possible.

[0149] If a bell or the like is attached to the neck of the virtual robot 8, the sound of the bell may be output in conjunction with the movement of the neck of the actual robot 2.

[0150] In addition, in conjunction with the wearing of the Santa Claus-like clothing 30, sounds such as bells commonly heard on Christmas Day may be output in conjunction with the walking of the actual robot 2. Furthermore, Christmas songs may be played in conjunction with the landing of each foot 19.

[0151] Furthermore, as virtual items, bells with different pitches may be attached to each leg 29 of the virtual robot 8. Then, the sounds of the bells with different pitches may be output in conjunction with the walking of the actual robot 2. Alternatively, a chord of bell sounds may be output in conjunction with the walking of the actual robot 2.

[0152] In addition, the movement, behavior, mood, and personality of the virtual robot 8 may be controlled by virtual items attached to it. For example, the following examples can be given. When socks are attached to the feet 29 of virtual robot 8, the real robot 2 will start walking slowly and quietly. When the virtual robot 8 is fitted with a dress, the real robot 3 begins to behave gracefully, walking slowly instead of stomping its feet or running. When cat ears are attached to virtual robot 8, the sounds of real robot 2 change to cat sounds, and its movements also become more cat-like. When a sheep's hat is placed on virtual robot 8, the sound of real robot 2 bleating changes to that of a sheep. When a motorcycle helmet is placed on virtual robot 8, real robot 2 will mimic motorcycle sounds and output motorcycle-related sound effects. When a collar with a bell is attached to the virtual robot 8, the sound of the bell is emitted in conjunction with the walking of the real robot 2. When a soccer uniform is placed on virtual robot 8, real robot 2 will pretend to kick the ball. When virtual robot 8 is given a musical instrument, real robot 2 will occasionally play phrases that sound like it's practicing that instrument. When virtual robot 8 is given a whistle, real robot 2 occasionally emits a whistle sound. When high heels are attached to the feet 19 of virtual robot 8, the way real robot 2 walks changes. Furthermore, this technology can be applied in various forms, enabling the provision of a high-quality user experience.

[0153] (Second embodiment: Sound output corresponding to a virtual object) In this entertainment system 1, as sound output corresponding to virtual representations, it is possible to output sounds corresponding to virtual objects placed in the corresponding virtual space. For example, it is possible to output sounds corresponding to virtual objects placed on a map corresponding to the real space RS, which is generated based on sensing data detected by the sensor unit 9.

[0154] Furthermore, the concept of a virtual object includes virtual items; for example, shoes 28 worn by the virtual robot 8 are also included in the concept of a virtual object.

[0155] Figure 8 is a schematic diagram showing an example of a map corresponding to the real-world RS. In the example shown in Figure 8, a two-dimensional map 37 is displayed on the touch panel 7 of the user terminal 3. Map 37 is a map that corresponds to the space in the real-world RS where the user 6 is spending time with the actual robot 2 (for example, their home).

[0156] In the map 37 shown in Figure 8, gray blocks indicate areas 38 that the actual robot 2 cannot pass through. These impassable areas 38 correspond to areas in the real-world space RS where obstacles such as desks and walls exist.

[0157] Furthermore, in the map 37 shown in Figure 8, the areas 39 that the actual robot 2 can traverse are indicated by white blocks. These traversable areas 39 correspond to areas where there are no obstacles such as desks or walls. The traversable areas 39 may be further divided into more detailed categories, such as areas that are easy to traverse and areas that are difficult to traverse.

[0158] Furthermore, the map 37 shown in Figure 8 displays the current position of the real robot 2 using a virtual dog object (icon) 40. The triangular marks displayed near the virtual dog object 40 indicate the current orientation of the real robot 2.

[0159] Furthermore, the map 37 shown in Figure 8 displays a virtual house object 41. This virtual house object 41 indicates the location of the charging station where the actual robot 2 is charged.

[0160] Figure 9 is a flowchart showing an example of processing within user terminal 3 (application) to execute sound output corresponding to a virtual object.

[0161] User 6 places a puddle as a virtual object in a predetermined area of ​​the map 37 corresponding to the real-world space RS within the application (step 401). For example, User 6 places a virtual object 42 of a puddle as illustrated in Figure 8. Note that the method and GUI for User 6 to select the virtual object to be placed on the map 37, and the method and GUI for placing the selected virtual object on the map 37, are not limited and any method and GUI may be used.

[0162] Information about the placement of the puddle virtual object 42 is recorded in the cloud's DB4 (step 402). For example, the type of virtual object (= puddle) and the location information of the area where the virtual object is placed are associated and recorded in DB4.

[0163] As location information for the area where the virtual object is placed, for example, location information according to the self-position information in the real space RS used to generate map 37 is recorded. That is, location information of the area in the real space RS corresponding to the area where the virtual object 42 of the puddle on map 37 is placed (hereinafter referred to as the first corresponding area) is recorded.

[0164] The placement information of the puddle virtual object 42 (type of virtual object and location information of the area where it is placed) corresponds to the parameter information of the virtual representation related to the corresponding virtual space.

[0165] As illustrated in Figure 8, a virtual object 42 representing a puddle is displayed on the map 37 within the application (step 403).

[0166] Figure 10 is a flowchart showing an example of processing within the actual robot 2 to produce sound output corresponding to a virtual object.

[0167] The parameter information acquisition unit 25 of the actual robot 2 downloads information about the arrangement of virtual objects 42 of the puddle recorded in DB4 from DB4 (step 501). The download of information about the arrangement of virtual objects 42 of the puddle in step 501 corresponds to one embodiment of acquiring parameter information of the virtual representation as shown in step 101 in Figure 3.

[0168] The sound-related data acquisition unit 26 of the actual robot 2 determines whether or not there is sound-related data corresponding to the virtual object 42 of the puddle (step 502). In this embodiment, the sound-related data acquisition unit 26 refers to the placement information of the virtual object 41 of the puddle downloaded from DB4. It then determines whether or not there is sound-related data for outputting a sound corresponding to the "puddle" placed on the map 37.

[0169] In this embodiment, the sound-related data acquisition unit 26 determines whether or not sound-related data for outputting a sound corresponding to "puddle" is stored in DB4. If sound-related data corresponding to "puddle" is not stored, step 502 is rejected, and the output of the sound corresponding to the virtual object ends.

[0170] If sound-related data corresponding to "puddle" is stored, step 502 is affirmative, and the sound-related data corresponding to "puddle" stored in DB4 is downloaded (step 503). The sound-related data corresponding to "puddle" is, for example, sound-related data for outputting the sound of footsteps when walking through a puddle in real space RS (e.g., "splish, splash, splash").

[0171] Note that sound-related data may be pre-stored in the memory unit 16 of the actual robot 2. In this case, the sound-related data acquisition unit 26 reads the sound-related data corresponding to "puddle" from the memory unit 16.

[0172] Steps 502 and 503 correspond to one embodiment of acquiring sound-related data in step 102, as shown in Figure 3.

[0173] The action control unit 22 determines whether the actual robot 2 has entered the area where the "puddle" is located, i.e., the first corresponding area, in the real space RS (step 504).

[0174] If the actual robot 2 has not entered the first corresponding area, step 504 is negated, and the sound output corresponding to the virtual object ends.

[0175] When the actual robot 2 enters the first corresponding area, sound output corresponding to the virtual object is executed. In this embodiment, the process proceeds from step 504 to step 505, and the action control unit 22 and sound output control unit 24 output footstep sounds as if walking through a puddle, in conjunction with the landing of each foot 19 of the actual robot 2 shown in Figure 1.

[0176] In other words, the action control unit 22 determines an action that includes outputting the sound of footsteps as if walking through a puddle, in conjunction with the landing of each leg 19 of the actual robot 2.

[0177] The sound output control unit 24 generates sound data for footsteps when walking through a puddle based on the sound-related data downloaded from DB4. If the sound-related data is footstep sound data, that sound data is used as is. If the sound-related data is sound setting information, the sound output control unit 24 generates footstep sound data based on the sound setting information.

[0178] The sound output control unit 24 then outputs the sound of footsteps walking through a puddle from the speaker 10 in conjunction with the landing of each foot 19 of the actual robot 2, based on the detection results of the landing sensor unit built into the sensor unit 9.

[0179] In this embodiment, when the real robot 2 enters the first corresponding area of ​​the real space RS that corresponds to the placement area of ​​the virtual object 42 of a puddle placed by the user 6 in the corresponding virtual space map 37, footstep sounds such as "splish, splash, splash" are output in response to the real robot 2's walking (landing of each foot 19).

[0180] In this embodiment, walking of the actual robot 2 in the first corresponding region (landing of each foot 19) corresponds to one embodiment of a predetermined movement of the movable device in the first corresponding region according to this technology.

[0181] In the example shown in Figure 8, a virtual object 43 representing a drum is also placed on the map 37.

[0182] For example, in step 401 shown in Figure 9, user 6 places a virtual drum object 43 on map 37. In step 402, information about the placement of the virtual drum object 43 is recorded in the cloud DB4. In step 403, the virtual drum object 42 is displayed on map 37 within the application.

[0183] In step 501 shown in Figure 10, the parameter information acquisition unit 25 of the actual robot 2 downloads the information on the placement of the virtual drum object 43 recorded in DB4 from DB4.

[0184] In step 502, the sound-related data acquisition unit 26 of the actual robot 2 determines whether or not there is sound-related data corresponding to the virtual object 43 of the drum. If sound-related data for outputting sounds corresponding to "drum" is stored in DB4, that sound-related data is downloaded. The sound-related data corresponding to "drum" is, for example, sound-related data for outputting the sound of drumming (e.g., "boom boom boom").

[0185] In step 504, it is determined whether the actual robot 2 has entered the area where the "drum" is placed, i.e., the first corresponding area, in the real space RS. If the actual robot 2 has entered the first corresponding area, in step 505, the sound of the drum is output in conjunction with the landing of each leg 19 of the actual robot 2. However, the sound of the drum may also be output if the actual robot 2 enters the first corresponding area without being linked to the landing of each leg 19.

[0186] In this way, it is possible to change the sound corresponding to the virtual representation by placing virtual objects within map 37. Of course, if the virtual drum object 43 moves, the location where the drum sound is emitted will also change accordingly.

[0187] Furthermore, the movement, behavior, mood, and personality may be controlled by virtual objects placed within map 37. For example, it is possible to have the actual robot 2, upon entering the first corresponding area, perform a drumming motion and output the sound of a drum. It is also possible to set it so that, for example, when placed on a virtual drum object 43, the actual robot 2 will play the drum when it feels like it. It is also possible to have the type of drum change from a toy drum to a Japanese drum, and the sound produced, depending on the robot's proficiency (skill level) in playing the drum.

[0188] In this way, the entertainment system 1 makes it possible to place various virtual objects on the map 37. When the actual robot 2 enters a first corresponding area that corresponds to the area where the virtual objects are placed, the actual robot 2 can output a sound corresponding to the virtual object.

[0189] For example, it's possible to set it up so that when the real robot 2 walks in the corresponding first area according to the placement of virtual objects in the "musical score," music is played with its footsteps. For example, it's possible to set it up so that the musical score progresses according to the landing of each of the real robot 2's feet 19. For example, if the "musical score" for Twinkle Twinkle Little Star is placed, the notes "Do Do Sol Sol La La Sol" will be played in order with each landing. For example, if the real robot 2 walks in the order of right front foot 19b, left hind foot 19c, left front foot 19a, right hind foot 19d, the notes will be played as Do, Do, Sol, Sol according to each landing. Note that no sound will be played when landing during rests. Also, different music will be played depending on the type of "musical score." Such settings are also possible.

[0190] For example, the sound of footsteps walking on snow or tap-dancing flooring can be output in response to virtual objects such as "snowy road" or "tap-dancing flooring." The sound of fireworks being launched can be output in response to the virtual object "fireworks." Various settings like these are possible.

[0191] In this second embodiment, the sound output (operational sound) of the actual robot 2 changes depending on the virtual objects placed on the map 37 corresponding to the real-world space RV. For example, it is difficult to actually place items that specify puddles in the real-world space RS, but by applying this technology, it is possible to achieve a similar effect.

[0192] (Third embodiment: Sound output corresponding to a range of predetermined attributes) Instead of placing virtual objects on the map 37 as described in the second embodiment, it is also possible to set a portion of the map 37 as an area with predetermined attributes.

[0193] For example, a portion of the map 37 can be set as an area with the attribute of a "puddle." This makes it possible to provide a user experience similar to that of placing a virtual puddle object 42 on the map 37.

[0194] For example, user 6 sets a portion of the map 37 as an area with the attribute "puddle" within the application. Information about the "puddle" area is recorded in the cloud's DB4. For example, the attribute (=puddle) and the location information of the set area are associated and recorded in DB4. A message indicating that an area has been set as a "puddle" is displayed on the map 37.

[0195] The information about the "puddle" area (attributes and location information of the defined area) corresponds to the parameter information of the virtual representation related to the corresponding virtual space.

[0196] The parameter information acquisition unit 25 of the actual robot 2 downloads information about the "puddle" area recorded in DB4 from DB4. This download of information about the "puddle" area corresponds to one embodiment of acquiring the parameter information of the virtual representation in step 101 shown in Figure 3.

[0197] The sound-related data acquisition unit 26 of the actual robot 2 determines whether or not there is sound-related data corresponding to the attribute region of "puddle". If sound-related data corresponding to the attribute region of "puddle" is stored in DB4, that sound-related data is downloaded. Sound-related data corresponding to the attribute region of "puddle" is, for example, sound-related data for outputting the sound of footsteps when walking in a puddle in the real space RS (e.g., "splish, splash, splash").

[0198] Note that sound-related data may be pre-stored in the memory unit 16 of the actual robot 2. In this case, the sound-related data acquisition unit 26 reads the sound-related data corresponding to the area with the "puddle" attribute from the memory unit 16.

[0199] Determining whether sound-related data exists in the area of ​​the "puddle" attribute, and downloading said sound-related data, corresponds to one embodiment of acquiring sound-related data in step 102 shown in Figure 3.

[0200] The behavior control unit 22 determines whether the actual robot 2 has entered the area of ​​the real-world RS that corresponds to the area of ​​the "puddle" attribute on the map 37 (hereinafter referred to as the second corresponding area).

[0201] When the actual robot 2 enters the second corresponding area, sound output corresponding to the area with the "puddle" attribute is performed. In this embodiment, the sound of footsteps walking through a puddle is output in conjunction with the actual robot 2's walking (landing of each foot 19).

[0202] The attributes are not limited to a specific set; attributes such as "snowy road" or "flooring for tap dancing" can be set for areas on map 37. It is also possible to set attributes such as "fireworks" for areas on map 37.

[0203] In this embodiment, the walking of the actual robot 2 in the second corresponding region (landing of each foot 19) corresponds to one embodiment of a predetermined movement of the movable device in the second corresponding region according to this technology.

[0204] Thus, in the third embodiment, the sound output (operational sound) of the actual robot 2 changes depending on the region of a predetermined attribute set in the map 37 corresponding to the real space RV. This makes it possible to achieve the same effect as in the second embodiment, in which virtual objects are placed on the map 37.

[0205] In the second and third embodiments, a three-dimensional map (3D map) may be generated as a map corresponding to the real-world space RS. Furthermore, the map is not limited to being generated by the actual robot 2; it may also be explicitly created by the user 6 or created by other means. It is also possible to download and use map data such as floor maps corresponding to the real-world space RS from an external source via the network 5. It is also possible to obtain maps generated by other actual robots 2 via the network 5.

[0206] Furthermore, when the actual robot 2 approaches a corresponding area (the first corresponding area and the second corresponding area), a sound corresponding to the virtual representation may be output. For example, in the second embodiment, when the actual robot 2 approaches the first corresponding area, a sound corresponding to the virtual object may be output. Also, in the third embodiment, when the actual robot 2 approaches the second corresponding area, a sound corresponding to an area with a predetermined attribute may be output.

[0207] Whether or not the actual robot 2 has approached the corresponding area (the first corresponding area and the second corresponding area) can be determined, for example, by setting a threshold value for the distance between the actual robot 2 and the area (the first corresponding area and the second corresponding area). The specific value of the threshold value is not limited and can be set arbitrarily. For example, it can be set arbitrarily within the range of a few centimeters to a few meters. Of course, it is not limited to this range. In addition, other methods may be used to determine whether or not the actual robot 2 has approached the corresponding area (the first corresponding area and the second corresponding area).

[0208] Depending on the placement of the virtual object 42 representing the "puddle" and the setting of the attributes of the "puddle," the actual robot 2 may behave as if it is enjoying playing in the water in the corresponding areas (the first corresponding area and the second corresponding area). Also, assuming that the puddle dries up over time, the sound of footsteps may change from the sound of water to the sound of walking on dry ground.

[0209] Depending on the placement of the virtual "carpet" object or the setting of the "carpet" attribute area, it is possible to configure the actual robot 2 to walk slowly and quietly in the corresponding area (first corresponding area and second corresponding area). Furthermore, the sound of footsteps may change over time, assuming that the carpet becomes harder rather than softer.

[0210] (Fourth embodiment: Sound output corresponding to proficiency level) In this entertainment system 1, it is possible to output sounds corresponding to the proficiency level of a virtual object for a given learning target, as sound output corresponding to the virtual representation. For example, it is possible to output sounds corresponding to the proficiency level of a virtual robot 8 for a given learning target from the actual robot 2.

[0211] Examples of designated learning objectives include various forms of performance. These include performances using musical instruments, as well as performances without instruments such as singing, whistling, and handclaps. Tap dancing is also included as a designated learning objective. Furthermore, any other technique (skill) that uses sound to create expression can be adopted as a designated learning objective related to this technique.

[0212] Furthermore, it is possible to have the actual robot 2 output sounds corresponding to its level of proficiency in playing music or dancing.

[0213] Figure 11 is a schematic diagram showing an example of sound output corresponding to a level of proficiency. In the example shown in Figure 11, a song with a predetermined melody is used as the target of proficiency.

[0214] Figure 11A illustrates a case where the virtual robot 8 displayed on the user terminal 3 has a relatively high level of proficiency in singing. In this case, in the real space RS, the actual robot 2 outputs a sound corresponding to the relatively high level of proficiency. For example, the actual robot 2 outputs a sound that beautifully sings a predetermined melody.

[0215] Figure 11B illustrates a case where the virtual robot 8 displayed on the user terminal 3 has a relatively low level of proficiency in singing. In this case, in the real space RS, the actual robot 2 outputs sounds corresponding to the relatively low level of proficiency. For example, the actual robot 2 outputs sounds of poor singing, where the pitch (scale), rhythm, timing, etc., are off for a given melody.

[0216] For example, user 6 improves the singing proficiency of virtual robot 8 within the application. As a result, in the real-world RS, the poor singing sound output shown in Figure 11B improves to the good singing sound output shown in Figure 11A. Furthermore, in the virtual space VS, virtual robot 8 sings with the same level of proficiency as the real robot 2.

[0217] The level of proficiency in singing corresponds to the parameter information of the virtual representation associated with virtual robot 8.

[0218] One way to specify the songs to be learned is, for example, for user 6 to select the songs they want the real robot 2 and virtual robot 8 to learn within the application. Alternatively, it is possible to play songs using a music player in the virtual space VS to make the robots recognize the songs they want to learn.

[0219] Furthermore, by integrating with streaming services and other platforms, music can be played to virtual robot 8 within the virtual space VS. Virtual robot 8 then analyzes the music and obtains information about it by downloading metadata stored on a separate server. This method is also possible.

[0220] Furthermore, it is possible for user 6 to play an instrument in the virtual space VS and let the virtual robot 8 listen to it, thereby enabling the robot to recognize the music it wants to learn. For example, if you want the robot to learn a specific snare drum phrase or a specific piano melody, letting it listen to the instrument being played in the virtual space VS is an effective method.

[0221] Furthermore, in the real-world RS environment, it is possible to play acquired music through a speaker and have the robot 2 listen to it with a microphone, thereby allowing it to recognize the music it wants to learn. Also, in the real-world RS environment, it is possible for user 6 to play an actual musical instrument to allow the robot to recognize the music it wants to learn.

[0222] One way to improve the proficiency of virtual robot 8 is for user 6 to instruct virtual robot 8 to practice songs or other things within the application. For example, by pressing a specific button within the application, virtual robot 8 can be made to practice, thereby improving its proficiency. For example, the number of times a specific button is pressed is counted, and the more times it is pressed, the higher the proficiency. Alternatively, a calculation method can be adopted in which the longer the practice time of virtual robot 8 in the virtual space VS, the higher the proficiency. The calculated proficiency is recorded in DB4 in the cloud.

[0223] Furthermore, in the virtual space VS, it is possible to improve the proficiency of virtual robot 8 by having it listen to exemplary music (such as CD audio or user 6's performance) multiple times. The more times virtual robot 8 is played music, and the longer it is played, the higher its proficiency will be.

[0224] Within the virtual space VS, the proficiency of virtual robot 8 can be improved by having it meet other virtual robots that already have a high level of proficiency. The higher the proficiency of the other virtual robots, the higher the proficiency of virtual robot 8 will be. Also, the more times virtual robot 8 meets other virtual robots, the higher its proficiency will be.

[0225] In the virtual space VS, it is possible to improve the proficiency of virtual robot 8 by giving it a specific virtual item. For example, giving virtual robot 8 a musical note-shaped cookie. When virtual robot 8 eats the musical note-shaped cookie, its proficiency will improve. Such settings are also possible.

[0226] While Virtual Robot 8 is practicing songs or other things in the virtual space VS, Virtual Robot 8 may behave in a manner appropriate to the practice. Furthermore, the application may display a message indicating that the robot's proficiency has improved.

[0227] User 6 can also improve the learning progress of the real robot 2 in the real-world RS. For example, in the real-world RS, the user can play a model song or other example (such as a CD recording or User 6's performance). When the real robot 2 recognizes, via its microphone, that the designated song or other example is playing, it enters practice mode and performs actions corresponding to the practice. The more times and for the longer the real robot 2 is played the song or other example, the higher its learning progress will be.

[0228] An example of how to calculate the proficiency level of virtual robot 8 is described below. For example, virtual robot 8 enters practice mode to practice a song or the like. User 6 may instruct the virtual robot 8 to switch to practice mode, or virtual robot 8 may switch to practice mode automatically.

[0229] Within the system, two states are defined for the virtual robot 8: a practice state and a non-practice state.

[0230] The proficiency level of the virtual robot 8 changes through practice, other actions (encounters with other virtual robots, acquisition of specific items), and the passage of time. For example, the proficiency level is calculated based on the number of times user 6 instructed practice, the total time user 6 spent performing operations for practice, and the number of times user 6 demonstrated. In addition to practice, the proficiency level may also change directly based on the number of times the virtual robot has encountered other virtual robots or the number of times a specific item has been acquired. It is also possible to set the proficiency level to decrease based on the time elapsed since the last practice session.

[0231] As a method for outputting sound corresponding to the level of proficiency, any method that can express the progress of the actual robot 2 in terms of sound may be adopted.

[0232] In this embodiment, for example, DB4 has complete performance sound data (performance sound data when the performance is successful) pre-recorded in the memory unit 16 of the actual robot 2. For example, pitch, length, timing of sound production, timbre, etc., are recorded as complete performance sound data.

[0233] By modifying this complete performance sound data according to the level of proficiency, the system outputs sounds corresponding to that level of proficiency. For example, by changing the pitch and timing of the sounds, or by adding or deleting sounds from the performance sound data, the system outputs sound data corresponding to the level of proficiency.

[0234] If the user's proficiency level is low, numerous modifications are made, and as their proficiency increases, the data is brought closer to the original, perfect performance data. As proficiency increases, the difference from the original, perfect performance sound data decreases, and when proficiency reaches its maximum value (or exceeds a predetermined threshold), the sound is output exactly as it was in the original, perfect performance sound data.

[0235] This section explains a specific example of modifying complete performance sound data. Assume that the complete performance sound data includes the pitch, length, timing of sound production, etc. When playing back the sound corresponding to the level of proficiency, a modification process consisting of the following five modification items is performed. (1) Adding errors: Playing a sound at a different pitch than intended for a specific sound. (2) Playback delay: Insert a pause of about 1 second during sound playback. This can be after (1) or after the correct sound. (3) Fluctuation in playback speed: Instead of a constant speed, the playback speed is slowed down in sections where short sounds occur in succession. (4) Stopping playback: The song is stopped before it is finished playing. (5) Repeat: Play a section of the song multiple times.

[0236] For example, as proficiency increases, the number of modifications performed from the five modification items mentioned above can be reduced. Alternatively, the degree of modification for each modification item can be reduced as proficiency increases. Or, the probability of each modification item being performed can be reduced as proficiency increases.

[0237] For example, proficiency levels can be classified into six stages (lowest proficiency level 1 to highest proficiency level 6). If the proficiency level is the lowest level 1, all five modification items listed above will be modified. Each time the proficiency level increases by one, the number of modification items to be modified decreases by one (the items to be modified may be selected randomly). If the proficiency level is the highest level 6, no modifications will be made to any of the modification items, and the sound will be output based on the original complete performance sound data. This type of processing is also possible.

[0238] Furthermore, for proficiency levels 1 through 5, a parameter (1 / proficiency level) may be set for each modification item as the probability of execution. For example, at the lowest proficiency level of 1, all modification items are executed with 100% probability. At proficiency level 5, all modification items are executed with 20% probability. At the highest proficiency level of 6, no modifications are executed. Such processing is also possible.

[0239] Thus, in the fourth embodiment, the virtual robot 8 practices music and other pieces in the virtual space VS, and the sound played by the real robot 2 changes in accordance with the proficiency level of the virtual robot 8. This makes it possible to provide a very high level of user experience.

[0240] In this embodiment, the sound-related data includes successful performance sound data, which is the sound of a performance when the performance is successful, or sound setting information for generating the successful performance sound data. Then, by performing a modification process on the successful performance sound data based on the level of proficiency, a sound corresponding to the level of proficiency is output.

[0241] In this way, it is possible to dynamically perform modification processing according to the level of proficiency, which allows for diverse expressions at a lower cost compared to, for example, preparing multiple performance sound data corresponding to different levels of proficiency in advance.

[0242] For example, if multiple performance sound data sets corresponding to different skill levels are prepared in advance and selected appropriately according to the skill level, the output performance sound will be the same each time, regardless of whether the performance is poor or good. In this embodiment, by setting the selection and probability of the five modification items described above, it becomes possible to output different performance sounds each time, even for the same skill level. This enables diverse expression and a more enjoyable user experience.

[0243] Furthermore, regarding the application of this technology, it is also possible to adopt a method in which data for multiple performance sounds corresponding to different skill levels are prepared in advance, and the appropriate sound is selected and output according to the skill level. That is, the sound-related data may include data for multiple corresponding performance sounds corresponding to different skill levels, or sound setting information for generating data for multiple corresponding performance sounds corresponding to different skill levels. Then, by selecting one of the multiple corresponding performance sound data based on the skill level, the sound corresponding to that skill level may be output. By adopting this method, it is possible to simplify the processing.

[0244] The movements of the actual robot 2 may be modified in accordance with changes to the complete performance sound data. For example, the expressions of the actual robot 2's eyes, ears, and the expression of realizing it has made a mistake may be executed in conjunction with the output of the performance sound.

[0245] Furthermore, modification processing can be controlled based on the fatigue level and mood of the actual robot 2 (virtual robot 8). For example, it is possible to set it so that when it is tired, it makes many modifications and more mistakes. For example, it is possible to set it so that it makes more mistakes when the battery level is low.

[0246] It is also possible to change the learning rate using the unique parameters of the actual robot 2 (virtual robot 8). These unique parameters of the actual robot 2 (virtual robot 8) include, for example, parameters that are randomly set when the actual robot 2 is purchased, and parameters that are initially set.

[0247] For example, unique parameters include those set by user 6, such as gender. Parameters set based on the interaction between user 6 and the actual robot 2, such as startup time, frequency of voice recognition, and frequency of petting, are also included.

[0248] These unique parameters may change the way in which proficiency changes. For example, these unique parameters can change the time it takes to master music or dance. By increasing interaction with the actual robot 2 (petting it a lot, praising it a lot, etc.), it will improve with less practice. These unique parameters can determine which songs the robot is good at learning and which it is not. Such various settings become possible, enabling a wide range of expressions.

[0249] When outputting sounds corresponding to the level of proficiency, parameters related to performance expression, in addition to pitch, length, and timing of sound production, may also be controlled in accordance with the level of proficiency.

[0250] For example, in the case of singing, you can lower the pitch of high notes, or in the case of wind instruments, you can change the envelope (time-dependent variation) of the timbre to express the lack of proficiency in playing.

[0251] As proficiency increases and skills improve, techniques such as adding vibrato (a wide, expansive expression created by periodic changes in pitch) to guitar or vocals, or altering the timing of parts you want to emphasize (creating pauses), may be employed. Furthermore, it's possible to set the system so that high proficiency allows for very fast playing.

[0252] Furthermore, various performance expressions that cannot be fully expressed in musical notation may be realized in accordance with the level of proficiency. For example, by performing machine learning based on performance sound data of world-renowned performers, it is possible to obtain performance sound data for very high levels of proficiency, and it becomes possible to output extremely high-level and wonderful performance sounds from the actual robot 2.

[0253] Furthermore, it is possible to link the pauses and timing changes to the body movements of the actual robot 2.

[0254] In the entertainment system 1 and the actual robot 2 according to each of the above embodiments, parameter information of a virtual representation related to at least one of the virtual robot 8 corresponding to the actual robot 2 and the map 37, which is a corresponding virtual space corresponding to the real space RS, and sound-related data corresponding to the parameter information are acquired. Then, based on the parameter information and sensing data from the sensor unit 9, the output of sound corresponding to the virtual representation is controlled using the sound-related data. This makes it possible to provide a high-quality user experience to the user 6 using the actual robot 2.

[0255] In entertainment robots, using a virtual robot 8 with the same appearance as the real robot 2 in a virtual space VS allows for the realization of expressions that would be physically difficult or costly to achieve with the real robot 2.

[0256] For example, in the real-world RS environment, it becomes possible to realize things that are difficult to achieve with the actual robot 2, such as the virtual robot 8 entering a swimming pool or being in outer space. It also becomes possible to dress the virtual robot 8 in virtual clothes, feed it virtual food, and have it play virtual musical instruments. As a result, user 6 can have a wider range of satisfying experiences than when spending time only with the actual robot 2.

[0257] However, the richer the representation of the virtual robot 8 in the virtual space VS becomes, the more difficult it becomes for the user 6 to perceive a connection with the actual robot 2 in the real space RS.

[0258] In this entertainment system 1, parameter changes in the virtual space VS can be reflected in the representation of the real robot 2 in the real space RS, allowing the user 6 to feel a stronger connection between the real robot 2 and the virtual robot 8.

[0259] <Other Embodiments> This technology is not limited to the embodiments described above, and various other embodiments can be realized.

[0260] In each of the above embodiments, augmented reality (AR) display may be possible on the user terminal 3. For example, it is possible to superimpose the display of virtual objects in real time onto an image acquired by the smartphone camera. Alternatively, an HMD such as AR glasses can be used as the user terminal 3 to superimpose the display of virtual objects onto the user's field of view 6.

[0261] For example, in the shoe fitting described in the first embodiment, when the actual robot 2 is photographed through the smartphone camera, an AR display is executed that makes it appear as if the robot is wearing the shoes 28 selected by the user 6.

[0262] Furthermore, when arranging the virtual object 42 of the "puddle" described in the second embodiment, or when setting the attribute area of ​​the "puddle" described in the third embodiment, viewing the first corresponding area and the second corresponding area through the smartphone camera will result in an AR display that makes it appear as if there is a puddle on the floor. For example, such an AR display can be realized by linking it with self-localization technology using the smartphone camera.

[0263] Furthermore, after the actual robot 2 leaves the first and second corresponding areas that correspond to the "puddle," wet footprints on the ground may be displayed using AR.

[0264] In the sound output corresponding to the level of proficiency in playing described in the third embodiment, it is possible to superimpose a virtual instrument onto the actual robot 2 using AR display via the smartphone camera, making it appear as if the robot 2 is actually playing an instrument.

[0265] These various AR displays make it possible to provide an even higher quality user experience.

[0266] The above example uses a real-world dog-shaped robot capable of autonomous walking (robot 2). Of course, the application of this technology is not limited to real-world robots of this form (robot 2).

[0267] For example, this technology can also be applied to stationary, movable devices. For instance, in the case of a stationary, rotating real robot, an item such as a bell can be attached to the corresponding virtual robot. This would change the sound that corresponds to the rotation of the real robot. If an item like a shoe with a bell is attached, the frequency of the bell ringing will change according to the rotation speed. Such settings are also possible.

[0268] By attaching actual physical clothing to the real robot 2, the clothing may also be reflected in the virtual robot 8 in the virtual space VS. In this case, attaching clothing to the real robot 2 may add sounds that are linked to movements such as walking. Also, in the case of a car, the driving sound may change by changing the car's settings in the application. For example, the driving sound may change by setting virtual tires.

[0269] The actual robot 2 may be a flying object (flying robot) capable of flight. For example, this technology can be applied to drones, etc. In such a flying object, for example, user 6 can use user terminal 3 to change the parameter information of a virtual representation related to the virtual robot 8 corresponding to the flying object in an application. It is also possible to set it so that the sound emitted by the actual robot 2 (flying object) during flight changes in accordance with the change in the parameter information.

[0270] For example, within the application, a virtual item resembling fairy wings can be attached to virtual robot 8 (the flying object). Then, when the real robot 2 (the flying object) flies in the real-world RV, a cute sound reminiscent of a fairy flying will be played. Such fun settings are also possible.

[0271] For example, the combination of wearing the shoes 28 illustrated in the first embodiment and the arrangement of the virtual object of the "puddle" described in the second embodiment may be combined. And, a special sound or the like may be output according to the combination of the virtual item worn on the virtual robot 8 and the virtual object arranged on the map 37.

[0272] Regarding the map 37 corresponding to the real space RS, when making a sound with AR glasses or a speaker, the sound may be emitted when a person enters an area where a sound is set, such as a virtual puddle. Also, in the case of the real robot 2 having a display area such as a display, expressions such as a change in color when approaching the "puddle" may be made. Also, the movement may be changed according to the season, time, date, etc. For example, when it rains, a puddle can be created in the virtual space VS, and processing such as a change in sound in that area is also possible.

[0273] Regarding the output of the footsteps of the real robot 2, when a person is watching TV, talking, listening to music, etc., it may be determined according to the environment, such as not outputting sound and keeping quiet, and the presence or absence of sound playback may be determined (not making much noise at night or during certain time periods).

[0274] In the above, the sound corresponding to the virtual expression was output by the speaker 10 mounted on the real robot 2. However, the present technology is not limited to this, and it is also possible to apply this technology to a speaker (sound output unit) configured separately from the real robot 2.

[0275] For example, when a very small real robot is used, it may be impossible to mount a speaker. In such a case, it is effective to output the sound corresponding to the virtual expression from a speaker (sound output unit) configured separately from the real robot 2.

[0276] Alternatively, a speaker may be built into the floor to output a variety of sounds corresponding to virtual expressions according to the walking of the real robot 2, and it is possible to provide a high-quality user experience.

[0277] Figure 12 is a block diagram showing an example of the hardware configuration of a computer 60 that can be used to construct an entertainment system 1 related to this technology.

[0278] The computer 60 includes a CPU 61, ROM 62, RAM 63, an input / output interface 65, and a bus 64 connecting these to each other. The input / output interface 65 is connected to a display unit 66, an input unit 67, a storage unit 68, a communication unit 69, and a drive unit 70, among others. The display unit 66 is a display device using, for example, liquid crystal, EL, etc. The input unit 67 is, for example, a keyboard, pointing device, touch panel, or other operating device. If the input unit 67 includes a touch panel, the touch panel may be integrated with the display unit 66. The storage unit 68 is a non-volatile storage device, such as an HDD, flash memory, or other solid-state memory. The drive unit 70 is a device capable of driving a removable recording medium 71, such as an optical recording medium or magnetic recording tape. The communication unit 69 is a modem, router, or other communication device that can connect to a LAN, WAN, etc., and communicates with other devices. The communication unit 69 may use either wired or wireless communication. The communication unit 69 is often used separately from the computer 60. Information processing by the computer 60 having the hardware configuration described above is realized through the cooperation of software stored in the memory unit 68 or ROM 62, etc., and the hardware resources of the computer 60. Specifically, the information processing method related to this technology is realized by loading the programs that constitute the software, stored in ROM 62, etc., into RAM 63 and executing them. The program is installed on the computer 60, for example, via a recording medium 71. Alternatively, the program may be installed on the computer 60 via a global network or the like. In addition, any non-transient storage medium that is readable by a computer may be used.

[0279] Multiple computers connected to each other via a network or the like may collaborate to execute the information processing method (control method for movable devices) and program related to this technology, thereby constructing an information processing system related to this technology. In other words, the information processing method, information processing system, and program related to this technology can be executed not only in a computer system composed of a single computer, but also in a computer system in which multiple computers operate in conjunction. In this disclosure, "system" means a collection of multiple components (devices, modules (parts), etc.), regardless of whether all components are located in the same enclosure. Therefore, multiple devices housed in separate enclosures and connected via a network, and a single device containing multiple modules in one enclosure, are both considered systems.

[0280] The execution of the information processing method, information processing system, and program related to this technology by a computer system includes both cases where, for example, the acquisition of parameter information of a virtual representation, the acquisition of sound-related data, the generation of a map which is a corresponding virtual space, and the output of sound corresponding to the virtual representation are performed by a single computer, and cases where each process is performed by different computers. Furthermore, the execution of each process by a predetermined computer includes having part or all of the process executed by another computer and obtaining the results. In other words, the information processing method, information processing system, and program related to this technology can also be applied to cloud computing configurations in which a single function is shared and processed collaboratively by multiple devices via a network.

[0281] The entertainment system, actual robot, user terminal, database, GUI configuration within the application, acquisition of virtual representation parameter information, acquisition of sound-related data, generation of a map representing the corresponding virtual space, and output of sound corresponding to the virtual representation, as described with reference to each drawing, are merely embodiments and can be arbitrarily modified without departing from the spirit of this technology. In other words, other arbitrary configurations and algorithms may be adopted to implement this technology.

[0282] In this disclosure, words such as "about," "abbreviated," "almost," and "approximately" may be used as appropriate to facilitate understanding of the explanation. However, there is no clear distinction defined between when these words are used and when they are not. In other words, in this disclosure, concepts that define shape, size, positional relationship, state, etc., such as "center," "central," "uniform," and "equal," include concepts such as "substantially centered," "substantially central," "substantially uniform," and "substantially equal." For example, this includes states that fall within a predetermined range (e.g., a range of ±10%) based on criteria such as "perfectly centered," "perfectly central," "perfectly uniform," and "perfectly equal." Therefore, even if words like "abbreviated," "almost," or "approximately" are not added, the concept may still be included in what would typically be expressed with such words. Conversely, a state expressed with such words does not necessarily exclude a complete state.

[0283] In this disclosure, expressions using "greater than A" such as "greater than A" and "less than A" are expressions that comprehensively include both concepts that include cases where something is equivalent to A and concepts that do not include cases where something is equivalent to A. For example, "greater than A" is not limited to cases where something is not equivalent to A, but also includes "greater than or equal to A". Similarly, "less than A" is not limited to "less than A", but also includes "less than or equal to A". When implementing this technology, you should appropriately adopt specific settings and other elements from the concepts included in "greater than A" and "less than A" so that the effects described above are achieved.

[0284] It is also possible to combine at least two of the feature features of the present technology described above. In other words, the various feature features described in each embodiment may be combined arbitrarily, regardless of the specific embodiment. Furthermore, the various effects described above are merely examples and not limiting, and other effects may also be exhibited.

[0285] Furthermore, this technology can also be configured as follows. (1) A movable device constructed in real space, One or more sensors, An acquisition unit that acquires parameter information of a virtual representation related to at least one of a corresponding virtual object corresponding to the movable device constructed in a virtual space and a corresponding virtual space corresponding to the real space, and sound-related data corresponding to the parameter information, Based on the acquired parameter information and sensing data from one or more sensors, an output control unit controls the output of sound corresponding to the virtual representation using the acquired sound-related data. A movable device equipped with the following. (2) The movable device described in (1), further, Equipped with an audio output section, The output control unit controls the output of sound corresponding to the virtual representation from the sound output unit. A movable device. (3) A movable device as described in (1) or (2), The parameter information includes information about the virtual item attached to the corresponding virtual object, The output control unit causes the sound corresponding to the virtual item to output. A movable device. (4)(3) The movable device described above, The one or more sensors mentioned above acquire data relating to the movement of the movable device. The output control unit causes the sound corresponding to the virtual item to be output in accordance with the predetermined movement of the movable device. A movable device. (5) The movable device according to (4), wherein the output control unit outputs a sound corresponding to the virtual item according to the movement of the movable device. Movable device. (6) The movable device according to any one of (1) to (5), wherein the parameter information includes at least one of information on a virtual object arranged in the corresponding virtual space and information on a region of a predetermined attribute set in the corresponding virtual space, and the output control unit outputs at least one of a sound corresponding to the virtual object and a sound corresponding to the region of the predetermined attribute. Movable device. (7) The movable device according to (6), wherein the one or more sensors acquire data related to the movement of the movable device, and the output control unit outputs a sound corresponding to the virtual object when the movable device enters or approaches a first corresponding region of the real space corresponding to a region in the corresponding virtual space where the virtual object is arranged, and outputs a sound corresponding to the region of the predetermined attribute when the movable device enters or approaches a second corresponding region of the real space corresponding to the region of the predetermined attribute in the corresponding virtual space, at least one of which is executed. (8) The movable device according to (7), wherein the output control unit outputs a sound corresponding to the virtual object according to a predetermined movement of the movable device in the first corresponding region, and outputs a sound corresponding to the region of the predetermined attribute according to a predetermined movement of the movable device in the second corresponding region, at least one of which is executed. (9) The movable device according to any one of (1) to (8), wherein the corresponding virtual space is a map corresponding to the real space generated based on sensing data by the one or more sensors. Movable device. (10) A movable device described in any one of (1) to (9), The parameter information includes the proficiency level of the corresponding virtual object with respect to a predetermined learning target, The output control unit causes the unit to output a sound corresponding to the level of proficiency. A movable device. (11)(10) A movable device as described above, The aforementioned proficiency level includes the proficiency level of the corresponding virtual object related to performance. A movable device. (12)(11) A movable device as described above, The sound-related data corresponding to the parameter information includes successful performance sound data, which is the sound of the performance when the performance is successful, or sound setting information for generating the successful performance sound data. The output control unit performs modification processing on the data of the successful performance sound based on the proficiency level, thereby outputting a sound corresponding to the proficiency level. A movable device. (13)(11) A movable device as described above, The sound-related data corresponding to the parameter information includes data for a plurality of corresponding performance sounds corresponding to the proficiency level, or sound setting information for generating data for a plurality of corresponding performance sounds corresponding to the proficiency level. The output control unit, based on the proficiency level, selects one of the data for the plurality of corresponding performance sounds to output a sound corresponding to the proficiency level. A movable device. (14) A movable device described in any one of (1) to (13), The output control unit executes the output of the sound corresponding to the virtual representation when the conditions for executing the output of the sound corresponding to the virtual representation are met. A movable device. (15) A movable device described in any one of (1) to (14), It is configured as a mobile device capable of autonomous movement. A movable device. (16) The system acquires parameter information of a virtual representation related to at least one of a corresponding virtual object constructed in a virtual space in relation to a movable device constructed in real space, and a corresponding virtual space in relation to the real space, and sound-related data corresponding to the parameter information. Based on the acquired parameter information and sensing data from one or more sensors of the movable device, the sound output corresponding to the virtual representation is controlled using the acquired sound-related data. A method of information processing performed by a computer system. [Explanation of Symbols]

[0286] RS... Real space VS...Virtual Space 1…Entertainment System 2…Actual robots 3…User terminal 4...DB 5…Network 6...User 8…Virtual robots 19...Foot of a real robot 28... shoes 29…Foot of a virtual robot 30...Clothes 31…Installation data 32... Item Data 37…Map 40-43...Virtual Objects 60… Computer

Claims

1. A movable device constructed in real space, One or more sensors, An acquisition unit that acquires parameter information of a virtual representation related to at least one of a corresponding virtual object corresponding to the movable device constructed in a virtual space and a corresponding virtual space corresponding to the real space, and sound-related data corresponding to the parameter information, Based on the acquired parameter information and sensing data from one or more sensors, an output control unit controls the output of sound corresponding to the virtual representation using the acquired sound-related data. A movable device equipped with the following.

2. A movable device according to claim 1, further, Equipped with an audio output section, The output control unit controls the output of sound corresponding to the virtual representation from the sound output unit. A movable device.

3. A movable device according to claim 1, The parameter information includes information about the virtual item attached to the corresponding virtual object, The output control unit causes the sound corresponding to the virtual item to output. A movable device.

4. A movable device according to claim 3, The one or more sensors mentioned above acquire data relating to the movement of the movable device. The output control unit causes the virtual item to output a sound corresponding to the virtual item in accordance with a predetermined movement of the movable device. A movable device.

5. A movable device according to claim 4, The output control unit causes the virtual item to output a sound corresponding to the movement of the movable device. A movable device.

6. A movable device according to claim 1, The parameter information includes at least one of the following: information about virtual objects placed in the corresponding virtual space, and information about areas of predetermined attributes set in the corresponding virtual space. The output control unit causes at least one of the sound corresponding to the virtual object and the sound corresponding to the area of ​​the predetermined attribute to output. A movable device.

7. A movable device according to claim 6, The one or more sensors mentioned above acquire data relating to the movement of the movable device. The output control unit performs at least one of the following: when the movable device enters or approaches a first corresponding area in the real space that corresponds to the area in the corresponding virtual space where the virtual object is located, it outputs a sound corresponding to the virtual object; and when the movable device enters or approaches a second corresponding area in the real space that corresponds to the area of ​​the predetermined attribute in the corresponding virtual space, it outputs a sound corresponding to the area of ​​the predetermined attribute. A movable device.

8. A movable device according to claim 7, The output control unit performs at least one of the following: outputting a sound corresponding to the virtual object in accordance with a predetermined movement of the movable device in the first corresponding area, and outputting a sound corresponding to the area of ​​a predetermined attribute in accordance with a predetermined movement of the movable device in the second corresponding area. A movable device.

9. A movable device according to claim 1, The corresponding virtual space is a map that corresponds to the real space, generated based on sensing data from one or more sensors. A movable device.

10. A movable device according to claim 1, The parameter information includes the proficiency level of the corresponding virtual object with respect to a predetermined learning target, The output control unit causes the unit to output a sound corresponding to the level of proficiency. A movable device.

11. A movable device according to claim 10, The aforementioned proficiency level includes the proficiency level of the corresponding virtual object related to performance. A movable device.

12. A movable device according to claim 11, The sound-related data corresponding to the parameter information includes successful performance sound data, which is the sound of the performance when the performance is successful, or sound setting information for generating the successful performance sound data. The output control unit performs modification processing on the data of the successful performance sound based on the proficiency level, thereby outputting a sound corresponding to the proficiency level. A movable device.

13. A movable device according to claim 11, The sound-related data corresponding to the parameter information includes data for a plurality of corresponding performance sounds corresponding to the proficiency level, or sound setting information for generating data for a plurality of corresponding performance sounds corresponding to the proficiency level. The output control unit, based on the proficiency level, selects one of the data for the plurality of corresponding performance sounds to output a sound corresponding to the proficiency level. A movable device.

14. A movable device according to claim 1, The output control unit executes the output of the sound corresponding to the virtual representation when the conditions for executing the output of the sound corresponding to the virtual representation are met. A movable device.

15. A movable device according to claim 1, It is configured as a mobile device capable of autonomous movement. A movable device.

16. The system acquires parameter information of a virtual representation related to at least one of a corresponding virtual object constructed in a virtual space in relation to a movable device constructed in real space, and a corresponding virtual space in relation to the real space, and sound-related data corresponding to the parameter information. Based on the acquired parameter information and sensing data from one or more sensors on the movable device, the sound output corresponding to the virtual representation is controlled using the acquired sound-related data. A method of information processing performed by a computer system.

Images