Entertainment systems and robots

Abstract

An entertainment system 10 comprises a user terminal 11, a robot 13, and a control device 14. The user terminal 11 causes a display part to display content for a user to select a character from multiple characters for which different personalities are defined. The control device 14 determines an operation mode of the robot 13 on the basis of a parameter corresponding to the personality of the character selected in the user terminal 11.

Description

Technical Field

[0001] The present invention relates to data processing technology, and particularly to entertainment systems and robots.

Background Art

[0002] In recent years, various entertainment robots such as pet robots have been provided. Some of the entertainment robots are capable of a certain degree of autonomous behavior.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Conventional entertainment robots have been created with the personality of a single character. The inventor has developed a technology to improve the user experience by an entertainment robot.

[0004] The present invention has been made based on the above problem recognition by the inventor, and one object is to provide a technology for improving the user experience by an entertainment robot.

Means for Solving the Problems

[0005] In order to solve the above problems, an entertainment system according to an aspect of the present invention includes a user terminal, a robot, and an operation determination unit. The user terminal includes a display control unit that causes the display unit to display content for allowing the user to select any one of a plurality of characters with different personalities. The operation determination unit determines the operation mode of the robot based on parameters corresponding to the personality of the character selected on the user terminal.

[0006] Another aspect of the present invention is a robot. This robot includes a reception unit that receives information from a user terminal about a character selected by the user from among a plurality of characters, each with a defined personality, and an operation determination unit that determines the operation mode of the robot based on parameters corresponding to the personality of the character selected by the user.

[0007] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, computer programs, recording media on which computer programs are recorded in a readable manner, data structures, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0008] According to the present invention, the user experience provided by entertainment robots can be improved. [Brief explanation of the drawing]

[0009] [Figure 1] This figure shows an example of the configuration of the information processing system in the embodiment. [Figure 2] This figure shows an example of the external shape of an HMD (Head-Mounted Display). [Figure 3] This is a block diagram of the functional blocks of the HMD. [Figure 4] This is a block diagram showing the functional blocks of the user terminal. [Figure 5] This figure shows an example of character data. [Figure 6] This is a block diagram showing the functional blocks of a robot. [Figure 7] This is a block diagram showing the functional blocks of a control device. [Figure 8] This figure shows an example of a character selection screen. [Figure 9] This figure shows an example of an AR image. [Figure 10] This figure shows an example of a game screen generated by the game generation unit. [Figure 11]This is a block diagram showing the functional block of the control device in the second embodiment. [Figure 12] This is an example of the first game image. [Figure 13] This is an example of a second game image. [Modes for carrying out the invention]

[0010] <First Example> The outline of the first embodiment will be described. In recent years, various entertainment robots, such as pet robots, have been offered, but until now, entertainment robots have been characterized by the built-in personality of a single character. In the first embodiment, we propose an entertainment system that allows the entertainment robot to perform actions according to the personality of a character desired by the user from among multiple characters with different personalities. The characters in this embodiment can also be called "avatars" as representations of the user or their friends.

[0011] Figure 1 shows an example configuration of the entertainment system 10 of the first embodiment. The entertainment system 10 is an information processing system that provides users with an entertainment experience using a robot. The entertainment system 10 comprises a user terminal 11, HMD 100a, and processing unit 128a operated by the user, and an HMD 100b, processing unit 128b, robot 13, and control unit 14 operated by other users registered as friends by the user (hereinafter referred to as "friends"). These devices have wired communication and / or wireless communication functions and are connected via a communication network 15 which may include LAN, WAN, the Internet, etc.

[0012] The user terminal 11 is a mobile phone equipped with a mobile operating system, such as a smartphone or tablet.

[0013] HMD100a is a head-mounted display device that a user wears on the head. The processing device 128a is an information processing device that controls the display of images (such as VR (Virtual Reality) images, AR (Augmented Reality) images, etc.) in HMD100a. The "image" in the embodiments includes still images and moving images. The processing device 128a receives an operation on the robot 13 input by the user and transmits information regarding the operation (hereinafter also referred to as "user operation information") to the control device 14. The user operation information includes voice data indicating the user's voice collected by the microphone of HMD100a.

[0014] HMD100b is a head-mounted display device that a friend wears on the head. The processing device 128b is an information processing device that controls the display of images in HMD100b. The processing device 128b receives an operation on the robot 13 input by the friend and transmits information regarding the operation (hereinafter also referred to as "friend operation information") to the control device 14. The friend operation information includes voice data indicating the friend's voice collected by the microphone of HMD100b.

[0015] Hereinafter, when not particularly distinguishing between HMD100a and HMD100b, it is referred to as "HMD100". Also, when not particularly distinguishing between the processing device 128a and the processing device 128b, it is referred to as "processing device 128". The processing device 128 may be a PC, or may be a stationary or portable game. HMD100 and the processing device 128 may be integrally configured. For example, HMD100 may include the functions of the processing device 128. In this case, a configuration may be adopted in which HMD100 and the control device 14 communicate via the communication network 15.

[0016] Robot 13 is a self-propelled robot with a full width of 192 mm, a full height of 167 mm, and a depth of 166 mm. Robot 13 includes a traveling drive unit 20, a neck drive unit 22, a right camera 24a, a left camera 24b, a right microphone 26a, a left microphone 26b, and a speaker 28. Hereinafter, when the right camera 24a and the left camera 24b are not particularly distinguished, they are referred to as "camera 24". Also, when the right microphone 26a and the left microphone 26b are not particularly distinguished, they are referred to as "microphone 26". The speaker 28 outputs voice data received from an external device.

[0017] The traveling drive unit 20 includes three omni wheels. Each omni wheel moves forward and backward by the rotation of the main body (wheel) on the axis, moves left and right by the rotation of the circumferential barrel, and can move in multiple directions by the combination of these. The neck drive unit 22 rotates the head 21 of the robot 13 in three axes. The three axes include a pitch axis for rotating the neck in the vertical direction, a roll axis for rotating the neck in the tilting direction, and a yaw axis for rotating the neck horizontally.

[0018] The right camera 24a and the left camera 24b are provided on the head 21. The right camera 24a and the left camera 24b are arranged at a predetermined interval in the horizontal direction. The right camera 24a and the left camera 24b form a stereo camera. The right camera 24a captures a right-eye image at a predetermined cycle, and the left camera 24b captures a left-eye image at a predetermined cycle. The height (eye height) of the mounting positions of the right camera 24a and the left camera 24b is 144 mm.

[0019] The right microphone 26a and the left microphone 26b are provided on the head 21. The right microphone 26a and the left microphone 26b are arranged at a predetermined interval in the horizontal direction. The right microphone 26a and the left microphone 26b form a stereo microphone. By arranging them at a predetermined interval in the horizontal direction, the time when the sound reaches varies according to the position of the sound source. The difference in the arrival time of the sound appears as a phase difference between the sound signals generated by the right microphone 26a and the left microphone 26b. In order to increase the phase difference between the sound signals of the right microphone 26a and the left microphone 26b, it is preferable to arrange the right microphone 26a and the left microphone 26b as far apart as possible.

[0020] The control device 14 is an information processing device that determines and controls the movements of the robot 13. The detailed configuration of the control device 14 will be described later.

[0021] Figure 2 shows an example of the external shape of the HMD100. In this example, the HMD100 consists of an output mechanism 110 and a mounting mechanism 112. The mounting mechanism 112 includes a mounting band 108 that wraps around the head when worn by the user to secure the HMD100 to the head. The mounting band 108 is made of a material or structure that allows its length to be adjusted to the user's head circumference.

[0022] The output mechanism 110 includes a housing 114 shaped to cover the left and right eyes when the HMD 100 is worn by the user, and has a display panel 102 positioned directly in front of the eyes. The display panel 102 may be an LCD panel or an organic EL panel. Inside the housing 114, there is a pair of left and right optical lenses positioned between the display panel 102 and the user's eyes to expand the user's field of view.

[0023] The HMD100 further includes earphones 104 that are inserted into the user's ears when worn. The earphones 104 are an example of an audio output means, and the HMD100 may also include headphones. In this case, the HMD100 and headphones may be integrated or separate components.

[0024] The HMD100 transmits sensor information detected by the posture sensor and audio data encoded from the audio signal from the microphone 106 to the control unit 14 via the processing unit 128. The HMD100 also receives image data and audio data generated by the control unit 14 (robot 13) via the processing unit 128 and outputs them from the display panel 102 and earphone 104.

[0025] The HMD100 shown in Figure 2 is an immersive (non-transparent) display device that completely covers both eyes, but a transparent display device may also be used. Furthermore, the shape may be a hat type as shown, or it may be a glasses type. The HMD100 may consist not only of a dedicated head-mounted display device, but also of a terminal device having a display panel, microphone, and speaker, and a housing that fixes the display panel of the terminal device in front of the user's eyes. The terminal device may be, for example, a smartphone or portable game console, which has a relatively small display panel.

[0026] Figure 3 is a block diagram showing the functional blocks of the HMD100. Each block shown in the block diagrams of this specification can be realized in hardware terms by components such as a computer processor, CPU, and memory, as well as electronic circuits and mechanical devices, and in software terms by computer programs, etc. However, here we are depicting functional blocks that are realized through the cooperation of these components. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways by combinations of hardware and software.

[0027] The control unit 120 is the main processor that processes and outputs various signals and data such as image signals, audio signals, and sensor information, as well as commands. The memory unit 122 temporarily stores the data and commands processed by the control unit 120. The attitude sensor 124 detects attitude information such as the rotation angle and tilt of the HMD 100 at predetermined intervals. The attitude sensor 124 includes at least a 3-axis accelerometer and a 3-axis gyroscope. The attitude sensor 124 also includes a sensor that detects the position (or change thereof) of the HMD 100 in the height direction. The microphone 106 converts the user's voice into an electrical signal to generate an audio signal.

[0028] The communication control unit 126 transmits and receives signals and data to and from the control device 14 via a network adapter or antenna, either by wired or wireless communication. In this embodiment, the communication control unit 126 transmits and receives signals and data to and from the control device 14 via the processing unit 128. The communication control unit 126 receives from the control unit 120 attitude information detected by the attitude sensor 124 (for example, the height position of the HMD 100 and / or changes in height position), and encoded audio data from the microphone 106, and transmits it to the control device 14.

[0029] Furthermore, the communication control unit 126 receives image data and audio data transmitted from the control device 14 and supplies them to the control unit 120. When the control unit 120 receives the image data and audio data transmitted from the control device 14, it supplies the image data to the display panel 102 for display and supplies the audio data to the earphone 104 for audio output.

[0030] Figure 4 is a block diagram showing the functional blocks of the user terminal 11. The user terminal 11 includes a display unit 30, a camera 32, a character data storage unit 34, an operation reception unit 36, an App control unit 38, a display content generation unit 40, a display control unit 42, and a character data transmission unit 44.

[0031] The display unit 30 displays various information. The display unit 30 may include a liquid crystal panel or an organic EL panel. The display unit 30 also includes touch panel (in other words, touchscreen) functionality. The camera 32 is an imaging device that captures images of the surroundings of the user terminal 11 (for example, objects present around the user terminal 11).

[0032] The character data storage unit 34 stores data (hereinafter also referred to as "character data") relating to multiple characters that are selectable by the user and each has a defined personality. Figure 5 shows an example of character data. Character A is assigned a curious personality, Character B is assigned a friendly personality, Character C is assigned a brave personality, and Character D is assigned a timid personality. Various attribute values ​​(also referred to as "parameters") are predetermined for each character to match the assigned personality.

[0033] Parameter (1) "Pathfinding - Distance from Obstacles" determines the width of the distance that the robot 13 maintains between itself and obstacles (e.g., people or objects) detected along the path. For example, a value corresponding to "narrow" may be 20 cm, a value corresponding to "medium" may be 40 cm, and a value corresponding to "wide" may be 60 cm.

[0034] Parameter (2) "Pathfinding - Path Smoothness" determines the curvature of the path that the robot 13 will travel. For example, the value corresponding to "straight" may be a relatively small curvature value (e.g., curvature 0), the value corresponding to "smooth" may be a relatively large curvature value, and the value corresponding to "medium" may be a curvature value intermediate between "straight" and "smooth". The path that the robot 13 will take to reach the goal is determined according to parameters (1) and (2).

[0035] Parameter (3) "Pathfinding - Detour (Person)" determines how often the robot 13 reacts to a person it detects on its movement path. Reacting may involve stopping near the detected person, moving around the detected person, capturing an image of the detected person with the camera 24, or a combination of these. For example, a value corresponding to "none" may be a frequency of 0 (no detours), a value corresponding to "frequent" may be a frequency of 50%, and a value corresponding to "moderate" may be a frequency of 25%.

[0036] Parameter (4) "Pathfinding - Detours (Objects)" determines how often the robot 13 reacts to objects other than people when it detects them on its movement path. Examples of reactions are the same as for people. For example, a value corresponding to "none" may be a frequency of 0 (no detours), a value corresponding to "frequent" may be a frequency of 50%, and a value corresponding to "moderate" may be a frequency of 25%. According to parameters (3) and (4), the robot 13's movement path is updated, or in other words, a new movement path including detours is determined.

[0037] Parameter (5) "Pathfinding - Goal Location (Person)" determines the preferred goal location when setting a path; in other words, it determines the distance from the robot 13 to the person who will be the preferred goal location. For example, a value corresponding to "close" may be a relatively short distance (e.g., less than 3 meters), a value corresponding to "far" may be a relatively long distance (e.g., 6 meters or more), and a value corresponding to "medium" may be an intermediate distance (e.g., 3 meters or more but less than 6 meters).

[0038] Parameter (6) "Velocity - Change in Moving Speed ​​(Acceleration)" determines the acceleration of the robot 13 during movement. For example, a value corresponding to "Large" may be a relatively large acceleration, a value corresponding to "Small" may be a relatively small acceleration, and a value corresponding to "Medium" may be an acceleration between "Large" and "Small". Also, a value corresponding to "Large (Human Only)" may be a relatively large acceleration when the goal is a human, and a relatively small acceleration when the goal is an object other than a human.

[0039] Parameter (7) "Speed ​​- Normal movement speed" determines the speed at which the robot 13 moves (e.g., maximum speed). For example, a value corresponding to "Fast" may be a relatively fast speed, a value corresponding to "Slow" may be a relatively slow speed, and a value corresponding to "Medium" may be a speed in between "Fast" and "Slow".

[0040] Parameter (8) "Speed ​​- Swing Speed" determines the swing speed of the robot 13 (i.e., the rotation speed of the neck drive unit 22). For example, a value corresponding to "Fast" may be a relatively fast speed, a value corresponding to "Slow" may be a relatively slow speed, and a value corresponding to "Medium" may be a speed in between "Fast" and "Slow".

[0041] Parameter (9) "Target Range - Person" determines the width of the range in which the robot 13 detects people in its surroundings. For example, a value corresponding to "wide" may be a relatively wide range (e.g., a range with a radius of 4 meters), and a value corresponding to "narrow" may be a relatively narrow range (e.g., a range with a radius of 2 meters).

[0042] Parameter (10) "Target Range - Obstacles" determines the width of the range in which the robot 13 detects objects other than people in its surroundings. For example, a value corresponding to "wide" may be a relatively wide range (e.g., a range with a radius of 4 meters), and a value corresponding to "narrow" may be a relatively narrow range (e.g., a range with a radius of 2 meters).

[0043] Parameter (11) "Focus Range - Person / Object Priority" determines the priority between detecting people in the surrounding area and detecting objects other than people in the surrounding area. For example, a value corresponding to "Person = Object" may indicate that people and objects in the surrounding area are detected with the same priority. A value corresponding to "Person > Object" may indicate that people in the surrounding area are detected with priority over objects. Although not shown in Figure 5, a character assigned an unfriendly personality may have parameter (11) set to "Person < Object". A value corresponding to "Person < Object" may indicate that objects are detected with priority over people in the surrounding area.

[0044] The character data stored in the character data storage unit 34 further includes images of each character. The user can select from multiple characters, including a character representing the user (user avatar) and characters representing friends (friend avatars). The character data for each character, including the user's character and the friend's character, includes images of each character, as well as information necessary for remotely controlling the robot 13. This information may include, for example, identification information for the processing unit 128 (HMD100), which is the destination for images and sounds collected by the robot 13 and the source of operation information for the robot 13, as well as information necessary for communication with the processing unit 128 (HMD100).

[0045] Returning to Figure 4, in this embodiment, an application program (hereinafter referred to as "Entertainment App"), which implements multiple modules corresponding to the operation reception unit 36, App control unit 38, display content generation unit 40, display control unit 42, and character data transmission unit 44, is installed in the storage of the user terminal 11. The Entertainment App may be stored on a recording medium and installed in the user terminal 11 via that recording medium. Alternatively, the Entertainment App may be downloaded from a server via a network and installed in the user terminal 11. The CPU of the user terminal 11 reads the Entertainment App into main memory and executes it, thereby performing the functions of the multiple functional blocks described above.

[0046] The operation reception unit 36 ​​receives operations entered by the user on the display unit 30 (touch panel). The operation reception unit 36 ​​notifies the App control unit 38 of the information regarding the received operations. The App control unit 38 controls the operation of the entertainment app in accordance with the user operations received by the operation reception unit 36.

[0047] The display content generation unit 40 generates content (images, etc.) to be displayed on the display unit 30. The display control unit 42 displays the content (images, etc.) generated by the display content generation unit 40 on the display unit 30. For example, the display content generation unit 40 generates content (referred to as the "character selection screen" in the first embodiment) that allows the user to select one of several characters, each with a different personality. The display control unit 42 displays the character selection screen on the display unit 30.

[0048] The character data transmission unit 44 reads the data of the character selected on the character selection screen (hereinafter also referred to as "selected character data") from the character data storage unit 34 and transmits the selected character data to the control device 14.

[0049] Figure 6 is a block diagram showing the functional blocks of the robot 13. The robot 13 includes an input system 50 that receives and processes input from the outside, and an output system 52 that processes output to the outside. The input system 50 includes an operation command receiving unit 54, a drive control unit 56, and an audio processing unit 58. The output system 52 includes an image processing unit 60, an audio processing unit 62, and a transmission unit 64.

[0050] The operation command receiving unit 54 receives operation commands transmitted from the control device 14. The operation command includes at least one of a drive command that instructs the operation mode of the travel drive unit 20 and the neck drive unit 22, and audio data to be output from the speaker 28.

[0051] The drive control unit 56 controls the operation of the travel drive unit 20 and the neck drive unit 22 according to the drive command included in the operation command. In other words, the drive control unit 56 operates the travel drive unit 20 and the neck drive unit 22 in the manner indicated by the drive command (e.g., speed, distance, angle, etc.). The voice processing unit 58 performs voice processing based on the voice data included in the operation command and outputs voice based on the voice data from the speaker 28.

[0052] The right camera 24a and the left camera 24b are directed in directions controlled by the neck drive unit 22 and capture images within their respective fields of view. The right camera 24a and the left camera 24b may be positioned far enough apart to represent the average distance between an adult's eyes. The right-eye image data captured by the right camera 24a and the left-eye image data captured by the left camera 24b are provided to the HMD 100 and may be displayed on the right and left halves of the display panel 102, respectively. These images form parallax images as seen from the right and left eyes, and by displaying them in areas that divide the display panel 102 into two sections, the image can be viewed in stereoscopic form. In addition, since the user views the display panel 102 through an optical lens, the image processing unit 60 may generate image data with optical distortion caused by the lens corrected in advance and supply it to the HMD 100. When the right-eye image data and left-eye image data are not specifically distinguished, they are collectively referred to as "image data".

[0053] The right microphone 26a and the left microphone 26b convert sounds from the robot 13's surroundings into electrical signals to generate audio signals. The audio processing unit 62 generates audio data by encoding the audio signals output from the right microphone 26a and the left microphone 26b. The transmission unit 64 transmits the imaging data supplied from the image processing unit 60 and the audio data supplied from the audio processing unit 62 to the control device 14.

[0054] Figure 7 is a block diagram showing the functional blocks of the control device 14. The control device 14 includes a character data storage unit 70, a game storage unit 72, a character data receiving unit 74, an image capture data receiving unit 76, an object identification unit 78, an action determination unit 80, an action command transmission unit 82, a game generation unit 84, a game provisioning unit 86, an audio data receiving unit 88, a robot surrounding data transmission unit 90, and a robot operation receiving unit 92.

[0055] The character data storage unit 70 stores the data of the character selected by the user, including the selected character data (image, various parameters, etc.) transmitted from the user terminal 11. The game storage unit 72 stores the game application data (hereinafter also referred to as "game data") generated by the game generation unit 84, which will be described later.

[0056] In this embodiment, an application program, which implements multiple modules corresponding to a character data receiving unit 74, an imaging data receiving unit 76, an object identification unit 78, an action determination unit 80, an action command transmission unit 82, a game generation unit 84, a game provisioning unit 86, an audio data receiving unit 88, a robot surrounding data transmission unit 90, and a robot operation receiving unit 92, is installed in the storage of the control device 14 via a recording medium or network. The CPU of the control device 14 reads the application program into main memory and executes it, thereby performing the functions of the multiple functional blocks.

[0057] The character data receiving unit 74 receives the selected character data transmitted from the user terminal 11, which is the data of the character selected by the user. The character data receiving unit 74 stores the received selected character data in the game memory unit 72.

[0058] The imaging data receiving unit 76 receives imaging data transmitted from the robot 13. In other words, the imaging data receiving unit 76 acquires image data of real-world objects captured by the camera 24 of the robot 13.

[0059] The object identification unit 78 identifies what the object captured by the camera 24 of the robot 13 is by known methods. In other words, the object identification unit 78 identifies the type of object that appears in the image data transmitted from the robot 13. For example, the object identification unit 78 identifies whether the object in the image data is a person or something other than a person. The object identification unit 78 may also use template matching to identify the type of object based on the shape, pattern, color, etc., of the object in the image data.

[0060] The motion determination unit 80 determines the operation mode of the robot 13 based on parameters corresponding to the personality of the character selected in the user terminal 11. Specifically, the motion determination unit 80 determines the operation mode of the robot 13 based on the image of the object captured in the imaging data transmitted from the robot 13, the type of the object identified by the object identification unit 78, and the character data stored in the character data storage unit 70. The operation mode of the robot 13 includes the operation mode of the travel drive unit 20 (direction of movement, speed of movement, etc.) and the operation mode of the neck drive unit 22 (rotation angle, rotation speed, etc.).

[0061] The motion command transmission unit 82 transmits motion commands (drive commands) to the robot 13 that indicate the mode of operation of the robot 13 determined by the motion determination unit 80, thereby operating the robot 13 in the mode determined by the motion determination unit 80.

[0062] The game generation unit 84 generates a game that displays an AR space (AR image) that includes both real-world objects captured by the camera 24 of the robot 13, that is, real-world objects captured in the image data transmitted from the robot 13, and virtual objects corresponding to those real-world objects. The game generation unit 84 stores the generated game data in the game storage unit 72.

[0063] The game provision unit 86 provides game data generated by the game generation unit 84 and stored in the game storage unit 72 to an external device. For example, the game generation unit 84 may generate a game in which the user explores an AR space by displaying an AR image for the right eye and an AR image for the left eye with a parallax between them. The game provision unit 86 may then provide the game data to the processing unit 128, thereby displaying the 3D game in which the user explores an AR space on the HMD 100.

[0064] The voice data receiving unit 88 receives voice data transmitted from the robot 13. The robot surrounding data transmitting unit 90 transmits robot surrounding data to the processing unit 128, which includes at least one of the imaging data acquired by the imaging data receiving unit 76 and the voice data acquired by the voice data receiving unit 88. As a result, the image of the area around the robot 13 captured by the robot 13 and the sound of the area around the robot 13 collected by the robot 13 are played back on the HMD 100.

[0065] The robot operation reception unit 92 accepts remote control requests for the robot 13 from the user or a friend. Specifically, the robot operation reception unit 92 receives user operation information transmitted from the processing unit 128a and friend operation information transmitted from the processing unit 128b.

[0066] When the operation determination unit 80 receives user operation information (or friend operation information), it generates an operation command that includes a drive command to cause the robot 13 to perform an action corresponding to the operation indicated by the user operation information (or friend operation information). The operation determination unit 80 also generates an operation command that includes the voice data contained in the user operation information (or friend operation information).

[0067] The operation of the entertainment system 10 of the first embodiment, configured as described above, will now be explained. The user launches an entertainment app on the user terminal 11. The display content generation unit 40 of the user terminal 11 generates data for the character selection screen, and the display control unit 42 displays the character selection screen on the display unit 30.

[0068] Figure 8 shows an example of a character selection screen. The character selection screen 131 is configured to display multiple characters with different personalities (i.e., candidate characters) in a switchable manner depending on the user's swipe gesture. In Figure 8, the multiple characters include character 130a (character B in Figure 5), character 130b (character A in Figure 5), character 130c (character C in Figure 5), and character 130d (Takeo), who represents a friend. When no particular distinction is made between characters 130a, 130b, 130c, and 130d, they are referred to as "character 130".

[0069] Icon 132 indicates that robot 13 is an autonomously operating character, or a character that is remotely controlled. Icon 134 indicates that the character is automatically generated in the game.

[0070] When a user presses the install button 136 on the character selection screen 131 where a specific character is displayed, the App control unit 38 of the user terminal 11 activates the camera 32. The display content generation unit 40 of the user terminal 11 determines the image captured by the camera 32, which shows objects around the user terminal 11, as the display content, and the display control unit 42 displays the image on the display unit 30.

[0071] The user places the robot 13 within the field of view of the camera 32. The display content generation unit 40 detects, using known object recognition processing (such as template matching), when the robot 13 is captured by the camera 32, that is, when the robot 13 is visible in the image captured by the camera 32. When the display content generation unit 40 detects that the robot 13 has been captured by the camera 32, it generates an AR image that displays the image of the robot 13 along with the image of the character 130 selected by the user on the character selection screen 131. The display control unit 42 displays the AR image on the display unit 30.

[0072] Next, the display content generation unit 40 generates an AR image that includes content in which the character 130 selected by the user possesses the robot 13. Figure 9 shows an example of an AR image. The AR image 138 may be a video showing the character 130 entering the robot 13 from outside. Alternatively, the AR image 138 may be an image that further displays objects that associate the robot 13 and the character 130. In this way, by displaying an AR image that includes content in which the character 130 selected by the user possesses the robot 13, it is possible to clearly show the user that the robot 13 will subsequently perform actions according to the personality of the character 130.

[0073] The character data transmission unit 44 of the user terminal 11 transmits character data related to the character 130 selected by the user to the control device 14 in synchronization with the display of the AR image 138 (i.e., the display of the content in which the character 130 selected by the user possesses the robot 13).

[0074] The character data receiving unit 74 of the control device 14 receives character data transmitted from the user terminal 11 and stores it in the character data storage unit 70. The operation determination unit 80 of the control device 14 decides to activate the camera 24 of the robot 13, and the operation command transmission unit 82 transmits an operation command to the robot 13 instructing it to activate the camera 24. The robot 13 activates the camera 24, and the transmission unit 64 of the robot 13 transmits the image data (image data) captured by the camera 24 to the control device 14.

[0075] The image data receiving unit 76 of the control device 14 receives the image data transmitted from the robot 13. The object identification unit 78 of the control device 14 identifies the type of object (e.g., person or object) that appears in the image of the image data and is present around the robot 13. The operation determination unit 80 of the control device 14 determines the operation mode of the robot 13 based on the attributes of the object present around the robot 13 (e.g., position, shape, pattern, color, type) and parameters corresponding to the personality of the character selected by the user (i.e., character data stored in the character data storage unit 70).

[0076] A first example of the process for determining the operation mode of the robot 13 will be described. The operation determination unit 80 determines the operation mode of the robot 13 regarding pathfinding based on parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 determines the goal position (for example, a position close to a person) according to parameter (5) "Pathfinding - Goal position (person)" in Figure 5. The operation determination unit 80 also determines the movement mode (movement path) to the goal according to parameters (1) "Pathfinding - Distance from obstacles", parameter (2) "Pathfinding - Smoothness of the path", parameter (3) "Pathfinding - Detour (person)", and parameter (4) "Pathfinding - Detour (object)" in Figure 5.

[0077] A second example of the process for determining the operation mode of the robot 13 will be described. The operation determination unit 80 determines the operation mode of the robot 13 based on parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 determines the movement speed of the robot 13 (e.g., the rotational speed of the running drive unit 20) according to parameter (7) "speed - normal movement speed" in Figure 5. The operation determination unit 80 also determines the acceleration until the above movement speed is reached (e.g., the rotational acceleration of the running drive unit 20) according to parameter (6) "speed - change in movement speed (acceleration)" in Figure 5. The operation determination unit 80 also determines the rotational speed of the neck drive unit 22 according to parameter (8) "speed - head swing speed" in Figure 5.

[0078] A third example of the process for determining the operation mode of the robot 13 will be described. The operation determination unit 80 determines an operation mode for determining the object that the robot 13 will focus on from among the objects identified by the object identification unit 78, based on parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 considers people within the range defined by parameter (9) "Target range - Person" in Figure 5 as target of attention candidates. The operation determination unit 80 also considers objects within the range defined by parameter (10) "Target range - Obstacle" in Figure 5 as target of attention candidates. Furthermore, the operation determination unit 80 determines the target of attention from among the target of attention candidates, people and objects, according to parameter (11) "Target range - Person / Object priority" in Figure 5.

[0079] If the action determination unit 80 has determined a target of interest, it may decide to have the robot 13 perform a targeting action (for example, imaging with the camera 24) on the person or object of interest.

[0080] The motion determination unit 80 generates an action command that instructs the determined motion mode. The action command transmission unit 82 of the control device 14 transmits the action command generated by the motion determination unit 80 to the robot 13. The drive control unit 56 of the robot 13 controls the operation of the travel drive unit 20 and the neck drive unit 22 according to the action command. As a result of the above processing, the robot 13 autonomously explores its surroundings in a manner that corresponds to the personality (each parameter of the character data) of the character selected by the user.

[0081] Thus, according to the entertainment system 10 of the first embodiment, the user can have the robot 13 perform actions corresponding to the personality of the character desired by the user from among multiple characters with different personalities, thereby improving the user experience provided by the robot 13 (entertainment robot).

[0082] Furthermore, if the user selects a second character (a different character from the first character) after selecting a first character, the control device 14 will control the robot 13's operation based on the character data of the second character, instead of controlling the robot 13's operation based on the character data of the first character. In other words, the control device 14 dynamically switches the parameters for controlling the robot 13's operation in response to the user's change in character selection. This allows the user to arbitrarily switch the operation mode of a single robot 13.

[0083] Next, we will explain the operation when character 130b (character A), which is assigned icon 134, is selected on the character selection screen 131 in Figure 8. In this case, the robot 13 autonomously searches its surroundings in a manner corresponding to the personality assigned to character A, and captures images of surrounding objects with the camera 24 at predetermined intervals. The transmission unit 64 of the robot 13 transmits the image data from the camera 24 to the control device 14 at predetermined intervals.

[0084] The image data receiving unit 76 of the control device 14 receives the image data transmitted from the robot 13, and the object identification unit 78 identifies the type of object captured in the image data, in other words, the type of real-world object captured by the camera 24 of the robot 13. The game generation unit 84 generates a game that displays an AR space that includes real-world objects captured by the camera 24 of the robot 13 (hereinafter also called "real objects") and virtual objects corresponding to those real-world objects (hereinafter also called "virtual objects").

[0085] Figure 10 shows an example of a game image generated by the game generation unit 84. In the game image 140, a skateboard, shelves, blocks, and a ball are placed as real objects 142. These are objects that actually exist in the room where the robot 13 is placed. The game image 140 also includes virtual objects 144, such as a companion character, a windmill, a fence, clouds, and character 130 (a character selected by the user). This game could be, for example, a game in which character 130 explores an AR space where real objects 142 and virtual objects 144 are mixed.

[0086] The game generation unit 84 stores data indicating the correspondence between any combination of shape, pattern, color, type, etc. of real objects 142 captured in the image data and virtual objects 144. The data indicating the correspondence may, for example, be data that associates (1) a skateboard as a real object 142 with a companion character as a virtual object 144, (2) a shelf as a real object 142 with a windmill as a virtual object 144, or (3) a block as a real object 142 with a fence as a virtual object 144. The game generation unit 84 places a virtual object 144 corresponding to a certain real object 142 in its vicinity.

[0087] The game generation unit 84 stores the generated game data in the game storage unit 72. The game provision unit 86 transmits the game data stored in the game storage unit 72 to the requesting external device in response to a request from the external device. According to the entertainment system 10 of the first embodiment, a game can be provided to the user that displays an AR space in which real objects, captured from the viewpoint of the robot 13 (the position of the camera 24), are placed along with virtual objects corresponding to those real objects, thereby providing a novel user experience using the robot 13.

[0088] Next, we will explain the operation when character 130d (friend avatar) is selected on the character selection screen 131 in Figure 8. In this case, the operation determination unit 80 determines the operation mode of the robot 13 based on the friend operation information transmitted from the friend's device.

[0089] Specifically, when character 130d (friend avatar) is selected on the character selection screen 131, the character data storage unit 70 of the control device 14 stores the data necessary for communication with the processing device 128b as data for that character. The control device 14 (robot operation reception unit 92, robot surrounding data transmission unit 90) uses the data stored in the character data storage unit 70 to establish a connection with the processing device 128b.

[0090] The imaging data receiving unit 76 of the control device 14 receives imaging data (right eye image and left eye image) transmitted from the robot 13 at predetermined intervals. The audio data receiving unit 88 receives audio data transmitted from the robot 13 at predetermined intervals. The robot surrounding data transmitting unit 90 transmits the imaging data and audio data transmitted from the robot 13 to the processing unit 128b.

[0091] The processing unit 128b transfers the imaging data and audio data transmitted from the control unit 14 to the HMD 100b. The HMD 100b displays the imaging data (right eye image and left eye image) transmitted from the processing unit 128b on the display panel 102 (right eye display panel and left eye display panel). The friend can see the surroundings of the robot 13 in real time and hear the sounds around the robot 13 in real time.

[0092] The Friend inputs operations for the robot 13 (operations that instruct the driving mode of the locomotion drive unit 20 and the driving mode of the neck drive unit 22) into the processing unit 128b. The processing unit 128b transmits Friend operation information, which indicates the operations for the robot 13 input by the Friend, to the control device 14. The Friend operation information includes audio data indicating the voice spoken by the Friend, which is collected by the microphone 106 of the HMD 100b.

[0093] The robot operation reception unit 92 of the control device 14 receives friend operation information transmitted from the processing unit 128b. The operation determination unit 80 generates an operation command that instructs the operation content indicated by the friend operation information (such as the drive mode of the travel drive unit 20 or the drive mode of the neck drive unit 22). The operation command transmission unit 82 transmits the operation command to the robot 13. The friend operation information may also include the movement of the head of the friend wearing the HMD 100b, and the operation determination unit 80 may determine the drive mode of the neck drive unit 22 of the robot 13 to match the movement of the friend's head.

[0094] As a result of the above processing, the operations that the friend inputs for robot 13 are reflected in the robot 13's movements in real time. The operation determination unit 80 also generates an operation command that includes voice data contained in the friend's operation information. The operation command transmission unit 82 transmits this operation command to robot 13. As a result, the voice spoken by the friend is output in real time from the robot 13's speaker 28.

[0095] Thus, according to the entertainment system 10 of the first embodiment, the robot 13 operates in response to remote control by a friend and also plays the friend's voice. Meanwhile, the friend can view images and sounds around the robot 13 in real time through the HMD 100b. This allows people around the robot 13 and the friend to communicate in real time.

[0096] Although not shown in Figure 8, the characters selectable on the character selection screen 131 may include a character representing the user (user avatar). If a user avatar is selected on the character selection screen 131, the operation determination unit 80 determines the operation mode of the robot 13 based on user operation information transmitted from the user's device. The specific processing when a user avatar is selected on the character selection screen 131 is the same as the processing when a friend avatar is selected on the character selection screen 131, with HMD 100b replaced by HMD 100a and processing unit 128b replaced by processing unit 128a.

[0097] <Second Example> The following description of the second embodiment will focus on the differences in configuration from the first embodiment, and descriptions of common configurations will be omitted as appropriate. It goes without saying that the configuration of the second embodiment can be combined with the configurations of the first embodiment and its modified examples in any way.

[0098] The second embodiment will now be outlined. In recent years, a technology called tele-existence has emerged, which uses a robot placed in a remote location as a digital avatar. The robot in the remote location transmits image and sound data of its surroundings to the user, and the user plays it back, allowing the user to experience a sense of presence as if they were actually there. Head-mounted displays (HMDs) are also being used in various fields. By equipping the HMD with head tracking functionality and updating the display screen in conjunction with the user's head posture, the sense of immersion in the visual world can be enhanced.

[0099] In the second embodiment, an AR image is displayed on the HMD worn by the user, showing real-world objects captured by the robot's camera and virtual objects placed by the user. The height position of the HMD can change significantly, for example, when the user stands up. The height direction is, for example, perpendicular to the floor surface on which the robot or user moves. On the other hand, the robot is usually configured so that the height position of the camera does not change beyond a predetermined range. Therefore, if the height position of the HMD changes, it can cause discomfort to the user viewing the AR image and may cause so-called VR sickness.

[0100] Therefore, in the entertainment system of the second embodiment, when the height position of the HMD changes, an image is generated in which real-world objects are removed and which shows the appearance of virtual objects as seen from a new viewpoint corresponding to the change in the height position of the HMD (a so-called VR image), and the system switches from displaying the AR image to displaying the VR image. In other words, in the entertainment system of the second embodiment, the HMD's video see-through is turned on as the default mode, but when the height position of the HMD changes, the HMD's video see-through is switched off.

[0101] The configuration of the entertainment system 10 in the second embodiment is the same as the configuration of the entertainment system 10 in the first embodiment shown in Figure 1. As previously described, the height (eye level) of the mounting position of the camera 24 on the robot 13 is 144 mm. The robot 13 is configured so that the position of the camera 24 in the height direction does not change beyond a predetermined range. For example, even if the neck drive unit 22 of the robot 13 rotates on the pitch axis or roll axis, the change in eye level is configured to be only a few centimeters. Note that the robot 13 may also be configured so that the position of the camera 24 in the height direction does not change.

[0102] Figure 11 is a block diagram showing the functional blocks of the control device 14 of the second embodiment. The control device 14 of the second embodiment includes, as functional blocks similar to the control device 14 of the first embodiment, a character data storage unit 70, a character data receiving unit 74, an imaging data receiving unit 76, an object identification unit 78, an action determination unit 80, an action command transmission unit 82, an audio data receiving unit 88, a robot surrounding data transmission unit 90, and a robot operation receiving unit 92. The control device 14 of the second embodiment further includes a game operation receiving unit 150, a game image generation unit 152, a game image transmission unit 154, a sensor information receiving unit 156, and a motion detection unit 158.

[0103] An application program, which implements multiple modules corresponding to the character data receiving unit 74, image capture data receiving unit 76, object identification unit 78, motion determination unit 80, motion command transmission unit 82, voice data receiving unit 88, robot surrounding data transmission unit 90, robot operation receiving unit 92, game operation receiving unit 150, game image generation unit 152, game image transmission unit 154, sensor information receiving unit 156, and motion detection unit 158, may be installed in the control device 14 via a recording medium or network. The CPU of the control device 14 may perform the functions of the multiple functional blocks by reading the application program into main memory and executing it.

[0104] The game operation reception unit 150 receives user operation information from the processing unit 128a, which indicates the user's actions on the game.

[0105] The game image generation unit 152 generates game images based on the imaging data acquired by the robot 13 by the imaging data reception unit 76 and the user operation information acquired by the game operation reception unit 150. The game images include a game image for the right eye and a game image for the left eye, which have parallax between them.

[0106] In the second embodiment, the game image transmission unit 154 of the control device 14, the processing unit 128a, and the control unit 120 of the HMD 100a work together to function as a display control unit that displays game images on the display panel 102 of the HMD 100a. Specifically, the game image transmission unit 154 transmits data of game images (game images for the right eye and game images for the left eye) generated by the game image generation unit 152 to the processing unit 128a. The processing unit 128a passes the game images to the HMD 100a, which displays the game image for the right eye on the right eye display panel and the game image for the left eye on the left eye display panel. As a result, the user can view the content depicted in the game images in 3D.

[0107] The sensor information receiving unit 156 acquires posture information detected by the posture sensor 124 of the HMD 100a. The motion detection unit 158 ​​uses known head tracking technology to detect the posture of the HMD 100a worn on the user's head based on the posture information acquired by the sensor information receiving unit 156.

[0108] In the second embodiment, the motion detection unit 158 ​​detects changes (amount of change) in the height position of the HMD 100a. The motion detection unit 158 ​​inputs the amount of change in the height position of the HMD 100a to the game image generation unit 152. The game image generation unit 152 generates a game image that takes into account the amount of change in the height position of the HMD 100a.

[0109] The operation of the entertainment system 10 of the second embodiment, configured as described above, will now be explained. Here, a user avatar is selected from among several characters, the user remotely controls robot 13, and video and audio of the robot 13's surroundings are played on the HMD 100a. The user inputs an operation to switch to game mode into the processing unit 128a, and the processing unit 128a transmits user operation information indicating that operation to the control device 14.

[0110] When the control device 14 receives user operation information instructing it to switch to game mode, the game image generation unit 152 of the control device 14 generates a first game image in which virtual objects are displayed along with real-world objects (real objects) that are captured in the imaging data acquired by the imaging data receiving unit 76 (i.e., images captured by the camera 24 of the robot 13). The first game image is the default game image and is a game image with the HMD 100a's video see-through enabled.

[0111] Figure 12 shows an example of the first game image. The first game image 160 in the figure includes a sofa 162, a door 164, a table 166, and a cup 168 as real objects 142. The first game image 160 also includes multiple blocks 170 and multiple enemy characters 172 as virtual objects 144. The first game image is an AR image in which virtual objects 144 are added to the real world space captured by the camera 24 of the robot 13.

[0112] The user inputs an operation to the processing unit 128a to place a virtual object (block 170 or enemy character 172) in the real-world space captured by the robot 13, and the processing unit 128a transmits user operation information indicating the above operation to the control device 14. The game operation reception unit 150 of the control device 14 receives the user operation information indicating the above operation.

[0113] The game image generation unit 152 of the control device 14 places the virtual object 144 in the space reflected in the image data acquired by the image data receiving unit 76, according to the user operation information indicating the above operation. The game image generation unit 152 generates a first game image showing the appearance of the virtual object 144 from the viewpoint of the robot 13's camera 24, or in other words, a first game image showing the appearance of the virtual object 144 as seen from the robot 13's eye level. The game image transmission unit 154 of the control device 14 transmits the first game image to the processing unit 128a, thereby displaying the first game image on the HMD 100a.

[0114] The sensor information receiving unit 156 of the control device 14 acquires the attitude information of the HMD 100a detected by the attitude sensor 124 of the HMD 100a. Based on the attitude information of the HMD 100a, the motion detection unit 158 ​​of the control device 14 detects the amount of change if the height position of the HMD 100a changes.

[0115] The game image generation unit 152 of the control device 14 generates a second game image in place of the first game image when the height position of the HMD 100a changes beyond a predetermined range. The game image transmission unit 154 transmits the second game image to the processing unit 128a, thereby displaying the second game image on the HMD 100a in place of the first game image. The predetermined range, which is the threshold for switching from the first game image to the second game image, may be the range in which the height position (i.e., eye level) of the camera 24 in the robot 13 can change. Alternatively, the predetermined range may be determined based on the developer's knowledge or experiments using the entertainment system 10.

[0116] Figure 13 shows an example of a second game image. The second game image 161 in the figure includes multiple blocks 170 and multiple enemy characters 172 as virtual objects 144. Also, unlike the first game image 160, the second game image 161 does not include real objects 142 (such as the cup 168). In other words, the second game image is an image in which real-world objects have been removed and is a VR image that shows the appearance of the virtual objects 144 as seen from a new viewpoint corresponding to the change in the height position of the HMD 100a. Furthermore, the second game image is a game image with the video see-through of the HMD 100a disabled.

[0117] The second game image 161 in Figure 13 is typically displayed when a user who was viewing the first game image 160 in a seated position stands up, and the height position of the HMD 100a changes from bottom to top. When the height position of the HMD 100a changes from bottom to top, the game image generation unit 152 of the control device 14 generates a second game image in which real-world objects have been removed, and which shows the appearance of virtual objects 144 as seen from a viewpoint higher than the viewpoint in the first game image. The second game image 161 in Figure 13 can also be called an overhead view. The viewpoint at the higher position may be the viewpoint of the standing user, or in other words, the user's eye level.

[0118] The first game image 160 can be said to be an image based on the viewpoint of the robot 13, while the second game image 161 can be said to be an image based on the viewpoint of the user. When generating the first game image 160, the game image generation unit 152 may place a virtual camera at the center of the coordinate system and align the direction of the optical axis of the virtual camera with the optical axis direction of the camera 24 of the robot 13. The game image generation unit 152 may place the image of the real object 142 captured by the camera 24 of the robot 13 and the virtual object 144 placed by user operation in the above coordinate system, capture them with the virtual camera, and generate the first game image 160 based on the capture results.

[0119] On the other hand, when generating the second game image 161, the game image generation unit 152 may change the position of the virtual camera to a higher position than before, in accordance with the amount of change in the height position of the HMD 100a. The game image generation unit 152 may capture images of the real object 142 and virtual object 144, which are relatively lower in position, from the relatively higher virtual camera, and generate the second game image 161 based on the imaging results.

[0120] According to the entertainment system 10 of the second embodiment, even if the height position of the HMD 100 changes, it is possible to suppress the user from feeling discomfort while viewing the game image and to suppress the occurrence of so-called VR sickness in the user. This improves the user experience through the cooperation of tele-existence and the HMD 100. The technology described in the second embodiment is not limited to the case where the camera 24 is mounted on the robot 13. It is broadly useful when the image (video) captured by the camera 24 is displayed on the HMD 100 and there are limitations on the change in the height position of the camera 24 (or the height position of the camera 24 does not change).

[0121] The present invention has been described above based on the first and second embodiments. These embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combination with their respective components and processing steps, and that such modifications also fall within the scope of the present invention.

[0122] A first modified example will be described. In the first embodiment, data (image, various parameters, etc.) of a character selected by the user was transmitted from the user terminal 11 to the control device 14. As a modified example, the character data storage unit 70 of the control device 14 may store multiple character data for multiple characters that can be selected by the user, similar to the character data storage unit 70 of the user terminal 11 in the first embodiment. The user terminal 11 may notify the control device 14 of the identification information (name, ID, etc.) of the character selected by the user. The operation determination unit 80 of the control device 14 may determine the operation mode of the robot 13 by referring to the character data of the character identified by the identification information transmitted from the user terminal 11 among the multiple character data stored in the character data storage unit 70.

[0123] A second modified example will be described. In the second embodiment, the first and second game images were displayed on the user's HMD 100a, but the same configuration can be used to display these game images on a friend's HMD 100b. In this case, the "processing device 128a" described in the second embodiment should be replaced with "processing device 128b," and the "user operation information" described in the second embodiment should be replaced with "friend operation information."

[0124] A third modification will now be described. Although not mentioned in the second embodiment, the game image generation unit 152 of the control device 14 may, even when the height position of the HMD 100 changes from top to bottom, generate a second game image in place of the first game image, which is a second game image in which the real objects have been removed and which shows the appearance of the virtual objects as seen from a new viewpoint corresponding to the change in the height position of the HMD 100. In this modification, the second game image may show the appearance of the virtual objects as seen from a viewpoint lower than the viewpoint in the first game image.

[0125] A fourth modification will now be described. The entertainment system 10 in each embodiment may not include the control device 14. In this case, the functions of the control device 14 described in the first and second embodiments (e.g., the operation determination unit 80) may be implemented in the robot 13 or in the user terminal 11 (entertainment app). Alternatively, some of the functions of the control device 14 described in the first and second embodiments may be implemented in the robot 13, and the remaining functions of the control device 14 may be implemented in the user terminal 11 (entertainment app). In this modification, the user terminal 11 and the robot 13 may communicate via the communication network 15.

[0126] For example, the robot 13 of the fourth modified example may include a character data receiving unit (corresponding to the character data receiving unit 74 of the control device 14 in the embodiment) and an operation determination unit (corresponding to the operation determination unit 80 of the control device 14 in the embodiment). The character data receiving unit may receive information from the user terminal 11 about a character selected by the user from among a plurality of characters, each with a different personality. The operation determination unit may determine the operation mode of the robot 13 based on parameters corresponding to the personality of the character selected by the user.

[0127] A fifth modified example will now be described. In the above embodiment, an image captured by a camera 24 installed at the viewpoint position of the robot 13 (referred to here as the "robot viewpoint image") was displayed on the HMD 100 (HMD 100a, HMD 100b). In the modified entertainment system 10, the robot 13 or the control device 14 may transmit the robot viewpoint image to the user's or friend's terminal (such as a smartphone, for example, the user terminal 11), and the robot viewpoint image may be displayed on the user's or friend's terminal. In this case, the user's or friend's terminal (such as a smartphone, for example, the user terminal 11) may transmit user operation information or friend operation information to the control device 14, or it may transmit it directly to the robot 13. In this way, the robot 13 may be controlled using the user's or friend's terminal.

[0128] A sixth modified example will be described. The character data stored in the character data storage unit 34 of the user terminal 11 may include the following parameters as characteristics of the character: (1) A parameter that controls whether or not the character preferentially finds toys such as balls (in other words, whether or not it is more likely to react to toys such as balls). In the character data of a character with a toy-loving personality, a value indicating preferential reaction to toys such as balls may be set as the parameter value based on image recognition technology. (2) A parameter that controls whether or not the character reacts sensitively to ambient sounds (by increasing microphone sensitivity above normal) and starts dancing in response to detected sounds. In the character data of a character with a dance-loving personality, a value indicating increased microphone sensitivity above normal may be set as the parameter value.

[0129] (3) A parameter that controls whether or not to make the voice louder than usual. In the character data of a character that has the characteristic of having a loud voice, a value indicating that the sound output intensity from the speaker 28 should be made louder than usual may be set as the parameter value. In the entertainment system 10 of this modified example, the operation determination unit 80 of the control device 14 may determine the sound operation mode (i.e., the operation mode of the microphone 26 and speaker 28 of the robot 13) based on parameters predetermined as the characteristics of the character selected at the user terminal 11.

[0130] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiments resulting from these combinations possess the combined effects of each of the embodiments and modifications. Furthermore, it will be understood by those skilled in the art that the functions to be performed by each component described in the claims can be achieved by each component shown in the embodiments and modifications individually or in combination thereof. [Industrial applicability]

[0131] This invention can be applied to entertainment systems and robots. [Explanation of Symbols]

[0132] 10 Entertainment system, 11 User terminal, 13 Robot, 14 Control device, 24 Camera, 30 Display unit, 32 Camera, 42 Display control unit, 76 Image data receiving unit, 78 Object identification unit, 80 Action determination unit, 84 Game generation unit, 92 Robot operation receiving unit, 152 Game image generation unit, 154 Game image transmission unit, 158 Motion detection unit.

Claims

1. User terminal and Robots and, It comprises an operation determination unit, The user terminal includes a display control unit that displays content on the display unit that allows the user to select one of several characters, each with a defined personality. The operation determination unit determines the operation mode of the robot based on parameters corresponding to the personality of the character selected on the user terminal. The operation determination unit determines the mode of operation of the robot and the mode of pathfinding for the robot based on parameters predetermined as the characteristics of the character selected in the user terminal. Entertainment system.

2. The user terminal further includes an imaging unit that captures images of the surroundings, The display control unit of the user terminal, when one of the characters is selected and the robot is captured by the imaging unit, causes the display unit to display an AR (Augmented Reality) image containing the selected character possessing the robot. The entertainment system according to claim 1.

3. The robot further includes an imaging unit that images its surroundings, The entertainment system further includes an identification unit that identifies an object captured by the robot's imaging unit, The operation determination unit determines an action relating to determining the object that the robot will focus on from among the objects identified by the identification unit, based on parameters predetermined as the individuality of the character selected in the user terminal. The entertainment system according to claim 1 or 2.

4. The operation determination unit determines the sound operation mode based on parameters predetermined as the personality of the character selected in the user terminal. An entertainment system according to any one of claims 1 to 3.

5. The robot further includes an imaging unit that images its surroundings, The robot further comprises a game generation unit that generates a game that displays an AR space including real-world objects captured by the robot's imaging unit and virtual objects corresponding to the real-world objects. An entertainment system according to any one of claims 1 to 4.

6. The system further includes a reception unit that receives operation information regarding operations entered by the user's friend from the friend's device. The aforementioned multiple characters include the character representing the friend, When a character representing the friend is selected on the user terminal, the operation determination unit determines the robot's operation mode based on the operation information transmitted from the friend's device. An entertainment system according to any one of claims 1 to 5.

7. A reception unit that receives information from the user's terminal about a character selected by the user from among multiple characters with different defined personalities, An action determination unit that determines the robot's operating mode based on parameters corresponding to the character's characteristics selected by the user, Equipped with, The operation determination unit determines the mode of operation of the robot and the mode of pathfinding for the robot based on parameters predetermined as the characteristics of the character selected in the user terminal. robot.

Images