Information Processing Apparatus, Information Processing Method, and Computer Program

Abstract

This control device 14 acquires an image captured by a camera of a robot 13, and generates a first image in which a virtual object appears together with a real-world object appeaing in said image, the first image showing the exterior appearance of the virtual object as viewed from the position of the camera of the robot 13. An HMD 100 displays the first image generated by the control device 14. When the position of the HMD 100 in the height direction changes, the control device 14 generates a second image in which the real-world thing is erased and which shows the exterior appearance of the virtual image as seen from a new viewpoint that corresponds to a change of the position of the HMD 100 in the height direction. The HMD 100 displays the second image in place of the first image.

Description

Technical Field

[0001] The present invention relates to data processing technology, and more particularly to an information processing apparatus, an information processing method, and a computer program.

Background Art

[0002] In recent years, a technology called tele-presence has emerged, which uses a robot placed at a remote location as one's own avatar. By having the robot at the remote location transmit surrounding image data and audio data to the user for playback on the user side, the user can experience a sense of presence as if they were at the location of the robot.

[0003] Also, head-mounted displays (HMDs) are being used in various fields. By equipping the HMD with a head-tracking function and updating the display screen in conjunction with the user's head posture, the sense of immersion in the virtual world can be enhanced.

Summary of the Invention

Problems to be Solved by the Invention

[0004] The inventor has focused on the potential of the combination of tele-presence and head-mounted displays, and has developed a technology to improve the user experience through the combination of tele-presence and head-mounted displays.

[0005] The present invention has been made based on the above problem recognition by the inventor, and one object is to provide a technology for improving the user experience through the combination of tele-presence and head-mounted displays.

Means for Solving the Problems

[0006] To solve the above problems, an information processing apparatus according to an aspect of the present invention includes an acquisition unit that acquires an image in which a real-world object imaged by an imaging device is reflected, and a virtual object together with the real-world object reflected in the image acquired by the acquisition unit. An image generation unit that generates a first image showing the appearance of the virtual object with the position of the imaging device as a viewpoint, and a display control unit that causes the head-mounted display to display the first image generated by the image generation unit, and a detection unit that detects a change in the position of the head-mounted display in the height direction. When the position of the head-mounted display in the height direction changes, the image generation unit generates a second image in which the real-world object is erased and that shows the appearance of the virtual object seen from a new viewpoint corresponding to the change in the position of the head-mounted display in the height direction, and the display control unit causes the head-mounted display to display the second image instead of the first image.

[0007] Another aspect of the present invention is an information processing method. This method includes a step of acquiring an image in which a real-world object imaged by an imaging device is reflected, and a first image in which a virtual object is reflected together with the real-world object reflected in the image acquired in the acquiring step, and that shows the appearance of the virtual object with the position of the imaging device as a viewpoint, a step of causing the head-mounted display to display the first image generated by the generating function, a step of detecting a change in the position of the head-mounted display in the height direction, and when the position of the head-mounted display in the height direction changes, a second image in which the real-world object is erased and that shows the appearance of the virtual object seen from a new viewpoint corresponding to the change in the position of the head-mounted display in the height direction, and a step of causing the head-mounted display to display the second image instead of the first image, which are executed by a computer.

[0008] Note that any combination of the above components, and those obtained by converting the expression of the present invention among a system, a computer program, a recording medium on which the computer program is readable, a data structure, etc. are also effective as aspects of the present invention.

Advantages of the Invention

[0009] According to the present invention, the user experience can be improved by the cooperation between the telemetry distance and the head-mounted display.

Brief Description of the Drawings

[0010]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Modes for Carrying Out the Invention

[0011] <First Embodiment> The outline of the first embodiment will be described. In recent years, various entertainment robots such as pet robots have been provided. However, the conventional entertainment robots have been created with the personality of a single character. In the first embodiment, an entertainment system is proposed that causes an entertainment robot to execute an operation according to the personality of a character desired by a user from among a plurality of characters having different personalities. Note that the characters in the embodiment can also be called "avatars" as avatars of users or friends.

[0012] FIG. 1 shows a configuration example of the entertainment system 10 according to the first embodiment. The entertainment system 10 is an information processing system that provides a user with an entertainment experience using a robot. The entertainment system 10 includes a user terminal 11 operated by a user, an HMD 100a, a processing device 128a, an HMD 100b operated by another user (hereinafter referred to as a "friend") registered by the user as a friend, a processing device 128b, a robot 13, and a control device 14. These devices have functions of wired communication and / or wireless communication, and are connected via a communication network 15 that can include a LAN, WAN, Internet, or the like.

[0013] The user terminal 11 is a mobile phone equipped with a mobile operating system, such as a smartphone or a tablet terminal.

[0014] The HMD 100a is a head-mounted display device worn by the user on the head. The processing device 128a is an information processing device that controls the display of images (e.g., VR (Virtual Reality) images, AR (Augmented Reality) images, etc.) on the HMD 100a. The "image" in the embodiment includes still images and moving images. The processing device 128a receives an operation on the robot 13 input by the user, and transmits information regarding the operation (hereinafter also referred to as "user operation information") to the control device 14. The user operation information includes voice data indicating the user's voice collected by the microphone of the HMD 100a.

[0015] HMD100b is a head-mounted display device worn on the head by a friend. The processing device 128b is an information processing device that controls the display of images in HMD100b. The processing device 128b receives an operation on the robot 13 input by the friend, and transmits information regarding the operation (hereinafter also referred to as "friend operation information") to the control device 14. The friend operation information includes voice data indicating the voice of the friend collected by the microphone of HMD100b.

[0016] Hereinafter, when HMD100a and HMD100b are not particularly distinguished, they are referred to as "HMD100". Also, when the processing device 128a and the processing device 128b are not particularly distinguished, they are referred to as "processing device 128". The processing device 128 may be a PC, or may be a stationary or portable game. HMD100 and the processing device 128 may be integrally configured. For example, HMD100 may include the functions of the processing device 128. In this case, HMD100 and the control device 14 may be configured to communicate via the communication network 15.

[0017] The robot 13 is a self-propelled robot with a full width of 192 mm, a full height of 167 mm, and a depth of 166 mm. The robot 13 includes a traveling drive unit 20, a neck drive unit 22, a right camera 24a, a left camera 24b, a right microphone 26a, a left microphone 26b, and a speaker 28. Hereinafter, when the right camera 24a and the left camera 24b are not particularly distinguished, they are referred to as "camera 24". Also, when the right microphone 26a and the left microphone 26b are not particularly distinguished, they are referred to as "microphone 26". The speaker 28 outputs voice data received from an external device.

[0018] The traveling drive unit 20 includes three omni wheels. Each omni wheel moves forward and backward by the rotation of the main body (wheel) on the axis, moves left and right by the rotation of the circumferential barrel, and can move in multiple directions by the combination of these. The neck drive unit 22 rotates the head 21 of the robot 13 in three axes. The three axes include a pitch axis for rotating the neck in the vertical direction, a roll axis for rotating the neck in the tilting direction, and a yaw axis for rotating the neck sideways.

[0019] The right camera 24a and the left camera 24b are provided on the head 21. The right camera 24a and the left camera 24b are arranged at a predetermined interval in the horizontal direction. The right camera 24a and the left camera 24b constitute a stereo camera. The right camera 24a captures an image for the right eye at a predetermined cycle, and the left camera 24b captures an image for the left eye at a predetermined cycle. The height (eye height) of the attachment positions of the right camera 24a and the left camera 24b is 144 mm.

[0020] The right microphone 26a and the left microphone 26b are provided on the head 21. The right microphone 26a and the left microphone 26b are arranged at a predetermined interval in the horizontal direction. The right microphone 26a and the left microphone 26b constitute a stereo microphone. By arranging them at a predetermined interval in the horizontal direction, the time when the sound reaches varies according to the position of the sound source. The difference in the arrival time of the sound appears as a phase difference between the sound signals generated by the right microphone 26a and the left microphone 26b. In order to increase the phase difference between the sound signals of the right microphone 26a and the left microphone 26b, it is preferable to arrange the right microphone 26a and the left microphone 26b as far apart as possible.

[0021] The control device 14 is an information processing device that determines the operation of the robot 13 and controls the operation of the robot 13. The detailed configuration of the control device 14 will be described later.

[0022] Figure 2 shows an example of the external shape of the HMD 100. In this example, the HMD 100 is composed of an output mechanism unit 110 and a wearing mechanism unit 112. The wearing mechanism unit 112 includes a wearing band 108 that wraps around the user's head to fix the HMD 100 to the head. The wearing band 108 is made of a material or has a structure that allows its length to be adjusted according to the user's head circumference.

[0023] The output mechanism unit 110 includes a housing 114 shaped to cover the left and right eyes when the user wears the HMD 100, and a display panel 102 is provided inside at a position facing the eyes. The display panel 102 may be a liquid crystal panel, an organic EL panel, or the like. Inside the housing 114, a pair of left and right optical lenses are provided, which are located between the display panel 102 and the user's eyes and expand the user's viewing angle.

[0024] The HMD 100 further includes earphones 104 that are inserted into the user's ears when worn. The earphones 104 are an example of audio output means, and the HMD 100 may include headphones. At this time, the HMD 100 and the headphones may be integrally configured or separate.

[0025] The HMD 100 transmits sensor information detected by the attitude sensor and audio data obtained by encoding the audio signal from the microphone 106 to the control device 14 via the processing device 128. Also, the HMD 100 receives image data and audio data generated by the control device 14 (robot 13) via the processing device 128 and outputs them from the display panel 102 and the earphones 104.

[0026] Note that the HMD 100 shown in FIG. 2 shows an immersive (non-transmissive) display device that completely covers both eyes, but it may also be a transmissive display device. Also, as for the shape, it may be a hat type as shown in the figure, or it may be a glasses type. Note that the HMD 100 may be composed not only of a dedicated head-mounted display device, but also of a terminal device having a display panel, a microphone, and a speaker, and a housing that fixes the display panel of the terminal device in front of the user's eyes. The terminal device may be, for example, a smartphone or a portable game machine, which has a relatively small display panel.

[0027] FIG. 3 is a block diagram showing the functional blocks of the HMD 100. Each block shown in the block diagrams in this specification can be realized, in terms of hardware, by elements and electronic circuits such as a computer processor, CPU, memory, and mechanical devices, and can be realized, in terms of software, by a computer program or the like. Here, however, the functional blocks realized by their cooperation are depicted. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by a combination of hardware and software.

[0028] The control unit 120 is a main processor that processes and outputs various signals and data such as image signals, audio signals, and sensor information, as well as instructions. The storage unit 122 temporarily stores data and instructions processed by the control unit 120. The attitude sensor 124 detects attitude information such as the rotation angle and inclination of the HMD 100 at a predetermined period. The attitude sensor 124 includes at least a three-axis acceleration sensor and a three-axis gyro sensor. Further, the attitude sensor 124 includes a sensor that detects the position (or its change) in the height direction of the HMD 100. The microphone 106 converts the user's voice into an electrical signal to generate an audio signal.

[0029] The communication control unit 126 transmits and receives signals and data to and from the control device 14 by wired or wireless communication via a network adapter or antenna. In the embodiment, the communication control unit 126 transmits and receives signals and data to and from the control device 14 via the processing device 128. The communication control unit 126 receives from the control unit 120 the attitude information detected by the attitude sensor 124 (e.g., the position in the height direction of the HMD 100 and / or the change in the position in the height direction), and the audio data obtained by encoding the audio signal from the microphone 106, and transmits them to the control device 14.

[0030] In addition, the communication control unit 126 receives the image data and audio data transmitted from the control device 14 and supplies them to the control unit 120. When the control unit 120 receives the image data and audio data transmitted from the control device 14, it supplies the image data to the display panel 102 for display and supplies the audio data to the earphone 104 for audio output.

[0031] FIG. 4 is a block diagram showing the functional blocks of the user terminal 11. The user terminal 11 includes a display unit 30, a camera 32, a character data storage unit 34, an operation reception unit 36, an App control unit 38, a display content generation unit 40, a display control unit 42, and a character data transmission unit 44.

[0032] The display unit 30 displays various information. The display unit 30 may include a liquid crystal panel or an organic EL panel. In addition, the display unit 30 includes the function of a touch panel (in other words, a touch screen). The camera 32 is an imaging device that images the state around the user terminal 11 (for example, an object existing around the user terminal 11).

[0033] The character data storage unit 34 stores data on a plurality of characters that can be selected by the user and that have different personalities defined (hereinafter also referred to as "character data"). FIG. 5 shows an example of character data. A curious personality is assigned to character A, an amiable personality is assigned to character B, a brave personality is assigned to character C, and a timid personality is assigned to character D. For each character, various attribute values (also referred to as "parameters") are predetermined so as to be consistent with the assigned personality.

[0034] The parameter (1) "path search - distance from an obstacle" determines the width of the distance that the robot 13 secures with respect to an obstacle (for example, a person or an object) detected on the path. For example, the value corresponding to "narrow" may be 20 cm, the value corresponding to "medium" may be 40 cm, and the value corresponding to "wide" may be 60 cm.

[0035] Parameter (2) "Path Search - Smoothness of Path" determines the curvature of the path along which robot 13 moves. For example, the value corresponding to "Straight line" may be a relatively small curvature value (e.g., curvature 0), the value corresponding to "Smooth" may be a relatively large curvature value, and the value corresponding to "Medium" may be a curvature value intermediate between "Straight line" and "Smooth". According to parameters (1) and (2), the path when robot 13 moves to the goal is determined.

[0036] Parameter (3) "Path Search - Detour (People)" determines the frequency of reacting to a person when robot 13 detects a person on the moving path. Reacting may mean stopping near the detection target, moving around the detection target, imaging the detection target with camera 24, or a combination of these. For example, the value corresponding to "None" may be a frequency of 0 (no detour), the value corresponding to "High" may be a frequency of 50%, and the value corresponding to "Medium" may be a frequency of 25%.

[0037] Parameter (4) "Path Search - Detour (Objects)" determines the frequency of reacting to an object other than a person when robot 13 detects an object other than a person on the moving path. Examples of reactions are the same as for people. For example, the value corresponding to "None" may be a frequency of 0 (no detour), the value corresponding to "High" may be a frequency of 50%, and the value corresponding to "Medium" may be a frequency of 25%. According to parameters (3) and (4), the moving path of robot 13 is updated. In other words, a new moving path including detours is determined.

[0038] Parameter (5) "Path Search - Goal Position (People)" determines the priority goal position at the time of path setting. In other words, it determines the distance from robot 13 to the person who is preferentially set as the goal position. For example, the value corresponding to "Near" may be a relatively short distance (e.g., less than 3 meters), the value corresponding to "Far" may be a relatively long distance (e.g., 6 meters or more), and the value corresponding to "Medium" may be an intermediate distance (e.g., 3 meters or more and less than 6 meters).

[0039] Parameter (6) "Speed - Change in Moving Speed (Acceleration)" determines the acceleration of the robot 13 during movement. For example, the value corresponding to "Large" may be a relatively large acceleration, the value corresponding to "Small" may be a relatively small acceleration, and the value corresponding to "Medium" may be an acceleration intermediate between "Large" and "Small". Also, the value corresponding to "Large (for Humans Only)" may be a relatively large acceleration when the goal is a human, and a relatively small acceleration when the goal is an object other than a human.

[0040] Parameter (7) "Speed - Normal Moving Speed" determines the speed (e.g., maximum speed) of the robot 13 during movement. For example, the value corresponding to "Fast" may be a relatively fast speed, the value corresponding to "Slow" may be a relatively slow speed, and the value corresponding to "Medium" may be a speed intermediate between "Fast" and "Slow".

[0041] Parameter (8) "Speed - Head Swing Speed" determines the speed of the robot 13's head swing (i.e., the rotational speed of the head drive unit 22). For example, the value corresponding to "Fast" may be a relatively fast speed, the value corresponding to "Slow" may be a relatively slow speed, and the value corresponding to "Medium" may be a speed intermediate between "Fast" and "Slow".

[0042] Parameter (9) "Target Detection Range - For Humans" determines the width of the range within which the robot 13 detects surrounding humans. For example, the value corresponding to "Wide" may be a relatively wide range (e.g., a range with a radius of 4 meters), and the value corresponding to "Narrow" may be a relatively narrow range (e.g., a range with a radius of 2 meters).

[0043] Parameter (10) "Target Detection Range - For Obstacles" determines the width of the range within which the robot 13 detects objects other than surrounding humans. For example, the value corresponding to "Wide" may be a relatively wide range (e.g., a range with a radius of 4 meters), and the value corresponding to "Narrow" may be a relatively narrow range (e.g., a range with a radius of 2 meters).

[0044] The parameter (11) "Target range - Person / Object Priority" determines the priority between detecting surrounding people and detecting objects other than surrounding people. For example, the value corresponding to "Person = Object" may be a value indicating that surrounding people and objects are detected with the same priority. The value corresponding to "Person > Object" may be a value indicating that surrounding people are detected preferentially over objects. Although not shown in FIG. 5, for a character assigned a surly personality, "Person < Object" may be set as the parameter (11). The value corresponding to "Person < Object" may be a value indicating that objects are detected preferentially over surrounding people.

[0045] The character data stored in the character data storage unit 34 further includes images of each character. In addition, the plurality of characters selectable by the user includes a character indicating the user himself / herself (user avatar) and a character indicating a friend (friend avatar). The character data regarding each of the character indicating the user himself / herself and the character indicating a friend includes, in addition to the image of each character, information necessary for remotely operating the robot 13. The information necessary for remotely operating the robot 13 may include, for example, the identification information of the processing device 128 (HMD100) which is the transmission destination of the images and sounds collected by the robot 13 and which is the transmission source of the operation information for the robot 13, and information necessary for communication with the processing device 128 (HMD100).

[0046] Returning to FIG. 4, in the embodiment, an application program (hereinafter "Entertainment App") in which a plurality of modules corresponding to the operation reception unit 36, the App control unit 38, the display content generation unit 40, the display control unit 42, and the character data transmission unit 44 are implemented is installed in the storage of the user terminal 11. The Entertainment App may be stored in a recording medium and installed in the user terminal 11 via the recording medium. Alternatively, the Entertainment App may be downloaded from a server via a network and installed in the user terminal 11. The CPU of the user terminal 11 reads the Entertainment App into the main memory and executes it to exhibit the functions of the above plurality of functional blocks.

[0047] The operation reception unit 36 receives the operations input by the user on the display unit 30 (touch panel). The operation reception unit 36 notifies the App control unit 38 of the information regarding the received operations. The App control unit 38 controls the operation of the entertainment App according to the user operations received by the operation reception unit 36.

[0048] The display content generation unit 40 generates the content (such as images) to be displayed on the display unit 30. The display control unit 42 causes the content (such as images) generated by the display content generation unit 40 to be displayed on the display unit 30. For example, the display content generation unit 40 generates content that allows the user to select one of a plurality of characters with different personalities (referred to as the "character selection screen" in the first embodiment). The display control unit 42 causes the character selection screen to be displayed on the display unit 30.

[0049] The character data transmission unit 44 reads out the data of the character selected on the character selection screen (hereinafter also referred to as "selected character data") from the character data storage unit 34 and transmits the selected character data to the control device 14.

[0050] FIG. 6 is a block diagram showing the functional blocks of the robot 13. The robot 13 includes an input system 50 that receives and processes external inputs, and an output system 52 that processes outputs to the outside. The input system 50 includes an operation command reception unit 54, a drive control unit 56, and an audio processing unit 58. The output system 52 includes an image processing unit 60, an audio processing unit 62, and a transmission unit 64.

[0051] The operation command reception unit 54 receives the operation commands transmitted from the control device 14. The operation commands include at least one of a drive command that instructs the operation modes of the traveling drive unit 20 and the neck drive unit 22, and audio data to be output from the speaker 28.

[0052] The drive control unit 56 controls the operations of the traveling drive unit 20 and the neck drive unit 22 according to the drive commands included in the operation commands. In other words, the drive control unit 56 operates the traveling drive unit 20 and the neck drive unit 22 in the manner (such as speed, distance, angle, etc.) indicated by the drive commands. The audio processing unit 58 executes audio processing based on the audio data included in the operation commands, and causes the audio based on the audio data to be output from the speaker 28.

[0053] The right camera 24a and the left camera 24b are directed in the direction controlled by the neck drive unit 22 and image the inside of their respective viewing angles. The right camera 24a and the left camera 24b may be arranged apart from each other at an interval corresponding to the average distance between the eyes of an adult. The right-eye image data captured by the right camera 24a and the left-eye image data captured by the left camera 24b may be provided to the HMD 100 and displayed on the right half and the left half of the display panel 102, respectively. These images form parallax images seen from the right eye and the left eye, and by displaying them in the regions formed by dividing the display panel 102 into two parts respectively, the images can be viewed stereoscopically. In addition, since the user views the display panel 102 through the optical lens, the image processing unit 60 may generate image data in which the optical distortion by the lens is corrected in advance and supply it to the HMD 100. When the right-eye image data and the left-eye image data are not particularly distinguished, they are collectively referred to as "imaging data".

[0054] The right microphone 26a and the left microphone 26b convert the sounds around the robot 13 into electrical signals to generate audio signals. The audio processing unit 62 generates audio data obtained by encoding the audio signals output from the right microphone 26a and the left microphone 26b. The transmission unit 64 transmits the imaging data supplied from the image processing unit 60 and the audio data supplied from the audio processing unit 62 to the control device 14.

[0055] FIG. 7 is a block diagram showing the functional blocks of the control device 14. The control device 14 includes a character data storage unit 70, a game storage unit 72, a character data reception unit 74, an imaging data reception unit 76, an object identification unit 78, an action determination unit 80, an action command transmission unit 82, a game generation unit 84, a game providing unit 86, an audio data reception unit 88, a robot surrounding data transmission unit 90, and a robot operation reception unit 92.

[0056] The character data storage unit 70 stores the data of the character selected by the user, that is, the selected character data (such as images and various parameters) transmitted from the user terminal 11. The game storage unit 72 stores the data of the game application (hereinafter also referred to as "game data") generated by the game generation unit 84 described later.

[0057] In the embodiment, an application program in which a plurality of modules corresponding to the character data reception unit 74, the imaging data reception unit 76, the object identification unit 78, the action determination unit 80, the action command transmission unit 82, the game generation unit 84, the game providing unit 86, the audio data reception unit 88, the robot surrounding data transmission unit 90, and the robot operation reception unit 92 are implemented is installed in the storage of the control device 14 via a recording medium or a network. The CPU of the control device 14 reads the above application program into the main memory and executes it to exert the functions of the above plurality of functional blocks.

[0058] The character data reception unit 74 receives the data of the character selected by the user, that is, the selected character data transmitted from the user terminal 11. The character data reception unit 74 stores the received selected character data in the game storage unit 72.

[0059] The imaging data reception unit 76 receives the imaging data transmitted from the robot 13. That is, the imaging data reception unit 76 acquires the image data in which the real-world objects imaged by the camera 24 of the robot 13 are reflected.

[0060] The object recognition unit 78 identifies what the object imaged by the camera 24 of the robot 13 is by a known method. In other words, the object recognition unit 78 identifies the type of the object reflected in the imaging data transmitted from the robot 13. For example, the object recognition unit 78 identifies whether the object reflected in the imaging data is a person or an object other than a person. The object recognition unit 78 may use template matching to identify the type of the object based on the shape, pattern, color, etc. of the object reflected in the imaging data.

[0061] The motion determination unit 80 determines the motion mode of the robot 13 based on the parameters corresponding to the personality of the character selected on the user terminal 11. Specifically, the motion determination unit 80 determines the motion mode of the robot 13 based on the image of the object reflected in the imaging data transmitted from the robot 13, the type of the object identified by the object recognition unit 78, and the character data stored in the character data storage unit 70. The motion mode of the robot 13 includes the motion mode of the traveling drive unit 20 (such as the moving direction, moving speed, etc.) and the motion mode of the neck drive unit 22 (such as the rotation angle, rotation speed, etc.).

[0062] The motion command transmission unit 82 transmits a motion command (drive command) indicating the motion mode of the robot 13 determined by the motion determination unit 80 to the robot 13, thereby operating the robot 13 in the mode determined by the motion determination unit 80.

[0063] The game generation unit 84 generates a game that displays an AR space (AR image) including both the real-world object imaged by the camera 24 of the robot 13, that is, the real-world object reflected in the imaging data transmitted from the robot 13, and the virtual object corresponding to the real-world object. The game generation unit 84 stores the generated game data in the game storage unit 72.

[0064] The game providing unit 86 provides the game data generated by the game generation unit 84 and stored in the game storage unit 72 to an external device. For example, the game generation unit 84 may generate a game that displays AR images for the right eye and the left eye with a parallax from each other and explores the AR space. The game providing unit 86 may cause the HMD 100 to display a 3D game that explores the AR space by providing the game data to the processing device 128.

[0065] The voice data receiving unit 88 receives the voice data transmitted from the robot 13. The robot surrounding data transmitting unit 90 transmits robot surrounding data including at least one of the imaging data acquired by the imaging data receiving unit 76 and the voice data acquired by the voice data receiving unit 88 to the processing device 128. Thereby, an image showing the surroundings of the robot 13 imaged by the robot 13 and the voice around the robot 13 collected by the robot 13 are reproduced on the HMD 100.

[0066] The robot operation receiving unit 92 receives a remote operation on the robot 13 by the user or a friend. Specifically, the robot operation receiving unit 92 receives the user operation information transmitted from the processing device 128a and also receives the friend operation information transmitted from the processing device 128b.

[0067] When the user operation information (or friend operation information) is received, the operation determination unit 80 generates an operation command including a drive command for causing the robot 13 to execute an operation according to the operation indicated by the user operation information (or friend operation information). Further, the operation determination unit 80 generates an operation command including the voice data included in the user operation information (or friend operation information).

[0068] The operation of the entertainment system 10 according to the first embodiment having the above configuration will be described. The user launches the entertainment App on the user terminal 11. The display content generation unit 40 of the user terminal 11 generates the data of the character selection screen, and the display control unit 42 causes the character selection screen to be displayed on the display unit 30.

[0069] FIG. 8 shows an example of the character selection screen. The character selection screen 131 is configured to switch and display a plurality of characters with different personalities (i.e., candidate characters for selection) according to a swipe operation or the like. In FIG. 8, as the plurality of characters, it includes character 130a (character B in FIG. 5), character 130b (character A in FIG. 5), character 130c (character C in FIG. 5), and character 130d (Takeo) indicating a friend. When not particularly distinguishing character 130a, character 130b, character 130c, and character 130d, they are referred to as "character 130".

[0070] The icon 132 indicates that the robot 13 operates autonomously or the robot 13 operates by remote control. Also, the icon 134 indicates that it is a character that automatically generates a game.

[0071] When a pressing operation on the installation button 136 is input on the character selection screen 131 where a specific character is displayed, the App control unit 38 of the user terminal 11 activates the camera 32. The display content generation unit 40 of the user terminal 11 determines an image showing the objects around the user terminal 11 captured by the camera 32 as the display content, and the display control unit 42 causes the image to be displayed on the display unit 30.

[0072] The user places the robot 13 within the angle of view of the camera 32. When the robot 13 is imaged by the camera 32, that is, when the robot 13 appears in the image captured by the camera 32, the display content generation unit 40 detects this by known object recognition processing (such as template matching). When the display content generation unit 40 detects that the robot 13 has been imaged by the camera 32, it generates an AR image that displays the image of the robot 13 together with the image of the character 130 selected by the user on the character selection screen 131. The display control unit 42 causes the display unit 30 to display the above AR image.

[0073] Subsequently, the display content generation unit 40 generates an AR image that includes content in which the character 130 selected by the user possesses the robot 13. FIG. 9 shows an example of the AR image. The AR image 138 may be a video of the character 130 entering the robot 13 from the outside of the robot 13. Also, the AR image 138 may be an image that further displays an object associating the robot 13 and the character 130. By thus displaying an AR image that includes content in which the character 130 selected by the user possesses the robot 13, it can be made easily understandable to the user that hereafter, the robot 13 will perform operations according to the personality of the character 130.

[0074] In synchronization with the display of the AR image 138 (that is, the display of the content in which the character 130 selected by the user possesses the robot 13), the character data transmission unit 44 of the user terminal 11 transmits character data regarding the character 130 selected by the user to the control device 14.

[0075] The character data receiving unit 74 of the control device 14 receives the character data transmitted from the user terminal 11 and stores it in the character data storage unit 70. The operation determination unit 80 of the control device 14 determines the activation of the camera 24 of the robot 13, and the operation command transmission unit 82 transmits an operation command instructing the activation of the camera 24 to the robot 13. The robot 13 activates the camera 24, and the transmission unit 64 of the robot 13 transmits the image data (captured data) captured by the camera 24 to the control device 14.

[0076] The captured data receiving unit 76 of the control device 14 receives the captured data transmitted from the robot 13. The object identification unit 78 of the control device 14 identifies the type (for example, human or object) of the object that appears in the image of the captured data and exists around the robot 13. The operation determination unit 80 of the control device 14 determines the operation mode of the robot 13 based on the attributes (for example, position, shape, pattern, color, type) of the objects existing around the robot 13 and the parameters corresponding to the personality of the character selected by the user (that is, the character data stored in the character data storage unit 70).

[0077] A first example of the determination process of the operation mode of the robot 13 will be described. The operation determination unit 80 determines the mode regarding the path search of the robot 13 based on the parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 determines the goal position (for example, a position close to a person) according to the parameter (5) "Path search - Goal position (person)" in FIG. 5. Also, the operation determination unit 80 determines the movement mode (movement path) to the goal according to the parameters (1) "Path search - Distance from obstacles", (2) "Path search - Smoothness of the path", (3) "Path search - Detour (person)", and (4) "Path search - Detour (object)" in FIG. 5.

[0078] A second example regarding the determination process of the operation mode of the robot 13 will be described. The operation determination unit 80 determines an aspect regarding the operation speed of the robot 13 based on parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 determines the moving speed of the robot 13 (for example, the rotational speed of the traveling drive unit 20) according to the parameter (7) "Speed - Normal Moving Speed" in FIG. 5. Further, the operation determination unit 80 determines the acceleration (for example, the acceleration of the rotation of the traveling drive unit 20) until reaching the above moving speed according to the parameter (6) "Speed - Change in Moving Speed (Acceleration)" in FIG. 5. Further, the operation determination unit 80 determines the rotational speed of the neck drive unit 22 according to the parameter (8) "Speed - Neck Swing Speed" in FIG. 5.

[0079] A third example regarding the determination process of the operation mode of the robot 13 will be described. The operation determination unit 80 determines an aspect regarding the operation of determining the target that the robot 13 pays attention to from among the objects identified by the object identification unit 78 based on parameters predetermined as the personality of the character selected by the user terminal 11. For example, the operation determination unit 80 sets a person within the range defined by the parameter (9) "Attention Target Range - Person to Person" in FIG. 5 as a candidate for the attention target. Further, the operation determination unit 80 sets an object within the range defined by the parameter (10) "Attention Target Range - Obstacle" in FIG. 5 as a candidate for the attention target. Further, the operation determination unit 80 determines the attention target from among the candidate people and objects of the attention target according to the parameter (11) "Attention Target Range - Person / Object Priority" in FIG. 5.

[0080] When the operation determination unit 80 determines the attention target, it may determine to cause the robot 13 to perform an attention operation (for example, imaging by the camera 24) on the person or object that is the attention target.

[0081] The operation determination unit 80 generates an operation command for instructing the determined operation mode. The operation command transmission unit 82 of the control device 14 transmits the operation command generated by the operation determination unit 80 to the robot 13. The drive control unit 56 of the robot 13 controls the operations of the traveling drive unit 20 and the neck drive unit 22 according to the operation command. As a result of the above processing, the robot 13 autonomously explores the surroundings in a manner corresponding to the personality (each parameter of the character data) of the character selected by the user.

[0082] As described above, according to the entertainment system 10 of the first embodiment, the robot 13 can be made to execute an operation corresponding to the personality of the character desired by the user from among a plurality of characters having different personalities, and the user experience by the robot 13 (entertainment robot) can be improved.

[0083] Note that when a second character (a character different from the first character) is newly selected by the user after the first character is selected by the user, the control device 14 controls the operation of the robot 13 based on the character data of the second character instead of controlling the operation of the robot 13 based on the character data of the first character. That is, the control device 14 dynamically switches the parameters for controlling the operation of the robot 13 in response to a change in the character selection by the user. Thereby, the user can arbitrarily switch the operation mode of the single robot 13.

[0084] Next, the operation when the character 130b (character A) to which the icon 134 is assigned on the character selection screen 131 in FIG. 8 is selected will be described. In this case, the robot 13 autonomously explores the surroundings in a manner corresponding to the personality assigned to character A, and images the surrounding objects with the camera 24 at a predetermined cycle. The transmission unit 64 of the robot 13 transmits the imaging data by the camera 24 to the control device 14 at a predetermined cycle.

[0085] The imaging data receiving unit 76 of the control device 14 receives the imaging data transmitted from the robot 13, and the object identification unit 78 identifies the objects shown in the imaging data, that is, the types of real-world objects imaged by the camera 24 of the robot 13. The game generation unit 84 generates a game that displays an AR space including real-world objects (hereinafter also referred to as "real objects") imaged by the camera 24 of the robot 13 and virtual objects (hereinafter also referred to as "virtual objects") corresponding to the real-world objects.

[0086] FIG. 10 shows an example of a game image generated by the game generation unit 84. In the game image 140, as real objects 142, a scooter, a shelf, a block, and a ball are arranged. These are objects that actually exist in the room where the robot 13 is placed. Further, the game image 140 includes, as virtual objects 144, a friend character, a windmill, a fence, clouds, and a character 130 (a character selected by the user). This game may be, for example, a game in which the character 130 explores an AR space in which real objects 142 and virtual objects 144 are mixed.

[0087] The game generation unit 84 stores data indicating the correspondence relationship between any combination of the shape, pattern, color, type, etc. of the real object 142 shown in the imaging data and the virtual object 144. The data indicating the correspondence relationship may be, for example, (1) associating the scooter as the real object 142 with the friend character as the virtual object 144, (2) associating the shelf as the real object 142 with the windmill as the virtual object 144, and (3) associating the block as the real object 142 with the fence as the virtual object 144. The game generation unit 84 arranges the virtual object 144 corresponding to the real object 142 at a position near the real object 142.

[0088] The game generation unit 84 stores the generated game data in the game storage unit 72. The game providing unit 86 transmits the game data stored in the game storage unit 72 to the requesting external device in response to a request from the external device. According to the entertainment system 10 of the first embodiment, it is possible to provide the user with a game that displays an AR space in which virtual objects corresponding to the real objects are arranged together with the real objects imaged from the perspective of the robot 13 (the position of the camera 24), and to provide a novel user experience using the robot 13.

[0089] Next, the operation when the character 130d (friend avatar) is selected on the character selection screen 131 in FIG. 8 will be described. In this case, the operation determination unit 80 determines the operation mode of the robot 13 based on the friend operation information transmitted from the friend's device.

[0090] Specifically, when the character 130d (friend avatar) is selected on the character selection screen 131, data necessary for communication with the processing device 128b is stored in the character data storage unit 70 of the control device 14 as data of the character. The control device 14 (the robot operation reception unit 92, the robot surrounding data transmission unit 90) establishes a connection with the processing device 128b using the data stored in the character data storage unit 70.

[0091] The imaging data reception unit 76 of the control device 14 receives imaging data (right-eye image and left-eye image) transmitted from the robot 13 at a predetermined cycle. The audio data reception unit 88 receives audio data transmitted from the robot 13 at a predetermined cycle. The robot surrounding data transmission unit 90 transmits the imaging data and audio data transmitted from the robot 13 to the processing device 128b.

[0092] The processing device 128b transfers the captured image data and audio data transmitted from the control device 14 to the HMD 100b. The HMD 100b displays the captured image data (right-eye image and left-eye image) transmitted from the processing device 128b on the display panel 102 (right-eye display panel and left-eye display panel). The friend can view the situation around the robot 13 in real time and can also hear the sounds around the robot 13 in real time.

[0093] The friend inputs an operation on the robot 13 (an operation for instructing the driving mode of the traveling drive unit 20 and the driving mode of the neck drive unit 22) to the processing device 128b. The processing device 128b transmits friend operation information indicating the operation on the robot 13 input by the friend to the control device 14. The friend operation information includes audio data indicating the voice uttered by the friend collected by the microphone 106 of the HMD 100b.

[0094] The robot operation reception unit 92 of the control device 14 receives the friend operation information transmitted from the processing device 128b. The operation determination unit 80 generates an operation command for instructing the operation content (driving mode of the traveling drive unit 20 and driving mode of the neck drive unit 22) indicated by the friend operation information. The operation command transmission unit 82 transmits the operation command to the robot 13. Note that the friend operation information may include the movement of the head of the friend wearing the HMD 100b, and the operation determination unit 80 may determine the driving mode of the neck drive unit 22 of the robot 13 so as to match the movement of the head of the friend.

[0095] As a result of the above processing, the operation on the robot 13 input by the friend is reflected in the operation of the robot 13 in real time. Also, the operation determination unit 80 generates an operation command including the audio data included in the friend operation information. The operation command transmission unit 82 transmits the operation command to the robot 13. Thereby, the voice uttered by the friend is output from the speaker 28 of the robot 13 in real time.

[0096] As described above, according to the entertainment system 10 of the first embodiment, the robot 13 operates in response to remote operation by a friend and also plays back the friend's voice. On the other hand, the friend can view the images and sounds around the robot 13 in real time through the HMD 100b. As a result, people around the robot 13 and the friend can communicate in real time.

[0097] Although not shown in FIG. 8, the characters selectable on the character selection screen 131 may include a character (user avatar) representing the user himself / herself. When the user avatar is selected on the character selection screen 131, the operation determination unit 80 determines the operation mode of the robot 13 based on the user operation information transmitted from the user's device. The specific processing when the user avatar is selected on the character selection screen 131 is the same as the processing when a friend avatar is selected on the character selection screen 131, and is the processing in which the HMD 100b is replaced with the HMD 100a and the processing device 128b is replaced with the processing device 128a.

[0098] <Second Embodiment> Hereinafter, the second embodiment will be described focusing on the configurations different from those of the first embodiment, and the description of the common configurations will be omitted as appropriate. It goes without saying that the configuration of the second embodiment can be combined arbitrarily with the configurations of the first embodiment and the modified example.

[0099] The outline of the second embodiment will be described. In recent years, a technology called telepresence has emerged in which a robot placed in a remote location is used as one's own avatar. By having the robot in the remote location transmit surrounding image data and audio data to the user and play it back on the user side, the user can experience a sense of presence as if they were at the location of the robot. In addition, head-mounted displays (HMDs) are being used in various fields. By providing the HMD with a head tracking function and updating the display screen in conjunction with the posture of the user's head, the immersion in the video world can be enhanced.

[0100] In the second embodiment, an AR image that shows a real-world object reflected in an image captured by a camera of a robot and a virtual object arranged by a user is displayed on an HMD worn by the user. When the user stands up or the like, the position of the HMD in the height direction can change significantly. The height direction is, for example, the direction perpendicular to the floor surface on which the robot or the user moves. On the other hand, the robot is usually configured such that the position of the camera in the height direction does not change beyond a predetermined range. Therefore, when the position of the HMD in the height direction changes, it may give a sense of discomfort to the user viewing the AR image and cause so-called VR sickness to the user.

[0101] Therefore, in the entertainment system of the second embodiment, when the position of the HMD in the height direction changes, an image in which the real-world object is erased and that shows the appearance of a virtual object as seen from a new viewpoint corresponding to the change in the position of the HMD in the height direction (so-called VR image) is generated, and the display is switched from the display of the AR image until then to the display of the VR image. That is, in the entertainment system of the second embodiment, while the video see-through of the HMD is turned on as the default mode, when the position of the HMD in the height direction changes, the video see-through of the HMD is switched off.

[0102] The configuration of the entertainment system 10 of the second embodiment is the same as the configuration of the entertainment system 10 of the first embodiment shown in FIG. 1. As described above, the height (eye height) of the mounting position of the camera 24 on the robot 13 is 144 mm. The robot 13 is configured such that the position of the camera 24 in the height direction does not change beyond a predetermined range. For example, even when the neck drive unit 22 of the robot 13 rotates about the pitch axis or the roll axis, the change in the eye height is configured to be several centimeters. Note that the robot 13 may be configured such that the position of the camera 24 in the height direction does not change.

[0103] FIG. 11 is a block diagram showing the functional blocks of the control device 14 of the second embodiment. The control device 14 of the second embodiment includes, as functional blocks similar to those of the control device 14 of the first embodiment, a character data storage unit 70, a character data reception unit 74, an imaging data reception unit 76, an object identification unit 78, a motion determination unit 80, a motion command transmission unit 82, an audio data reception unit 88, a robot surrounding data transmission unit 90, and a robot operation reception unit 92. The control device 14 of the second embodiment further includes a game operation reception unit 150, a game image generation unit 152, a game image transmission unit 154, a sensor information reception unit 156, and a motion detection unit 158.

[0104] An application program in which a plurality of modules corresponding to the character data reception unit 74, the imaging data reception unit 76, the object identification unit 78, the motion determination unit 80, the motion command transmission unit 82, the audio data reception unit 88, the robot surrounding data transmission unit 90, the robot operation reception unit 92, the game operation reception unit 150, the game image generation unit 152, the game image transmission unit 154, the sensor information reception unit 156, and the motion detection unit 158 are implemented may be installed in the control device 14 via a recording medium or a network. The CPU of the control device 14 may exhibit the functions of the plurality of functional blocks by reading the application program into the main memory and executing it.

[0105] The game operation reception unit 150 receives, from the processing device 128a, user operation information indicating a user's operation on the game input to the processing device 128a.

[0106] The game image generation unit 152 generates a game image based on the imaging data captured by the robot 13 acquired by the imaging data reception unit 76 and the user operation information acquired by the game operation reception unit 150. The game image includes a right-eye game image and a left-eye game image having a parallax from each other.

[0107] In the second embodiment, the game image transmission unit 154 of the control device 14, the processing device 128a, and the control unit 120 of the HMD 100a function as a display control unit that causes the game image to be displayed on the display panel 102 of the HMD 100a by cooperating with each other. Specifically, the game image transmission unit 154 transmits the data of the game images (the right-eye game image and the left-eye game image) generated by the game image generation unit 152 to the processing device 128a. The processing device 128a passes the game image to the HMD 100a, and the HMD 100a displays the right-eye game image on the right-eye display panel and the left-eye game image on the left-eye display panel. Thereby, the user can view the content depicted in the game image stereoscopically.

[0108] The sensor information reception unit 156 acquires the attitude information detected by the attitude sensor 124 of the HMD 100a. The motion detection unit 158 uses a known head tracking technique to detect the attitude of the HMD 100a worn on the user's head based on the attitude information acquired by the sensor information reception unit 156.

[0109] In the second embodiment, the motion detection unit 158 detects the change (change amount) in the position of the HMD 100a in the height direction. The motion detection unit 158 inputs the change amount of the position of the HMD 100a in the height direction to the game image generation unit 152. The game image generation unit 152 generates a game image taking into account the change amount of the position of the HMD 100a in the height direction.

[0110] The operation of the entertainment system 10 of the second embodiment having the above configuration will be described. Here, it is assumed that a user avatar is selected from among a plurality of characters, the user remotely operates the robot 13, and the video and audio around the robot 13 are reproduced on the HMD 100a. The user inputs an operation for instructing a switch to the game mode to the processing device 128a, and the processing device 128a transmits user operation information indicating the operation to the control device 14.

[0111] When the game image generation unit 152 of the control device 14 receives user operation information instructing a switch to a game mode, it generates a first game image in which virtual objects (virtual objects) are reflected together with real-world objects (real objects) reflected in the imaging data acquired by the imaging data reception unit 76 (that is, the image captured by the camera 24 of the robot 13). The first game image is the default game image and is a game image with the video see-through of the HMD 100a enabled.

[0112] FIG. 12 shows an example of the first game image. The first game image 160 in the figure includes a sofa 162, a door 164, a table 166, and a cup 168 as real objects 142. The first game image 160 also includes a plurality of blocks 170 and a plurality of enemy characters 172 as virtual objects 144. The first game image is an AR image in which the virtual object 144 is added to the real-world space imaged by the camera 24 of the robot 13.

[0113] The user inputs an operation to place a virtual object (block 170 or enemy character 172) in the real-world space imaged by the robot 13 to the processing device 128a, and the processing device 128a transmits user operation information indicating the above operation to the control device 14. The game operation reception unit 150 of the control device 14 receives the user operation information indicating the above operation.

[0114] The game image generation unit 152 of the control device 14 places the virtual object 144 in the space reflected in the imaging data acquired by the imaging data reception unit 76 according to the user operation information indicating the above operation. The game image generation unit 152 generates a first game image showing the appearance of the virtual object 144 with the position of the camera 24 of the robot 13 as the viewpoint, in other words, generates a first game image showing the appearance of the virtual object 144 as seen from the eye level of the robot 13. The game image transmission unit 154 of the control device 14 transmits the first game image to the processing device 128a to display the first game image on the HMD 100a.

[0115] The sensor information receiving unit 156 of the control device 14 acquires the attitude information of the HMD 100a detected by the attitude sensor 124 of the HMD 100a. The movement detection unit 158 of the control device 14 detects the amount of change when the position of the HMD 100a in the height direction changes based on the attitude information of the HMD 100a.

[0116] When the position of the HMD 100a in the height direction changes beyond a predetermined range, the game image generation unit 152 of the control device 14 generates a second game image instead of the first game image. The game image transmission unit 154 transmits the second game image to the processing device 128a, thereby causing the HMD 100a to display the second game image instead of the first game image. The above-mentioned predetermined range, which is the threshold for switching from the first game image to the second game image, may be a range in which the position in the height direction (i.e., the eye height) of the camera 24 in the robot 13 can change. Alternatively, the above-mentioned predetermined range may be an appropriate range determined based on the developer's knowledge or experiments using the entertainment system 10.

[0117] FIG. 13 shows an example of the second game image. The second game image 161 in the figure includes a plurality of blocks 170 and a plurality of enemy characters 172 as virtual objects 144. Further, unlike the first game image 160, the second game image 161 does not include real objects 142 (such as the cup 168). That is, the second game image is an image in which real-world objects are erased, and is a VR image showing the appearance of the virtual object 144 as seen from a new viewpoint corresponding to the change in the position of the HMD 100a in the height direction. Further, the second game image is a game image in which the video see-through of the HMD 100a is disabled.

[0118] The second game image 161 in FIG. 13 is typically displayed when the user who was viewing the first game image 160 in a sitting posture stands up, and accordingly, the position of the HMD 100a in the height direction changes from bottom to top. When the position of the HMD 100a in the height direction changes from bottom to top, the game image generation unit 152 of the control device 14 generates a second game image in which real-world objects are erased and which shows the appearance of the virtual object 144 visible from a viewpoint above the viewpoint in the first game image. The second game image 161 in FIG. 13 can also be said to be an overhead image. Note that the upper viewpoint may be the viewpoint of the standing-up user, in other words, it may be the eye height of the user.

[0119] The first game image 160 can be said to be an image based on the viewpoint of the robot 13, while the second game image 161 can be said to be an image based on the viewpoint of the user. When generating the first game image 160, the game image generation unit 152 may arrange a virtual camera at the center of the coordinate system and align the direction of the optical axis of the virtual camera with the optical axis direction of the camera 24 of the robot 13. The game image generation unit 152 may arrange the image of the real object 142 captured by the camera 24 of the robot 13 and the virtual object 144 arranged by the user operation in the above coordinate system and then capture them with the virtual camera, and generate the first game image 160 based on the imaging result.

[0120] On the other hand, when generating the second game image 161, the game image generation unit 152 may change the position of the virtual camera to a position higher than before according to the amount of change in the position of the HMD 100a in the height direction. The game image generation unit 152 may capture the relatively lower-position real object 142 and the virtual object 144 from a relatively upper-position virtual camera, and generate the second game image 161 based on the imaging result.

[0121] According to the entertainment system 10 of the second embodiment, even when the position of the HMD 100 in the height direction changes, it is possible to suppress the user who views the game image from feeling discomfort and to suppress the so-called VR sickness from occurring in the user. Thereby, the user experience by the cooperation of the tele distance and the HMD 100 can be improved. Note that the technology described in the second embodiment is not limited to the case where the camera 24 is mounted on the robot 13. It is widely useful when the image (video) captured by the camera 24 is displayed on the HMD 100 and there is a limit to the change in the position of the camera 24 in the height direction (or the position of the camera 24 in the height direction does not change).

[0122] As described above, the present invention has been described based on the first embodiment and the second embodiment. These embodiments are examples, and it is understood by those skilled in the art that various modifications are possible for the combination of each of these components and each processing process, and such modifications are also within the scope of the present invention.

[0123] A first modification will be described. In the first embodiment, the data (image, various parameters, etc.) of the character selected by the user is transmitted from the user terminal 11 to the control device 14. As a modification, the character data storage unit 70 of the control device 14 may store a plurality of character data regarding a plurality of characters that can be selected by the user, similar to the character data storage unit 70 of the user terminal 11 in the first embodiment. The user terminal 11 may notify the control device 14 of the identification information (name, ID, etc.) of the character selected by the user. The operation determination unit 80 of the control device 14 may refer to the character data of the character specified by the identification information transmitted from the user terminal 11 among the plurality of character data stored in the character data storage unit 70, and determine the operation mode of the robot 13.

[0124] A second modification example will be described. In the second embodiment, the first game image and the second game image are displayed on the user's HMD 100a. However, the same configuration can also be realized when these game images are displayed on the friend's HMD 100b. In this case, the "processing device 128a" described in the second embodiment may be replaced with the "processing device 128b", and the "user operation information" described in the second embodiment may be replaced with the "friend operation information".

[0125] A third modification example will be described. Although not mentioned in the second embodiment, the game image generation unit 152 of the control device 14 may generate, instead of the first game image, a second game image in which the real object is erased and which shows the appearance of the virtual object visible from a new viewpoint corresponding to the change in the height direction position of the HMD 100 even when the height direction position of the HMD 100 changes from top to bottom. The second game image in this modification example may show the appearance of the virtual object visible from a viewpoint lower than the viewpoint in the first game image.

[0126] A fourth modification example will be described. The entertainment system 10 of each embodiment may be configured not to include the control device 14. In this case, the functions of the control device 14 described in the first embodiment and the second embodiment (for example, the operation determination unit 80, etc.) may be implemented in the robot 13 or may be implemented in the user terminal 11 (entertainment App). Alternatively, a part of the functions of the control device 14 described in the first embodiment and the second embodiment may be implemented in the robot 13, and the remaining functions of the control device 14 may be implemented in the user terminal 11 (entertainment App). In this modification example, the user terminal 11 and the robot 13 may be configured to communicate via the communication network 15.

[0127] For example, the robot 13 of the fourth modification example may include a character data reception unit (corresponding to the character data reception unit 74 of the control device 14 in the embodiment) and an operation determination unit (corresponding to the operation determination unit 80 of the control device 14 in the embodiment). The character data reception unit may receive information about a character selected by the user from among a plurality of characters with different personalities from the user terminal 11. The operation determination unit may determine the operation mode of the present robot 13 based on parameters corresponding to the personality of the character selected by the user.

[0128] A fifth modification example will be described. In the above embodiment, an image captured by the camera 24 installed at the viewpoint position of the robot 13 (referred to herein as a "robot viewpoint image") is displayed on the HMD 100 (HMD 100a, HMD 100b). In the entertainment system 10 of the modification example, the robot 13 or the control device 14 may transmit the robot viewpoint image to the terminal of the user or a friend (such as a smartphone, for example, the user terminal 11) and display the robot viewpoint image on the terminal of the user or a friend. In this case, the terminal of the user or a friend (such as a smartphone, for example, the user terminal 11) may transmit user operation information or friend operation information to the control device 14, or may transmit it directly to the robot 13. In this way, the robot 13 may be controlled using the terminal of the user or a friend.

[0129] A sixth modification will be described. The character data stored in the character data storage unit 34 of the user terminal 11 may include the following parameters as the characteristics of the character. (1) A parameter for controlling whether to preferentially find toys such as balls (in other words, whether to be sensitive to toys such as balls). In the character data of a character with a personality who likes toys, a value indicating preferentially reacting to toys such as balls based on image recognition technology may be set as the parameter value. (2) A parameter for controlling whether to react sensitively to surrounding sounds (increasing the microphone sensitivity more than usual) and start dancing in accordance with the detected sound. In the character data of a character with a personality who likes dancing, a value indicating increasing the microphone sensitivity more than usual may be set as the parameter value.

[0130] (3) A parameter for controlling whether to make the voice louder than usual. In the character data of a character with the personality of having a loud voice, a value indicating making the voice output intensity from the speaker 28 louder than usual may be set as the parameter value. In the entertainment system 10 of this modification, the operation determination unit 80 of the control device 14 may determine the operation mode related to sound (that is, the operation mode of the microphone 26 and the speaker 28 of the robot 13) based on the parameters predetermined as the characteristics of the character selected in the user terminal 11.

[0131] Any combination of the above-described embodiments and modifications is also useful as an embodiment of the present invention. The new embodiments generated by the combination have the effects of the combined embodiments and modifications. Also, it should be understood by those skilled in the art that the functions to be performed by each constituent element described in the claims are realized by a single one of the constituent elements shown in the embodiments and modifications or by their cooperation.

Industrial Applicability

[0132] The present invention can be applied to an information processing apparatus.

Explanation of Reference Numerals

[0133] 10 Entertainment system, 11 User terminal, 13 Robot, 14 Control device, 24 Camera, 30 Display unit, 32 Camera, 42 Display control unit, 76 Imaging data reception unit, 78 Object identification unit, 80 Motion determination unit, 84 Game generation unit, 92 Robot operation reception unit, 152 Game image generation unit, 154 Game image transmission unit, 158 Motion detection unit.

Claims

1. An acquisition unit that acquires an image showing a real-world object imaged by an imaging device; A first image that shows a virtual object together with the real-world object shown in the image acquired by the acquisition unit, and that shows the appearance of the virtual object with the position of the imaging device as the viewpoint, and an image generation unit that generates the first image; A display control unit that causes the head-mounted display to display the first image generated by the image generation unit; A detection unit that detects a change in the position of the head-mounted display in the height direction; Comprising; When the position of the head-mounted display in the height direction changes beyond a predetermined range, The image generation unit generates a second image in which the real-world object is erased, and that shows the appearance of the virtual object as seen from a new viewpoint corresponding to the change in the position of the head-mounted display in the height direction; The display control unit causes the head-mounted display to display the second image instead of the first image; An information processing device.

2. When the position of the head-mounted display in the height direction changes upward beyond the predetermined range from below, the image generation unit generates a second image in which the real-world object is erased, and that shows the appearance of the virtual object as seen from a viewpoint above the viewpoint in the first image; The display control unit causes the head-mounted display to display, instead of the first image, a second image in which the real-world object is erased, and that shows the appearance of the virtual object as seen from a viewpoint above the viewpoint in the first image; The information processing device according to claim 1.

3. The imaging device is attached to a robot, The robot is configured such that the position of the imaging device in the height direction does not change beyond a predetermined range; The information processing device according to claim 1 or 2.

4. A step of acquiring an image showing a real-world object imaged by an imaging device; A step of generating a first image that shows a virtual object together with the real-world object shown in the image acquired in the acquiring step, and that shows the appearance of the virtual object with the position of the imaging device as the viewpoint; A step of causing the head-mounted display to display the first image generated by the generating function; A step of detecting a change in the position of the head-mounted display in the height direction; When the position of the head-mounted display in the height direction changes beyond a predetermined range, generating a second image in which the real-world object is erased and showing the appearance of the virtual object as seen from a new viewpoint corresponding to the change in the position of the head-mounted display in the height direction; A step of causing the head-mounted display to display the second image instead of the first image; An information processing method executed by a computer.

5. A function of acquiring an image showing a real-world object imaged by an imaging device; A function of generating a first image showing a virtual object together with the real-world object imaged in the image acquired by the acquiring function and showing the appearance of the virtual object with the position of the imaging device as the viewpoint; A function of causing the head-mounted display to display the first image generated by the generating function; A function of detecting a change in the position of the head-mounted display in the height direction; Implemented on a computer, When the position of the head-mounted display in the height direction changes beyond a predetermined range, The generating function generates a second image in which the real-world object is erased and showing the appearance of the virtual object as seen from a new viewpoint corresponding to the change in the position of the head-mounted display in the height direction; The displaying function causes the head-mounted display to display the second image instead of the first image; A computer program.

Images