Head-mounted display device

Abstract

To provide a head-mounted display device (HMD) with a non-transmissive display, which enables a user to use an information processing apparatus as desired even when he / she experiences contents with the device.SOLUTION: A head-mounted display device includes a non-transmissive display for displaying a VR image, a camera, and a control unit. When a user experiences VR contents, the control unit analyzes a captured image generated by the camera (step S106), and makes the display show the VR image over an image of an information processing apparatus (step S109) when the user is determined to hold the information processing apparatus from the captured image (step S107:Yes).SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a head mounted display device (HMD) for experiencing virtual reality (VR) content and the like, which allows the user to start using an information processing device or the like while experiencing the content. [Background technology]

[0002] VR technology is known, which allows users to experience an artificially created virtual reality space as if it were real space. VR technology is applied to, for example, flight simulators, tourist guides, and games in which many users participate via a network and build a virtual world. Users wearing an HMD experience VR content by viewing VR content images (also referred to as VR images) displayed on the HMD's display. VR images are supplied to the HMD from a server or other device and updated according to the user's movements. This type of VR content is known as immersive content.

[0003] For example, Patent Document 1 discloses that an HMD is linked to an information processing device such as a smartphone, allowing a user to recognize the background in front of them through a semi-transparent display provided in the HMD. Specifically, Patent Document 1 discloses that when a call arrives on the smartphone, an incoming call icon is displayed on the semi-transparent display, and the user can operate the smartphone through the semi-transparent display without removing the HMD. At this time, the smartphone's device information is displayed on the smartphone screen as a two-dimensional code or the like, and the HMD captures and reads the device information displayed on the smartphone with a camera. If the read device information matches the device information registered in the HMD, the smartphone's authentication procedure is reduced, improving usability. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6300705 Summary of the Invention [Problem to be solved by the invention]

[0005] However, some HMDs are equipped with non-transparent displays. When using an HMD with a non-transparent display to experience or view immersive content such as VR content, the user cannot see the information processing device such as a smartphone. As a result, for example, when the user wants to view information on the information processing device or receives an incoming call, the user cannot perform necessary operations on the information processing device in a timely manner.

[0006] Therefore, an object of the present invention is to allow a user to use an information processing device whenever he or she wishes, even when experiencing content through an HMD equipped with a non-transmissive display. [Means for solving the problem]

[0007] In order to solve the above problem, for example, the configuration described in the claims is adopted. The present application includes a plurality of means for solving the above problem, and as an example, a head-mounted display device includes a non-transparent display that displays a VR image, a camera, and a control device, and the control device analyzes a captured image generated by the camera while the user is experiencing VR content, and when it determines from the captured image that the user is holding an information processing device, causes the display to display the VR image and an image of the information processing device in a superimposed manner. [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a head-mounted display device that allows a user to use an information processing device whenever he or she wants to, even when experiencing VR content. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a configuration diagram showing an example of a VR experience system including a head-mounted display device according to a first embodiment of the present invention. [Figure 2] 1 is an external view showing an example of a head-mounted display device according to a first embodiment of the present invention. [Figure 3A] 1 is a block diagram showing an example of a head-mounted display device according to a first embodiment of the present invention. [Figure 3B] FIG. 4 is a block diagram showing another example of the head-mounted display device according to the first embodiment of the present invention. [Figure 4] FIG. 1 is a block diagram illustrating an example of a configuration of an information processing device. [Figure 5] FIG. 10 is a flow diagram showing an example of processing when a user experiencing VR content uses an information processing device. [Figure 6] FIG. 10 is a diagram showing a first example of a display image of a head-mounted display device in which a VR image and an image of an information processing device are superimposed. [Figure 7] FIG. 10 is a diagram showing a second example of a display image of the head-mounted display device in which a VR image and an image of the information processing device are superimposed. [Figure 8] FIG. 10 is a diagram showing a third example of a display image of a head-mounted display device in which a VR image and an image of an information processing device are superimposed. [Figure 9] FIG. 10 is an operational sequence diagram showing an example of processing when an incoming call is received according to the second embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing an example of a display image when a call arrives. [Figure 11] FIG. 11 is an operational sequence diagram showing an example of processing at the time of an incoming call according to the third embodiment of the present invention. [Figure 12A] 10 is a diagram illustrating an example of a display image when the information processing device 3 is within the imaging range of the camera. FIG. [Figure 12B] 10 is a diagram illustrating an example of a display image when the information processing device 3 is within the imaging range of the camera. FIG. [Figure 12C]It is a diagram illustrating a display image when the information processing apparatus 3 exists within the imaging range of the camera. [Figure 13A] It is a diagram illustrating a display image when the information processing apparatus according to Embodiment 2 of the present invention does not exist within the imaging range of the camera. [Figure 13B] It is a diagram illustrating a display image when the information processing apparatus according to Embodiment 2 of the present invention does not exist within the imaging range of the camera. [Figure 14] It is a flowchart showing an example of a method for detecting a user's cooperation instruction according to Embodiment 4 of the present invention. [Figure 15] It is a flowchart showing another example of a method for detecting a user's cooperation instruction according to Embodiment 4 of the present invention. [Figure 16] It is a flowchart showing an example of cooperation processing by a VR somatosensory application according to Embodiment 4 of the present invention. [Figure 17] It is a diagram showing an example of an object used during VR content somatosensation according to Embodiment 5 of the present invention. [Figure 18A] It is a diagram showing another example of an object used during VR content somatosensation according to Embodiment 5 of the present invention. [Figure 18B] It is a block diagram showing an example of a communication module.

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all the drawings for explaining the embodiments, the same members are generally denoted by the same reference numerals, and repeated explanations thereof are omitted.

[0011] (Embodiment 1) Embodiment 1 will be described using FIGS. 1 to 8.

[0012] <VR Somatosensory System Configuration> Fig. 1 is a configuration diagram showing an example of a VR experience system including a head-mounted display device according to embodiment 1 of the present invention. The VR experience system in Fig. 1 includes a head-mounted display device 2 worn by a user 1, an information processing device 3, a network 6, an access point 5 to the network 6, a VR service server 7, etc.

[0013] In response to access from the user 1 or the head-mounted display device 2, the VR service server 7 transmits VR content (including VR images and VR audio) to the head-mounted display device 2 via the network 6 and the access point 5. The user 1 wearing the head-mounted display device 2 experiences the VR content that the head-mounted display device 2 has received from the VR service server 7 and downloaded.

[0014] There may be multiple VR service servers 7. The VR service server 7 may be any server that includes a storage for storing VR content and a communication device for connecting to the head-mounted display device 2. The VR service server 7 may be, for example, a personal computer.

[0015] The head mounted display device 2 and the information processing device 3 transmit and receive various information to and from a VR service server 7 via an access point 5 and a network 6.

[0016] Specifically, the head mounted display device 2 transmits various pieces of information to the VR service server 7 by transmitting the various pieces of information to the access point 5 via a network signal 4a. The head mounted display device 2 receives various pieces of information from the VR service server 7 via a network signal 4c transmitted from the access point 5. The network signals 4a, 4b, and 4c are, for example, Wi-Fi (registered trademark) signals.

[0017] The information processing device 3 is, for example, a smartphone, a tablet terminal, a wearable terminal, etc., but is not limited to these. The connection between the information processing device 3 and the head-mounted display device 2 may be performed by transmitting and receiving various information using the above-mentioned network signals 4a and 4b, or by transmitting and receiving information using short-range communication 8. The short-range communication 8 is, for example, Bluetooth (registered trademark).

[0018] Fig. 2 is an external view showing an example of a head-mounted display device according to Embodiment 1 of the present invention, Fig. 3A is a block diagram showing an example of a head-mounted display device according to Embodiment 1 of the present invention.

[0019] 2, the head-mounted display device 2 includes cameras 20a and 20b, displays 22a and 22b, speakers 23a and 23b, a microphone 24, proximity communication receiving devices (receiving devices) 25a, 25b, and 25c, a sensor group 210, and a control device 220. These elements are connected to one another via an internal bus 200.

[0020] The head mounted display device 2 also includes holders 26a and 26b. The user 1 uses the holders 26a and 26b to wear the head mounted display device 2. The head mounted display device 2 is fixed to the head by holder 26a and fixed above the nose by holder 26b.

[0021] The cameras 20a and 20b are attached so as to capture images in front of the head-mounted display device 2, i.e., in the line of sight of the user 1. In FIG. 2, the camera 20a is placed at a position corresponding to the left eye of the user 1, and the camera 20b is placed at a position corresponding to the right eye of the user 1. Here, two cameras are provided, but three or more cameras may be provided. The cameras 20a and 20b capture images in the line of sight of the user 1 and output the captured image data to the control device 220 via the internal bus 200.

[0022] 3A, the sensor group 210 includes, for example, a direction sensor 211, a gyro sensor 212, an acceleration sensor 213, a distance sensor 214, and a position sensor (not shown). Based on the sensing results of these sensors, the position of the head mounted display device 2 (i.e., the user 1), the posture of the user 1 (for example, the tilt of the head), the line of sight of the user 1, movement (change) of the line of sight of the user 1, etc. are detected. Each sensor outputs the sensing result to the control device 220 via the internal bus 200.

[0023] The distance sensor 114 is a sensor that measures the distance from the head mounted display device 2 (i.e., the user 1) to the information processing device 3, which is an object. Any sensor that can estimate the position of an object in three dimensions can be used as the distance sensor 214. Examples of the distance sensor 214 include three-dimensional distance sensors such as LiDAR (Light Detection And Ranging) and TOF (Time of Flight) sensors. Note that, as shown in FIGS. 2 and 3A, if the head mounted display device 2 is provided with a proximity communication receiving device for distance measurement, the distance sensor 214 may not be necessary.

[0024] The displays 22a and 22b display content such as VR content (VR images) and guidance content (guidance images) for the head-mounted display device 2. The displays 22a and 22b are non-transmissive and are configured, for example, with a display panel (e.g., a curved panel or a flat panel) such as a liquid crystal or organic electro-luminescence (EL) panel, and lenses. The display 22a is a display corresponding to the left eye of the user 1, and the display 22b is a display corresponding to the right eye of the user 1. The displays 22a and 22b receive display image data (e.g., VR image data, guidance image data, etc.) output from the control device 220 via the internal bus 200 and display VR content, etc. corresponding to each eye of the user 1, respectively.

[0025] The speakers 23a and 23b receive output audio data output from the control device 220 via the internal bus 200, and output various sounds such as VR content (VR audio), guidance content (guidance audio) for the head-mounted display device 2, and operation sounds based on the output audio data. The speaker 23a is a speaker corresponding to the left ear of the user 1, and the speaker 23b is a speaker corresponding to the right ear of the user 1. The speakers 23a and 23b output sounds corresponding to the respective ears of the user 1.

[0026] The microphone 24 acquires sounds such as sounds emitted by the user and environmental sounds, and outputs the acquired sounds to the control device 220 via the internal bus 200 as input sound data.

[0027] The proximity communication receiving devices 25a, 25b, and 25c are receiving devices that receive proximity communication signals (position detection signals) transmitted from the information processing device 3. The proximity communication receiving devices 25a, 25b, and 25c output the received proximity communication signals to the control device 220 via the internal bus 200. The proximity communication signals are used for information and data communication with other devices such as the information processing device 3, and for measuring the distance from the head-mounted display device 2 (i.e., the user 1) to the information processing device 3.

[0028] The control device 220 is a functional block that controls each element included in the head mounted display device 2, performs image analysis processing, voice recognition processing, etc. As shown in Fig. 3A, the control device 220 includes a communication device 221, a computer 222, a memory 223, an image memory 224, a storage device 225, etc. The elements within the control device 220 are connected to each other via an internal bus 200.

[0029] The communication device 221 covers multiple communication methods, such as Wi-Fi (registered trademark), fourth generation (4G), and fifth generation (5G) mobile communications, and selects an appropriate communication method to connect to the network 6 and the information processing device 3. The communication device 221 may also have a proximity communication function. In this case, the communication device 221 may have a proximity communication receiver for receiving data separate from the proximity communication receivers 25a to 25c for measuring distance, or the proximity communication receivers 25a to 25c may be used for receiving data. The communication device 221 may also have a configuration that allows it to be connected to the VR service server 7 in a proprietary manner.

[0030] The computer 222 is configured with a processor such as a CPU (Central Processing Unit). The computer 222 reads and executes various programs stored in the memory 223, and realizes functional blocks that perform various processes on the computer 222. The computer 222 also accesses the memory 223 and the image memory 224 to write and read programs and data. The computer 222 may include a graphics processor that mainly performs image processing.

[0031] The memory 223 is a volatile memory such as a RAM (Random Access Memory). The memory 223 temporarily stores various programs that are read from the storage device 225 and deployed. The memory 223 also stores various data such as parameters used in the various programs, and calculation results in the computer 222. The memory 223 outputs and stores various information such as program data and calculation results based on instructions from the computer 222 or a functional block implemented on the computer 222, for example.

[0032] The image memory 224 is a volatile memory such as a RAM, and mainly temporarily stores various data related to image processing (for example, display image data, captured image data, etc.). The image memory 224 also mainly outputs and stores various data related to image processing, for example, based on instructions from the computer 222 or a functional block implemented on the computer 222. In FIG. 3A, the image memory 224 is provided separately from the memory 223, but the image memory 234 may be provided within the memory 223. In this case, the memory 233 temporarily stores various data related to image processing.

[0033] The control device 220 measures (calculates) the distance from the head mounted display device 2 to the information processing device 3 using the proximity communication signals received by the proximity communication receiving devices 25a, 25b, and 25c. The control device 220 estimates the distance between the head mounted display device 2 and the information processing device 3, for example, by performing processing such as detecting the phase difference between the proximity communication receiving devices 25a, 25b, and 25c based on the proximity communication signals input from the proximity communication receiving devices 25a, 25b, and 25c. Furthermore, the control device 220 can detect the three-dimensional position of the information processing device 3 using the proximity communication signals received by the proximity communication receiving devices 25a, 25b, and 25c.

[0034] By providing three or more proximity communication receiving devices, the control device 220 can estimate the three-dimensional position of the target object. Note that if the distance to the information processing device 3 is not to be estimated, the number of proximity communication receiving devices 25a to 25c does not need to be multiple.

[0035] The control device 220 outputs output audio data to the speakers 23a and 23b. The control device 220 also performs analysis processing of images captured by the cameras 20a and 20b and recognition processing of audio acquired by the microphone 24. The control device 220 also performs processing of network communication, short-range communication, etc.

[0036] The storage device 225 includes a nonvolatile memory such as a flash memory. The storage device 225 includes various storage areas, such as a storage area 225a that stores a basic operation program for performing basic control of the head mounted display device 2, a storage area 225b that stores a linkage processing program that links the head mounted display device 2 with the information processing device 3, a storage area 225c that stores an image analysis program, and a storage area 225d that stores a voice recognition program. The programs stored in the various storage areas are expanded in the memory 223 and executed by the computer 222. As a result, each functional block is realized on the computer 222.

[0037] The content images displayed on the displays 22a and 22b are mainly VR images, but in this embodiment, an image of the information processing device 3 cut out from an image captured by a camera may be superimposed on the VR image. The image data of these images is stored in, for example, the image memory 224, and is read from the image memory 224 in response to an instruction from the control device 220 and output to the display 22.

[0038] <<Other configuration examples of head-mounted display devices>> Fig. 3B is a block diagram showing another example of the head-mounted display device according to embodiment 1 of the present invention. In Fig. 3B, the same elements as in Fig. 3A are given the same reference numerals, and descriptions of parts that overlap with Fig. 3A will be omitted. Fig. 3B differs from Fig. 3A in that a storage area 225e for storing VR content is provided in the storage device 225.

[0039] 3B, the communication device 221 downloads VR content (images, audio) from the VR service server 7 via the network 6 under the control of the computer 222. The computer 222 stores the downloaded VR content in the storage area 225e of the storage device 225.

[0040] The computer 222 reads out the VR content from the storage area 225e as needed, outputs the VR image to the displays 22a and 22b, and outputs the VR sound to the speakers 23a and 23b, thereby allowing the user 1 to experience the VR content. Then, the computer 222 performs update processing of the VR image and the VR sound according to the movement of the user 1.

[0041] <Information processing device> Next, an information processing device will be described. Fig. 4 is a block diagram showing an example of the configuration of an information processing device. In Fig. 4, the configuration of the information processing device 3 will be described using a smartphone as an example. The information processing device 3 shown in Fig. 4 includes a camera 30, an input-integrated display 31, a sensor group 32, a microphone 33, a speaker 34, a communication device 35, a computer 36, a memory 37, an image memory 38, and a storage device 39.

[0042] The camera 30 is provided, for example, on the surface opposite to the input-integrated display 31. The camera 30 may also be provided on the side of the input-integrated display 31. The camera 30 outputs captured image data to the computer 36, the storage device 39, etc. via the internal bus 300.

[0043] The input-integrated display 31 is a display with an input function that is based on a so-called touch operation. The input-integrated display 31 displays, for example, an operation screen, an incoming call screen, a captured image, etc. of the information processing device 3. The information processing device 3 is operated by a touch operation via the input-integrated display 31.

[0044] The sensor group 32 includes, for example, a direction sensor, a gyro sensor, an acceleration sensor, a position sensor (all of which are not shown), etc. Based on the sensing results of these sensors, the position and attitude of the information processing device 3 are detected. Each sensor outputs the sensing result to the computer 36 via the internal bus 300.

[0045] The microphone 33 acquires sounds such as sounds emitted by the user and environmental sounds, and outputs the acquired sounds to the computer 36 via the internal bus 300 as input sound data.

[0046] The speaker 34 receives output audio data output from the computer 36 via the internal bus 300, and outputs various sounds based on the output audio data, such as call audio, application audio, guidance audio of the information processing device 3, operation sounds, etc.

[0047] The communication device 35 connects to, for example, a fourth-generation (4G) or fifth-generation (5G) mobile communication network. The communication device 35 also connects to the network 6 and the head-mounted display device 2 via, for example, Wi-Fi (registered trademark). The communication device 35 also has a proximity communication function and transmits a proximity communication signal to the head-mounted display device 2.

[0048] The computer 36 is configured with a processor such as a CPU. The computer 36 reads and executes various programs stored in the memory 37, and realizes functional blocks that perform various processes on the computer 36. The computer 36 also accesses the memory 37 and the image memory 38 to write and read programs and data.

[0049] The memory 37 is a volatile memory such as a RAM. The memory 37 temporarily stores various programs read from the storage device 39 and deployed therein. The memory 37 also stores various data such as parameters used in the various programs, and calculation results in the computer 36. The memory 37 outputs and stores various information such as program data and calculation results based on instructions from the computer 36 or functional blocks implemented on the computer 36, for example.

[0050] Image memory 38 is a volatile memory such as a RAM, and mainly temporarily stores various data related to image processing (for example, display image data, captured image data, etc.). Image memory 38 also mainly outputs and stores various data related to image processing, for example, based on instructions from computer 36 or a functional block implemented on computer 36. As shown in FIG. 4, image memory 38 may be provided separately from memory 37, or image memory 38 may be provided within memory 37. In this case, various data related to image processing is temporarily stored in memory 37.

[0051] Storage device 39 includes a nonvolatile memory such as a flash memory, etc. As shown in Fig. 4, storage device 39 includes, for example, a storage area 391 for storing a basic operation program, a storage area 392 for storing a call application, and a storage area 393 for storing other applications such as an SNS application.

[0052] <Processing flow when starting to use an information processing device> The head-mounted display device 2 of the present embodiment allows the user 1 experiencing VR content to use the information processing device 3 by superimposing an image of the information processing device on a VR content image.

[0053] 5 is a flow diagram showing an example of a cooperative process by the VR experience application according to the first embodiment of the present invention. FIG. 5 shows an example of a process when a user experiencing VR content uses an information processing device. When the VR experience application is launched and processing begins (step S100), the control device 220 transmits a network signal 4a from the communication device 221 to access the VR service server 7 (step S101). At this time, the network signal 4a includes various information such as address information of the VR service server 7 on the network and VR content designation information that designates VR content to be downloaded. When accessed by the head-mounted display device 2 (control device 220), the VR service server 7 transmits predetermined VR content designated by the VR content designation information to the head-mounted display device 2.

[0054] When the control device 220 receives the VR content, it stores the received VR content in, for example, the memory 223 or the image memory 224. If the storage device 225 is provided with a storage area 225e (FIG. 3B) for storing VR content, the control device 220 may store the received VR content in the storage area 225e. In this case, the VR content may be downloaded in advance and stored in the storage area 225e, and predetermined VR content may be read from the storage area 225e when starting to use the head-mounted display device 2. In this case, the processing of step S101 is performed only when the necessary VR content has not been stored.

[0055] Next, in step S102, predetermined VR content is output to provide the VR content to the user 1. The control device 225 outputs VR initial image data (VR image data) at the start of the VR content to the displays 22a and 22b, and causes the displays 22a and 22b to display VR initial images (VR images). The control device 225 also outputs VR audio data to the speakers 23a and 23b, and causes the speakers 23a and 23b to output VR audio associated with the VR image.

[0056] Next, in step S103, the control device 220 continues to acquire movement information of the head mounted display device 2, i.e., movement information of the user 1, based on the sensing information of each sensor of the sensor group 210. Then, in step S104, the control device 220 generates VR updated image data (VR image data) based on the movement information acquired in step S103.

[0057] Then, in step S105, the control device 220 outputs the VR updated image data generated in step S104 to the displays 22a and 22b, and displays VR updated images (VR images) that match the user 1's field of view, line of sight, head and body tilt, etc. on the displays 22a and 22b, respectively.

[0058] Next, in step S106, the captured image data input from the cameras 20a and 20b is analyzed (image analysis). The control device 220 analyzes the captured image data and extracts the information processing device 3. The control device 220 may also extract the hand of the user 1 from the captured image data. Note that the capturing of images by the cameras 20a and 20b and the analysis of the captured image data may be performed asynchronously.

[0059] In step S107, based on the analysis result of the captured image data in step S106, it is determined whether or not the user 1 is holding the information processing device 3. Specifically, in step S107, it is determined whether or not the user 1 has moved the information processing device 3 in front of the camera 20a or 20b. If it is difficult to detect the hand of the user 1 in the image analysis, the control device 220 may determine whether or not the user 1 has moved the information processing device 3 in front of a camera based on whether or not the information processing device 3 is captured in the captured image.

[0060] In addition, the control device 220 may detect that the user 1 has moved the information processing device 3 by recognizing from the captured image that the information processing device 3 is displaying an image that is to be displayed when vibration is detected by a vibration sensor or the like.

[0061] Alternatively, the control device 220 may determine whether the user 1 has moved the information processing device 3 in front of the camera based on a combination of the position of the information processing device 3 in the captured image and the gaze direction of the user 1 extracted based on the sensing result. In this case, the control device 220 may determine that the user 1 has moved the information processing device 3 in front of the camera if the gaze of the user 1 is directed toward the information processing device 3, or may determine that the user 1 has not moved the information processing device 3 in front of the camera if the gaze of the user 1 is not directed toward the information processing device 3.

[0062] As described above, if it is not possible to determine that the user 1 wishes to control the information processing device 3, for example, if the information processing device 3 is left on a desk, even if the information processing device 3 is extracted in the captured image, it will not be displayed on the head-mounted display device 2, and therefore the user 1 will not be prevented from viewing the content.

[0063] If the user 1 is holding the information processing device 3, that is, if it is determined that the user 1 has moved the information processing device 3 in front of the camera (Yes), the control device 220 executes the process of step S108. In step S108, the control device 220 cuts out an image of the information processing device 3 from the captured image and generates image data for the information processing device 3. Furthermore, if the control device 220 is able to acquire the display screen data of the information processing device 3 by requesting the information processing device 3 to transmit display screen data, the control device 220 transforms the display screen data to fit the display screen of the image data of the information processing device 3 and superimposes it on the image data of the information processing device 3. Note that, to acquire the display screen data of the information processing device 3, it is assumed that device registration and authentication have been performed in advance between the information processing device 3 and the head-mounted display device 2, and mutual authentication is performed via close proximity wireless communication or the like.

[0064] In step S109, the control device 220 superimposes the image data of the information processing device 3 generated in step S108 on the VR image. Specifically, the control device 220 generates composite image data by synthesizing the image data of the information processing device 3 generated in step S108 with the VR update image data (VR image data). Then, the control device 220 outputs the generated composite image data to the displays 22a and 22b, and displays the composite image in which the VR image and the image of the information processing device 3 are superimposed on the displays 22a and 22b, respectively. This allows the user to view the information processing device 3 and start using it while experiencing the VR content.

[0065] In step S110, it is determined whether the user 1 has issued an instruction to pause the progress of the VR content. If the user 1 has issued an instruction to pause the progress of the VR content (Yes), the control device 220 proceeds to step S111 and pauses the progress of the VR content. This allows the user 1 to concentrate on using the information processing device 3. Then, when the user 1 issues an instruction to resume the progress of the VR content, the progress of the VR content is resumed, and the process of step S112 is executed.

[0066] On the other hand, if the user 1 has not issued an instruction to pause the progress of the VR content in step S110 (No), the user 1 uses the information processing device 3 while the VR content continues to progress.

[0067] In step S112, the control device 220 determines whether to end the VR experience application that executes the VR content. The determination of whether to end the VR experience application may be based on whether or not an end instruction has been issued by the user 1, or may be determined based on whether or not the VR content has ended.

[0068] If the VR experience application is to be continued (No), the process returns to step S103, and steps S103 to S112 are executed again. On the other hand, if the VR experience application is to be ended (Yes), the control device 220 ends the VR experience application (step S113).

[0069] On the other hand, in step S107, if the user 1 is not holding the information processing device 3, that is, if the control device 220 determines that the user 1 has not moved the information processing device 3 in front of the camera (No), the process proceeds to step S112. The subsequent processes are as already described.

[0070] Incidentally, there may be cases where the user 1 does not issue an instruction to resume the progress of the VR content in step S111. For this reason, between steps S111 and S112, a step in which the control device 220 superimposes a guidance image for resuming the progress of the VR content on the VR image and the image of the information processing device 3 may be provided. Alternatively, the control device 220 may count the time since the user 1 stopped the progress of the VR content, and when a predetermined time has elapsed, the progress of the VR content may be forcibly resumed. Alternatively, the control device 220 may count the time since the user 1 stopped the progress of the VR content, and when a predetermined time has elapsed, the control device 220 may proceed to step S113 and terminate the VR experience application.

[0071] <<Example of display image>> Here, a specific example of a display image displayed on the head mounted display device 2 will be described. Fig. 6 is a diagram showing a first example of a display image on the head mounted display device in which a VR image and an image of an information processing device are superimposed. Fig. 6 shows a VR image 50, imaging ranges 51 of cameras 20a and 20b, and an image 52 of the information processing device 3. Note that the imaging range 51 is shown for the purpose of explanation, but is not displayed on the head mounted display device 2.

[0072] The control device 220 recognizes the information processing device 3 within the imaging range 51 by analyzing the captured image data, and cuts out an image 52 of the information processing device 3 (which may include the user's hand holding it) from the captured image. Then, the control device 220 superimposes the cut-out image 52 of the information processing device 3 on the VR image. Note that the present invention is also applicable to a transparent head-mounted display device, and in the case of a transparent head-mounted display device 2, the control device 220 may make the VR image of the area corresponding to the image 52 of the information processing device 3 transparent, so that the VR image is not displayed in the transparent area.

[0073] Furthermore, the control device 220 may not display the hand of the user 1 holding the information processing device 3, or may replace the hand of the user 1 with another image, such as a generated graphic image of a hand. The user 1 can confirm that the information processing device 3 has been recognized by the head-mounted display device 2, and can perform various operations, such as authentication, while viewing the image 52 of the information processing device 3. Note that when the user 1 operates the information processing device 3, they may operate it with the fingers on the opposite side of the hand holding the information processing device 3. Therefore, by displaying an image of the opposite hand and fingers in addition to the hand holding the information processing device 3, it is expected that the operation will be easier.

[0074] Fig. 7 is a diagram showing a second example of a display image of a head-mounted display device in which a VR image and an image of an information processing device are superimposed. In Fig. 7, an overlay image 53 of the information processing device 3 in which an image 52 of the information processing device 3 in Fig. 6 and a display image 53a of the information processing device 3 are superimposed is displayed. At this time, the image of the hand of the user 1 is displayed taking into consideration its positional relationship with the information processing device 3.

[0075] The control device 220 acquires display image data relating to a display image from the information processing device 3, for example, via network communication or near-field communication. Here, it is assumed that the head mounted display device 2 and the information processing device 3 have been authenticated and communication between the two devices has been established. When the head mounted display device 2 requests the information processing device 3 to transmit display screen data, the information processing device 3 transmits the display image data to the head mounted display device 2.

[0076] The control device 220 generates composite image data by superimposing the VR image data, the image data of the information processing device 3, and the received display image data. Then, the control device 220 outputs the generated composite image data to the displays 22a and 22b, and causes the displays 22a and 22b to display a composite image by superimposing the VR image 50, the image 52 of the information processing device 3, and the display image 53a. Here, the composite image by superimposing the image 52 of the information processing device 3 and the display image 53a is the overlay image 53 of the information processing device 3.

[0077] The head mounted display device 2 acquires the latest display image of the information processing device 3 by acquiring the display image of the information processing device 3 while recognizing the information processing device 3 within the captured image.

[0078] 7, instead of the display image captured by the camera, the display image 53a generated from the display image data of the information processing device 3 is used. This makes it possible to improve the definition of the display image in the composite image.

[0079] In addition, if it is difficult to recognize the hand of user 1, the control device 220 may determine that, for example, when the information processing device 3 is lifted, a lock image of the information processing device 3 is displayed and user 1 is holding the information processing device 3 and performing a lock screen display operation, and recognize that user 1 has picked up the information processing device 3.

[0080] FIG. 8 is a diagram showing a third example of a display image of a head-mounted display device in which a VR image and an image of an information processing device are superimposed. In FIG. 8, in order to make the display image of the information processing device 3 easier to see, an enlarged display image 53b of a display image 53a of the information processing device 3 is displayed near the image 52 of the information processing device 3 (on the right side in FIG. 8). The display position of the enlarged display image 53b is not limited to the right side of the image 52. The size of the enlarged display image 53b is arbitrary and can be changed to a size that is easy for the user 1 to see. The display position and size may be controlled by the user 1 moving the information processing device 3 left and right, up and down, and by moving the information processing device 3 forward and backward to change the display position and size. In addition, FIG. 8 shows an example in which an operation pointer 54 is superimposed on the information processing device 3 in correspondence with the operating finger of the user 1.

[0081] <Other> In this embodiment, for example, when user 1 lowers his / her hand and the information processing device 3 moves out of the camera's imaging range, the image (52, 53, 53b) of the information processing device 3 may be erased from the composite image, or the image (52, 53, 53b) of the information processing device 3 may continue to be displayed until user 1 issues an instruction to erase it.

[0082] Furthermore, for example, if the head-mounted display device 2 and the information processing device 3 have mutually approved each other and the head-mounted display device 2 can authenticate the information processing device 3, even if the information processing device 3 cannot be confirmed in the image captured by the camera, the control device 220 may determine that the user 1 is holding the information processing device 3 in his / her hand and performing a lock screen display operation, for example, by lifting the information processing device 3, and may request the information processing device 3 to send display image data.

[0083] <Major Effects of This Embodiment> According to this embodiment, even when the user 1 is experiencing immersive content such as VR content, the user 1 can pick up and use the external information processing device 3 without removing the head-mounted display device 2.

[0084] Furthermore, according to the present embodiment, the display image 53a of the information processing device 3 can be superimposed on the VR image, thereby making it possible to improve the definition of the display image.

[0085] Furthermore, according to the present embodiment, the enlarged display image 53b of the information processing device 3 can be superimposed on the VR image, thereby making it possible to improve the visibility of the displayed image.

[0086] The above has been explained using an example of a function that is thought to be particularly effective when the user 1 is experiencing VR content, but in an HMD equipped with a non-transmissive display, as long as the surroundings cannot be seen when wearing the HMD, the present invention can achieve the same effect regardless of the content, for example, even when a menu is displayed or nothing is displayed. In the case of a transmissive display, the present invention is effective when content that significantly obstructs the field of view is displayed.

[0087] (Embodiment 2) Next, a description will be given of embodiment 2. In this embodiment, a process will be described in the case where an incoming call is received by the information processing device 3 while the user 1 is experiencing VR content.

[0088] Fig. 9 is an operation sequence diagram showing an example of processing at the time of an incoming call according to the second embodiment of the present invention. Fig. 9 shows an interrupt sequence in which an incoming call is received by the information processing device 3 and the user 1 is notified to use the information processing device 3. Specifically, Fig. 9 shows an operation sequence diagram in which an incoming call is received by the information processing device 3 while the user 1 is experiencing VR content, and the information processing device 3 notifies the head-mounted display device 2 of the incoming call, which is used as a trigger to cause the user 1 to start using the information processing device 3.

[0089] 9 shows the relationship between the head-mounted display device 2, the information processing device 3, and the VR service server 7. When the user 1 is experiencing VR content, the head-mounted display device 2 executes a cooperative process using a VR experience application 227.

[0090] The head mounted display device 2 (control device 220) transmits movement data of the head mounted display device 2 detected based on, for example, a sensing result to the VR service server 7 (step S207). Upon receiving the movement data from the head mounted display device 2, the VR service server 7 transmits image data of the VR image that is updated in accordance with the movement to the head mounted display device 2 (step S201).

[0091] When receiving the update VR image data from the VR service server 7, the head mounted display device 2 displays an update VR image based on the update VR image data on the displays 22a and 22b (step S202). Steps S200 to S202 are repeatedly executed.

[0092] When the information processing device 3 receives an incoming call, the information processing device 3 activates a call application and enters an incoming call status (step S207). Then, the information processing device 3 transmits, as an interrupt request, a request to display an incoming call icon indicating that there is an incoming call, to the head mounted display device 2 (step S208).

[0093] In step S203, upon receiving the display request for the incoming call icon, the head mounted display device 2 reads image data of the incoming call icon stored in, for example, the memory 223 or the image memory 224, and generates composite image data by combining the VR image data for update with the image data of the incoming call icon. Then, the head mounted display device 2 displays, on the displays 22a and 22b, a composite image in which the incoming call icon is superimposed on the VR image based on the composite image data.

[0094] Fig. 10 is a diagram showing an example of a display image when a call is received. Fig. 10 shows a composite image in which a VR image 50 and an incoming call icon 55 are superimposed. As shown in Fig. 10, the incoming call icon 55 is superimposed on a partial area of the VR image 50. The display location of the incoming call icon 55 is not limited to the example of Fig. 10.

[0095] When the user 1 recognizes the incoming call icon 55, the user 1 moves the information processing device 3 in front of the head-mounted display device 2 (S204). The incoming call icon 55 is recognized, for example, by detecting the movement of the user 1's fingers through analysis of the captured image and comparing the detected finger movement with a predetermined movement pattern. Alternatively, the incoming call icon 55 may be recognized by acquiring the user 1's speech with the microphone 24 and comparing the user 1's speech with a predetermined speech pattern.

[0096] Then, the head mounted display device 2 analyzes the captured image data to extract the information processing device 3, and superimposes the image of the information processing device 3 on the VR image, for example, as shown in Figures 6 to 8 (step S205). At this time, processing such as pausing the progress of the VR content may be performed (step S206).

[0097] Then, the user 1 operates the information processing device 3 to make a call (step S209). When the call ends, the information processing device 3 transmits a request to delete the incoming call icon to the head mounted display device 2 (step S210). Note that step S211 may be performed at any timing after the call starts.

[0098] When the head mounted display device 2 receives the request to delete the incoming call icon, it ends the synthesis of the update VR image data with the image data of the incoming call icon, and deletes the incoming call icon 55 displayed on the displays 22a and 22b (step S211).

[0099] It is also possible to set an icon display time in advance, and have the incoming call icon 55 automatically disappear when the icon display time has elapsed since it was displayed. In this case, step S210 can be omitted. In this case, step S211 is executed at any timing after step S204, depending on the icon display time.

[0100] Then, the head mounted display device 2 cancels the pause of the VR content, executes steps S200 to S202, and resumes the progress of the VR content.

[0101] In addition to the call application, the present embodiment can also be applied to an SNS application, an email application, etc. In this case, when an incoming call or email is received in the SNS application, the email application, etc., the information processing device 3 transmits a request to display an incoming call icon or a received email icon.

[0102] The user 1 can pick up the information processing device 3, check the received content of an SNS application, an email application, or the like, and reply to the received content.

[0103] According to the present embodiment, in addition to the effect of the first embodiment, it becomes possible to accept an interrupt from the information processing device 3 even when the user 1 is experiencing VR content.

[0104] (Embodiment 3) Next, a third embodiment will be described. In this embodiment, it is assumed that the user 1 does not know the location of the information processing device 3. In this case, even if the user 1 recognizes that there is an incoming call from the incoming call icon 55, there may be cases where the user 1 is unable to operate the information processing device 3 immediately. Therefore, in this embodiment, a method for making the user 1 recognize the location of the information processing device 3 when there is an incoming call will be described.

[0105] Fig. 11 is an operational sequence diagram showing an example of processing when an incoming call is received according to the third embodiment of the present invention. Fig. 11 is similar to Fig. 9, but differs from Fig. 9 in that step S312 is added between step S204 and step S205.

[0106] When the user 1 recognizes the incoming call icon 55 in step S204, step S312 is executed. In step S312, the head-mounted display device 2 generates a composite image by further superimposing the captured images from the cameras 20a and 20b on a composite image in which the VR image and the incoming call icon 55 are superimposed, and displays the composite image with the captured images superimposed on the displays 22a and 22b. When the composite image with the captured images superimposed is displayed on the displays 22a and 22b, the user 1 searches for the information processing device 3 while looking at the captured image superimposed on the VR image. When the user 1 recognizes the location of the information processing device 3, he or she grabs the information processing device 3 with his or her hand and performs an operation such as unlocking.

[0107] When the user 1 recognizes the position of the information processing device 3 and grasps the information processing device 3 with his / her hand, step S205 is executed, and a composite image in which the VR image and the image of the information processing device 3 are superimposed is displayed on the displays 22a and 22b.

[0108] <Example of display image> Here, a specific example of a display image displayed on the head mounted display device 2 in this embodiment will be described.

[0109] <<When the information processing device is within the imaging range>> First, a case where the information processing device is present within the captured image of the camera will be described. Figures 12A, 12B, and 12C are diagrams illustrating display images when the information processing device 3 is present within the captured image range of the camera. Note that an incoming call icon 55 is also superimposed on the display images of Figures 12A, 12B, and 12C.

[0110] First, in the example of Fig. 12A, the captured image of the camera is not processed, and an image in which the VR image 50 and the unprocessed captured image 351A are superimposed is displayed on the displays 22a and 22b. According to the example of Fig. 12A, it is possible to search for the information processing device 3 without preventing a decrease in visibility of the VR image 50.

[0111] 12B, an image in which the VR image 50 and the transparent image 351B are superimposed is displayed on the displays 22a and 22b. The transparent image 351B is generated by performing a transparency process on the captured image to make transparent the area other than the information processing device 3. The transparent image 351B includes the image 52 of the information processing device 3.

[0112] In the example of Fig. 12C, an image in which the VR image 50 and the line drawing processed image 351C are superimposed is displayed on the displays 22a and 22b. The line drawing processed image 351C is an image obtained by performing line drawing processing on the captured image. Edge detection is performed on the line drawing processed image 351C to detect edge processing devices, etc. In Fig. 12C, the process of line drawing processing is indicated by dashed lines.

[0113] According to the examples of FIGS. 12B and 12C, it is possible to search for the information processing device 3 without interfering with the visibility of the VR image 50.

[0114] <<When the information processing device is not within the imaging range>> Next, a case where the information processing device 3 is not within the imaging range of the camera will be described. Figures 13A and 13B are diagrams illustrating examples of display images when the information processing device according to Embodiment 2 of the present invention is not within the imaging range of the camera.

[0115] When the information processing device 3 is not present within the imaging range of the camera, the head-mounted display device 2 recognizes the position of the information processing device 3 by, for example, three-dimensional position detection using proximity communication. Then, in an area outside the imaging range 51 of the camera image, a mark 356 indicating the direction in which the information processing device 3 exists is superimposed on the VR image 50 and displayed. For example, in FIGS. 13A and 13B, the information processing device 3 is located on the right side of the imaging range 51, and therefore the mark 356 is displayed on the right side of the imaging range 51.

[0116] 13A and 13B, an image of the information processing device is shown as the mark 356, but an icon may be used as the mark 356. Also, an icon (for example, an arrow) indicating the direction in which the information processing device 3 is located may be used as the mark 356. In this case, the position of the mark is not particularly limited.

[0117] 13A is similar to Fig. 12A, and an image in which the VR image 50 and the captured image 351A are superimposed is displayed on the displays 22a and 22b. Furthermore, outside the captured image 351A (imaging range 51), a mark 356 indicating the direction in which the information processing device 3 exists is displayed superimposed on the VR image 50.

[0118] The user 1 points the head mounted display device 2 in the direction in which the mark 356 is displayed or in the direction indicated by the mark 356 until the information processing device 3 enters the captured image 351A. Then, when the information processing device 3 enters the imaging range 51, the user 1 picks up the information processing device 3 and operates the information processing device 3 while looking at the captured image 351A.

[0119] 13B, the three-dimensional position of the information processing device 3 is constantly detected by, for example, near-field communication while the user 1 is experiencing the VR content, regardless of whether or not there is a notification from the information processing device 3 or whether or not there is detection of the user 1's intention to use the information processing device 3. Then, a mark 356 is displayed on the displays 22a and 22b superimposed on the VR image.

[0120] The user 1 moves the head mounted display device 2 while looking at the mark 356 so that the information processing device 3 is in the vicinity of the front of the head mounted display device 2. Then, the user 1 picks up the information processing device 3 and operates the information processing device 3 while looking at the captured image 351A.

[0121] In addition, when the head-mounted display device 2 recognizes that the user 1 has reached out and touched the information processing device 3 through analysis of the captured image (captured image data), it can display the information processing device 3 and the hand superimposed on each other to notify the user 1 that it has recognized the information processing device 3.

[0122] According to the present embodiment, even if the user 1 does not know the location of the information processing device 3, the user 1 can find the information processing device 3. Therefore, the user 1 can pick up and use the information processing device 3 without putting on or taking off the head-mounted display device 2.

[0123] (Fourth embodiment) Next, a fourth embodiment will be described. In this embodiment, a collaboration process is performed to allow the user 1 to use the information processing device 3 in response to a collaboration instruction from the user 1. That is, in this embodiment, a collaboration process between the head mounted display device 2 and the information processing device 3 is started based on an active collaboration instruction from the user 1. The collaboration instruction from the user 1 conveys to the head mounted display device 2 that the user 1 intends to use the information processing device 3.

[0124] 14 is a flow diagram showing an example of a method for detecting a user's cooperation instruction according to Embodiment 4 of the present invention. In the example of Fig. 14, a cooperation instruction from user 1 is detected from a gesture of user 1 (including a movement pattern of the fingers of user 1) in images captured by cameras 20a and 20b.

[0125] When the image analysis application is started, the cameras 20a and 20b start capturing images (step S300). The head-mounted display device 2 (control device 220) receives captured image data generated by the cameras 20a and 20b (step S301), analyzes the captured image data, and extracts feature points of the captured image (step S302). The extracted feature points are, for example, the hands and fingers of the user 1.

[0126] Then, the head mounted display device 2 detects gestures (movement patterns) of feature points using the analysis results of the multiple captured image data (step S303). The head mounted display device 2 compares the gestures of feature points detected in step S303 with pre-registered gestures for matching (step S304). The gestures for matching may include gestures other than those corresponding to collaboration instructions from the user 1. In the pattern matching, the detected gestures of feature points are compared with the registered gestures for matching to determine what the user 1 wants to do.

[0127] Then, when a cooperation instruction from the user 1 is detected by the pattern matching in step S304, the head mounted display device 2 outputs a command to perform cooperation processing to the VR experience application 227 (step S305). On the other hand, when a cooperation instruction from the user 1 is not detected in the pattern matching in step S304, step S305 is ignored.

[0128] In step S306, the head mounted display device 2 determines whether or not to end the image analysis application. If the motion pattern detection application is to be ended (Yes), the process proceeds to step S307, where the image analysis application is ended. On the other hand, if the image analysis application is not to be ended (No), the processes of steps S301 to S306 are performed again.

[0129] 15 is a flow diagram showing another example of a method for detecting a user's cooperation instruction according to Embodiment 4 of the present invention. In FIG. 15, a method for detecting a cooperation instruction from user 1 from a voice uttered by user 1 is shown.

[0130] When the voice recognition application is started, voice acquisition by the microphone 24 starts (step S320). The head-mounted display device 2 (control device 220) takes in input voice data generated by the microphone 24 (step S321), analyzes the input voice data, and extracts features of the input voice (step S322). The features of the input voice extracted here are, for example, words, phrases, sentences, etc. of the input voice.

[0131] Then, the head mounted display device 2 compares the input voice feature extracted in step S322 with pre-registered features for matching (step S323). The features for matching may include features other than those corresponding to the collaboration instruction of the user 1. In the pattern matching, the extracted features of the input voice are compared with the registered features for matching to determine what the user 1 wants to do.

[0132] Then, when a cooperation instruction from the user 1 is detected by the pattern matching in step S323, the head mounted display device 2 outputs a command to perform cooperation processing to the VR experience application 227 (step S324). On the other hand, when a cooperation instruction from the user 1 is not detected in the pattern matching in step S323, step S324 is ignored.

[0133] In step S325, the head mounted display device 2 determines whether or not to end the image analysis application. If the voice recognition application is to be ended (Yes), the process proceeds to step S326, where the voice recognition application is ended. On the other hand, if the voice recognition application is not to be ended (No), the processes of steps S321 to S325 are performed again.

[0134] Fig. 16 is a flow diagram showing an example of a collaboration process by a VR experience application according to embodiment 4 of the present invention. Since Fig. 16 is similar to Fig. 5, the following mainly describes the differences from Fig. 5.

[0135] When the VR updated image is displayed in step S105, the process proceeds to step S420. In step S420, it is determined whether or not there is a command to perform a cooperative process based on a cooperative instruction from user 1.

[0136] When there is a command to perform cooperative processing based on a cooperative instruction from user 1, a predetermined cooperative signal is input to control device 220 (computer 222) from, for example, memory 223 or a register (not shown) in computer 222. Control device 220 determines, based on the cooperative signal, whether or not there is a command to perform cooperative processing based on a cooperative instruction from user 1.

[0137] If a link signal has been input (Yes), the control device 220 recognizes that there is a command to perform link processing based on a link instruction from the user 1. That is, in this case, the control device 220 determines that the user 1 intends to use the information processing device 3. Then, the process proceeds to step S421.

[0138] On the other hand, if the link signal has not been input (No), the control device 220 recognizes that there is no command to perform link processing based on the link instruction from the user 1. That is, in this case, the control device 220 determines that the user 1 has no intention of using the information processing device 3. Then, the process proceeds to step S112.

[0139] In step S421, control device 220 displays the VR image and the captured images from cameras 20a and 20b in a superimposed manner. In this way, the VR image and the captured images are displayed in a superimposed manner based on the cooperation instruction from user 1. The processing from step S421 onwards is the same as that in FIG. 5.

[0140] According to this embodiment, it is possible to detect an active cooperation instruction from the user 1 and start using the information processing device 3 while experiencing VR content.

[0141] (Embodiment 5) Next, a description will be given of embodiment 5. In this embodiment, the user 1 experiencing VR content can use an object other than the information processing device 3.

[0142] Fig. 17 is a diagram showing an example of an object used during the experience of VR content according to the fifth embodiment of the present invention. While Fig. 17 shows a mug 60 as an example of the object, other objects may also be used. In this embodiment, the shape of the object is registered in advance.

[0143] The head-mounted display device 2 (control device 220) analyzes the captured image using an image analysis application. When the head-mounted display device 2 detects the mug 60 from the captured image, the head-mounted display device 2 displays, for example, an image of the mug 60 superimposed on the VR image. This allows the user 1 to take a coffee break, for example, using the mug 60 while experiencing the VR content.

[0144] 18A is a diagram showing another example of an object used during the experience of VR content according to embodiment 5 of the present invention. In FIG. 18A, the object is shown as a mug 60 to which a communication module 61 is attached. Even if the position of mug 60 is unknown, head-mounted display device 2 can detect the position of mug 60 based on the proximity communication signal (position detection signal) transmitted from communication module 61.

[0145] 18B is a block diagram showing an example of a communication module. As shown in FIG. 18B, a communication module 61 includes a proximity communication device 62 and a microcomputer 63. The proximity communication device 62 and the microcomputer 63 are connected to each other via an internal bus 600.

[0146] As shown in FIG. 18B, the microcomputer 63 includes, for example, an MPU (Micro Processor Unit) 631, a memory 632, and a storage device 633.

[0147] Head-mounted display device 2 receives the proximity communication signal transmitted from proximity communication device 62 and detects the three-dimensional position of mug 60. Based on the detected three-dimensional position of mug 60, head-mounted display device 2 displays VR image 50 superimposed on a mark indicating the direction of mug 60.

[0148] When user 1 finds mug 60, he or she picks up mug 60 and takes a coffee break. Note that user 1 may pause the progress of the VR content when using mug 60.

[0149] According to this embodiment, the user 1 can use objects other than the information processing device 3 while experiencing the VR content.

[0150] It should be noted that the present invention is not limited to the above-described embodiment, and includes various modifications. For example, the above-described embodiment has been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to an embodiment having all of the described configurations.

[0151] Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations. All of these belong to the scope of the present invention, and furthermore, the numerical values, messages, etc. appearing in the text and figures are merely examples, and the use of different ones does not impair the effects of the present invention.

[0152] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example, by designing them as integrated circuits. They may also be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program. Furthermore, the scope of software implementation is not limited, and hardware and software may be used together. Note that the components and relative sizes shown in the drawings are simplified and idealized to clearly explain the invention, and may actually be more complex in actual implementation. [Explanation of symbols]

[0153] 1...user, 2...head-mounted display device, 3...information processing device, 4a to 4c...network signal, 5...access point, 6...network, 7...VR service server, 8...proximity communication, 20a, 20b...camera, 210...sensor group, 22a, 22b...display, 24...microphone, 25a, 25b, 25c...proximity communication receiving device, 220...control device, 221...communication device, 222...computer, 223...memory, 224...image memory, 225...storage device, 50...VR image, 51...imaging range, 52, 352...image of information processing device, 53...overlay image, 55...incoming call icon, 351A...captured image, 351B...transparent image, 351C...line drawing processed image, 356...mark, 60...mug, 61...communication module.

Claims

1. A non-transparent display; A camera and a communication device connected to the information processing device; a memory for storing VR content including VR image data and incoming call image data indicating that an incoming call has been received; a control device; Equipped with The control device Displaying the VR image data read from the memory on the display; When a notification indicating that the information processing device has received an incoming call is received from the information processing device via the communication device while the VR image data is being displayed on the display, the incoming call image data read from the memory, the VR image data, and the captured image data input from the camera are superimposed and displayed on the display. Head-mounted display device.

2. The head-mounted display device according to claim 1, When the captured image data input from the camera is analyzed and the information processing device is extracted, image data of the information processing device is generated, and the generated image data of the information processing device is superimposed on the VR image data and displayed on the display. Head-mounted display device.

3. The head-mounted display device according to claim 1, when the control device receives a request to delete the incoming image data from the information processing device via the communication device, the control device terminates display of the incoming image data. Head-mounted display device.

4. The head-mounted display device according to claim 1, the control device terminates the display of the incoming image data when a predetermined time has elapsed since the incoming image data was displayed on the display; Head-mounted display device.

5. The head-mounted display device according to claim 1, the communication device has a near field communication function, If the information processing device is not included in the captured image data input from the camera, the position of the information processing device is detected using the near-field communication function, and image data indicating the direction of the information processing device is superimposed on the VR image data and displayed on the display. Head-mounted display device.

6. The head-mounted display device according to any one of claims 1 to 5, The control device downloads the VR content from a server via the communication device and stores the downloaded VR content in the memory. Head-mounted display device.

7. The head-mounted display device according to any one of claims 1 to 5, The incoming call image data is image data of an incoming call icon. Head-mounted display device.

Images