Information processing device, control method for information processing device, and program

Abstract

To provide a technique capable of reducing user's efforts and more appropriately control display and sound output.SOLUTION: An information processing device connected to a display device worn on a user's head and controlling a sound output device includes: determination means for determining whether a display mode of the display device, which includes a first mode and a second mode in which visibility of the user's surroundings is different from each other or whether user's surroundings can be viewed, is different from each other, has been switched from the first mode to the second mode; and switching means for switching a sound output mode of a sound output device, which includes a third mode and a fourth mode in which audibility of sound around the user is different from each other, from the third mode to the fourth mode, when the first mode is switched to the second mode. The display mode is switched depending on whether it is determined that the user may come into contact with an object or a type of an application that has been started.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, a control method for an information processing device, and a program. [Background technology]

[0002] VR (Virtual Reality) technology is known as a technology that allows users to experience virtual space. MR (Mixed Reality) technology is also known as a technology that seamlessly fuses real space and virtual space in real time.

[0003] As a device that allows users to experience such technology, for example, a head-mounted device such as an HMD (Head Mounted Display) is used. The HMD works in conjunction with external earphones to display images (video) and play audio through the earphones.

[0004] Patent Document 1 describes earphones that have an external sound capture function that makes it easier to hear sounds around the earphones, and a noise cancellation function that makes it harder to hear sounds around the earphones. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2023-29976 A Summary of the Invention [Problem to be solved by the invention]

[0006] Both VR technology and noise cancellation technology have the advantage of making it easier for users to immerse themselves in the content (images / audio), while MR technology and ambient sound capture technology have the advantage of allowing users to experience the content (images / audio) in a state where it is blended with their surroundings (reality) (enabling them to sense danger and communicate with those around them).

[0007] However, the HMD (display device) and the earphones (audio output device) are separate devices, and therefore, in order to adjust both the display and audio output to an appropriate state, the user must manually adjust both devices, which is time-consuming for the user.

[0008] Therefore, an object of the present invention is to provide a technique that can more appropriately control display and audio output while reducing the effort required by the user. [Means for solving the problem]

[0009] One aspect of the present invention is a method for producing a composition comprising the steps of: An information processing device that is connected to a display device worn on a user's head and controls an audio output device, a determination means for determining whether a display mode of the display device, which includes a first mode and a second mode in which the visibility of the surroundings of the user is different from each other or in which the visibility of the surroundings of the user is different from each other, has been switched from the first mode to the second mode; a switching means for switching an audio output mode of the audio output device, which includes a third mode and a fourth mode in which the audibility of sounds around the user is different from each other, from the third mode to the fourth mode when the display mode is switched from the first mode to the second mode. and, having The display mode is switched depending on whether it is determined that the user may come into contact with an object, or whether the launched application is an application for the first mode or an application for the second mode. The present invention is an information processing device characterized by the above. One aspect of the present invention is a method for producing a composition comprising the steps of: An information processing device that controls a display device mounted on a user's head and is connected to an audio output device, a determination means for determining whether or not an audio output mode of the audio output device, which includes a third mode and a fourth mode in which the audibility of sounds around the user is different from each other, has been switched from the third mode to the fourth mode; a switching means for switching a display mode of the display device, which includes a first mode and a second mode in which visibility of a state around the user is different from each other or in which whether or not the state around the user is visible is different from each other, from the first mode to the second mode when the audio output mode is switched from the third mode to the fourth mode; The information processing device is characterized by having:

[0010] One aspect of the present invention is a method for producing a composition comprising the steps of: A method for controlling an information processing device that is connected to a display device worn on a user's head and controls an audio output device, comprising: a determination step of determining whether a display mode of the display device, which includes a first mode and a second mode in which the visibility of the state around the user is different from each other or in which the visibility of the state around the user is different from each other, has been switched from the first mode to the second mode; a switching step of switching an audio output mode of the audio output device, which includes a third mode and a fourth mode in which audibility of sounds around the user is different from each other, from the third mode to the fourth mode when the display mode is switched from the first mode to the second mode; having The display mode is switched depending on whether it is determined that the user may come into contact with an object, or whether the launched application is an application for the first mode or an application for the second mode. The control method is characterized by the above. One aspect of the present invention is a method for producing a composition comprising the steps of: A method for controlling an information processing device that controls a display device worn on a user's head and is connected to an audio output device, comprising the steps of: a determination step of determining whether or not an audio output mode of the audio output device, which includes a third mode and a fourth mode in which the audibility of sounds around the user is different from each other, has been switched from the third mode to the fourth mode; a switching step of switching a display mode of the display device, which includes a first mode and a second mode in which visibility of a state around the user is different from each other or in which whether or not the state around the user is visible is different from each other, from the first mode to the second mode when the audio output mode is switched from the third mode to the fourth mode; The control method is characterized by having the following features. Effect of the Invention

[0011] According to the present invention, it is possible to more appropriately control display and audio output while reducing the effort required of the user. [Brief description of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an information processing system according to a first embodiment. [Diagram 2] 1 is a diagram showing a configuration of an HMD and the like according to a first embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a display mode according to the first embodiment. [Figure 4] 4 is a flowchart of a control process of an audio output mode according to the first embodiment. [Diagram 5] 10 is a flowchart of a control process of a display mode according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in each embodiment, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the attached drawings, the same reference numbers are given to the same or similar configurations, and duplicated explanations are omitted. Note that, hereinafter, sounds around the user (sounds around the HMD 100; sounds other than sounds emitted by the earphones 120) are referred to as "external sounds."

[0014] <Embodiment 1> An information processing system 1 according to a first embodiment will be described with reference to Fig. 1. The information processing system 1 is an XR system for providing a user with VR (Virtual Reality) that allows the user to experience a virtual space, and a simulated experience (Mixed Reality: MR) that combines the real world and the virtual world. The information processing system 1 includes an HMD 100, an information processing device 110, and earphones 120. The information processing device 110 is connected to the HMD 100 and the earphones 120. For this reason, the HMD 100 is connected to the earphones 120 via the information processing device 110.

[0015] The HMD 100 is a head-mounted display device (electronic device) that is worn on the user's head. The HMD 100 displays content such as CG (computer graphics) in a form that corresponds to the posture of the HMD 100. This allows the user to experience VR. If an image that combines the "image captured by the HMD 100 of the area in front of the user" with the content is used as the displayed image, a video see-through MR experience is possible.

[0016] The information processing device 110 is an electronic device (control device) that performs various processes. The various processes include control of the HMD 100, processing of captured images captured by a camera mounted on the HMD 100, generation of content such as CG, and generation of display images to be displayed on the HMD 100. The information processing device 110 is configured by a computer including a CPU (processor) and a memory. For example, the information processing device 110 is a smartphone, a tablet terminal, a PC (personal computer), a game device, or the like. The information processing device 110 is connected to the HMD 100 wirelessly or by wire. The information processing device 110 generates a display image and transmits the display image to the HMD 100. Note that each component of the information processing device 110 may be built into the HMD 100.

[0017] The earphones 120 are playback devices (audio output devices) capable of playing back (outputting) audio corresponding to audio data attached to the content of the HMD 100. Note that instead of the earphones 120, any audio output device (such as headphones) that is worn on the ears can also be used.

[0018] (Internal structure of HMD) The internal configuration of the HMD 100 will be described with reference to Fig. 2. The HMD 100 includes an HMD control unit 201, an imaging unit 202, and an image display unit 203.

[0019] The HMD control unit 201 controls each component of the HMD 100. When the HMD control unit 201 acquires a display image from the information processing device 110, the HMD control unit 201 displays the display image on the image display unit 203. By wearing the HMD 100, the user can view the display image displayed on the image display unit 203.

[0020] The imaging unit 202 includes, for example, two cameras (imaging devices). The two cameras are disposed near the positions of the left and right eyes of the user when wearing the HMD 100 in order to capture an image of a space similar to the space the user normally sees. Images of a subject (a range in front of the user) captured by the two cameras are output to the information processing device 110. In addition, the two cameras in the imaging unit 202 can acquire distance information indicating the distance from the two cameras to the subject by measuring distances using a stereo camera.

[0021] The image display unit 203 displays a display image. The image display unit 203 has, for example, a display panel (such as a liquid crystal panel or an organic EL panel). When the user wears the HMD 100, the display panels are disposed in front of the left and right eyes of the user.

[0022] (Internal configuration of information processing device) The internal configuration of the information processing device 110 will be described with reference to Fig. 2. The information processing device 110 includes a control unit 211, a content database (DB) 212, a communication unit 213, and a position and orientation estimation unit 214.

[0023] The control unit 211 generates a display image based on the image data of the content, and transmits the display image to the HMD control unit 201.

[0024] If a CG image is used as the display image, the user can experience a virtual space (VR space) in which CG is displayed. On the other hand, if an "image in which a captured image of the space in front of the user captured by the imaging unit 202 and CG are combined" is used as the display image, the user can experience a mixed reality space (MR space) in which CG is blended into the real space.

[0025] The content DB 212 is a memory unit (storage unit) that stores "contents that combine CG image data and audio data" etc. The control unit 211 can switch the content (such as CG used to generate a display image) read from the content DB 212.

[0026] The communication unit 213 transmits audio data of the content (audio data corresponding to a displayed image) by communicating with the earphone 120. The communication unit 213 is also capable of acquiring the state of the earphone 120 and setting the earphone 120.

[0027] The position and orientation estimation unit 214 acquires position and orientation information by estimating the position and orientation of the HMD 100. The position and orientation information indicates the absolute three-dimensional position and orientation of the HMD 100 in real space. A method for estimating the position and orientation is used that allows the absolute position and orientation of the HMD 100 in real space to be observed. For example, the position and orientation estimation unit 214 may use an optical sensor that measures the position and orientation of the HMD 100 based on multiple markers attached to a measurement target. The optical sensor measures the position and orientation of the HMD 100 by analyzing (integrating) marker information acquired by multiple cameras installed on a ceiling or the like.

[0028] In addition, a method for estimating position and orientation using Simultaneous Localization and Mapping (SLAM) is available. SLAM tracks natural object features such as corners in captured images in each frame to estimate the position and orientation of the object in real space. This is a technology for estimating information on the three-dimensional position and orientation of an imaging device in a virtual reality environment. The position and orientation estimation unit 214 may measure the position and orientation of the HMD 100 using markers or image features that appear in an image captured by the imaging unit 202 of the HMD 100, without using an optical sensor. Here, by using the measured position and orientation, it is possible to generate a CG image that depicts a predetermined CG model in a predetermined position and orientation.

[0029] (Internal structure of earphones) The internal configuration of the earphones 120 will be described with reference to Fig. 2. The earphones 120 include a control unit 221, an audio output unit 222, a communication unit 223, a processing unit 224, and an operation unit 225.

[0030] The control unit 221 controls each component of the earphone 120. The control unit 221 is configured, for example, by a small general-purpose processor, a memory, and a program. The control unit 221 may be configured by a dedicated microprocessor or an ASIC (Application Specific Integrated Circuit), etc.

[0031] The audio output unit 222 outputs audio (emits sound) in accordance with the audio data processed by the processing unit 224. The audio output unit 222 is, for example, a small speaker.

[0032] The communication unit 223 performs wireless communication with the information processing device 110 (the communication unit 213).

[0033] The processing unit 224 performs external sound processing on the audio data of the content received via the communication unit 223 according to the audio output mode. The processing unit 224 also supplies the audio data after the external sound processing to the audio output unit 222. When the audio output mode is the "external sound capture mode", the processing unit 224 performs "processing to make the external sound easier to hear by adding the sound (external sound) around the HMD 100 to the audio data of the content" as "external sound processing". When the audio output mode is the "noise cancellation mode", the processing unit 224 performs "processing to make the external sound (i.e., noise) harder to hear in the audio data of the content (noise cancellation processing)" as "external sound processing". Specifically, when the audio output mode is the "noise cancellation mode", the processing unit 224 adds a sound that makes the external sound harder to hear (a sound that cancels the external sound) to the audio data of the content. Note that when the audio output mode is the "noise cancellation mode", the external sound is not added to the audio data of the content. The external sound may be acquired by a microphone of the HMD 100 or a microphone of the earphone 120.

[0034] The earphone 120 can switch the audio output mode between an "external sound capture mode" and a "noise cancellation mode". The audio output mode may include a mode that can adjust the "output level of external sound (volume of external sound output from the audio output unit 222)" or the "noise cancellation level (level of canceling external sound)".

[0035] The operation unit 225 detects an operation for switching the audio output mode of the earphone 120 (operation by the user).

[0036] In the first embodiment, an example in which the information processing device 110 and the earphones 120 are connected wirelessly will be described, but the information processing device 110 and the earphones 120 may be connected by wire.

[0037] (HMD display mode) A description will be given of a plurality of display modes of the HMD 100 and images displayed on the image display unit 203 in each display mode with reference to Fig. 3. The display modes include a VR mode and an MR mode.

[0038] In the VR mode, the display image is a VR image including only CG, and does not include an image captured by the imaging unit 202 of the real space. In the VR mode, the user can be immersed in the content, but cannot check the surroundings of the user from the display image. In other words, the VR mode is a mode in which the surroundings of the user cannot be viewed.

[0039] In the MR mode, the display image is an MR image including an image captured by the imaging unit 202 of the real space. Therefore, the user can check the surroundings by the display image. The MR mode can further include the following MR modes M1 to M4.

[0040] In the MR mode M1, the display image is a composite image in which CG is composited (superimposed) on a captured image.

[0041] In the MR mode M2, CG is displayed, but a part of the CG is masked. Therefore, the displayed image is an image in which the masked CG and the captured image are combined. The masking process is, for example, a process of extracting an area based on color, such as a chromakey compositing technique.

[0042] In the MR mode M3, the displayed image is an image in which only the contours (edges) of objects captured in the captured image are superimposed on the CG.

[0043] In the MR mode M4, the display image is an image in which a CG image with increased transparency (set to semi-transparent or full-transparent) is synthesized with a captured image.

[0044] For this reason, the area of ​​the captured image (CG) relative to the area of ​​the displayed image (display surface) is different in the MR modes M1 to M4. It can be said that the MR mode M4 is the "display mode in which the user can most easily visually recognize the surroundings of the HMD 100", followed by the MR mode M2, the MR mode M1, and the MR mode M3 in that order.

[0045] An example of controlling the audio output mode of the earphone 120 in accordance with the operation of the HMD 100 will be described with reference to the flowchart of FIG.

[0046] In step S401, the control unit 211 determines whether or not a display mode switching instruction has been issued to the HMD 100. If it is determined that a display mode switching instruction has been issued, the process proceeds to step S402. If it is determined that a display mode switching instruction has not been issued, the process proceeds to step S403.

[0047] For example, when a new application is started, if the application is a VR application, the control unit 211 determines that an instruction to switch to the VR mode has been given. When a new application is started, if the application is an MR application, the control unit 211 determines that an instruction to switch to the MR mode has been given. The VR application is an application for the VR mode and is an application for viewing VR images. The MR application is an application for the MR mode and is an application for viewing MR images.

[0048] Furthermore, switching between the VR mode and the MR mode may be possible in response to a user's operation of a switch arranged on the HMD 100. The switch may be a button-type switch, a dial-type switch, or a slide-type switch.

[0049] Furthermore, the control unit 211 determines that an instruction to switch the display mode has been given in response to a user changing the ON / OFF state of an imaging function (a function for capturing an image of real space) of the imaging unit 202 by a menu setting or a button operation of the HMD 100. If the imaging function has been changed to ON, the control unit 211 determines that an instruction to switch to the MR mode has been given. If the imaging function has been changed to OFF, the control unit 211 determines that an instruction to switch to the VR mode has been given.

[0050] The control unit 211 may determine whether or not an instruction to switch the display mode has been issued in response to an instruction generated inside the HMD 100, regardless of an instruction from the user. For example, the control unit 211 may determine whether or not an instruction to switch the display mode has been issued in response to a change in ON / OFF of a safety function for checking the surroundings of the HMD 100. The safety function is a function that is turned on to notify the user of a danger when the user moves outside a predetermined moving area and there is a possibility (danger) that the user will come into contact (crash) with an object (person or object). When the safety function is turned on to notify the user of the danger, the control unit 211 determines that an instruction to switch to the MR mode has been issued. When the safety function is turned off because the danger has been avoided, the control unit 211 determines that an instruction to switch to the VR mode has been issued.

[0051] In step S402, the control unit 211 switches to a display mode corresponding to the switching instruction given in step S401. That is, the control unit 211 changes the display of the HMD 100 in response to the display mode switching instruction, thereby changing whether the surrounding state is visible or not, or changing the visibility of the surrounding state.

[0052] The display mode may be gradually switched over time or according to the amount of operation. For example, the control unit 211 may control the speed of switching according to the amount of operation by the user for instructing the display mode to be switched (such as the degree of pressing a button-type switch, the degree of rotation of a dial-type switch, or the degree of sliding a slide-type switch). For example, when switching from the MR mode to the VR mode, the control unit 211 controls the display so that the area (area) of the CG gradually expands (the area of ​​the real image narrows) from the center of the display surface of the image display unit 203 or the user's gaze point. When switching from the VR mode to the MR mode, the control unit 211 controls the display so that the area of ​​the real image gradually expands (the area of ​​the CG narrows) from the periphery of the display surface.

[0053] In step S403, the control unit 211 determines whether the display mode of the HMD 100 is the MR mode. If it is determined that the display mode of the HMD 100 is the MR mode, the process proceeds to step S404. If it is determined that the display mode of the HMD 100 is not the MR mode (the display mode of the HMD 100 is the VR mode), the process proceeds to step S405.

[0054] In step S404, the control unit 211 sets the audio output mode of the earphones 120 to the ambient sound capture mode. That is, the control unit 211 causes the earphones 120 to emit ambient sound. As a result, the audio output mode of the earphones 120 switches to the ambient sound capture mode in conjunction with the display mode of the HMD 100 being set to the MR mode. This makes it possible to switch the audio output mode without the user's effort, and makes it easier for the user to check the surrounding state (situation) by audio in addition to the image (display).

[0055] In step S405, the control unit 211 sets the sound output mode of the earphones 120 to a noise canceling mode (active noise canceling mode). In other words, the control unit 211 makes the earphones 120 emit sounds that cancel out external sounds. Thus, in conjunction with the change of the display mode of the HMD 100 to the VR mode, the audio output mode of the earphones 120 is switched to the noise cancellation mode. This allows the user to switch the audio output mode without any effort, and allows the user to be immersed in the content through audio in addition to images (display).

[0056] It is not necessary for the control unit 211 to discontinuously switch the sound output mode of the earphone 120 between the external sound capture mode and the noise cancellation mode. For example, the control unit 211 may switch the noise cancellation (or external sound output) level according to the ratio (area ratio) of the captured image (CG) in the display image (display surface). At this time, the CG ratio may take into consideration the transparency of the CG. For example, if the transparency of a certain CG is 20%, in the CG area, the CG ratio is set to 80% and the captured image ratio is set to 20%. Also, when the display mode is gradually switched, the sound output mode may also be gradually switched according to the speed of switching the display mode.

[0057] Here, the audio output mode set in step S404 may be any audio output mode as long as the audio output mode has a higher audibility of the surrounding sounds (the user can easily perceive the surrounding sounds) than the audio output mode set in step S405. For example, the noise cancellation mode may be set to a noise cancellation mode with a noise cancellation level of L1 in step S404, and the noise cancellation mode may be set to a noise cancellation mode with a noise cancellation level of L2 (L2>L1) in step S405. Here, the higher the noise cancellation level value, the more the processing is performed to cancel the surrounding sounds (to make them harder to hear). Note that the directivity of the noise cancellation may be changed according to the change in the value of the noise cancellation level. When the noise cancellation level value is set high, the directivity may be set high (for example, unidirectional), and when the noise cancellation level value is set low, the directivity may be set low (for example, omnidirectional).

[0058] Furthermore, when the power supply of the HMD 100 is turned off or when the image display of the HMD 100 is stopped due to power saving control, the control unit 211 may automatically switch the audio output mode of the earphones 120 to the ambient sound capture mode. When the control unit 211 determines that the user has stopped viewing the image display unit 203 of the HMD 100 by removing the HMD 100 or flipping up the HMD 100, for example, the control unit 211 may switch the audio output mode of the earphones 120 to the ambient sound capture mode.

[0059] As described above, according to the first embodiment, when the display mode of the HMD 100 is switched, the audio output mode of the earphones 120 is also switched. Therefore, even if the user does not perform an operation to switch the audio output mode of the earphones 120, the audio output can be switched to a more appropriate one.

[0060] <Embodiment 2> A process for controlling the display mode of the HMD 100 in response to the operation of the earphones 120 according to the second embodiment will be described with reference to the flowchart of FIG.

[0061] In step S501, the control unit 211 determines whether or not an instruction to switch the audio output mode has been issued to the earphones 120. The instruction to switch the audio output mode can be issued, for example, by the user operating the operation unit 225. If it is determined that an instruction to switch the audio output mode has been issued, the process proceeds to step S502. If it is determined that an instruction to switch the audio output mode has not been issued, the process proceeds to step S503.

[0062] In step S502, the control unit 211 switches to an audio output mode in response to the audio output mode switching instruction. That is, the control unit 211 changes the sound emitted by the earphone 120 in response to the audio output mode switching instruction, thereby improving the audibility (ease of hearing) of surrounding sounds. Change the length.

[0063] The audio output mode may be gradually switched over time or according to the amount of operation. For example, the control unit 211 may control the speed of switching according to the amount of operation by the user for instructing the switching of the audio output mode (such as the degree of pressing a button-type switch, the degree of rotation of a dial-type switch, or the degree of sliding a slide-type switch). For example, when switching from the noise cancellation mode to the external sound capture mode, the control unit 211 controls the audio output so that the noise cancellation level is gradually reduced and then the external sound capture mode is set. When switching from the external sound capture mode to the noise cancellation mode, the control unit 211 controls the audio output so that the output level of the external sound is gradually reduced and then the noise cancellation mode is set.

[0064] In step S503, the control unit 211 determines whether the sound output mode of the earphones 120 is the ambient sound capture mode. If it is determined that the sound output mode of the earphones 120 is the ambient sound capture mode, the process proceeds to step S504. If it is determined that the sound output mode of the earphones 120 is not the ambient sound capture mode (the sound output mode is the noise cancellation mode), the process proceeds to step S505.

[0065] In step S504, the control unit 211 sets the display mode of the HMD 100 to the MR mode. As a result, the display mode of the HMD 100 is switched to the MR mode in conjunction with the change of the sound output mode of the earphones 120 to the external sound capture mode. This makes it possible to switch the display mode without the user's effort, and makes it easier for the user to check the surrounding state (situation) by images (displays) in addition to sounds.

[0066] In step S505, the control unit 211 sets the display mode of the HMD 100 to the VR mode. As a result, the display mode of the HMD 100 is switched to the VR mode in conjunction with the change of the audio output mode of the earphones 120 to the noise cancellation mode. This makes it possible to switch the display mode without the user's effort, and the user can be immersed in the content through images (display) in addition to audio.

[0067] Furthermore, when the audio output mode is gradually switched, the display mode may also be gradually switched in step S504 or S505 depending on the speed at which the audio output mode is switched.

[0068] Note that, when a safety function for checking the surroundings is on (when the user is at risk of contact with an object), even if an instruction to switch to the noise cancellation mode is given, the control unit 211 may maintain the display mode in the MR mode. Then, when the control unit 211 determines that there is no possibility of contact (collision) between the user and the object (danger has been avoided; the safety function has been changed to off), the control unit 211 may switch the display mode to the VR mode.

[0069] Furthermore, when the user removes the earphones 120, the control unit 211 may automatically change the display mode to the MR mode so that the user can check the surroundings using the HMD 100 as well.

[0070] The display mode set in step S504 may be any display mode as long as the display mode has higher visibility of the surrounding state (situation) (the user can easily visually recognize the surrounding state) than the display mode set in step S505. For example, the display mode may be set to MR mode M4 in step S504 and set to MR mode M3 in step S505.

[0071] As described above, according to the second embodiment, when the audio output mode of the earphones 120 is switched, the display mode of the HMD 100 is also switched. Therefore, even if an operation for switching the display mode of the HMD 100 is not performed, the display can be switched to a more appropriate display.

[0072] <Variation 1> In the first and second embodiments, the HMD 100 has been described as a video see-through type HMD, but it may be an optical see-through type HMD. Some optical see-through type HMDs can change the dimming level (transmittance; transmission level) of the display surface and adjust the sense of immersion. In an optical see-through type HMD, the user can see the surroundings (surrounding conditions) through the light-transmitting display surface.

[0073] When the display mode is in a high dimming level (low transmittance) state and the surroundings become difficult to see, the control unit 211 may automatically switch the earphones 120 to a noise cancellation mode or increase the noise cancellation level, thereby enhancing the sense of immersion both visually and aurally.

[0074] Conversely, when the display mode has a low dimming level (high transmittance) on the display surface and the surroundings are easily visible, control unit 211 may automatically switch earphones 120 to the ambient sound capture mode or increase the ambient sound capture level, making it easier to check the surroundings (situation).

[0075] According to each embodiment and each modified example described above, when one of the display mode and the audio output mode is switched, the other of the display mode and the audio output mode is switched. This reduces the user's effort and makes it possible to more appropriately control (adjust) the display and audio output.

[0076] In each of the above embodiments, the display mode and the audio output mode are controlled so that the VR mode and the noise cancellation mode are set as a set, and the MR mode and the external sound capture mode are set as a set. However, if the user desires, the display mode and the audio output mode may be controlled so that the VR mode and the external sound capture mode are set as a set, and the MR mode and the noise cancellation mode are set as a set.

[0077] That is, for example, when the display mode is switched to the MR mode, the control unit 211 may switch the audio output mode to the noise cancellation mode. Also, when the audio output mode is switched to the external sound capture mode, the control unit 211 may switch the display mode to the VR mode. This allows the user to check the state (situation) around him / her by either audio output or display. Then, by either audio output or display, it is possible to ensure a certain degree of immersion in the content. That is, it becomes possible to adjust the balance between the immersion and the safety of the user.

[0078] Although the present invention has been described in detail based on the preferred embodiments, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0079] Also, in the above, "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2" may be read as "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2." Conversely, "If A is greater (higher) than B, proceed to step S1, and if A is equal to or less than B, proceed to step S2" may be read as "If A is equal to or greater than B, proceed to step S1, and if A is smaller (lower) than B, proceed to step S2." This may result in a contradiction. Unless otherwise stated, "A or more" may be read as "bigger (higher; longer; more) than A" and "A or less" may be read as "smaller (lower; shorter; less) than A." And "bigger (higher; longer; more) than A" may be read as "A or more," and "smaller (lower; shorter; less) than A" may be read as "A or less."

[0080] Each functional unit in each of the above embodiments (variations) may or may not be individual hardware. The functions of two or more functional units may be realized by common hardware. Each of a plurality of functions of one functional unit may be realized by individual hardware. Two or more functions of one functional unit may be realized by common hardware. Furthermore, each functional unit may or may not be realized by hardware such as an ASIC, FPGA, or DSP. For example, the device may have a processor and a memory (storage medium) in which a control program is stored. Then, the functions of at least some of the functional units of the device may be realized by the processor reading and executing the control program from the memory.

[0081] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions.

[0082] The disclosure of the above embodiments includes the following configurations, methods, and programs. (Configuration 1) An information processing device that is connected to a display device worn on a user's head and controls an audio output device, a determination means for determining whether a display mode of the display device, which includes a first mode and a second mode in which the visibility of the state around the user is different from each other, or in which the visibility of the state around the user is different from each other, has been switched from the first mode to the second mode; a switching means for switching an audio output mode of the audio output device, the audio output mode including a third mode and a fourth mode having different audibility of sounds around the user, from the third mode to the fourth mode when the display mode is switched from the first mode to the second mode; having The display mode is switched depending on whether it is determined that the user may come into contact with an object, or whether the launched application is an application for the first mode or an application for the second mode. 23. An information processing apparatus comprising: (Configuration 2) An information processing device that controls a display device mounted on a user's head and is connected to an audio output device, a determination means for determining whether or not an audio output mode of the audio output device, which includes a third mode and a fourth mode in which the audibility of sounds around the user is different from each other, has been switched from the third mode to the fourth mode; a switching means for switching a display mode of the display device, which includes a first mode and a second mode in which visibility of a state around the user is different from each other or in which whether or not the state around the user is visible is different from each other, from the first mode to the second mode when the audio output mode is switched from the third mode to the fourth mode; 13. An information processing device comprising: (Configuration 3) The second mode is a mode in which the state around the user is not visible or a mode in which the visibility of the state around the user is lower than that in the first mode; The fourth mode is a mode in which the audibility of sounds around the user is lower than that of the third mode, the switching means does not switch the audio output mode from the first mode to the second mode when there is a possibility that the user will come into contact with an object, even if the audio output mode is switched from the third mode to the fourth mode. 3. The information processing device according to configuration 2. (Configuration 4) The second mode is a mode in which the state around the user is not visible or a mode in which the visibility of the state around the user is lower than that in the first mode; The fourth mode is a mode in which the audibility of sounds around the user is lower than that of the third mode. 4. The information processing device according to any one of configurations 1 to 3. (Configuration 5) The first mode is a mode in which the state around the user is not visible, or a mode in which the state around the user is less visible than in the second mode; The third mode is a mode in which the audibility of sounds around the user is lower than that of the fourth mode. 3. The information processing device according to configuration 1 or 2. (Configuration 6) In the third mode and the fourth mode, the sound emitted from the audio output device is different depending on the control of the output of the sound around the user from the audio output device or the control of canceling the sound around the user. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 7) one of the third mode and the fourth mode is a mode in which the audio output device does not emit sounds around the user; The other of the third mode and the fourth mode is a mode in which the audio output device emits sounds around the user. 7. The information processing device according to configuration 6. (Configuration 8) At least one of the third mode and the fourth mode is a mode in which the audio output device cancels sounds around the user. 7. The information processing device according to configuration 6. (Configuration 9) the third mode and the fourth mode are both modes in which the audio output device cancels out sounds around the user; The third mode and the fourth mode have different levels of cancellation. 9. The information processing device according to configuration 8. (Configuration 10) the third mode and the fourth mode are both modes in which the audio output device emits sounds around the user; and The output level of the sound around the user emitted by the audio output device is different between the third mode and the fourth mode. 7. The information processing device according to configuration 6. (Configuration 11) In the first mode and the second mode, the display of the display device is different depending on the control of the display of a captured image of the surroundings of the user or the control of a transparency level of a display surface of the display device. 6. The information processing device according to any one of configurations 1 to 5. (Configuration 12) the display device has a display surface that transmits light; The user can view the surroundings of the user through the display surface, The first mode and the second mode have different transmission levels of the display surface. 12. The information processing device according to configuration 11. (Configuration 13) one of the first mode and the second mode is a mode for displaying an image including the captured image, The other of the first mode and the second mode is a mode in which an image including the captured image is not displayed. 12. The information processing device according to configuration 11. (Configuration 14) The first mode and the second mode are both display modes for displaying a display image in which CG (computer graphics) is synthesized with the captured image, The ratio of the area of ​​the captured image to the area of ​​the displayed image is different between the first mode and the second mode. 12. The information processing device according to configuration 11. (Method 1) A method for controlling an information processing device that is connected to a display device worn on a user's head and controls an audio output device, comprising: a determination step of determining whether a display mode of the display device, which includes a first mode and a second mode in which the visibility of the state around the user is different from each other or in which the visibility of the state around the user is different from each other, has been switched from the first mode to the second mode; a switching step of switching an audio output mode of the audio output device, which includes a third mode and a fourth mode in which audibility of sounds around the user is different from each other, from the third mode to the fourth mode when the display mode is switched from the first mode to the second mode; having The display mode is switched depending on whether it is determined that the user may come into contact with an object, or whether the launched application is an application for the first mode or an application for the second mode. A control method comprising: (Method 2) A method for controlling an information processing device that controls a display device worn on a user's head and is connected to an audio output device, comprising the steps of: a determination step of determining whether or not an audio output mode of the audio output device, which includes a third mode and a fourth mode in which the audibility of sounds around the user is different from each other, has been switched from the third mode to the fourth mode; a switching step of switching a display mode of the display device, which includes a first mode and a second mode in which visibility of a state around the user is different from each other or in which whether or not the state around the user is visible is different from each other, from the first mode to the second mode when the audio output mode is switched from the third mode to the fourth mode; A control method comprising the steps of: (program) A program for causing a computer to function as each of the means of the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0083] 100: HMD, 110: information processing device, 120: earphones, 211: Control unit

Claims

1. An information processing device connected to a display device worn on the user's head and controlling an audio output device, The display device has a display mode including a first mode and a second mode in which the visibility of the user's surroundings differs from each other, or whether the user's surroundings are visible or not differs from each other, and the audio output device has an audio output mode including a third mode and a fourth mode in which the audibility of the sounds around the user differs from each other. The aforementioned information processing device is A first determination means for determining whether the display mode has been switched from the first mode to the second mode, When it is determined that the display mode has been switched from the first mode to the second mode, a first switching means switches the audio output mode from the third mode to the fourth mode, A second determination means for determining that the user has removed the display device or has stopped viewing the display surface of the display device, The system includes a second switching means for switching the audio output mode to the third mode when it is determined that the user has removed the display device or has stopped viewing the display surface. An information processing device characterized by the following:

2. An information processing device that controls a display device worn on the user's head and is connected to an audio output device, The display device has a display mode including a first mode and a second mode in which the visibility of the user's surroundings differs from each other, or whether the user's surroundings are visible or not differs from each other, and the audio output device has an audio output mode including a third mode and a fourth mode in which the audibility of the sounds around the user differs from each other. The aforementioned information processing device is A determination means for determining whether the audio output mode has been switched from the third mode to the fourth mode, If it is determined that the audio output mode has been switched from the third mode to the fourth mode, the display mode is switched from the first mode to the second mode. A switching mechanism is provided. An information processing device characterized by the following:

3. The display mode is switched depending on whether or not it is determined that the user may come into contact with an object, or whether the launched application is an application for the first mode or an application for the second mode. The information processing apparatus according to feature 1.

4. The second mode is a mode in which the user's surroundings are not visible, or a mode in which the visibility of the user's surroundings is lower than in the first mode. The fourth mode is a mode in which the audibility of sounds around the user is lower than that of the third mode. Even if the switching means determines that the audio output mode has been switched from the third mode to the fourth mode, if there is a possibility that the user may come into contact with an object, it will not switch the display mode from the first mode to the second mode. The information processing apparatus according to feature 2.

5. The second mode is a mode in which the user's surroundings are not visible, or a mode in which the visibility of the user's surroundings is lower than in the first mode. The fourth mode is a mode in which the audibility of sounds around the user is lower than that of the third mode. The information processing apparatus according to any one of claims 1 to 4.

6. The first mode is a mode in which the user's surroundings are not visible, or a mode in which the visibility of the user's surroundings is lower than that of the second mode. The third mode is a mode in which the audibility of sounds around the user is lower than that of the fourth mode. The information processing apparatus according to claim 1 or 2.

7. In the third mode and the fourth mode, the sound emitted from the audio output device differs depending on the control of the output of ambient sounds from the audio output device or the control of canceling ambient sounds from the user. The information processing apparatus according to any one of claims 1 to 4.

8. One of the third mode and the fourth mode is a mode in which the audio output device does not emit sounds from the user's surroundings. The other of the third mode and the fourth mode is a mode in which the audio output device emits sounds from the user's surroundings. The information processing apparatus according to feature 7.

9. At least one of the third mode and the fourth mode is a mode in which the audio output device cancels out sounds around the user. The information processing apparatus according to feature 7.

10. Both the third mode and the fourth mode are modes in which the audio output device cancels out sounds around the user. The third mode and the fourth mode have different levels of cancellation. The information processing apparatus according to feature 9.

11. In both the third and fourth modes, the audio output device is located around the user. This is a mode that emits ambient sounds. In the third mode and the fourth mode, the output level of the sound surrounding the user emitted by the audio output device is different. The information processing apparatus according to feature 7.

12. In the first mode and the second mode, the display of the display device differs depending on the control of the display of the captured image of the user's surroundings, or the control of the transparency level of the display surface of the display device. The information processing apparatus according to any one of claims 1 to 4.

13. The display device has a display surface that transmits light, The user is able to see their surroundings through the display surface. The transparency level of the display surface differs between the first mode and the second mode. The information processing apparatus according to feature 12.

14. One of the first mode and the second mode is a mode that displays an image including the captured image, The other of the first mode and the second mode is a mode in which the image including the captured image is not displayed. The information processing apparatus according to feature 12.

15. Both the first mode and the second mode are display modes that display a display image obtained by combining computer graphics (CG) with the captured image, In the first mode and the second mode, the ratio of the area of ​​the captured image to the area of ​​the displayed image is different. The information processing apparatus according to feature 12.

16. A control method for an information processing device that is connected to a display device worn on the user's head and controls an audio output device, The display device has a display mode including a first mode and a second mode in which the visibility of the user's surroundings differs from each other, or whether the user's surroundings are visible or not differs from each other, and the audio output device has an audio output mode including a third mode and a fourth mode in which the audibility of the sounds around the user differs from each other. The control method described above is A first determination step of determining whether the display mode has been switched from the first mode to the second mode, When it is determined that the display mode has been switched from the first mode to the second mode, a first switching step is performed to switch the audio output mode from the third mode to the fourth mode, A second determination step of determining that the user has removed the display device or has stopped viewing the display surface of the display device, A second switching step is performed to switch the audio output mode to the third mode when it is determined that the user has removed the display device or has stopped viewing the display surface, Equipped with A control method characterized by the following:

17. A method for controlling an information processing device that controls a display device worn on the user's head and is connected to an audio output device, The display device has different visibility of the user's surroundings, or the user The device has a display mode including a first mode and a second mode in which the visibility of the surrounding environment is different from that of the device, and the audio output device has an audio output mode including a third mode and a fourth mode in which the audibility of the sounds surrounding the user is different from that of the device. The control method described above is A determination step to determine whether the audio output mode has been switched from the third mode to the fourth mode, The system includes a switching step of switching the display mode from the first mode to the second mode when it is determined that the audio output mode has been switched from the third mode to the fourth mode. A control method characterized by the following:

18. A program for causing a computer to function as one of the means of an information processing apparatus according to any one of claims 1 to 4.

Images