Information processing device, information processing method and program

Abstract

To make outside sound audible when it is preferable to be able to hear the outside sound, even if a viewer is experiencing virtual reality and composite reality.SOLUTION: An information processing device acquires mixed information on a real image of a real space and a virtual image, and virtual sound. Then, the information processing device acquires a mixed ratio of the virtual sound and outside sound in the real space, based on the mixed information, and adjusts relative volume between the virtual sound and the outside sound based on the mixed ratio.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention relates to information processing technology for processing sounds when experiencing virtual reality or mixed reality.

Background Art

[0002] As a form of display device, there is a head-mounted device (head-mounted display, hereinafter referred to as HMD) that can be worn on the head to view images so that viewers can enjoy images full of a sense of presence. As applications of the HMD, there are VR (Virtual Reality) applications that display only images of the virtual reality world, and MR (Mixed Reality) applications that synthesize images of the virtual reality world with images of the real world around the viewer, etc.

[0003] In addition, there are sound output devices such as headphone devices and earphone devices with a noise cancellation function that can adjust the intensity of external sounds. Hereinafter, noise cancellation will be referred to as "NC", and headphone devices and earphone devices with an NC function will be referred to as "NC earphones". According to NC earphones, it is also possible to substantially block external sounds. For example, when NC earphones are used in VR applications, by blocking external sounds from the surroundings, viewers can easily obtain a sense of immersion. On the other hand, in the case of MR applications, depending on the content displayed on the HMD, it may be desirable to hear the sounds in the surrounding real space.

[0004] Patent Document 1 describes a method that enables detection of a talking operation from a person in the surroundings for a viewer whose visual and auditory information from the surroundings is restricted by an HMD and NC earphones, and adjusts so that external sounds can be heard when the talking operation is detected. By adjusting so that external sounds can be heard, viewers can easily respond to conversations from the surroundings.

Prior Art Documents

Patent Documents

[0005] [Patent Document 1] Japanese Patent Publication No. 2017-69687 [Overview of the Initiative] [Problems that the invention aims to solve]

[0006] However, in the technology described in Patent Document 1, if no speaking action is detected during viewing, external sounds will not be heard.

[0007] Therefore, the present invention aims to ensure that external sounds can be heard when it is preferable for the viewer to hear external sounds, even when they are experiencing virtual reality or mixed reality. [Means for solving the problem]

[0008] The information processing apparatus of the present invention is characterized by comprising: information acquisition means for acquiring mixed information of a real image and a virtual image in real space; virtual sound acquisition means for acquiring a virtual sound; ratio acquisition means for acquiring the mixing ratio of the virtual sound and external sound in real space based on the mixed information; and adjustment means for adjusting the relative volume between the virtual sound and the external sound based on the mixing ratio. [Effects of the Invention]

[0009] According to the present invention, even when viewers are experiencing virtual reality or mixed reality, external sounds will be audible when it is preferable for them to hear them. [Brief explanation of the drawing]

[0010] [Figure 1] This figure shows an example of the hardware configuration of an information processing device. [Figure 2] This figure shows an example of how an HMD (Head-Mounted Display) is used for mixed reality (MR) applications. [Figure 3] This figure shows the functional configuration of the information processing device in the first embodiment. [Figure 4] This is a flowchart of the information processing in the first embodiment. [Figure 5]This is a flowchart of the information processing in the second embodiment. [Figure 6] This is a detailed flowchart from spatial information acquisition to external sound synthesis. [Figure 7] This is a flowchart of the information processing in the third embodiment. [Figure 8] This is a flowchart of the information processing in the fourth embodiment. [Figure 9] This is a flowchart of the information processing in the fifth embodiment. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. The following embodiments are not limiting to the present invention, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. The configuration of the embodiments may be modified or changed as appropriate depending on the specifications of the apparatus to which the present invention is applied and various conditions (usage conditions, usage environment, etc.). Furthermore, parts of each embodiment described later may be combined as appropriate. In each of the following embodiments, the same components and processes will be denoted by the same reference numerals.

[0012] <Hardware configuration of the information processing device according to the first embodiment> The hardware configuration of the information processing device of the first embodiment will be described below with reference to Figure 1. The CPU 101 uses RAM 102 as work memory to execute programs stored in ROM 103 or hard disk drive (HDD) 105, and controls the operation of each block described later via the system bus 114. The programs executed by the CPU 101 include information processing programs according to this embodiment, as described later. The HDD interface (hereinafter referred to as "I / F") 104 connects to secondary storage devices such as HDD 105 and optical disc drives. HDDI / F 104 is an I / F such as Serial ATA (SATA). The CPU 101 can read data from and write data to HDD 105 via HDDI / F 104. Furthermore, the CPU 101 can load data stored in HDD 105 into RAM 102, and conversely, it can save data loaded into RAM 102 to HDD 105. The CPU 101 can then execute the programs loaded into RAM 102.

[0013] Input I / F 106 connects to input devices 107 such as a keyboard, mouse, digital camera, scanner, and accelerometer. Input I / F 106 can also connect to a stereo camera equipped on a head-mounted display (HMD) as an input device 107. Input I / F 106 is a serial bus interface such as USB or IEEE 1394. The CPU 101 can read data from the input devices 107 via Input I / F 106. Output I / F 108 connects the information processing device 100 to the display device 109, which functions as an HMD. Output I / F 108 is an image output interface such as DVI or HDMI (registered trademark). The CPU 101 can display images from the virtual reality world or mixed reality world on the HMD by sending these images to the HMD via Output I / F 108.

[0014] The audio input I / F 110 connects an audio input device 111 capable of collecting sounds such as a microphone or a directional microphone. The audio input I / F 110 is a serial bus I / F such as USB or IEEE1394, for example. The audio output I / F 112 connects an audio output device 113 that outputs sounds such as a headset device or a speaker. The connection between the audio output I / F 112 and the audio output device 113 may be not only a wired connection but also a wireless connection. The audio output device 113 may have a configuration separate from the information processing device 100, and the information processing device 100 can control the audio output device 113 by controlling it or transmitting a control signal to it. As a result, the CPU 101 of the information processing device can integrally control the HMD and the audio output device 113 (such as a headset device). The audio output I / F 110 is a serial bus I / F such as USB or IEEE1394, for example. The CPU 101 sends synthesized sound data obtained by synthesizing external ambient sound data in the real world around the viewer with virtual sound data in the virtual reality world or virtual sound data in the virtual reality world to a headset device or the like via the audio output I / F 112, and causes the headset device or the like to output those sounds.

[0015] Note that the information processing device 100 may not include the HDD 105 or the display device 109. Also, the information processing device 100 may or may not include an HMD. When the information processing device 100 does not include an HMD, the information processing device 100 receives data from the HMD and sends data to the HMD by connecting to an external HMD through the input I / F 106. The components of the information processing device 100 may include configurations other than the components described above, but the illustration and description thereof are omitted here.

[0016] Here, before explaining the detailed operations and processes of the information processing device 100, an example in the case where an image of a Mixed Reality (MR) world in which an image of the virtual reality world is synthesized with an image of the surrounding real space is provided to the viewer will be explained using FIG. 2.

[0017] As shown in Fig. 2(A), assume that the viewer is wearing the HMD1, which is a head-mounted device, and the headset device 2. The HMD1 is equipped with a camera (imaging device), not shown in the figure, and the camera acquires real image data obtained by imaging the real space around the viewer. The HMD1 is also equipped with a microphone, and it is assumed that the microphone can acquire ambient sound data, which is the sound of the real space around the viewer. Note that the microphone does not necessarily have to be mounted on the HMD1. In that case, the microphone is connected to the HMD or the information processing device 100 by wire or wirelessly.

[0018] The image 3 in Fig. 2(B) shows a real image (referred to as real image 3) obtained by the camera of the HMD imaging the real space. The HMD1 also acquires a virtual image 4. The virtual image 4 is a virtual reality image, and hereinafter will be referred to as VR image 4. Then, the HMD1 displays a composite reality image 5 (referred to as MR image 5), which is a composite of the real image 3 and the VR image 4, on the built-in display as a display image. As a result, the viewer wearing the HMD1 can view the MR image 5 in which the real image 3 and the VR image 4 are combined, that is, can experience a composite reality feeling. Note that Fig. 2 shows an example of an MR application in which the HMD1 displays the MR image 5. However, when the HMD1 is used for a virtual reality (VR) application, only the VR image of the virtual reality world is displayed as the display image on the HMD1.

[0019] Headphone device 2 is an audio output device worn by the viewer on their ears. The audio output device may also be an earphone device. Here, if HMD1 is used to display only images of the virtual reality world (VR images), outputting virtual sound data, which is the sound of the virtual reality world (hereinafter referred to as VR sound), from headphone device 2 makes it easier for the viewer to immerse themselves in the virtual reality world. In particular, when displaying only VR images, if information processing is performed to block out external sounds, for example, by using a noise cancellation function (NC function), the viewer can easily achieve a sense of immersion in the virtual reality world. On the other hand, if HMD1 is used to combine images of the virtual reality world (VR images) with images of the real world (real images) surrounding the viewer, depending on the content of the displayed image on HMD1, it is often desirable to hear external sounds from the surrounding real space.

[0020] The following describes an information processing device of a first embodiment that, when used to display only VR images, allows viewers to easily gain a sense of immersion in the virtual reality world, while when used to display both VR images and real images, allows external sounds from the surrounding real space to be heard in accordance with the display on the HMD1.

[0021] <Functional configuration of the information processing device according to the first embodiment> Figure 3 is a functional block diagram showing the functional configuration of the information processing device 100 according to the first embodiment. As shown in Figure 3, the information processing device 100 includes a VR image acquisition unit 11, a mixed information acquisition unit 12, a real image acquisition unit 13, an output image acquisition unit 14, a mixing ratio calculation unit 15, a VR sound acquisition unit 16, an external sound acquisition unit 17, a sound synthesis unit 18, and an output unit 19. Each of these functional units is realized by the CPU 101 executing the information processing program according to this embodiment, but some or all of each functional unit may be realized by hardware configurations such as circuits. In the following description, unless otherwise specified, image data handled by the information processing device 100 will be simply referred to as "images," and sound data handled by the information processing device 100 will be simply referred to as "sound" (or "external sound" in the case of external sound data). The information processing device 100 will also be simply referred to as the information processing device.

[0022] The VR image acquisition unit 11 is a virtual image acquisition unit that acquires VR images, which are virtual image data generated by rendering, or pre-prepared VR images. Since the VR image acquisition unit 11 is a functional unit configured by the CPU 101 executing the information processing program of this embodiment, VR image rendering and the like are performed by the CPU 101, but for example, a GPU (not shown) may also perform these tasks. The VR image acquisition unit 11 outputs the acquired VR images to the output image acquisition unit 14. Details of the operation of the VR image acquisition unit 11 will be described later.

[0023] The real image acquisition unit 13 acquires a real image of the real space captured by the camera (imaging device). The real image acquisition unit 13 then outputs the real image to the output image acquisition unit 14. In this embodiment, the camera is assumed to be included in the input device 107.

[0024] The mixed information acquisition unit 12 outputs the mixed information to the output image acquisition unit 14 and the mixed ratio calculation unit 15. In this embodiment, mask image data (referred to as a mask image) is used as the mixed information. The mask image is used to specify the range of the VR image that is superimposed on the real image captured using a camera in real space. Details of the mixed information and mask image in this embodiment will be described later.

[0025] The mixing ratio calculation unit 15 is a ratio acquisition unit that calculates the mixing ratio between the real image and the VR image based on the mixing information acquired by the mixing information acquisition unit 12. The mixing ratio calculation unit 15 outputs the calculated mixing ratio information to the sound synthesis unit 18. Details such as examples of mixing ratios will be described later.

[0026] The output image acquisition unit 14 synthesizes the VR image and the real image based on the mixed information acquired by the mixed information acquisition unit 12, and acquires the resulting synthesized image data (referred to as the synthesized image) as the output image. Since the output image acquisition unit 14 is a functional unit configured by the CPU 101 executing an information processing program, the generation of the synthesized image is performed by the CPU 101, although this may be performed by a GPU (not shown), for example. The output image acquisition unit 14 then outputs the resulting image to the output unit 19.

[0027] The VR sound acquisition unit 16 is a virtual sound acquisition unit that generates VR sound, which is virtual reality sound data, or acquires VR sound that has been prepared in advance and associated with a VR image. Since the VR sound acquisition unit 16 is a functional unit that is composed of the CPU 101 executing an information processing program, the generation of VR sound is performed by the CPU 101, but it may also be performed by a GPU (not shown). The VR sound acquisition unit 16 then outputs the VR sound to the sound synthesis unit 18. Details of the operation of the VR sound acquisition unit 16 will be described later.

[0028] The external sound acquisition unit 17 acquires external sounds from the surrounding real space using a sound input device 111 such as a microphone. The external sound acquisition unit 17 then outputs the acquired external sounds to the sound synthesis unit 18. The sound synthesis unit 18 adjusts the relative volume between the external sound and the VR sound based on the mixing ratio information, and outputs the adjusted sound to the subsequent output unit 19. As will be described in detail later, the sound synthesis unit 18 performs processing to adjust the sound so that the viewer can hear the external sound, or to adjust it to reduce the external sound, based on the mixing ratio information. Sound adjustment based on the mixing ratio includes generating a synthesized sound by directly combining the external sound with the VR sound, generating a synthesized sound by adjusting it to reduce the external sound and then combining it with the VR sound, and adjusting the volume of the VR sound so that the external sound in the real world can be heard (lowering the volume). In other words, the sound synthesis unit 18 adjusts the relative volume between the external sound and the VR sound (hereinafter referred to as sound adjustment) based on the mixing ratio information, and outputs the adjusted sound to the subsequent output unit 19. Adjustment to reduce external sound includes adjustments that almost completely block out external sound by so-called noise cancellation. Furthermore, if adjustments are made to the volume of VR sound based on the mixing ratio, for example, by lowering the volume of external sounds, the volume of external sounds will become relatively louder compared to the VR sound, making it easier for the user to hear external sounds.

[0029] The output unit 19 outputs the output image from the output image acquisition unit 14 to the display device 109 (HMD1 in this embodiment). The output unit 19 also outputs the synthesized sound from the sound synthesis unit 18 to the sound output device 113 (headphone device 2 in this embodiment).

[0030] The aforementioned VR image, mask image (mixing information), mixing ratio, output image, and VR sound data may be acquired in advance and stored on HDD105 or similar, and the information processing device may read them from HDD105 as needed. Alternatively, the information processing device may acquire the VR image, mask image, output image, mixing ratio, and VR sound data from a cloud (not shown) via a communication device (not shown) as needed.

[0031] In this embodiment, the VR image, mask image, real image, and output image are assumed to be images of the same resolution, but they do not necessarily have to be the same resolution. Each of these images may be held at a different resolution, for example, and the information processing device may adjust the resolution to match during calculations by scaling or other processes. Also, while the VR sound and external sound are assumed to be sound data with the same sample rate, they do not necessarily have to be at the same sample rate; they may be combined after being made to the same sample rate through interpolation or resampling processes.

[0032] Furthermore, while VR images, real images, and output images are assumed to be 3-channel image data of the three primary colors RGB, they do not necessarily have to be 3 channels. For example, they could be 1-channel monochrome image data, or 5 channels including color difference, etc. Furthermore, while external sounds, VR sounds, and synthesized sounds are assumed to be 1-channel monaural data, they do not necessarily have to be 1-channel; 2-channel stereo data or even 5-channel or other spatial audio data are also acceptable.

[0033] <Information processing according to the first embodiment> Figure 4 is a flowchart showing the information processing flow performed by the information processing device of the first embodiment. Hereafter, each processing step (process) will be represented by adding "S" before the symbol, and the description of the processing step will be omitted. In S101, the VR image acquisition unit 11 acquires a VR image. As mentioned above, a VR image is virtual reality image data, an image that represents something different from the real world surrounding the viewer. VR images can be, for example, images created by rendering three-dimensional model data from a virtual viewpoint using known rendering techniques, or images created by taking pictures of the real world, such as a different location or time, with a camera, etc.

[0034] Next, in S102, the VR sound acquisition unit 16 acquires VR sound. As mentioned above, VR sound is sound data from the virtual reality world. VR sound is generated as sound heard at the listening point (virtual viewpoint) based on the positional relationship, including the distance and direction, between the position of the virtual sound source set in the three-dimensional model data and the virtual listening point (virtual viewpoint in this embodiment). If the VR image is taken in a situation different from the real world surrounding the viewer, the VR sound will be, for example, sound acquired by collecting ambient sounds in that situation using a microphone or the like. In addition, sounds such as background music may be used directly as VR sound.

[0035] Next, in S103, the mixed information acquisition unit 12 acquires a mask image as mixed information. The mask image is an image in which the pixel values ​​of the pixel positions to be superimposed on the real image as a VR image are set to 1, and the pixel values ​​of the other pixel positions are set to 0. In other words, in the mask image, the image area with a pixel value of 1 corresponds to the VR image, and the image area with a pixel value of 0 corresponds to the image other than the VR image.

[0036] Next, in S104, the mixing ratio calculation unit 15 calculates the mixing ratio based on the mixing information acquired by the mixing information acquisition unit 12 in S103. Then, the mixing ratio calculation unit 15 sets the value of a predetermined flag based on that mixing ratio. For example, if there is a proportion in the mixing ratio that corresponds to the real image, that is, if there is an image region in the mask image that corresponds to the real image, the mixing ratio calculation unit 15 sets a predetermined flag to 1, which indicates whether or not to synthesize the real image. On the other hand, if there is no proportion in the mixing ratio that corresponds to the real image, that is, if there is no image region in the mask image that corresponds to the real image, the mixing ratio calculation unit 15 sets a predetermined flag to 0. In other words, the mixing ratio calculation unit 15 sets a predetermined flag to 1 if there is an image region other than the VR image in the mask image, and sets a predetermined flag to 0 if all image regions are regions that correspond to the VR image. In the following explanation, this predetermined flag will be called the MR flag.

[0037] Next, in S105, the information processing device determines whether the MR flag is 1 or 0. If the MR flag is 1, the processing of the information processing device proceeds to S106 or later. On the other hand, if the MR flag is 0, the processing of the information processing device proceeds to S108 or later.

[0038] If the MR flag is determined to be 0 in S105 and the process proceeds to S108, the external sound acquisition unit 17 acquires external sounds from the surrounding real space using a sound input device 111 such as a microphone. Next, in S109, the sound synthesis unit 18 synthesizes the external sound acquired by the external sound acquisition unit 17 and the VR sound acquired by the VR sound acquisition unit 16, but performs sound adjustment processing to reduce the external sound. In this embodiment, a known noise cancellation method is used as the method to reduce the external sound. That is, in the external sound synthesis processing in S109, sound adjustment processing is performed to almost completely block the external sound by adding a sound with the opposite phase to the external sound acquired in S108, and then the sound is synthesized with the VR sound. As a result, the external sound and the sound with the opposite phase cancel each other out, and the external sound is almost completely blocked. The sound synthesis unit 18 then outputs the sound after the sound adjustment processing, that is, the synthesized sound which has had the external sound blocked and consists almost only of the VR sound, to the output unit 19.

[0039] Then, after processing in S109, the process proceeds to S110, where the output unit 19 outputs the output image to the display device 109, i.e., the HMD, and outputs the synthesized sound to the sound output device 113, i.e., the headphone device. In other words, if the MR flag is determined to be 0 in S105 and processing proceeds from S108 onwards, then the process proceeds to S110, the HMD displays an output image consisting only of VR images, and the headphone device outputs synthesized sound consisting only of VR sounds after external sounds have been blocked out. Note that it is not necessarily required to consist only of VR sounds; the VR sounds should be relatively stronger than other sounds.

[0040] On the other hand, if the MR flag is determined to be 1 in S105 and the process proceeds to S106, the real image acquisition unit 13 acquires a real image of the real space captured by the camera, and the external sound acquisition unit 17 acquires external sounds of the real space collected by the sound input device 111 such as a microphone.

[0041] Next, in S107, the output image acquisition unit 14 generates a composite image as the output image by combining the real image acquired by the real image acquisition unit 13 and the VR image acquired by the VR image acquisition unit 11. In other words, the output image at this time is an MR image, which is a composite of the real image and the VR image. When combining the real image and the VR image, the output image acquisition unit 14 acquires a mask image from the mixed information acquisition unit 12 and multiplies the VR image by this mask image at each pixel position. Furthermore, the output image acquisition unit 14 adds the result of this multiplication to the real image at each pixel position. As a result, the VR image within the range specified to be superimposed by the mask image is superimposed onto the real image. At this time, the output image acquisition unit 14 may perform known blending processing to smooth the transition at the boundary between the real image and the VR image.

[0042] In S107, the sound synthesis unit 18 generates a synthesized sound as the output sound by combining the external sound acquired by the external sound acquisition unit 17 and the VR sound acquired by the VR sound acquisition unit 16. In other words, the output sound at this time is a sound (referred to as MR sound) which is a combination of the collected external sound as is and the VR sound. Thus, if the MR flag is 1 in S105, the sound adjustment process to reduce the external sound in S109 is not performed, and an MR sound, which is the external sound combined with the VR sound as is, is output. In addition, in S107, the sound synthesis unit 18 may perform sound adjustment, for example, by amplifying the collected external sound and then synthesizing it with the VR sound. That is, as long as both the external sound and the VR sound can be heard, the form of sound synthesis is not limited.

[0043] After processing in S107, when the process proceeds to S110, the output unit 19 outputs an output image (MR image) which is a composite of the VR image and the real image to the HMD's display device 109. Also, if the process proceeds from S107 to S110, the output unit 19 outputs an output sound (MR sound) which is a composite of the VR sound and the ambient sound to the headphone device worn by the user. In other words, if the MR flag is determined to be 1 in S105 and the process proceeds from S106 onwards, and the process proceeds to S110, the headphone device outputs a composite sound that includes not only the VR sound but also the ambient sound. As a result, the viewer can hear the VR sound output from the headphone device while also hearing the ambient sounds of the surrounding real space.

[0044] Subsequently, in S111, the information processing device determines whether or not to terminate the processing of this flowchart. If it does not terminate, it returns to S101. For example, if a viewer is watching a video, processing corresponding to the next frame will be performed in S101. On the other hand, if it determines that the processing should be terminated, for example, by a termination command from the viewer, the information processing device terminates the processing of this flowchart.

[0045] As described above, according to the information processing device of the first embodiment, when only VR images are displayed on the HMD, external sounds can be almost completely blocked, allowing viewers to easily immerse themselves in the virtual reality world. On the other hand, when an image combining VR images and real images is displayed on the MMD, external sounds are not reduced, allowing viewers to hear ambient sounds. Furthermore, the information processing device of the first embodiment automatically switches whether or not to block external sounds depending on whether only VR images are displayed on the HMD or whether an image combining VR images and real images is displayed. Therefore, according to the first embodiment, viewers do not need to perform any operation to switch whether or not to block external sounds themselves, saving them time and effort.

[0046] In Figure 4, when a composite image of a VR image and a real image is displayed, the viewer can hear both VR sound and ambient sound. On the other hand, when only a VR image is displayed, ambient sound is largely blocked by noise cancellation, allowing the viewer to hear only VR sound. This embodiment is not limited to this example. For example, if an earphone device is used that allows ambient sound to reach the viewer's ears when the noise cancellation function is off, the ambient sound acquisition process in S106 and the ambient sound synthesis process in S107 of Figure 4 do not need to be performed. In this example as well, when a composite image of a VR image and a real image is displayed, the viewer can hear the VR sound output from the earphone device while also hearing ambient sound from the surrounding real space.

[0047] Furthermore, in the first embodiment, an example was given in which the mixed information acquisition unit 12 acquires a mask image as mixed information in S103, but this embodiment is not limited to this example. For example, other information besides a mask image may be used as long as it is possible to determine whether the HMD is for VR use, which displays only VR images, or for MR use, which displays a composite image of VR images and real images. For example, if the viewing application programs that run on the HMD are divided into VR and MR versions, and it is possible to distinguish between these uses, the mixed information acquisition unit 12 may acquire information indicating which use the application program is for as mixed information. In this case, the mixed information acquisition unit 12 may acquire information indicating which type of application program it is as mixed information. In this case, in S105, the mixed ratio calculation unit 15 sets the MR flag to 0 if the application program is for VR use, and sets the MR flag to 1 if the application program is for MR use.

[0048] Furthermore, in the first embodiment, an example was described in which external noise is almost completely blocked by noise cancellation, which adds sound with the opposite phase to the external sound. However, external noise may be blocked by methods other than noise cancellation. For example, the viewer's ears may be covered with an object made of a material that does not easily transmit external sound, and then, when the MR flag is 1, external sound may be collected with a microphone or the like in S106, and this external sound may be synthesized into VR sound in S107. In this example, the processes in S108 and S109 of the flowchart in Figure 4 are unnecessary.

[0049] Furthermore, while the first embodiment described an example of reducing ambient noise when only VR images are displayed, it is not always necessary to reduce ambient noise in this case; the sound may be adjusted so that the viewer can hear ambient noise. This is because, for example, when the viewer moves, they may want to be aware of their surroundings. In this case, for example, in S103, the mixed information acquisition unit 12 acquires the value of an acceleration sensor mounted on the HMD, for example, and includes it in the mixed information. Then, in S104, the mixed ratio calculation unit 15 calculates the mixed ratio taking into account the value of the acceleration sensor. As a result, for example, in S109, the sound synthesis unit 18 can synthesize ambient noise into the VR sound based on the value of the acceleration sensor. That is, for example, if the sound synthesis unit 18 determines from the value of the acceleration sensor that the viewer has moved, it generates a synthesized sound without making any adjustments to reduce ambient noise. As a result, the viewer can be aware of their surroundings by ambient noise when they move.

[0050] Furthermore, determining whether or not the viewer is active can be done, for example, by training a classifier that identifies whether or not the viewer is active from the values ​​of the accelerometer using a known machine learning process. In this case, the mixing ratio calculation unit 15 uses the classifier to calculate the mixing ratio, including a flag indicating whether or not the viewer is active. Then, if the sound synthesis unit 18 identifies that the viewer is active based on the flag included in the mixing ratio, it refrains from synthesizing a sound that is in the opposite phase to the external sound into the VR sound. On the other hand, if the sound synthesis unit 18 identifies that the viewer is not active based on the flag included in the mixing ratio, it reduces the external sound by synthesizing a sound that is in the opposite phase to the external sound into the VR sound. Note that using the values ​​of the accelerometer as a method for identifying whether or not the viewer is active is not mandatory; other methods may be used as long as the viewer's activity can be detected. For example, a real-world area that the viewer is viewing can be set in advance, and it may be sensed using a known sensing technique when the viewer leaves that area. Then, if the viewer leaves the set real-world area, the external sound may not be reduced, or conversely, the external sound may be reduced.

[0051] In this embodiment, the information processing device handles both image data and sound data. Generally, the sampling rate of image data and the sampling rate of sound data are different, with the sampling rate of sound data often being higher. In this case, the processing of sound data in S108 and S109 and the processing of sound data in S106 and S107 may be looped until the next image data is sampled. Alternatively, the processing of sound data and image data may be performed in separate threads. In this case, the sound synthesis unit 18, which operates in a separate thread from the image data processing, acquires an MR flag. If the MR flag is 1, the external noise is not reduced; if it is 0, the external noise is reduced.

[0052] <Second Embodiment> In the first embodiment, an example was described in which the MR flag is determined to be 1 or 0 based on a mask image which is mixed information, and whether or not to adjust the external sound is switched according to the value of the MR flag. The following second embodiment describes an example in which spatial information is included in the mixed information. In the second embodiment, the spatial information is distance information from the viewer. The distance information in the second embodiment is information representing the distance from the viewer to the sound source in real space. Viewers may want to hear external sounds from nearby sound sources in real space. An example of such a situation is when a viewer is working on something with their hands while looking at a VR image. In such a hypothetical example, it is often desirable for the viewer to be able to hear external sounds from sound sources close to them.

[0053] The information processing device of the second embodiment includes distance information as spatial information in the mixed information to address such situations, and performs sound adjustment based on that distance information. The hardware configuration and functional configuration of the information processing device in the second embodiment are the same as those of the first embodiment. In the second embodiment, the same reference numerals are used for functional configurations and processing steps as in the example of the first embodiment, and their descriptions are omitted. The following description will mainly focus on the parts that differ from the first embodiment. The functional configuration of the information processing apparatus according to the second embodiment is the same as that shown in Figure 3 above. However, in the case of the second embodiment, the mixed information acquisition unit 12 outputs mixed information including spatial information to the output image acquisition unit 14 and the mixed ratio calculation unit 15.

[0054] <Information processing in the second embodiment> Figure 5 is a flowchart showing the flow of information processing performed by the information processing device according to the second embodiment. In the second embodiment, if the MR flag is 0 in S105, the information processing device proceeds to processing from S108 onwards, while if the MR flag is 1, it proceeds to processing from S201 onwards. After processing from S201 to S206, the information processing device proceeds to processing S110.

[0055] In the second embodiment, since the MR flag is 1 in S105, the process proceeds to S201, where the real image acquisition unit 13 acquires a real image of the real space captured by the camera. Next, in S202, the output image acquisition unit 14 generates a composite image by combining the real image acquired by the real image acquisition unit 13 and the VR image acquired by the VR image acquisition unit 11. The compositing process of the real image and the VR image is the same as the image compositing process in S107 described above.

[0056] Next, proceeding to S203, the mixed information acquisition unit 12 acquires distance information as spatial information. Figure 6 is a flowchart showing the details of the process in S203, the subsequent process in S204, and the process in S206 when acquiring distance information. Refer to Figure 6 to explain the detailed processing of S203.

[0057] In S2001, the mixed information acquisition unit 12 acquires distance setting information. Distance setting refers to information set by, for example, the viewer or system configurator as the distance at which external sounds are desirable to hear. In this embodiment, an example is given where the viewer sets the distance. In this example, the information processing device displays a user interface (UI) screen on the HMD, for example, and the mixed information acquisition unit 12 acquires the distance threshold Dth, which is arbitrarily set by the viewer via the input device 107, as distance setting information. The distance threshold Dth set by the viewer as distance setting corresponds to the distance from the HMD to an object in the real world. On the UI screen during distance setting, objects whose distance from the HMD is less than the distance threshold (less than Dth) are displayed, and objects whose distance is greater than or equal to the distance threshold (greater than or equal to Dth) are not displayed. The viewer sets the distance at which external sounds are desirable to hear while looking at the UI screen. Note that the method of setting distance and the definition of distance are not limited to settings using the UI. In cases where an object existing in real space, such as an object on a desk, becomes the sound source, the distance from the HMD to the desk may be measured using known techniques such as stereo matching, and the settings may be based on that measured distance. Furthermore, the distance does not necessarily have to be defined as the distance from the HMD; for example, it may be defined as the distance from the viewer's center of gravity.

[0058] Next, in S2002, the mixed information acquisition unit 12 acquires depth information from the real image. Depth information is information representing the depth to each pixel of an object in the real image, and can be obtained using known techniques such as stereo matching. Alternatively, the mixed information acquisition unit 12 may determine the depth by using a rangefinder such as Lidar (Light Detection and Ranging or Laser Imaging Detection and Ranging).

[0059] Next, proceeding to S204, the mixing ratio calculation unit 15 calculates the mixing ratio. The mixing ratio in the second embodiment is different from that of the first embodiment described above. Refer to Figure 6 to explain the detailed process of S204. In S2003, the mixing ratio calculation unit 15 calculates depth threshold information. Depth threshold information is information stored for each pixel of the actual image, indicating whether the depth to the object corresponding to each pixel is less than the distance threshold (less than Dth) or greater than or equal to the distance threshold (greater than or equal to Dth). The mixing ratio calculation unit 15 records the depth information for each pixel of an object in the actual image acquired in S2002 as 1 if it is less than the distance threshold Dth, and as 0 if it is greater than or equal to the distance threshold Dth, at the corresponding pixel position of the depth information.

[0060] Next, in S2004, the mixing ratio calculation unit 15 calculates the connected components of the depth threshold information. The mixing ratio calculation unit 15 considers adjacent positions in the depth threshold information as a single connected component and stores these connected components in a list. As will be described later, the list of connected components results in a mixing ratio in which external sounds from the region indicated by this connected component are synthesized, and external sounds from other regions are not synthesized (external sounds are reduced). Next, proceeding to S205, the external sound acquisition unit 17 acquires external sound, and then the information processing device proceeds to S206. The external sound acquisition process in the external sound acquisition unit 17 is the same as described in S108 above.

[0061] Proceeding to S206, the sound synthesis unit 18 synthesizes external sounds. The sound synthesis process in S206 differs from the external sound synthesis process in the first embodiment described above. The detailed processing of S206 will be explained with reference to Figure 6. In S2005, the sound synthesis unit 18 refers to the list calculated in S2004 to determine whether there are any connected components for which external sounds have not yet been synthesized. If there are any connected components for which external sounds have not yet been acquired, the sound synthesis unit 18 selects one of those connected components from the list and performs the subsequent processing in S2006 on that connected component. In other words, the processing in S206 is a loop process, and if there are no connected components in the list for which external sounds have not yet been synthesized, the information processing device proceeds to S110.

[0062] When the process proceeds to S2006, the sound synthesis unit 18 acquires the external sound at the selected connected component position from the external sound acquisition unit 17. As a method for acquiring the external sound at the connected component position, for example, a directional microphone can be used, and the external sound acquisition unit 17 can acquire the external sound from that directional microphone. Alternatively, for example, multiple microphones can be installed, and the sound source position can be estimated using a known sound source estimation method that estimates the sound source position from the phase difference between each microphone. If there is sound at the position indicated by the connected component, the sound at that position can be acquired.

[0063] In the second embodiment, when synthesizing external sound in S206, it is not necessary to synthesize the external sound at its original intensity. For example, in S204, instead of depth threshold information, the intensity may be adjusted using a distance-based gain, and the sound may be synthesized at the intensity adjusted based on that gain. In this case, the connected component may be obtained by calculating the connected component of pixels whose value is not zero.

[0064] In the second embodiment, when displaying only VR images on the HMD, external sounds are largely blocked by noise cancellation, similar to the example of the first embodiment described above. However, the viewer's ears may be physically covered, so that the viewer cannot hear external sounds. In this case, the external sound acquisition process at S108, the sound adjustment and external sound synthesis processes at S109 in the flowchart of Figure 5 are unnecessary.

[0065] As described above, the information processing device of the second embodiment includes distance information as spatial information in the mixed information, and can perform sound adjustment based on that distance information. In other words, in the case of the second embodiment, the mixing ratio calculation unit 15 calculates a mixing ratio that reduces external sounds from sound sources above a distance threshold in real space, while calculating a mixing ratio that does not reduce external sounds from sound sources below a distance threshold. This makes it possible for the listener to hear nearby sounds in real space. Also, in the second embodiment, as in the first embodiment described above, the listener is saved the trouble of having to adjust the external sound themselves.

[0066] <Third Embodiment> In the second embodiment, an example was described in which distance information as spatial information is included in the mixed information. In the third embodiment, an example is described in which the transparency information of the VR image is included in the mixed information. For example, even when displaying a VR image, viewers may want to temporarily check their surroundings. For instance, when a viewer moves, they may come into contact with other people approaching. In such cases, it is preferable for the viewer to be able to check their surroundings. Therefore, the information processing device of the third embodiment includes the transparency information of the VR image in the mixed information and adjusts the external sound based on that transparency information. The hardware and functional configurations of the information processing device in the third embodiment are the same as those of the first embodiment described above. In the third embodiment, the same functional configurations and processing steps as in the first and second embodiments are denoted by the same reference numerals and their descriptions are omitted. The following description will mainly focus on the differences.

[0067] <Functional configuration of the information processing device according to the third embodiment> The functional configuration of the information processing apparatus according to the third embodiment is the same as that shown in Figure 3 above. In the third embodiment as well, the mixed information acquisition unit 12 outputs the mixed information to the output image acquisition unit 14 and the mixed ratio calculation unit 15. In the third embodiment, the mixed information includes transparency information of the VR image. <Information processing in the third embodiment> Figure 7 is a flowchart showing the information processing flow executed by the information processing device of the third embodiment. In the third embodiment, if the MR flag is 1 in S105, the information processing device proceeds to processing from S301 onwards, while if the MR flag is 0, it proceeds to processing from S108 onwards. After processing from S301 to S306, the information processing device proceeds to processing S110.

[0068] When the MR flag is determined to be 1 and the process proceeds to S301, the mixed information acquisition unit 12 acquires the transparency information included in the mixed information. The transparency information is, for example, the value of the transparency set by the viewer or system setter. In this embodiment, the mixed information acquisition unit 12 acquires the transparency value set by the viewer, for example, via the input device 107, as the transparency information. In this embodiment, the transparency value Tv is used as the transparency information, and the transparency value Tv is assumed to be a value between 0 and 1.

[0069] Next, proceeding to S302, the mixing ratio calculation unit 15 calculates the mixing ratio based on the transmission information (transparency Tv). In the third embodiment, the transmission Tv is used directly as the mixing ratio. However, the transmission Tv does not necessarily have to be used as the mixing ratio. For example, the transmission Tv raised to a power may be used as the mixing ratio. In this case, the information processing device treats the transmission Tv raised to a power as if it were substituted for the transmission Tv, and the synthesis described later is performed.

[0070] Next, in S303, the real image acquisition unit 13 acquires a real image of the real space surrounding the viewer. Next, in S304, the output image acquisition unit 14 combines the VR image acquired by the VR image acquisition unit 11 and the real image acquired by the real image acquisition unit 13 according to the mixing ratio calculated in S302. At this time, the output image acquisition unit 14 multiplies the RGB values ​​of each pixel of the real image by the transparency Tv. The output image acquisition unit 14 also multiplies the RGB values ​​of each pixel of the VR image by (1-Tv). After that, the output image acquisition unit 14 combines the real image and the VR image after these multiplications to generate a composite image.

[0071] Next, in S305, the external sound acquisition unit 17 acquires external sounds. Next, in S306, the sound synthesis unit 18 synthesizes the external sound acquired by the external sound acquisition unit 17 and the VR sound acquired by the VR sound acquisition unit 16 to generate a synthesized sound. At this time, the sound synthesis unit 18 adjusts the sound such that the intensity of the external sound is multiplied by the transparency Tv, and then synthesizes the adjusted external sound with the VR sound. Note that when adjusting the intensity of the external sound, it is not necessary to multiply by Tv, but another predetermined constant may be used instead. In addition, the sound synthesis unit 18 may determine the maximum value of the sound adjustment based on the transparency Tv, and apply gain so that the maximum value becomes the maximum volume of the external sound.

[0072] As described above, the information processing device of the third embodiment can adjust external sound based on the transparency information included in the mixed information. In other words, in the third embodiment, the mixing ratio calculation unit 15 obtains a mixing ratio according to the transparency information. As a result, in the third embodiment, the viewer can hear the surrounding external sounds. Furthermore, in this embodiment, since the external sound is adjusted according to the transparency of the VR image, the effort required of the viewer to adjust the external sound can be reduced.

[0073] In the third embodiment, an example was given in which the viewer or system configurator sets the transparency information, but the embodiment is not limited to this example. For example, the mixed information acquisition unit 12 may acquire the amount of movement when the viewer is moving, that is, the amount of movement of the HMD, and automatically set the transparency according to the acquired amount of movement. Here, the amount of movement when the viewer is moving (the amount of movement of the HMD) can be determined, for example, from the output of the acceleration sensor. In this example, the mixed information acquisition unit 12 acquires transparency information in which the transparency increases as the amount of movement increases. The mixed ratio calculation unit 15 then calculates a mixed ratio such that the amount of external sound reduction decreases as the transparency of the transparency information increases, or in other words, a mixed ratio such that the amount of external sound reduction increases as the transparency of the transparency information increases. Alternatively, the mixed information acquisition unit 12 may acquire transparency information in which the transparency decreases as the amount of movement decreases. In this case, the mixed ratio calculation unit 15 calculates a mixed ratio such that the amount of external sound reduction increases as the transparency of the transparency information decreases, or in other words, a mixed ratio such that the amount of external sound reduction decreases as the transparency of the transparency information decreases.

[0074] In a third embodiment, for example, a designated area may be predetermined in the virtual reality space or the real world, and when the viewer leaves that designated area, the mixed information acquisition unit 12 may acquire predetermined transparency information. For example, when a viewer is experiencing a VR game using an HMD, a system has been put into practical use that determines the area in the real world where the VR game is played, and interrupts the VR game when the viewer leaves that area. In this way, when the viewer leaves the area set in the real world, for example, by making the VR game image transparent and displaying the real image, the viewer can check both the situation in the real world and the situation in the virtual reality world. And in situations where the viewer is checking the surroundings, it is desirable that external sounds can also be heard at the same time. To address this hypothetical example, the mixed information acquisition unit 12 may, for example, acquire transparency information that makes the VR image transparent when the viewer of the HMD leaves the designated area in the real world, and acquire transparency information that does not make the VR image transparent when the viewer is within the designated area. In this case, when the mixed ratio calculation unit 15 acquires transparency information that does not make the VR image transparent, it calculates a mixed ratio that reduces external sounds.

[0075] In the third embodiment, an example was shown in which the transparency of the VR image is set and then composited with the real image. However, as another example, the transparency of the real image may be set and then composited with the VR image. Furthermore, the transparency Tv does not necessarily have to be a value between 0 and 1. For example, it may be set to a value between 0 and 100, and after processing in S302, it may be scaled by dividing by the transparency Tv of 100. Furthermore, the transparency information does not necessarily have to be a numerical value representing the transparency of the entire VR image; for example, it could be a transparent image with the same resolution as the real image. In this case, for example, each pixel value could represent the transparency of each transparent image. Then, the transparency Tv could be a statistical value of those transparent images. For example, the statistical value could be the mean, median, maximum, minimum, etc., of the transparent images.

[0076] In the third embodiment, as in the example of the first embodiment described above, when the MR flag is 0 and only VR images are displayed and real images are not displayed, external sounds are largely blocked by noise cancellation. In contrast, as explained in the second embodiment, for example, the viewer's ears may be physically covered so that the viewer cannot hear external sounds. Also in the third embodiment, for example, if a headphone device is used that allows external sounds from the surrounding real space to reach the viewer's ears when noise cancellation processing to reduce external sounds is not performed, the external sound acquisition processing in S106 and the external sound synthesis processing in S107 may not be performed.

[0077] <Fourth Embodiment> In the second embodiment described above, distance information was used as spatial information, but in the fourth embodiment, an example in which directional information is used as spatial information will be explained. Viewers sometimes want to check the situation in the real world in a specific direction while also hearing external sounds from that direction. For example, by displaying a real image in the direction where a monitored object such as a baby or pet is located, while displaying a VR image in other directions, it becomes possible to check the situation of the monitored object in the real world. In this case, it is considered even more desirable to be able to hear sounds such as crying, in addition to seeing the real image of the baby or pet. Therefore, in the fourth embodiment, the directional information from which the real image is captured is used as spatial information.

[0078] The hardware and functional configurations of the information processing device in the fourth embodiment are the same as those of the first embodiment described above, so their explanations will be omitted. The functional configuration of the information processing device is generally the same as that of Figure 3 described above, but in the case of the fourth embodiment, the mixed information acquisition unit 12 acquires direction information in addition to the mixed information and outputs that information to the output image acquisition unit 14 and the mixing ratio calculation unit 15. Thus, in the case of the fourth embodiment, direction information is included in the mixed information. In the following, in the fourth embodiment, the same reference numerals are used for functional configurations and processing steps as in the examples of each embodiment described above, and their explanations will be omitted. The following will mainly describe the differences.

[0079] <Information processing in the fourth embodiment> Figure 8 is a flowchart showing the information processing flow executed by the information processing device of the fourth embodiment. In the fourth embodiment, if the MR flag is 0 in S105, the information processing device proceeds to processing from S108 onwards, while if the MR flag is 1, it proceeds to processing from S401 onwards. After processing from S401 to S406, the information processing device proceeds to processing S110.

[0080] If the MR flag is determined to be 1 and the process proceeds to S401, the real image acquisition unit 13 acquires a real image of the real space surrounding the viewer. Next, in S402, the output image acquisition unit 14 combines the VR image acquired by the VR image acquisition unit 11 with the real image acquired by the real image acquisition unit 13.

[0081] Next, proceeding to S403, the mixed information acquisition unit 12 acquires directional information as spatial information included in the mixed information. Here, the mixed information acquisition unit 12 acquires, for example, which direction (left or right) has a larger proportion of the real image when the HMD screen is divided left and right, as directional information. At this time, the mixed information acquisition unit 12 checks the number of pixels occupied by the VR image and the real image from the mask image acquired in S103 for both the left and right sides of the HMD screen. Then, the mixed information acquisition unit 12 determines that if the number of pixels of the real image is larger, that direction is the direction in which the real image has a larger proportion. Note that when dividing the HMD screen left and right, it does not have to be divided in the middle, but can be divided at any position. Also, the HMD screen can be divided not only into left and right halves, but also into any shape and number of divisions. Furthermore, for example, if multiple directional microphones for collecting external sounds in the real space are set up facing different directions, the HMD screen may be divided according to the direction of those directional microphones. Furthermore, if input is received to place the virtual display on, for example, the left half, the direction may be determined based on the viewer's input, such as making the right side the direction that occupies a larger proportion.

[0082] Next, in S404, the mixing ratio calculation unit 15 calculates the mixing ratio. For example, the mixing ratio calculation unit 15 calculates a mixing ratio such that external sounds from the direction in which the actual image is deemed to occupy a large proportion in S403 are synthesized at their original intensity, while external sounds from directions other than the direction in which the actual image is deemed to occupy a large proportion are not synthesized (the external sounds are reduced).

[0083] Next, in S405, the external sound acquisition unit 17 acquires external sound. In this case, as mentioned above, if multiple directional microphones are installed facing in different directions, the external sound acquisition unit 17 acquires the external sound collected by each of those directional microphones.

[0084] Next, in S406, the sound synthesis unit 18 synthesizes external sounds based on the mixing ratio calculated in S404. For example, the sound synthesis unit 18 synthesizes external sounds acquired from a directional microphone positioned in the direction to be synthesized at the same intensity into VR sounds.

[0085] As described above, the fourth embodiment of the information processing device allows for the adjustment of external sounds based on directional information as spatial information. In other words, in the fourth embodiment, the mixing ratio calculation unit 15 calculates a mixing ratio that reduces external sounds from sound sources in directions other than the direction corresponding to the directional information in real space, while calculating a mixing ratio that does not reduce external sounds from sound sources in the direction corresponding to the directional information. This allows the viewer to hear sounds from a specific direction and eliminates the need for the viewer to adjust external sounds themselves.

[0086] The mixing ratio calculated in S404 does not necessarily have to be a mixing ratio that synthesizes the external sound at its original intensity. For example, an image showing the mixing ratio with the same resolution as the real image may be prepared, and a mixing ratio that has a gain for the intensity of the external sound may be calculated by corresponding each pixel value. In this example, the mixing information acquisition unit 12 acquires direction information indicating the direction in which the image based on the real image occupies a larger proportion in the composite image, which is created by synthesizing the real image and the VR image based on the mixing information. The mixing ratio calculation unit 15 in this case calculates a normal distribution centered on the direction based on the proportion of the image based on the real image, that is, a normal distribution centered on the center of the region in the direction in which the real image is considered to occupy a larger proportion. The mixing ratio calculation unit 15 then uses the value of the normal distribution corresponding to each pixel position in the composite image as a gain to adjust the intensity of the external sound. In other words, the mixing ratio calculation unit 15 acquires a mixing ratio that adjusts the gain of the external sound using the value of the normal distribution as the gain. As a result, in S406, the sound synthesis unit 18 multiplies the external sound by the gain corresponding to each direction and then synthesizes it into a VR sound.

[0087] For example, when a sound-emitting object moves between a direction where external noise is not reduced and a direction where external noise is reduced, the sound emitted by the moving object will change significantly at the boundary. A similar situation can occur when the viewer is moving. Therefore, to prevent such abrupt changes in the volume of external noise, the external noise may be adjusted gradually over time. For example, suppose the HDMI screen is split into left and right halves, and external noise is reduced on the left side, while a sound source enters the left side from the right. In this case, the movement of the object can be detected using a known object detection method, and when detected, only that sound source can be separated using a known sound separation technique. Then, the sound data of that sound source can be synthesized so that the sound is gradually reduced over a predetermined period. This can be done, for example, by applying a gain that decreases sequentially from 1 to 0 to the intensity of the separated external noise.

[0088] In addition, in S403, the mixed information acquisition unit 12 may use known object recognition technology to recognize a region of a specific object and use that region as directional information. In that case, in S404, the mixed ratio calculation unit 15 calculates a mixed ratio that synthesizes the external sound of the sound source in that region.

[0089] In the fourth embodiment, as described above, the headphones may be used to physically cover the viewer's ears, so that the viewer cannot hear external sounds. Also in the fourth embodiment, if a headphone device is used in which external sounds from the surrounding real space can be heard by the viewer's ears when noise cancellation processing to reduce external sounds is not performed, the external sound acquisition processing in S106 and the external sound synthesis processing in S107 may not be performed.

[0090] <Fifth Embodiment> In the fourth embodiment, directional information was used as spatial information, but in the fifth embodiment, an example of using area information as spatial information will be described. Here, for example, if the proportion of the area occupied by the VR image among the images displayed on the HMD is large, it can be considered that the content is centered on the virtual reality space, and in this case, it is considered desirable to increase the amount of external sound reduction. On the other hand, if the proportion of the area occupied by the VR image is small, it can be considered that the content is not centered on the virtual reality space, and it is desirable that external sounds be heard. To address these situations, in the fifth embodiment, area information occupied by the VR image is used.

[0091] The hardware and functional configurations of the information processing device in the fifth embodiment are the same as those of the first embodiment described above, so their explanation will be omitted. The functional configuration of the information processing device is generally the same as that of Figure 3 described above, but in the fifth embodiment, the mixed information acquisition unit 12 acquires mixed information including area information and outputs that information to the output image acquisition unit 14 and the mixed ratio calculation unit 15. Thus, in the fifth embodiment, the mixed information includes area information. In the following, in the fifth embodiment, the same reference numerals are used for functional configurations and processing steps as in the examples of each embodiment described above, and their explanations will be omitted. The following will mainly describe the differences.

[0092] <Information processing in the fifth embodiment> Figure 9 is a flowchart showing the information processing flow executed by the information processing device of the fifth embodiment. In the fifth embodiment, if the MR flag is 0 in S105, the information processing device proceeds to processing from S108 onwards, while if the MR flag is 1, it proceeds to processing from S501 onwards. After processing from S501 to S506, the information processing device proceeds to processing S110.

[0093] If the MR flag is determined to be 1 and the process proceeds to S501, the real image acquisition unit 13 acquires a real image of the real space surrounding the viewer. Next, in S502, the output image acquisition unit 14 combines the VR image acquired by the VR image acquisition unit 11 with the real image acquired by the real image acquisition unit 13.

[0094] Next, proceeding to S503, the mixed information acquisition unit 12 acquires area information of the VR image and the real image. For example, the mixed information acquisition unit 12 counts the number of pixels with a value of 1 in the mask image acquired in S104, and uses the ratio AR, obtained by dividing the count of pixels with a value of 1 by the total number of pixels, as area information.

[0095] Next, in S504, the mixing ratio calculation unit 15 calculates the mixing ratio. In the fifth embodiment, the mixing ratio is such that a gain of ratio AR is applied to the intensity of the VR sound, and a gain of (1-AR) is applied to the intensity of the external sound.

[0096] Next, in S505, the external sound acquisition unit 17 acquires external sounds. Then, in S506, the sound synthesis unit 18 synthesizes external sounds based on the mixing ratio calculated in S504.

[0097] As described above, the fifth embodiment of the information processing device allows for adjustment of external sound based on area information as spatial information. Specifically, in the fifth embodiment, the mixing ratio calculation unit 15 calculates a mixing ratio that reduces external sound when the area occupied by the VR image is larger than that of the real image in the composite image obtained by combining the real image and the VR image displayed on the HMD. On the other hand, the mixing ratio calculation unit 15 calculates a mixing ratio that does not reduce external sound when the area occupied by the real image is larger than that of the VR image. As a result, viewers can hear sounds corresponding to the content displayed on the HMD, and the effort of adjusting external sound is eliminated.

[0098] Area information may also be calculated based on the field of view corresponding to the real image and the VR image. For example, if the field of view of the real image and the VR image are defined, their ratio can be used as the area information. As described above, according to the information processing device of the fifth embodiment, external sound can be adjusted based on area information included in spatial information. This makes it possible to adjust external sound according to the area occupied by the VR image, thus saving the viewer the trouble of adjusting the sound.

[0099] In the fifth embodiment, as described above, the headphones may be used to physically cover the viewer's ears, so that the viewer cannot hear external sounds. Also in the fifth embodiment, if a headphone device is used in which external sounds from the surrounding real space can be heard by the viewer's ears when noise cancellation processing to reduce external sounds is not performed, the external sound acquisition processing in S106 and the external sound synthesis processing in S107 may not be performed.

[0100] In the first to fifth embodiments described above, examples were given in which mask images, transparency information, and application program information were used as mixed information, and distance information, direction information, and area information were used individually as spatial information. However, one or more of these pieces of information may be combined as appropriate. In other words, one or more of these pieces of information may be combined as appropriate to perform image synthesis, external sound synthesis, sound adjustment, etc.

[0101] The present invention can also be realized by supplying a program that implements one or more of the functions of each of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The embodiments described above are merely examples of how the present invention can be implemented, and the technical scope of the invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various ways without departing from its technical concept or its main features.

[0102] This embodiment includes the following configurations, methods, and programs. (Composition 1) Information acquisition means for acquiring mixed information of real images and virtual images in real space, A means for acquiring virtual sound, Based on the aforementioned mixing information, a ratio acquisition means is provided to acquire the mixing ratio between the virtual sound and the external sound in the real space. An adjustment means for adjusting the relative volume between the virtual sound and the external sound based on the aforementioned mixing ratio, An information processing device characterized by having the following features. (Configuration 2) It has external sound acquisition means for acquiring sounds in the real space as external sounds, The information processing apparatus according to configuration 1, characterized in that the adjustment means generates a synthesized sound by combining the virtual sound and the external sound based on the mixing ratio. (Composition 3) A real image acquisition means that acquires the real image of the real space captured by an imaging device mounted on a head-mounted device, A virtual image acquisition means for acquiring the aforementioned virtual image, Output means for outputting a composite image, obtained by combining the real image and the virtual image based on the mixed information, to a display device provided by the head-mounted device, The information processing apparatus according to configuration 2, further comprising: (Composition 4) The aforementioned mixed information is information that represents the synthesis of the real image with the virtual image. The information processing apparatus according to configuration 3, characterized in that the ratio acquisition means acquires the mixing ratio for reducing the external sound to be synthesized with the virtual sound when the mixing information does not include information indicating that the real image is to be synthesized with the virtual image. (Composition 5) The aforementioned mixed information is information that represents the synthesis of the real image with the virtual image. The information processing apparatus according to configuration 3 or 4, characterized in that, when the ratio acquisition means includes information indicating that the real image is to be synthesized with the virtual image, it acquires the mixing ratio that does not reduce the external sound synthesized with the virtual sound. (Composition 6) The aforementioned mixed information is a mask image indicating an image region corresponding to the virtual image. The information processing apparatus according to configuration 4 or 5, characterized in that the ratio acquisition means determines, based on the mask image, whether or not the mixed information includes information indicating that the real image is combined with the virtual image. (Composition 7) The aforementioned mixed information is information indicating an application program that displays a composite image, which is a combination of a real image of the real space and a virtual image, on a display device provided by the head-mounted device. The information processing apparatus according to configuration 3, characterized in that the ratio acquisition means acquires the mixing ratio for reducing the external sound that is synthesized into the virtual sound when the mixing information indicates that it is not an application program that displays a composite image obtained by combining a real image of the real space and the virtual image. (Composition 8) The aforementioned mixed information is information indicating an application program that displays a composite image, which is a combination of a real image of the real space and a virtual image, on a display device provided by the head-mounted device. The information processing apparatus according to configuration 3 or 7, characterized in that, when the ratio acquisition means indicates that the mixed information is an application program that displays a composite image obtained by combining a real image of the real space and the virtual image, it acquires the mixing ratio that does not reduce the external sound that is synthesized with the virtual sound. (Composition 9) The mixed information includes spatial information corresponding to the real space in which the head-mounted device exists. The information processing apparatus according to any one of configurations 3 to 8, characterized in that the ratio acquisition means acquires the mixing ratio according to the spatial information. (Composition 10) The spatial information includes distance information from the head-mounted device in the real space, The information processing apparatus according to configuration 9, characterized in that the ratio acquisition means acquires the mixing ratio for reducing external noise from sound sources above a distance threshold represented by the distance information in the real space. (Composition 11) The spatial information includes distance information from the head-mounted device in the real space, The information processing apparatus according to configuration 9 or 10, characterized in that the ratio acquisition means acquires the mixing ratio that does not reduce external noise from sound sources below a distance threshold represented by the distance information in the real space. (Composition 12) The spatial information includes distance information from the head-mounted device in the real space, The information processing apparatus according to configuration 9, characterized in that the ratio acquisition means acquires the mixing ratio, which adjusts the gain of the external sound according to the distance information. (Composition 13) The spatial information includes directional information from the head-mounted device in the real space. The information processing apparatus according to configuration 9, characterized in that the ratio acquisition means acquires the mixing ratio for reducing external noise from sound sources in directions other than the direction corresponding to the direction information in the real space. (Composition 14) The spatial information includes directional information from the head-mounted device in the real space. The information processing apparatus according to configuration 9 or 13, characterized in that the ratio acquisition means acquires the mixing ratio that does not reduce external noise from a sound source in a direction corresponding to the direction information in the real space. (Composition 15) The spatial information includes directional information from the head-mounted device in the real space. The information acquisition means acquires direction information indicating the direction in which the image based on the real image occupies a larger proportion in the composite image obtained by combining the real image and the virtual image based on the mixed information. The information processing apparatus according to configuration 9, characterized in that the ratio acquisition means calculates a normal distribution centered on a direction based on the proportion occupied by the image based on the actual image, and acquires the mixing ratio for adjusting the gain of the external sound using the value of the normal distribution corresponding to each pixel position of the composite image as the gain. (Composition 16) The information processing apparatus according to any one of configurations 13 to 15, characterized in that the information acquisition means acquires direction information indicating the direction in which the image based on the actual image occupies a larger proportion in the composite image obtained by combining the actual image and the virtual image based on the mixed information. (Composition 17) The spatial information includes area information of an image based on the actual image and an image based on the virtual image. The information processing apparatus according to configuration 9, wherein the ratio acquisition means acquires the mixing ratio for reducing external noise when, in a composite image obtained by combining the real image and the virtual image displayed on a display device provided by the head-mounted device, the proportion of the area occupied by the image based on the virtual image is larger than that of the real image. (Composition 18) The spatial information includes area information of an image based on the actual image and an image based on the virtual image. The information processing apparatus according to configuration 9 or 17, wherein the ratio acquisition means acquires the mixing ratio that does not reduce external sound when, in a composite image obtained by combining the real image and the virtual image displayed on a display device provided by the head-mounted device, the proportion of the area occupied by the image based on the real image is larger than the proportion of the area occupied by the virtual image. (Composition 19) The information processing apparatus according to any one of configurations 1 to 19, characterized in that the ratio acquisition means calculates the mixing ratio in such a way that the change in the mixing ratio becomes gradual over time. (Composition 20) The mixed information includes transparency information that determines the transparency of the image based on the virtual image, The information processing apparatus according to configuration 3, characterized in that the ratio acquisition means acquires the mixing ratio according to the transmitted information. (Composition 21) The information acquisition means acquires the transparency information such that the transparency decreases as the amount of movement of the head-mounted device in the real space decreases, The information processing apparatus according to configuration 20, characterized in that the ratio acquisition means calculates a mixing ratio such that the amount of external sound reduction increases as the transmittance of the transmitted information decreases. (Composition 22) The information acquisition means acquires the transparency information, which indicates that the transparency increases as the amount of movement of the head-mounted device in the real space increases. The information processing apparatus according to configuration 20 or 21, characterized in that the ratio acquisition means acquires a mixing ratio such that the external sound is increased as the transparency of the transmitted information increases. (Composition 23) The information acquisition means acquires the transparency information that allows the image based on the virtual image to pass through when the head-mounted device is outside the designated area of ​​the real space, and acquires the transparency information that does not allow the image based on the virtual image to pass through when the head-mounted device is inside the designated area of ​​the real space. The information processing apparatus according to any one of configurations 20 to 22, characterized in that the ratio acquisition means acquires the mixing ratio for reducing external sound when the transparency information that prevents the image based on the virtual image from being transparent is acquired. (Composition 24) An information processing apparatus characterized by having an adjustment means for adjusting the relative volume between sound in real space and sound associated with a displayed image, based on at least one of the real image corresponding to the real space and the displayed image. (Method 1) An information processing method performed by an information processing device, An information acquisition process that obtains mixed information of real images and virtual images in the real space, The virtual sound acquisition process involves acquiring a virtual sound, Based on the aforementioned mixing information, a ratio acquisition step is performed to obtain the mixing ratio between the virtual sound and the external sound in the real space. An adjustment step to adjust the relative volume between the virtual sound and the external sound based on the aforementioned mixing ratio, An information processing method characterized by having the following features. (Program 1) A program that causes a computer to function as an information processing device described in any one of configurations 1 to 24. [Explanation of symbols]

[0103] 11: VR image acquisition unit, 12: Mixing information acquisition unit, 13: Real image acquisition unit, 14: Output image acquisition unit, 15: Mixing ratio calculation unit, 16: VR sound acquisition unit, 17: External sound acquisition unit, 18: Sound synthesis unit, 19: Output unit

Claims

1. an image acquisition means for acquiring a real image and a virtual image corresponding to a real space; a sound acquisition means for acquiring an external sound in the real space and a virtual sound associated with the virtual image; an adjustment means for adjusting a relative volume between the external sound and the virtual sound based on at least one of the real image and the virtual image; an output unit that outputs a composite image obtained by combining the real image and the virtual image based on mixed information to a display device provided in a head-mounted device; having the adjustment means generates a synthetic sound by synthesizing the virtual sound and the external sound based on at least one of the real image and the virtual image; the real image is an image obtained by capturing an image of the real space by an imaging device mounted on the head-mounted device, the blending information includes transparency information that determines a transparency of an image based on the virtual image; The adjustment means performs the adjustment in response to the transmission information, the mixed information includes the transparency information in which the transparency decreases as the amount of movement of the head-mounted device in the real space decreases, The information processing device, wherein the adjustment means performs control such that the amount of reduction of the external sound increases as the transparency of the transparency information decreases.

2. The image processing apparatus further includes an information acquiring means for acquiring mixed information representing a combination of the real image and the virtual image, The information processing apparatus according to claim 1 , wherein, when the mixing information includes information representing that the real image is to be combined with the virtual image, the adjustment means performs control not to reduce the external sound.

3. The mixed information is a mask image indicating an image area corresponding to the virtual image, 3 . The information processing apparatus according to claim 2 , wherein the adjustment means determines, based on the mask image, whether or not the mixed information includes information representing synthesis of the real image with the virtual image.

4. The mixed information is information indicating an application program for displaying a composite image obtained by combining a real image of the real space and the virtual image on a display device provided in the head-mounted device, The information processing device according to claim 1, characterized in that when the mixed information indicates an application program that displays a composite image obtained by combining a real image of the real space with the virtual image, the adjustment means performs control so as not to reduce the external sound.

5. The mixed information includes distance information from the head-mounted device to an object in the real space, The information processing apparatus according to claim 1 , wherein the adjustment means performs control to reduce external sounds from a sound source that is equal to or greater than a distance threshold represented by the distance information in the real space.

6. The information processing device according to claim 5, characterized in that the adjustment means performs control to adjust the gain of the external sound in accordance with the distance information.

7. The information processing device described in Claim 1, characterized in that the mixed information is the proportion of images based on the actual image in the synthetic image.

8. The mixed information includes area information of an image based on the real image and an image based on the virtual image, The information processing device according to claim 1, characterized in that the adjustment means performs control so as not to reduce the external sound when the proportion of the area occupied by the image based on the real image is larger than that of the virtual image in the synthetic image displayed on a display device equipped in the head-mounted device.

9. The information processing device according to claim 1, characterized in that the adjustment means adjusts the external sound so that it changes gradually over time.

10. An image acquisition means for acquiring a real image and a virtual image corresponding to a real space; a sound acquisition means for acquiring an external sound in the real space and a virtual sound associated with the virtual image; an adjustment means for adjusting a relative volume between the external sound and the virtual sound based on at least one of the real image and the virtual image; an output unit that outputs a composite image obtained by combining the real image and the virtual image based on mixed information to a display device provided in a head-mounted device; having the adjustment means generates a synthetic sound by synthesizing the virtual sound and the external sound based on at least one of the real image and the virtual image; the real image is an image obtained by capturing an image of the real space by an imaging device mounted on the head-mounted device, the blending information includes transparency information that determines a transparency of an image based on the virtual image; The adjustment means performs the adjustment in response to the transmission information, the mixed information includes the transparency information in which the transparency increases as the amount of movement of the head-mounted device in the real space increases, The information processing device, wherein the adjustment means performs control to increase the volume of the external sound as the transparency of the transparency information increases.

11. An information processing device as described in Claim 1, characterized in that when the head-mounted device is outside a designated area of ​​the real space, the adjustment means performs control to reduce the external sound.

12. An image acquisition step of acquiring a real image and a virtual image corresponding to a real space; a sound acquisition step of acquiring an external sound in the real space and a virtual sound associated with the virtual image; an adjustment step of adjusting a relative volume between the external sound and the virtual sound based on at least one of the real image and the virtual image; an output step of outputting a composite image obtained by combining the real image and the virtual image based on mixed information to a display device provided in a head-mounted device; having In the adjustment step, a synthetic sound is generated by synthesizing the virtual sound and the external sound based on at least one of the real image and the virtual image, the real image is an image obtained by capturing an image of the real space by an imaging device mounted on the head-mounted device, the blending information includes transparency information that determines a transparency of an image based on the virtual image; In the adjusting step, the adjustment is performed in accordance with the transmission information, the mixed information includes the transparency information in which the transparency decreases as the amount of movement of the head-mounted device in the real space decreases, The information processing method, wherein the adjustment step performs control such that the amount of reduction of the external sound increases as the transparency of the transmission information decreases.

13. A program that causes a computer to function as the information processing device described in claim 1.

Images