Information processing apparatus, control method of information processing apparatus and program

Abstract

To provide an information processing apparatus, a control method of the information processing apparatus and a program in which a fact being a sound in a real space can be more certainly understood.SOLUTION: An information processing apparatus 101 comprises: first acquisition means (a user information acquisition unit 203) that acquires user information associated with a user who visibly recognizes a space image including an image of a virtual space; second acquisition means (a virtual object information acquisition unit 205) that acquires virtual object information associated with the virtual object within the space image; third acquisition means (a real sound position estimation unit 202) that acquires positional information of a sound source of a sound when the sound is emitted in a real space; and determination means (a notification determination unit 207) that determines a notification method which notifies the user of the direction of the sound source in the real space on the basis of the user information acquired by the first acquisition means, the virtual object information acquired by the second acquisition means and the positional information acquired by the third acquisition means.SELECTED DRAWING: Figure 2

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, a control method for the information processing apparatus, and a program.

Background Art

[0002] In recent years, technologies that enable users to experience a space including the real space and the virtual space, such as augmented reality (AR) and mixed reality (MR), have been developing. For example, a head-mounted display (HMD) worn on the head can allow a user wearing the HMD to experience a composite space in which virtual objects are superimposed on a video image of the real space in front of the user's eyes. In addition, some HMDs can acquire the movements of a user's body and the movement of the user's line of sight. In this case, the HMD can synchronize the movements of the user's body and the movement of the user's line of sight with the movements in the composite space. As a result, the user can obtain a high sense of immersion in the composite space. Furthermore, some HMDs improve the sense of immersion by emitting sound. For example, Patent Document 1 discloses an apparatus that analyzes voices in the real space and detects a person who is speaking in the real space.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in the apparatus described in Patent Document 1, since all the sounds in the real space are notified to the user, the user may feel bothered. Also, when the sound in the composite space can be heard, it becomes difficult for the user to determine whether the sound heard is the sound in the real space or the sound in the composite space. And if this determination is incorrect, that is, if the sound heard by the user is determined to be the sound in the composite space when it is actually the sound in the real space, the user may miss the sound in the real space.

[0005] The present invention has been made in view of the above problems. An object of the present invention is to provide an information processing apparatus, a control method of the information processing apparatus, and a program that can more reliably grasp that the sound is in the real space.

Means for Solving the Problems

[0006] In order to achieve the above object, the information processing apparatus of the present invention includes: a first acquisition unit that acquires user information regarding a user who visually recognizes a space image including at least an image of a virtual space; a second acquisition unit that acquires virtual object information regarding a virtual object in the space image; a third acquisition unit that acquires position information of a sound source when a sound is generated in the real space; and a determination unit that determines a notification method for notifying the user of the direction of the sound source in the real space based on the user information acquired by the first acquisition unit, the virtual object information acquired by the second acquisition unit, and the position information acquired by the third acquisition unit.

Effects of the Invention

[0007] According to the present invention, it is possible to more reliably grasp that the sound is in the real space.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Mode for Carrying Out the Invention

[0009] Hereinafter, each embodiment of the present invention will be described in detail with reference to the drawings. However, the configurations described in the following embodiments are merely examples, and the scope of the present invention is not limited by the configurations described in each embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, an arbitrary component may be added. Further, any two or more configurations (features) in each embodiment can be combined.

[0010] <First Embodiment> Hereinafter, a first embodiment will be described with reference to FIGS. 1 to 6. FIG. 1 is a block diagram showing an example of the hardware configuration of an information processing apparatus according to the first embodiment. As shown in FIG. 1, the information processing apparatus 101 includes a CPU (Central Processing Unit) 102, a ROM (Read Only Memory) 103, and a RAM (Random Access Memory) 104. Further, the information processing apparatus 101 includes a communication unit 105, a sensing unit 106, an output unit 107, an input unit 108, and an imaging unit 110. These hardware configurations of the information processing apparatus 101 are communicably connected to each other via a bus 109. The CPU 102 is a computer that controls the information processing apparatus 101. The operations of the information processing apparatus 101 can be realized by programs loaded in the ROM 103 and the RAM 104. The programs include, for example, programs for causing the CPU 102 to execute each part and each means (control method of the information processing apparatus) of the information processing apparatus 101. Further, the RAM 104 is also used as a work memory for temporarily storing data of processes executed by the CPU 102.

[0011] Note that the number of CPUs 102 arranged is one in the configuration shown in FIG. 1, but is not limited thereto, and may be plural, for example. Further, in the information processing apparatus 101, when the RAM 104 is used as a primary storage area, a secondary storage area and a tertiary storage area may be further provided. The secondary storage area and the tertiary storage area are not particularly limited, and for example, an HDD (Hard Disk Drive), an SSD (Solid State Drive), or the like can be used. The hardware configurations of the information processing apparatus 101 are not limited to being connected to each other via the bus 109, and may be connected in multiple stages, for example. The information processing apparatus 101 may further include a GPU (Graphics Processing Unit) or the like.

[0012] The communication unit 105 is an interface for communicating with an external device. The sensing unit 106 acquires, for example, the user's line-of-sight information in the real space or acquires data for determining whether to notify sound or the like in the real space. The output unit 107 is composed of a liquid crystal display or the like. Thus, the output unit 107 functions as a display means for displaying various images or displaying the direction of the sound when the sound is emitted in the real space. Note that the image displayed by the output unit 107 is not particularly limited, and examples include an image of the real space, an image of the virtual space, and an image of a composite space including an image of the real space and an image of the virtual space. In this embodiment, the image of the composite space is used. Thereby, the user can experience MR (Mixed Reality). The input unit 108 is composed of a plurality of microphones with directivity or the like. Thus, when sound is emitted in the real space, the input unit 108 functions as a sound collection means for collecting the sound. In this embodiment, the information processing device 101 is a head-mounted display (HMD: Head Mounted Display) that can be detachably attached to the head of the user who uses the information processing device 101. Note that the information processing device 101 is not limited to a head-mounted display, and may be, for example, a desktop or notebook personal computer equipped with a web camera, a tablet terminal, a smartphone, or the like.

[0013] FIG. 2 is a block diagram showing an example of the software configuration (functional configuration) of the information processing apparatus shown in FIG. 1. As shown in FIG. 2, the information processing apparatus 101 includes an actual sound source information acquisition unit 201 and an actual sound position estimation unit (third acquisition means) 202. The information processing apparatus 101 includes a user information acquisition unit (first acquisition means) 203 and a user information storage unit 204. Further, the information processing apparatus 101 includes a virtual object information acquisition unit (second acquisition means) 205 and a virtual object information storage unit 206. Furthermore, the information processing apparatus 101 includes a notification determination unit (determination means) 207 and a notification unit (notification means) 208. The actual sound source information acquisition unit 201 acquires, as sound data (sound information), the sound from the sound source 303 collected by the input unit 108 when a sound is emitted from the sound source 303 (see FIGS. 3(a) and 3(b)) in the actual space. The actual sound position estimation unit 202 estimates the position of the sound source 303 based on the sound data (the sound collected by the input unit 108) acquired by the actual sound source information acquisition unit 201, and acquires the estimation result as the position information of the sound source 303 (third acquisition step). The method for estimating the position of the sound source 303 is not particularly limited, and examples thereof include a method of estimating the position of the sound source 303 based on the difference in reception timings of the sound from the sound source 303 received by a plurality of microphones constituting the input unit 108.

[0014] The user information acquisition unit 203 acquires user information regarding the user wearing the head-mounted display, that is, the user who visually recognizes the spatial image output by the output unit 107 (first acquisition step). The user information is not particularly limited, and includes, for example, at least one of the user's position information, the user's line-of-sight information, and gesture information regarding the user's gesture. The user's position information can be acquired by the user information acquisition unit 203 based on information from GPS (not shown), for example. The user's line-of-sight information can be acquired by the user information acquisition unit 203 based on information from a detection unit (not shown) that detects the user's line of sight, for example. The user's gesture information can be acquired by the user information acquisition unit 203 based on information from motion capture (not shown), for example. Then, the user information acquired by the user information acquisition unit 203 is stored in the user information storage unit 204.

[0015] The virtual object information acquisition unit 205 acquires virtual object information regarding a virtual object 308 (see FIG. 3(b)) displayed, for example, by CG (Computer Graphics) in the spatial image (second acquisition step). The virtual object information includes at least one of the position information, size, and orientation (tilt) of the virtual object 308 in the spatial image. Then, the virtual object information acquired by the virtual object information acquisition unit 205 is stored in the virtual object information storage unit 206.

[0016] The notification determination unit 207 determines a notification method (notification method) for notifying the user of the direction of the sound source 303 in the real space (determination step). This determination is made based on the position information of the sound source 303 estimated by the real sound position estimation unit 202, the user information stored in the user information storage unit 204, and the virtual object information stored in the virtual object information storage unit 206. Note that the determination of the notification method by the notification determination unit 207 will be described later with reference to FIG. 4. The notification unit 208 notifies the direction of the sound source 303 based on the determination result of the notification determination unit 207, that is, by the notification method determined by the notification determination unit 207.

[0017] FIG. 3 is a diagram for explaining an example of a notification method for notifying the direction of a sound source. FIG. 3(a) is a diagram showing the state of the real space 301. As shown in FIG. 3(a), in the real space 301, there are a user 302 wearing a head-mounted display configured by the information processing apparatus 101 and a sound source 303 that has emitted sound. The user 302 is facing the direction opposite to the sound source 303. In such a state, the sound source 303 is located outside the field of view of the user 302. FIG. 3(b) is a space image displayed on the output unit 107 of the information processing apparatus 101 in the state shown in FIG. 3(a). The user 302 can visually recognize this space image. As shown in FIG. 3(b), the space image 304 includes an arrow (marker) 305 displayed by CG. The arrow 305 is an image for notifying the direction of the sound source 303 with respect to the user 302. Thereby, the user 302 can grasp that the sound source 303 exists in the direction of the arrow 305, that is, the sound source 303 can be visually recognized if the direction of the arrow 305 is faced. Note that when the sound source 303 is not included in the space image 304, the arrow 305 preferably has a length proportional to the distance to the sound source 303. For example, when the distance to the sound source 303 is 3 m and when the distance to the sound source 303 is 30 m, the length of the arrow 305 can be made longer in the latter case than in the former case. Thereby, the user 302 can determine whether the sound source 303 is relatively close or relatively far. Note that the marker for notifying the direction of the sound source 303 is the arrow 305, but is not limited thereto, and for example, a character string indicating the direction of the sound source 303 may be used.

[0018] FIG. 3(c) is a diagram showing the state of the real space 306. As shown in FIG. 3(c), in the real space 306, there exist a user 302 and a sound source 303. The user 302 is facing the sound source 303. In such a state, the sound source 303 is located within the visual field of the user 302. FIGS. 3(d) to 3(f) are spatial images respectively displayed on the output unit 107 of the information processing apparatus 101 in the state shown in FIG. 3(c). The user 302 can visually recognize one of these spatial images. As shown in FIG. 3(d), the spatial image 307 includes the sound source 303, a virtual object 308 and an arrow 309 displayed in CG. The arrow 309 is an image for notifying the direction of the sound source 303 with respect to the user 302. The tip of this arrow 309 is in contact with the sound source 303. Thereby, the sound source 303 can be indicated by the arrow 309. Thereby, the user 302 can grasp that what the arrow 309 indicates is the sound source 303. In the spatial image 307, the sound source 303 and the virtual object 308 are spaced apart. Note that the virtual object 308 is not particularly limited, and for example, it may be an avatar of the user 302, a building such as a building, or a moving body such as an automobile.

[0019] As shown in FIG. 3(e), the spatial image 310 includes the sound source 303, the virtual object 308, and the arrow 309. This spatial image 310 is the same as the spatial image 307 except that the positional relationship between the sound source 303 and the virtual object 308 is different. In the spatial image 310, the sound source 303 and the virtual object 308 overlap (overlap), and the virtual object 308 is located in front of the sound source 303. In this case, it is preferable to adjust the transparency of the virtual object 308 to make it semi-transparent. Thereby, for example, when the virtual object 308 is a moving body, even if the virtual object 308 crosses in front of the sound source 303, it is possible to prevent the sound source 303 from being hidden by the virtual object 308.

[0020] As shown in FIG. 3(f), the spatial image 311 includes the sound source 303, the virtual object 308, and the arrow 309. This spatial image 311 is the same as the spatial image 310 except that the positional relationship between the sound source 303 and the virtual object 308 is different. In the spatial image 311, the sound source 303 and the virtual object 308 overlap (coincide), and the virtual object 308 is located behind the sound source 303. For example, when the virtual object 308 is a building, it can be grasped that the sound source 303 exists in front of the virtual object 308.

[0021] FIG. 4 is a flowchart showing the processing executed by the information processing apparatus. The program based on the flowchart shown in FIG. 4 is executed at the timing when the input unit 108 of the information processing apparatus 101 receives the sound from the sound source 303 in the real space. As shown in FIG. 4, in step S401, the real sound source information acquisition unit 201 acquires the sound data (sound source information) from the sound source 303 received by the input unit 108.

[0022] In step S402, the real sound position estimation unit 202 estimates the position of the sound source 303 based on the sound data acquired in step S401. This estimation result is used as the position information of the sound source 303. Note that the real sound position estimation unit 202 preferably acquires the position information of the sound source 30 when the sound emitted in the real space is equal to or greater than a threshold value (predetermined or more). Thereby, the notification target sound notified in step S409 or step S410 can be narrowed down from all the sounds in the real space. Note that the threshold value may be changed as appropriate. Also, the real sound position estimation unit 202 may acquire the position information of the sound source 303 when the sound emitted in the real space is a predetermined type of sound. This can also narrow down the notification target sound notified in step S409 or step S410 from all the sounds in the real space. Also, in step S402, the type of the sound source 303 may be estimated by machine learning and the image analysis technique may be used for the video from the user's viewpoint to specify the sound source position. In this case, the waveform and frequency of the sound are acquired.

[0023] In step S403, the user information acquisition unit 203 acquires the position information of the user as user information. Then, the user information acquisition unit 203 stores this user information in the user information storage unit 204.

[0024] In step S404, the virtual object information acquisition unit 205 acquires the position information, size, and orientation of the virtual object 308 as virtual object information. Then, the virtual object information acquisition unit 205 stores these virtual object information in the virtual object information storage unit 206.

[0025] In step S405, the notification determination unit 207 determines whether the sound source 303 exists (is included) within the user's field of view, that is, within the viewing angle (spatial image) of the imaging range that can be imaged by the imaging unit 110. This determination is made based on the position information of the sound source 303 estimated in step S402 and the position information of the user stored in the user information storage unit 204 in step S403. And as a result of the determination in step S405, if it is determined that the sound source 303 exists within the field of view, the process proceeds to step S406. On the other hand, if it is determined in step S405 that the sound source 303 does not exist within the field of view, the process proceeds to step S410.

[0026] In step S406, the notification determination unit 207 determines whether the virtual object 308 exists within the user's field of view. This determination is made based on the position information of the user stored in the user information storage unit 204 in step S403 and the virtual object information stored in the virtual object information storage unit 206 in step S404. And as a result of the determination in step S406, if it is determined that the virtual object 308 exists within the field of view, the process proceeds to step S407. On the other hand, if it is determined in step S406 that the virtual object 308 does not exist within the field of view, the process ends.

[0027] In step S407, the notification determination unit 207 determines whether or not the virtual object 308 and the sound source 303 overlap within the visual field. This determination is made based on the position information of the sound source 303 estimated in step S402. If, as a result of the determination in step S407, it is determined that they overlap, the process proceeds to step S408. Also, when the notification determination unit 207 determines that there is an overlapping state, it also determines the front-back relationship between the virtual object 308 and the sound source 303. Here, as an example, it is assumed that it is determined that the virtual object 308 is located in front of the sound source 303. On the other hand, if, as a result of the determination in step S407, it is determined that they do not overlap, the process proceeds to step S409. In the present embodiment, the notification determination unit 207 also functions as a determination means (judgment means) for making the determinations in step S405, step S406, and step S407. Note that in the information processing apparatus 101, the part that functions as the determination means may be provided separately from the notification determination unit 207. Also, determination means for making each determination in steps S405 to S407 may be provided respectively.

[0028] In step S408, the notification unit 208 semi-transmits the virtual object 308 determined to be in an overlapping state in step S407 and displays it on the output unit 107 (see FIG. 3(e)).

[0029] In step S409, the notification unit 208 displays an arrow 309 indicating the sound source 303 on the output unit 107 based on the position information of the sound source 303 estimated in step S402 (see FIG. 3(e)). After step S409 is executed, the process ends.

[0030] In step S410 after step S405 is executed, the notification unit 208 displays an arrow 305 facing the direction of the sound source 303 on the output unit 107 based on the position information of the sound source 303 estimated in step S402 (see FIG. 3(a)). After step S410 is executed, the process ends.

[0031] The information processing apparatus 101 capable of the above control can notify the user of a sound to be notified in the real space. As a result, it is suppressed that all sounds in the real space are notified to the user, and thus, for example, it is possible to reduce the user's feeling that the notification of all sounds is troublesome. Further, even if the user hears the sound from the head-mounted display, by checking the arrow displayed on the output unit 107, it is possible to accurately determine whether the sound is a sound in the real space or a sound from the head-mounted display. In this way, in the information processing apparatus 101, it is possible to more surely grasp that the sound is a sound in the real space.

[0032] FIG. 5 is a diagram showing an example of a table of the data structure stored in the user information storage unit. As shown in FIG. 5, the user information storage unit 204 stores the position information of the user. This position information includes 6-degree-of-freedom (Dof) information, that is, the position of the user's head and the orientation of the head using coordinates.

[0033] FIG. 6 is a diagram showing an example of a table of the data structure stored in the virtual object information storage unit. As shown in FIG. 6, the virtual object information storage unit 206 stores, as virtual object information, the name, position information, size, and inclination of the virtual object. The position information of the virtual object is indicated by the distance from the reference position using 6-degree-of-freedom information. The size of the virtual object is indicated by the distance from the center in the virtual object. The inclination of the virtual object indicates the rotation angle of the virtual object.

[0034] <Second Embodiment> Hereinafter, with reference to FIGS. 7 to 9, the second embodiment will be described. The description will focus on the differences from the above-described embodiments, and the description of the same matters will be omitted. This embodiment is the same as the first embodiment except for determining whether to perform a notification determination from the user operation acquired at the timing when the actual sound is heard. FIG. 7 is a block diagram showing an example of the software configuration (functional configuration) of the information processing apparatus according to the second embodiment. As shown in FIG. 7, the information processing apparatus 101 further includes a user operation determination unit 701 and a user operation information storage unit 702. The user operation determination unit 701 determines what operation the user has performed based on the transition of the position information of the user acquired by the user information acquisition unit 203. In the user operation information storage unit 702, the determination result of the user operation determination unit 701, that is, the operation information of the user is stored.

[0035] FIG. 8 is a diagram showing an example of a table of the data structure stored in the user operation information storage unit. As shown in FIG. 8, gesture information as the operation information of the user is stored in the user operation information storage unit 702. The gesture information includes an operation name (gesture name) and an operation (transition of the tilt of the head). For example, operation 1 represents a gesture in which the user looks down.

[0036] FIG. 9 is a flowchart showing the processing executed by the information processing apparatus. As shown in FIG. 9, in step S801, the actual sound source information acquisition unit 201 acquires sound data (sound source information) from the sound source 303 received by the input unit 108. This step S801 is the same as step S401 of the flowchart shown in FIG. 4.

[0037] In step S802, the actual sound position estimation unit 202 estimates the position of the sound source 303 used as the position information of the sound source 303 based on the sound data acquired in step S401. This step S802 is the same as step S402.

[0038] In step S803, the user information acquisition unit 203 acquires the position information of the user as user information and stores it in the user information storage unit 204. This step S803 is the same as step S403.

[0039] In step S804, the user motion determination unit 701 determines the motion of the user based on the transition of the position information of the user stored in the user information storage unit 204 in step S803, that is, the change over time.

[0040] In step S805, the notification determination unit 207 determines whether the gesture information pre-stored in the user motion information storage unit 702 matches the motion information of the user determined in step S804. If it is determined in step S805 that they match, the process proceeds to step S805. On the other hand, if it is determined in step S805 that they do not match, the process ends. Note that in step S805, it is determined whether the gesture information and the motion information of the user match, but it is not limited to this. For example, in step S805, it may be determined whether the captured image by the imaging unit 110 or the gesture of the user read from a controller (not shown) held by the user matches the gesture information.

[0041] In step S806, the notification unit 208 causes the output unit 107 to display that it is a real sound.

[0042] With such control, for example, even in a situation where it is relatively difficult for the user to recognize the sound in the real space, the notification of the sound can be performed.

[0043] The preferred embodiments of the present invention have been described above. However, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. The present invention can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors of a computer of the system or device read and execute the program. Further, the present invention can also be realized by a circuit (for example, ASIC) that realizes one or more functions. Further, the information processing device 101 is a head-mounted display having the CPU 102 to the imaging unit 110 in the above-described embodiment, but is not limited thereto. For example, in the information processing device 101, the sensing unit 106, the output unit 107, the input unit 108, and the imaging unit 110 may be omitted, and these may constitute a head-mounted display communicably connected to the information processing device 101. In this case, the information processing device 101 and the head-mounted display may be connected by wire or wirelessly. Further, in this case, the information processing device 101 can be used as a server to configure an information processing system with the server and the head-mounted display.

[0044] In this information processing system, even when, for example, the server is located outside Japan and the head-mounted display as a terminal device is located within Japan, each file and data can be transmitted from the server to the terminal device, and the terminal device can receive each file and data. Even when the server is located outside Japan in this way, the transmission and reception (communication) of files and data in this system are integrated, that is, they are performed without going through a separate operation by the user of the terminal device. And since the system functions when the terminal device within Japan receives each file and data, the communication can be regarded as having been performed within the country. In this system, for example, even when the server is located outside Japan and the terminal device is located within Japan, the terminal device can perform the main functions of this system, and the effects due to the functions can be manifested within Japan. For example, even if the server is located outside Japan, if the terminal device constituting this system is located within Japan, it is possible to use this system within the country using the terminal device. And the use of this system can, for example, have an impact on the economic interests of the patent holder.

[0045] The disclosure of each embodiment includes the following configurations, methods, and programs. (Configuration 1) A first acquisition means for acquiring user information regarding a user who visually recognizes a spatial image including at least an image of a virtual space, a second acquisition means for acquiring virtual object information regarding a virtual object within the spatial image, a third acquisition means for acquiring position information of a sound source when a sound is emitted in the real space, and a determination means for determining a notification method for notifying the direction of the sound source in the real space with respect to the user based on the user information acquired by the first acquisition means, the virtual object information acquired by the second acquisition means, and the position information acquired by the third acquisition means. The information processing apparatus is characterized by comprising these components. (Configuration 2) The information processing apparatus according to Configuration 1, characterized by comprising a notification means for notifying the direction of the sound source by the notification method determined by the determination means. (Configuration 3) It is provided with display means for displaying the spatial image, The notification means is the information processing apparatus according to Configuration 2, characterized in that the direction of the sound source is notified by a marker displayed in the spatial image by the display means. (Configuration 4) The information processing apparatus according to Configuration 3, characterized in that the marker is an arrow. (Configuration 5) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The notification means is the information processing apparatus according to Configuration 4, characterized in that when the sound source is included in the spatial image, it becomes the arrow indicating the sound source. (Configuration 6) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The notification means is the information processing apparatus according to Configuration 4, characterized in that when the sound source is not included in the spatial image, it becomes the arrow having a length proportional to the distance to the sound source. (Configuration 7) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The display means includes the virtual object and the sound source in the spatial image, and when the virtual object and the sound source overlap, the information processing apparatus according to Configuration 3, characterized in that the transparency of the virtual object is adjusted. (Configuration 8) The first acquisition means is the information processing apparatus according to any one of Configurations 1 to 7, characterized in that it acquires at least one of the user's position information, the user's line-of-sight information, and information regarding the user's gesture as the user information. (Configuration 9) The second acquisition means is the information processing apparatus according to any one of Configurations 1 to 8, characterized in that it acquires at least one of the position information, size, and orientation of the virtual object in the spatial image as the virtual object information. (Configuration 10) It is provided with sound collection means for collecting the sound when the sound is emitted in the real space, The third acquisition means is configured to estimate the position of the sound source based on the sound collected by the sound collection means and acquire the estimation result as the position information, according to any one of Configurations 1 to 9 of the information processing apparatus described above. (Configuration 11) The third acquisition means is configured to acquire the position information of the sound source when the sound generated in the real space is equal to or greater than a predetermined level, according to any one of Configurations 1 to 10 of the information processing apparatus described above. (Configuration 12) The third acquisition means is configured to acquire the position information of the sound source when the sound generated in the real space is a sound of a predetermined type, according to any one of Configurations 1 to 11 of the information processing apparatus described above. (Configuration 13) The determination means is configured such that it is also possible not to perform notification of the direction of the sound source as the notification method, according to any one of Configurations 1 to 12 of the information processing apparatus described above. (Configuration 14) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to any one of Configurations 1 to 13, further comprising determination means for determining whether or not the sound source is included in the spatial image. (Configuration 15) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to any one of Configurations 1 to 14, further comprising determination means for determining whether or not the virtual object is included in the spatial image. (Configuration 16) The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to any one of Configurations 1 to 15, further comprising determination means for determining whether or not the virtual object and the sound source are included in an overlapping state in the spatial image. (Configuration 17) The information processing apparatus according to any one of Configurations 1 to 16, further comprising display means for displaying the spatial image. (Configuration 18) The information processing apparatus according to Configuration 17, which is a head-mounted display. (Method 1) A method for controlling an information processing apparatus that processes information, comprising: a first acquisition step of acquiring user information regarding a user who visually recognizes a space image including at least an image of a virtual space; a second acquisition step of acquiring virtual object information regarding a virtual object in the space image; a third acquisition step of acquiring position information of a sound source when a sound is generated in the real space; a determination step of determining a notification method for notifying the user of the direction of the sound source in the real space based on the user information acquired in the first acquisition step, the virtual object information acquired in the second acquisition step, and the position information acquired in the third acquisition step. A method for controlling an information processing apparatus, characterized by comprising the above steps. (Program 1) A program for causing a computer to execute each means of the information processing apparatus according to any one of Configurations 1 to 18.

Description of Reference Numerals

[0046] 101 Information processing apparatus 202 Real sound position estimation unit 203 User information acquisition unit 205 Virtual object information acquisition unit 207 Notification determination unit 208 Notification unit 303 Sound source 304, 307, 310, 311 Space image 305, 309 Arrow 308 Virtual object

Claims

1. First acquisition means for acquiring user information regarding a user who visually recognizes a spatial image including at least an image of a virtual space; Second acquisition means for acquiring virtual object information regarding a virtual object in the spatial image; Third acquisition means for acquiring position information of a sound source when a sound is generated in the real space; Determination means for determining a notification method for notifying the direction of the sound source in the real space with respect to the user based on the user information acquired by the first acquisition means, the virtual object information acquired by the second acquisition means, and the position information acquired by the third acquisition means. An information processing apparatus characterized by comprising:

2. The information processing apparatus according to claim 1, further comprising notification means for notifying the direction of the sound source by the notification method determined by the determination means.

3. Comprising display means for displaying the spatial image, The information processing apparatus according to claim 2, wherein the notification means notifies the direction of the sound source by a marker displayed in the spatial image by the display means.

4. The information processing apparatus according to claim 3, wherein the marker is an arrow.

5. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 4, wherein when the sound source is included in the spatial image, the notification means becomes the arrow indicating the sound source.

6. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 4, wherein when the sound source is not included in the spatial image, the notification means becomes the arrow having a length proportional to the distance to the sound source.

7. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 3, wherein the display means includes the virtual object and the sound source in the spatial image, and when the virtual object and the sound source overlap, the transparency of the virtual object is adjusted.

8. The information processing apparatus according to claim 1, wherein the first acquisition means acquires at least one of the user's position information, the user's line-of-sight information, and information regarding the user's gesture as the user information.

9. The information processing apparatus according to claim 1, wherein the second acquisition means acquires, as the virtual object information, at least one of position information, size, and orientation of the virtual object in the spatial image.

10. Comprising sound collection means for collecting sound when sound is emitted in the real space, The information processing apparatus according to claim 1, wherein the third acquisition means estimates the position of the sound source based on the sound collected by the sound collection means and acquires the estimation result as the position information.

11. The information processing apparatus according to claim 1, wherein the third acquisition means acquires the position information of the sound source when the sound emitted in the real space is equal to or more than a predetermined level.

12. The information processing apparatus according to claim 1, wherein the third acquisition means acquires the position information of the sound source when the sound emitted in the real space is a sound of a predetermined type.

13. The information processing apparatus according to claim 1, wherein the determination means can also not perform notification in the direction of the sound source as the notification method.

14. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 1, comprising determination means for determining whether or not the sound source is included in the spatial image.

15. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 1, comprising determination means for determining whether or not the virtual object is included in the spatial image.

16. The spatial image is an image of a composite space including an image of the real space and an image of the virtual space, The information processing apparatus according to claim 1, comprising determination means for determining whether or not the virtual object and the sound source are included in an overlapping state in the spatial image.

17. The information processing apparatus according to claim 1, comprising display means for displaying the spatial image.

18. The information processing apparatus according to claim 17, which is a head-mounted display.

19. A method for controlling an information processing apparatus that processes information, A first acquisition step of acquiring user information regarding a user who visually recognizes a spatial image including at least an image of a virtual space, A second acquisition step of acquiring virtual object information regarding a virtual object in the spatial image; A third acquisition step of acquiring position information of a sound source of the sound when the sound is generated in the real space; A determination step of determining a notification method for notifying the user of the direction of the sound source in the real space with respect to the user based on the user information acquired in the first acquisition step, the virtual object information acquired in the second acquisition step, and the position information acquired in the third acquisition step. A control method for an information processing apparatus, characterized by comprising:

20. A program for causing a computer to execute each means of the information processing apparatus according to claim 1.

Images