Information processing device, information processing method, and program

Abstract

This technology, which outputs acoustic signals to sound-producing devices worn in the user's right and left ears, enhances the sense of immersion in the acoustic space. [Solution] In the information processing device 10, the first acquisition unit 30 acquires information about the acoustic space. The second acquisition unit 32 acquires positional information of the user's right and left ears. The determination unit 34 determines whether the user's right and left ears have entered the acoustic space based on the information acquired by the first acquisition unit 30 and the second acquisition unit 32. If the output unit 36 ​​determines that the user's right ear has entered the acoustic space, it outputs a first acoustic signal to the first sound-producing device 12 attached to the user's right ear, and if the output unit 36 ​​determines that the user's left ear has entered the acoustic space, it outputs a second acoustic signal to the second sound-producing device 14 attached to the user's left ear.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and a program.

Background Art

[0002] In an exhibition hall or the like, an automatic guidance system that automatically explains exhibits or the like using headphones or the like worn by a person is known (see, for example, Patent Document 1). This system includes a plurality of first devices installed on the exhibit itself or in its vicinity, each transmitting a different ID signal for each exhibit, means for recording guidance information for each exhibit by one or both of voice and image, means for receiving the ID signal transmitted by the first device, and a second device having means for recognizing the ID of the exhibit based on the received ID signal and reproducing the guidance information of the corresponding exhibit from the recorded guidance information.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] The present inventors have recognized that it is useful to enhance the sense of immersion in a preset acoustic space for a user wearing headphones or the like.

[0005] An object of the present invention is to enhance the sense of immersion in an acoustic space in a technique for outputting an acoustic signal to a sound-emitting device worn on each of a user's right ear and left ear.

Means for Solving the Problems

[0006] To solve the above problems, an information processing device in one aspect of the present invention includes: a first acquisition unit for acquiring information about the acoustic space; a second acquisition unit for acquiring positional information of the user's right and left ears; a determination unit for determining whether the user's right and left ears have entered the acoustic space based on the information acquired by the first and second acquisition units; and an output unit that, if it is determined that the user's right ear has entered the acoustic space, outputs a first acoustic signal to a first sound-producing device attached to the user's right ear, and if it is determined that the user's left ear has entered the acoustic space, outputs a second acoustic signal to a second sound-producing device attached to the user's left ear.

[0007] Another aspect of the present invention is an information processing method. This method is an information processing method performed by a computer and comprises: a first acquisition step of acquiring information about an acoustic space; a second acquisition step of acquiring positional information of the user's right and left ears; a determination step of determining whether the user's right and left ears have entered the acoustic space based on the information acquired in the first and second acquisition steps; and an output step of outputting a first acoustic signal to a first sound-producing device attached to the user's right ear if it is determined that the user's right ear has entered the acoustic space, and outputting a second acoustic signal to a second sound-producing device attached to the user's left ear if it is determined that the user's left ear has entered the acoustic space.

[0008] Furthermore, any combination of the above components, as well as conversions of the expression of the present invention between methods, apparatus, systems, etc., are also valid embodiments of the present invention. [Effects of the Invention]

[0009] According to the present invention, in a technology that outputs acoustic signals to sound-producing devices attached to the user's right and left ears, it is possible to enhance the sense of immersion in the acoustic space. [Brief explanation of the drawing]

[0010] [Figure 1] This is a diagram illustrating the schematic operation of the acoustic system according to the embodiment. [Figure 2]This diagram shows a situation where the user's head is outside the acoustic space. [Figure 3] This diagram shows a situation where the right ear side of the user's head is within the acoustic space, and the left ear side of the head is outside the acoustic space. [Figure 4] This diagram shows a situation where the user's entire head is immersed in the acoustic space. [Figure 5] This is a block diagram showing the acoustic system of the first configuration example of the embodiment. [Figure 6] Figure 5 is a diagram illustrating the operation of the acoustic system. [Figure 7] Figure 5 is a flowchart showing the processing of the acoustic system. [Figure 8] This is a diagram illustrating a second example configuration of the acoustic system. [Figure 9] This is a diagram illustrating a third example configuration of the acoustic system. [Modes for carrying out the invention]

[0011] Figure 1 is a diagram illustrating the schematic operation of the acoustic system of the embodiment. The acoustic system of the embodiment presents a virtual acoustic space 110 to user 100 by individually controlling the sounds presented to both of user 100's ears using spatial audio technology, thereby providing an intuitive and strong sense of self-awareness, as if "my body has actually entered that acoustic space 110." User 100 wears sound-producing devices such as headphones or earphones (not shown) on both ears, and the acoustic space 110 is presented when sound is output from the sound-producing devices. The acoustic space 110 can also be called an acoustic three-dimensional space.

[0012] The acoustic space 110 is superimposed on the real space as virtual reality (AR), and the inside of the acoustic space 110 is filled with acoustic expressions, and it is a space whose boundaries are defined by three-dimensional coordinates. It is walkable, and the user 100 can actively enter and exit the acoustic space 110. When the user 100 enters the acoustic space 110, a feeling that their own body is gradually entering the space can be obtained, so the sense of immersion is high. The user 100 can be clearly aware of where the boundary 112 of the acoustic space 110 is located in the real space. By actively entering and exiting the acoustic space 110, the user 100 can become aware of the shape and volume of the acoustic space 110, so a strong sense of presence can be given to the acoustic space 110.

[0013] The acoustic system sequentially detects the position and orientation of the head of the user 100 moving in the real space by using a walk-through type position information acquisition technology and coordinate tracking and head tracking technologies.

[0014] When the user enters the acoustic space 110 defined on the AR space, the acoustic system determines whether each of the left ear and the right ear has entered the acoustic space 110 by the position information acquisition technology and coordinate tracking and head tracking technologies.

[0015] The acoustic system individually controls the acoustic expressions to the left ear and the right ear by using earphones or headphones using spatial audio technology according to the determination result.

[0016] Next, referring to FIGS. 2 to FIG. 4, an example in which the user 100 enters the acoustic space 110 from the right side of the head will be described.

[0017] FIG. 2 shows a situation where the head of the user 100 is outside the acoustic space 110. The user 100 is wearing headphones 120. In this case, the acoustic system does not present acoustic expressions to both ears.

[0018] FIG. 3 shows a situation where the right ear side of the head of user 100 is inside the acoustic space 110 and the left ear side of the head is outside the acoustic space 110. In this case, the acoustic system presents an acoustic representation indicating that the user is in the acoustic space to the right ear of the user, and does not present an acoustic representation to the left ear.

[0019] FIG. 4 shows a situation where the entire head of user 100 is inside the acoustic space 110. In this case, the acoustic system presents an acoustic representation indicating that the user is in the acoustic space to each of the two ears.

[0020] As described above, according to the acoustic system of the embodiment, based on the active physical movement of the user himself and the time difference that occurs accordingly when each of the right ear and the left ear enters the acoustic space 110, it is possible to provide a highly immersive experience with a strong sense of self that can clearly obtain a feeling of immersing in the acoustic space 110.

[0021] Hereinafter, the configuration of the acoustic system will be described in more detail. FIG. 5 is a block diagram showing an acoustic system 1 according to a first configuration example of the embodiment. The acoustic system 1 includes an information processing device 10, a first sound emitting device 12, and a second sound emitting device 14.

[0022] The first sound emitting device 12 is attached to the right ear of the user. The second sound emitting device 14 is attached to the left ear of the user. The first sound emitting device 12 and the second sound emitting device 14 are configured as headphones 120 or earphones as shown in FIG. 2 etc. The first sound emitting device 12 emits sound based on a first acoustic signal supplied from the information processing device 10. The second sound emitting device 14 emits sound based on a second acoustic signal supplied from the information processing device 10.

[0023] The information processing device 10 is a portable terminal such as a smartphone, a tablet terminal, a wearable device, or a notebook computer. Hereinafter, an example where the information processing device 10 is a smartphone will be described. The information processing device 10 has an imaging unit 20, a GPS receiving unit 22, and a processing unit 24.

[0024] The imaging unit 20 includes a camera and periodically captures images at a predetermined frame rate, and sequentially supplies the captured image data to the processing unit 24. For example, the user moves with the information processing device 10, which is a smartphone, fixed to a predetermined position on their body, for example, on their chest, so that the imaging unit 20 can capture images in front of their body.

[0025] The GPS receiver 22 receives signals from GPS satellites, periodically derives the position of the information processing device 10, and sequentially supplies the derived position information to the processing unit 24.

[0026] The processing unit 24 includes a first acquisition unit 30, a second acquisition unit 32, a determination unit 34, and an output unit 36. The configuration of the processing unit 24 can be realized in hardware terms using the CPU, memory, and other LSIs of any computer, and in software terms using programs loaded into memory, etc., but here we are describing the functional blocks realized by the cooperation of these. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various ways using only hardware, only software, or a combination thereof.

[0027] The first acquisition unit 30 acquires information about the virtual acoustic space and supplies the acquired information to the determination unit 34. The acoustic space information includes three-dimensional coordinates that define the acoustic space within the real space. The shape of the acoustic space is a rectangular prism in the example of Figure 1, but it may be any three-dimensional shape. The acoustic space information may be acquired via a network such as the Internet. The acoustic space may be set by user operation.

[0028] The second acquisition unit 32 sequentially acquires the position information of the user's right and left ears using known tracking techniques, based on the image data supplied from the imaging unit 20 and the position information of the information processing device 10 supplied from the GPS receiving unit 22, and supplies the acquired information to the determination unit 34.

[0029] Furthermore, the second acquisition unit 32 may also receive directional information from the GPS receiver unit 22 and acquire the position information of the user's right and left ears based on the directional information and position information. In this case, image data does not need to be used.

[0030] Location information and tracking information may be obtained using other known technologies, such as those listed below. For example, information may be obtained using various sensors such as optical, inertial, mechanical, and magnetic sensors attached to the user. Alternatively, a tracker may be attached to the user, similar to motion capture, and the tracker may be read by an external sensor installed in the installation environment. Furthermore, the user may not wear any equipment for obtaining location information, and their position and posture may be detected by cameras or LiDAR sensors installed in the installation environment.

[0031] The determination unit 34 determines whether the user's right ear and left ear have entered the acoustic space based on the information acquired by the first acquisition unit 30 and the second acquisition unit 32, and supplies the determination result to the output unit 36.

[0032] The output unit 36 ​​outputs a first acoustic signal to the first sound-producing device 12 when the detection unit 34 determines that the user's right ear has entered the acoustic space. The first acoustic signal is pre-set for the acoustic space. In this case, the output unit 36 ​​may output the first acoustic signal so that the sound is heard by the user from the direction of the right ear. The first sound-producing device 12 emits sound based on the first acoustic signal.

[0033] If the detection unit 34 does not determine that the user's right ear has entered the acoustic space, the output unit 36 ​​does not output an acoustic signal to the first sound-producing device 12. In this case, the first sound-producing device 12 does not emit sound.

[0034] If the detection unit 34 determines that the user's left ear has entered the acoustic space, the output unit 36 ​​outputs a second acoustic signal to the second sound-producing device 14. The second acoustic signal is also pre-set for the acoustic space. In this case, the output unit 36 ​​may output the second acoustic signal so that the sound is heard by the user from the direction of the left ear. The second sound-producing device 14 emits sound based on the second acoustic signal.

[0035] If the detection unit 34 does not determine that the user's left ear has entered the acoustic space, the output unit 36 ​​does not output an acoustic signal to the second sound-producing device 14. In this case, the second sound-producing device 14 does not emit sound.

[0036] The first and second acoustic signals may be signals for reproducing, for example, voice, music, sound effects, etc., and can be appropriately determined depending on the situation in which the acoustic system 1 is used.

[0037] The above functions of the processing unit 24 can be implemented in various ways. For example, software such as Unity® may be used. In this case, the following implementation method may be adopted.

[0038] (1) Define the coordinates of the AR acoustic space in Unity. This AR acoustic space corresponds to the acoustic space of the real world.

[0039] (2) Define the camera in Unity as the user's head.

[0040] (3) In the camera on Unity, place sound source objects at positions corresponding to the left and right ears of the head.

[0041] (4) The user holds a smartphone and wears audio devices with spatial audio functionality in both ears.

[0042] (5) The smartphone detects location information and the smartphone's camera detects tracking information, etc.

[0043] (6) The information detected in (5) is synchronized with the camera in Unity.

[0044] (7) When each object in (3) enters the coordinates defined in (1), it plays an acoustic representation that indicates it is in the acoustic space.

[0045] (8) The user listens to the sound played back by the spatial audio equipment in (4).

[0046] Figure 6 is a diagram illustrating the operation of the acoustic system 1 in Figure 5. Figure 6 schematically shows the user's head 102 viewed from above. In the situation shown in Figure 6, the position P1 of the right ear is not within the acoustic space 110, so the first acoustic signal is not supplied to the first sound-producing device 12 for the right ear, and the first sound-producing device 12 does not emit sound.

[0047] On the other hand, since the position P2 of the left ear is within the acoustic space 110, a second acoustic signal is supplied to the second sound-producing device 14 for the left ear, and the second sound-producing device 14 emits sound.

[0048] Although not shown in the diagram, when both the position P1 of the right ear and the position P2 of the left ear are within the acoustic space 110, the first sound-emitting device 12 and the second sound-emitting device 14 each emit sound.

[0049] Figure 7 is a flowchart showing the processing of the acoustic system 1 in Figure 5. The first acquisition unit 30 acquires information about the acoustic space (S10). The second acquisition unit 32 acquires positional information of the user's right and left ears (S12).

[0050] If the user's right ear is within the acoustic space (Y in S14), the output unit 36 ​​outputs a first acoustic signal to the first sound-producing device 12. If the user's right ear is not within the acoustic space (N in S14), the process moves to S18.

[0051] If the user's left ear is within the acoustic space (Y in S18), the output unit 36 ​​outputs a second acoustic signal to the second sound-producing device 14 (S20), and the process returns to S12. If the user's left ear is not within the acoustic space (N in S18), the process returns to S12.

[0052] According to this embodiment, when the user's right ear enters the acoustic space 110, a sound is heard from the first sound-producing device 12 to the user's right ear, and when the user's left ear enters the acoustic space 110, a sound is heard from the second sound-producing device 14 to the user's left ear. As a result, when the user enters the acoustic space 110, they get the sensation that their body is gradually entering the acoustic space 110, thus achieving a high level of immersion.

[0053] Additionally, the following effects can be obtained: (1) Zoning can be presented in real space using sound representation.

[0054] (2) It is possible to provide individual acoustic information to specific users in specific zones within the real world.

[0055] (3) It can present acoustic spaces of various shapes.

[0056] (4) By giving direction to the acoustic expression itself within the acoustic space, it is possible to present the user with a sense of direction within the acoustic space.

[0057] (5) Since this can be achieved with earphones that output auditory information and input devices such as smartphones that acquire location information, the sound system 1 can be easily constructed without the use of special equipment, and an acoustic space can be experienced.

[0058] (6) There is no need to install special sound equipment or anything like that in the facility.

[0059] Next, we will describe another configuration example of sound system 1. The following explanation will focus on the differences from the first configuration example.

[0060] Figure 8 is a diagram illustrating a second configuration example of the acoustic system 1. Figure 8 schematically shows the user's head 102 as viewed from above. In this configuration example, acoustic signals are output so that the user can hear sound from the direction of each position that is inside the acoustic space 110, among a plurality of positions arranged in a ring around the user's head 102.

[0061] The second acquisition unit 32 acquires information on the positions P1 and P2 of the right and left ears, as well as information on a number of pre-set positions P3 to P12 around the user's head 102. In the example in Figure 8, information is acquired on the position P3 in front of the head 102, two positions P4 and P5 to the left front of the head 102, two positions P6 and P7 to the left rear of the head 102, a position P8 to the rear of the head 102, two positions P9 and P10 to the right rear of the head 102, and two positions P11 and P12 to the right front of the head 102.

[0062] The determination unit 34 further determines whether each of the multiple positions P3 to P12 around the head 102 has entered the acoustic space 110.

[0063] If the output unit 36 ​​determines that at least one of the multiple positions P3 to P12 has entered the acoustic space 110, it outputs an acoustic signal to at least one of the first sound-producing device 12 and the second sound-producing device 14 so that the user can hear the sound from the direction of each position that has been determined to be in the acoustic space 110. Known spatial audio technology can be used as the method for outputting such an acoustic signal.

[0064] In the situation shown in Figure 8, since the left ear's positions P2 and P4-P7 are within the acoustic space 110, a second acoustic signal is supplied to the second sound-producing device 14 for the left ear, and the second sound-producing device 14 emits sound so that the sound is heard from each of the directions of the left ear's positions P2 and P4-P7. In other words, the user perceives the sound as coming from each of the five directions indicated by the arrows in Figure 8.

[0065] In the second configuration example, the boundary of the acoustic space 110 feels softer compared to the first configuration example. In other words, the boundary of the acoustic space 110 is clearer in the first configuration example. On the other hand, the user has the sensation that their body is passing through the boundary and gradually entering it, resulting in a stronger sense of immersion with a greater sense of direction than in the first configuration example. Furthermore, a higher sense of immersion is obtained when entering the acoustic space 110 in the front-to-back direction.

[0066] Figure 9 is a diagram illustrating a third configuration example of the acoustic system 1. In this third configuration example, external ambient noise is suppressed for ears located within the acoustic space 110.

[0067] In the third configuration example, the first and second acoustic signals are acoustic signals for suppressing or canceling ambient noise, respectively. Known noise-canceling techniques can be used to suppress ambient noise. This allows for the cancellation of external ambient noise in the real world for ears entering the acoustic space 110. Therefore, ambient noise becomes less audible to ears entering the acoustic space 110. On the other hand, external ambient noise can be heard by ears not entering the acoustic space 110. This provides the user with an immersive experience in a quiet acoustic space 110.

[0068] In the situation shown in Figure 9, the right ear is inside the acoustic space 110, while the left ear is outside the acoustic space 110. Therefore, ambient sounds are difficult to hear with the right ear, but are audible with the left ear.

[0069] Furthermore, the first and second acoustic signals may each include an acoustic signal for suppressing ambient noise and an acoustic signal pre-set for the acoustic space 110. This allows the ears entering the acoustic space 110 to receive sounds pre-set for the acoustic space 110 while canceling out external ambient noise from the real world. As a result, ambient noise becomes less audible to the ears entering the acoustic space 110, while the sounds pre-set for the acoustic space 110 become easier to hear. In this case, the sense of immersion in the audible acoustic space 110 can be enhanced.

[0070] Furthermore, as a fourth configuration example of the sound system 1, the sense of immersion may be enhanced by reverberation feedback. By adding a reverberation effect as feedback to speech, etc., within the acoustic space, the feeling that the user is immersed in the surrounding space can be emphasized.

[0071] The sound system 1 of this embodiment can be used in a variety of fields. For example, the sound system 1 can be used in highly immersive experiential digital art, artwork guidance devices in museums and art galleries, exploratory recreation in amusement facilities, tourist information applications for foreign tourists, route guidance applications that enable visually impaired people to travel to their destinations, acoustic car navigation for drivers using vehicles equipped with 3D sound, and user interfaces in video games.

[0072] The present invention has been described above based on embodiments. These embodiments are illustrative, and it will be understood by those skilled in the art that various modifications are possible in combinations of these components and processing processes, and that such modifications also fall within the scope of the present invention.

[0073] For example, at least some of the functions of the information processing device 10 may be provided by the server. In this case, data is transmitted and received between the information processing device 10 and the server via wireless communication.

[0074] Furthermore, the sound system 1 of this embodiment may be applied to a virtual reality (VR) environment. For example, an acoustic space may be pre-set within a virtual space such as a game or metaverse, and the user may operate virtual objects such as characters or avatars within the virtual space. The output unit 36 ​​may output a first acoustic signal to the first sound-producing device 12 attached to the user's right ear when the right ear of the virtual object enters the acoustic space, and a second acoustic signal may be output to the second sound-producing device 14 attached to the user's left ear when the left ear of the virtual object enters the acoustic space. This modification can enhance the sense of immersion in the acoustic space within the virtual space.

[0075] Furthermore, the second and third configuration examples of the sound system 1 of the embodiment may be combined. The second and fourth configuration examples of the sound system 1 of the embodiment may also be combined. The second, third, and third configuration examples of the sound system 1 of the embodiment may also be combined. The new embodiments resulting from these combinations will possess the combined effects of each of the embodiments that are combined. [Explanation of symbols]

[0076] 1...Acoustic system, 10...Information processing device, 12...First sound-producing device, 14...Second sound-producing device, 20...Imaging unit, 22...GPS receiving unit, 24...Processing unit, 30...First acquisition unit, 32...Second acquisition unit, 34...Determination unit, 36...Output unit, 110...Acoustic space.

Claims

1. A first acquisition unit that acquires information about the acoustic space, A second acquisition unit that acquires location information of the user's right and left ears, A determination unit determines whether the user's right ear and left ear have entered the acoustic space, based on the information acquired by the first acquisition unit and the second acquisition unit. An output unit that, when it is determined that the user's right ear has entered the acoustic space, outputs a first acoustic signal to a first sound-producing device attached to the user's right ear, and when it is determined that the user's left ear has entered the acoustic space, outputs a second acoustic signal to a second sound-producing device attached to the user's left ear. An information processing device characterized by comprising:

2. The second acquisition unit further acquires information on multiple locations around the user's head, The determination unit further determines whether each of the plurality of positions has entered the acoustic space, The output unit is, If it is determined that the user's right ear has entered the acoustic space, the first sound-producing device is output a first acoustic signal so that the user can hear the sound from the direction of the right ear. If it is determined that the user's left ear has entered the acoustic space, the second acoustic signal is output to the second sound-producing device so that the user can hear the sound from the direction of the left ear. If it is determined that at least one of the plurality of positions has entered the acoustic space, an acoustic signal is output to at least one of the first sound-producing device and the second sound-producing device so that the user can hear the sound from the direction of each position that has been determined to have entered the acoustic space. The information processing apparatus according to feature 1.

3. The first acoustic signal and the second acoustic signal are, respectively, acoustic signals for suppressing ambient noise. The information processing apparatus according to feature 1.

4. The first acoustic signal and the second acoustic signal each include an acoustic signal for suppressing ambient noise and an acoustic signal pre-set in the acoustic space. The information processing apparatus according to feature 1.

5. A method of information processing performed by a computer, The first acquisition step involves obtaining information about the acoustic space, The second acquisition step involves obtaining location information for the user's right and left ears, A determination step is performed to determine whether the user's right ear and left ear have entered the acoustic space, based on the information obtained in the first acquisition step and the second acquisition step. An output step in which, when it is determined that the user's right ear has entered the acoustic space, a first acoustic signal is output to a first sound-producing device attached to the user's right ear, and when it is determined that the user's left ear has entered the acoustic space, a second acoustic signal is output to a second sound-producing device attached to the user's left ear. An information processing method characterized by comprising:

6. The first acquisition step involves obtaining information about the acoustic space, The second acquisition step involves obtaining location information for the user's right and left ears, A determination step is performed to determine whether the user's right ear and left ear have entered the acoustic space, based on the information obtained in the first acquisition step and the second acquisition step. An output step in which, when it is determined that the user's right ear has entered the acoustic space, a first acoustic signal is output to a first sound-producing device attached to the user's right ear, and when it is determined that the user's left ear has entered the acoustic space, a second acoustic signal is output to a second sound-producing device attached to the user's left ear. A program characterized by causing a computer to execute something.

Images