Audio augmented reality object playback device and audio augmented reality object playback method

Abstract

To reproduce audio AR objects while suppressing difficulty in hearing sounds in the real space. [Solution] An audio augmented reality object reproduction device (1) includes multiple microphones (22A, 23A), audio reproduction devices (22B, 23B) worn on the left and right ears, and a processor (51). The processor controls the output of stereophonic audio from an audio augmented reality object placed in real space and detects the direction of a sound source in real space collected by the microphone. If the object's placement direction falls within a predetermined directional range relative to the sound source direction in real space, the processor corrects the object's placement position so that it is away from the sound source direction in real space, and outputs stereophonic audio based on the corrected position.

Description

[Technical Field]

[0001] The present invention relates to an audio augmented reality object playback device and an audio augmented reality object playback method. [Background technology]

[0002] Mixed reality (MR), which allows users to experience a mix of real and virtual spaces, is being used in games, maintenance work, sales promotions, and other content.

[0003] Virtual objects that form a virtual space include display AR (Augmented Reality) objects created using computer graphics, audio augmented reality objects (hereinafter referred to as "audio AR objects"), and hybrid objects that link display AR objects and audio AR objects. "Audio AR objects" are also called sound AR or audio AR.

[0004] As an example of mixed reality, it is possible to listen to an audio AR object such as a bird chirping in a scene in real space, or to play with a pet as an AR object in real space (at home or while walking). In this example, the pet's cries are linked as an audio AR object to the displayed AR object of the pet. The audio AR object is placed in a three-dimensional real space, and the user hears it as 3D audio.

[0005] As for audio processing in three-dimensional space, Patent Document 1 discloses a stereophonic audio reproduction device and reproduction method, and Patent Document 2 discloses a method for detecting the direction of a sound source. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2018-64227 [Patent Document 2] Japanese Patent Application Laid-Open No. 2006-227328 Summary of the Invention [Problem to be solved by the invention]

[0007] In a complex space that includes audio AR objects, overlapping and mixing with real-world sounds can be an issue. When real-world sounds are noise, they can interfere with listening to the audio AR object. When real-world sounds are voices from a third party, listening to the audio AR object can cause the user to miss the real-world sound. These issues become particularly pronounced when the audio AR object and the real-world sound are in the same direction.

[0008] However, neither Patent Document 1 nor Patent Document 2 describes the mixing of an audio AR object and a sound in the real space, where the object overlaps or mixes with the sound.

[0009] The present invention has been made in consideration of the above points, and has an object to reproduce an audio AR object while suppressing difficulty in hearing sounds in the real space. [Means for solving the problem]

[0010] Among the inventions disclosed in this application, the outline of representative inventions will be briefly explained as follows.

[0011] The audio augmented reality object playback device of the present invention is an audio augmented reality object playback device comprising a plurality of microphones, an audio playback device worn on the left and right ears of a listener, and a processor connected to each of the plurality of microphones and the audio playback device, wherein the processor places an audio augmented reality object in a real space, acquires a stereophonic audio signal from the audio augmented reality object, and controls the audio playback device to output audio based on the stereophonic audio signal, detects the sound source direction of the sound in the real space collected by the plurality of microphones, and if the placement position direction of the audio augmented reality object is included in a predetermined same directional range with respect to the sound source direction of the sound in the real space, corrects the placement position of the audio augmented reality object so that the placement position of the audio augmented reality object is moved away from the sound source direction of the sound in the real space, and controls the audio playback device to output audio based on the stereophonic audio signal at the corrected placement position.

[0012] The audio augmented reality object playback method of the present invention is a method for playing an audio augmented reality object using a playback device, characterized in that a processor of the playback device executes the following steps: placing an audio augmented reality object in real space; acquiring a stereophonic audio signal from the audio augmented reality object and controlling the audio playback device to output audio based on the stereophonic audio signal; detecting the sound source direction of sound in the real space collected by multiple microphones; if the placement position direction of the audio augmented reality object is included in a predetermined same directional range with respect to the sound source direction of the sound in the real space, correcting the placement position of the audio augmented reality object so that the placement position of the audio augmented reality object is moved away from the sound source direction of the sound in the real space; and controlling the audio playback device to output audio based on the stereophonic audio signal at the corrected placement position. [Effects of the Invention]

[0013] According to the present invention, it is possible to reproduce an audio AR object while suppressing difficulty in hearing sounds in real space. Problems, configurations, and effects other than those described above will become clear from the following description of the embodiments. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a functional block diagram of an audio AR object playback device according to a first embodiment. [Figure 2A] FIG. 10 is a diagram illustrating a mixed reality space formed by placing an audio AR object in a real space. [Figure 2B] FIG. 10 is a diagram showing a state in which sound congestion occurs. [Figure 2C] FIG. 10 is an explanatory diagram illustrating how to correct the placement of an audio AR object. [Figure 3] FIG. 1 is a hardware configuration diagram of an audio AR object playback device. [Figure 4] 4 is a flowchart showing a processing flow of the audio object reproduction device according to the first embodiment. [Figure 5] FIG. 10 is a functional block diagram of an audio AR object playback device according to a second embodiment. [Figure 6] FIG. 10 is a hardware configuration diagram of an audio AR object reproduction device 1A according to a second embodiment. [Figure 7] 10 is a flowchart showing the flow of processing in an audio object reproduction device according to the second embodiment. [Figure 8] FIG. 10 is an explanatory diagram illustrating how the position of an audio AR object is corrected in accordance with the movement of a sound source in real space. [Figure 9] 1 is an external view of a head-mounted display that is an embodiment of an audio AR object playback device. [Figure 10] FIG. 1 is a diagram illustrating a hardware configuration of a head-mounted display. [Figure 11] 10 is a flowchart showing a processing flow of an audio object reproduction device according to the fourth embodiment. [Figure 12]13 is a flowchart showing another example of the processing flow of the audio object reproduction device according to the fourth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings. Throughout the drawings, components having the same functions are designated by the same reference numerals, and repeated explanations thereof will be omitted. In the following description, an audio augmented reality object will be referred to as an "audio AR object."

[0016] The audio playback technology of the present invention will contribute to the achievement of "9. Build resilient infrastructure, promote inclusive and sustainable industrialization, foster innovation and inclusive governance," one of the Sustainable Development Goals (SDGs) advocated by the United Nations.

[0017] [First embodiment of the present invention] A first embodiment of the present invention will be described with reference to FIGS.

[0018] FIG. 1 is a functional block diagram of an audio AR object reproduction device according to the first embodiment.

[0019] The audio AR object reproducing device 1 shown in FIG. 1 includes an audio collection and reproducing unit 2 and an information processing unit 3.

[0020] The sound collection and reproduction unit 2 is an integrated combination of a microphone and a sound reproduction device, and is configured using, for example, headphones with a microphone or earphones with a microphone.

[0021] The sound collection and reproduction unit 2 includes a first communicator 20, a motion sensor 21, a left microphone array 22A, a left sound reproduction unit 22B, a right microphone array 23A, and a right sound reproduction unit 23B.

[0022] The left microphone array 22A and the left audio reproduction unit 22B are worn on the left side of an MR listener (also referred to as a listener), and the right microphone array 23A and the right audio reproduction unit 23B are worn on the right side of the listener.

[0023] The information processing unit 3 may be a portable information terminal, such as a smartphone or a tablet, or may be configured using a wearable terminal, such as a smart watch.

[0024] The information processing unit 3 includes an input operation unit 30, a communication control unit 31, a level detection unit 32, a sound source direction detection unit 33, a main control unit 34, an audio AR object placement unit 35, a placement correction unit 36, a head-related transfer function determination unit 37, a convolution calculation unit 38, an audio AR object D / B 40A, and a head-related transfer function D / B 40B. Note that "D / B" is an abbreviation for database.

[0025] In the sound collection and playback unit 2, a listener of an audio AR object in an MR space wears the left sound playback unit 22B on their left ear and the right sound playback unit 23B on their right ear. The sound collection and playback unit 2 is a semi-closed or open type (open-air type), and the listener hears a combination of the stereophonic sound reproduced based on the stereophonic sound signal of the audio AR object and the sounds of the real space.

[0026] The sound collection and playback unit 2 is, for example, an air conduction headphone that listens to air conduction sound transmitted by air vibrations, which is worn so that it touches the surface of the ear, and in particular, open-ear headphones are sometimes worn without completely blocking the ear, so that ambient sounds pass around the headphones and enter the ear, picking up and hearing the ambient sounds.Also, bone conduction headphones that listen to bone conduction sound transmitted by bone vibrations are worn without blocking the ear at all, so that ambient sounds enter the ear directly and are heard.

[0027] The sound collection and reproduction unit 2 may be of an air conduction type or a bone conduction type. The left microphone array 22A and the right microphone array 23A are located near the left sound reproduction unit 22B and the right sound reproduction unit 23B, and collect sounds in the real space that are heard by the left and right ears.

[0028] The movement sensor 21 detects the movement of the listener's head and detects the direction in real space in which the listener is facing.

[0029] The motion sensor 21 is configured by, for example, a gyro sensor, a geomagnetic sensor, and an acceleration sensor, all of which are not shown.

[0030] A gyro sensor is a sensor that detects angular velocity in the direction of rotation, and detects vertical, horizontal, and diagonal orientations.

[0031] The geomagnetic sensor is a sensor that detects the magnetic force of the earth, and detects the direction in which the sound collection and reproduction unit 2 body is facing.

[0032] An acceleration sensor is a sensor that detects acceleration, which is the change in speed per second, and detects movement, vibration, shock, etc.

[0033] Therefore, by using a gyro sensor or geomagnetic sensor, and in some cases by using an acceleration sensor in combination, it is possible to detect the movement of the listener's head. In particular, by using a three-axis geomagnetic sensor that detects geomagnetism in the up and down directions in addition to the front and back and left and right directions, it is possible to detect head movement with higher accuracy by detecting changes in geomagnetism in response to head movement.

[0034] The first communicator 20 communicates with the information processing unit 3. The first communicator 20 receives stereophonic signals to be reproduced by the left audio reproduction unit 22B and the right audio reproduction unit 23B from the information processing unit 3. The first communicator 20 also transmits microphone array signals collected by the left microphone array 22A and the right microphone array 23A, and detection signals from the motion sensor 21, to the information processing unit 3.

[0035] The first communicator 20 is typically an electronic tag, but is not limited to this, and examples of such a device include Bluetooth (registered trademark), IrDA (Infrared Data Association), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), or wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g). Although wireless communication is used here to improve usability, wired communication may of course also be used.

[0036] When the listener operates the input operation unit 30, the audio AR object placement unit 35 of the information processing unit 3 places the audio AR object in the real space in response to the operation, thereby forming an MR space. The audio AR object is selected from the audio AR object D / B 40A.

[0037] The level detection unit 32 and sound source direction detection unit 33 receive the microphone array signal received by the communication control unit 31 as input, and the level detection unit 32 detects the volume of the sound in the real space. When the volume is higher than a predetermined level, the level detection unit 32 notifies the placement correction unit 36. The sound source direction detection unit 33 detects the sound source direction of the sound in the real space and notifies the placement correction unit 36.

[0038] When the level difference between the sound in the real space and the sound of the audio AR object is relatively small enough to cause sound mixing and the sound source direction of the sound in the real space overlaps with the audio AR object, the placement correction unit 36 ​​corrects the placement of the audio AR object so that the sound in the real space and the audio of the audio AR object do not overlap. The above "relatively small enough to cause sound mixing" will be described later.

[0039] The head-related transfer function determination unit 37 determines a head-related transfer function from the position of the audio AR object, which has been corrected as necessary, and the orientation of the listener. The head-related transfer function is obtained by searching for the head-related transfer function D / B 40B, or is calculated from the head-related transfer function obtained by searching for the head-related transfer function D / B 40B.

[0040] The convolution calculation unit 38 performs a convolution calculation or an equivalent calculation on the head-related transfer function and the audio signal of the audio AR object to obtain a stereophonic audio signal, and transmits it to the audio collection and playback unit 2 via the communication control unit 31.

[0041] The main control unit 34 controls the operation of each unit of the information processing unit 3, such as timing.

[0042] FIG. 2A is a diagram illustrating a mixed reality space formed by placing an audio AR object in a real space.

[0043] In FIG. 2A, a sound source 4A of the sound in the real space 4 and an audio AR object 5 exist in the real space 4, and a listener 6 is listening to the sound of the sound source 4A and the sound of the audio AR object 5.

[0044] The listener 6 wears the sound collection and reproduction unit 2 on his head, and wears a left microphone array 22A and a left sound reproduction unit 22B, and a right microphone array 23A and a right sound reproduction unit 23B on his left and right ears, respectively. From the perspective of the listener 6, the direction θ0 of the placement position of the sound AR object 5 and the direction θ1 of the sound source 4A in the real space 4 are separated and are heard separately, so there is no sound mixing.

[0045] Here, "sound mixing" means that sound mixing is determined to have occurred when the level difference between the sound level of the sound source 4A in the real space 4 and the level of the audio AR object 5 is less than a predetermined level difference, and the placement position direction θ0 of the audio AR object 5 is included in a predetermined same direction range that is determined to be the same direction as the direction θ1 of the sound source 4A relative to the listener 6. The same direction range is defined as the range from margin angles -θmg to +θmg, with the direction θ1 as the center.

[0046] The direction of the sound source 4A in the real space 4 as seen by the listener 6 is detected at the positions of the left microphone array 22A and the right microphone array 23A, but is converted into a value at the center of the listener 6.

[0047] 2B is a diagram showing a state in which sound mixing occurs. FIG. 2B shows a case in which sound mixing occurs between an audio AR object 5 and a sound in real space, in which a sound source 4A of the sound in real space approaches the audio AR object 5. In this situation, the direction of the audio AR object 5 and the direction of the sound source 4A of the sound in real space overlap as seen from the listener 6. As a result, the sound of the audio AR object and the sound in real space are mixed together.

[0048] 2C is an explanatory diagram illustrating how to correct the position of the audio AR object 5. In the situation where the audio AR object 5 and the sound source 4A in the real space overlap as described in FIG. 2B, the position of the audio AR object 5 is corrected to position 5A, thereby avoiding the mixing of the sound of the audio AR object 5 and the sound in the real space.

[0049] FIG. 3 is a diagram showing the hardware configuration of the audio AR object reproduction device 1. As shown in FIG.

[0050] The audio AR object reproducing device 1 is configured by connecting an audio collection and reproduction unit 2 and an information processing unit 3 via communication.

[0051] The sound collection and reproduction unit 2 is a non-sealed type, for example, a wireless earphone, and the information processing unit 3 is a portable information terminal such as a smartphone, which is held and used by the listener.

[0052] The sound collection and playback unit 2 is as described in FIG. 1, but the first communication device 20 supports close proximity wireless communication and is used by pairing with the information processing unit 3 via close proximity wireless communication.

[0053] The information processing unit 3 includes a second communication device 50 , a processor 51 , a RAM 52 , a ROM 53 , a camera 55 , a distance measurement sensor 56 , a motion sensor 57 , an image display / input operation unit 58 , and an audio unit 59 .

[0054] The second communication device 50 supports mobile communication, wireless LAN protocols, and close proximity wireless communication. The mobile communication and wireless LAN can be used to connect to an external network and download data on the audio AR object 5 and head-related transfer functions from an external server or the like.

[0055] The processor 51 is composed of, for example, a CPU (Central Processing Unit).

[0056] The information processing unit 3 also includes a RAM 52 and a ROM 53 .

[0057] The ROM 53 includes an area for storing a processing program, ie, a basic operation program 53A and an audio AR object reproduction program 53B, and a data storage area 54 for storing data necessary for the placement processing of the audio AR object 5.

[0058] The data storage area 54 includes an audio AR object D / B 40A and a head-related transfer function D / B 40B. The data storage area 54 may also store data necessary to execute the basic operation program 53A and the audio AR object playback program 53B.

[0059] The processor 51 loads the basic operation program 53A and the audio AR object reproduction program 53B into the RAM 52 and executes them.

[0060] The listener holds the information processing unit 3 and takes a picture of the real space with the camera 55. At the same time, the distance sensor 56 measures the distance to the real space.

[0061] The distance sensor 56 is a sensor that detects the distance to an object, the direction of the object, and the relative speed of the object by emitting radio waves, light waves, ultrasonic waves, etc. and detecting the waves reflected from the object.

[0062] The distance measurement sensor 56 can detect the distance to an object present around the listener 6, the relative speed of the object, and the direction in which the object exists, with the listener 6 as the reference point.

[0063] The image display / input operation unit 58 is a flat display with a touch sensor stacked on top of it, and the camera image is displayed on the flat display of the image display / input operation unit 58. The listener 6 touches a location on the display screen to specify where to place the audio AR object 5.

[0064] The specified placement location is accompanied by distance data, and the real space 4 is treated as a three-dimensional space, and the audio AR object 5 is placed therein.

[0065] Next, an example of detailed processing contents in the processor 51 of Figures 1 and 3 will be described with reference to Figure 4. Figure 4 is a flowchart showing the flow of processing in the audio object reproduction device according to the first embodiment.

[0066] When the processor 51 starts the program (S10), it acquires a camera image and distance data (S11).

[0067] The listener 6 selects an audio AR object from the audio AR object D / B 40A, and the processor 51 places the audio AR object at a specified location on the camera image (S12).

[0068] The processor 51 receives a motion sensor signal from the sound collection and playback unit 2 (S13), detects a change in the orientation of the listener 6 (S14), and if there is a change (S14: YES), corrects the placement position of the sound AR object 5 (S15).

[0069] The information processing unit 3 receives the microphone array signal from the sound collection and reproduction unit 2 (S16). The level detection unit 32 detects the level of the microphone array signal (S17).

[0070] The processor 51 calculates the level difference between the audio signal level of the audio AR object 5 and the level of the microphone array signal. If the level difference is less than a predetermined level difference threshold defined as a first condition for determining that sound mixing has occurred (S18: YES), the sound source direction detection unit 33 detects the sound source direction of the sound in the real space 4 (S19). The second condition for determining that sound mixing has occurred is whether the sound source direction of the sound in the real space 4 is the same as the direction of the audio AR object 5. The term "directional identity" as used here refers not only to the case where the sound source direction is the same, but also to the case where the direction of the audio AR object 5 is included in a range that is considered to be the same as the sound source direction of the sound in the real space 4.

[0071] If the level difference is less than the level difference threshold in S18 (S18: YES) and there is directional identity, i.e., the sound source direction roughly overlaps with the audio AR object 5 (S20: YES), the processor 51 corrects the placement position of the audio AR object 5 (S21).

[0072] If the level difference is equal to or greater than the level difference threshold (S18: NO), or if there is no directional identity (S20: NO), and if the placement position of the sound AR object 5 has been corrected, the processor 51 returns the sound AR object 5 to the position before correction. If the placement position of the sound AR object 5 has not been corrected, the processor 51 leaves it as is (S22).

[0073] The processor 51 uses the head-related transfer function D / B 40B to determine the head-related transfer function.

[0074] The processor 51 performs a convolution operation of the determined head-related transfer function and the audio signal of the audio AR object, or an equivalent operation (S24), reconstructs the stereophonic audio signal of the audio AR object 5 in three dimensions, and transmits it to the audio collection and playback unit 2 (S25).

[0075] If the processor 51 continues the program and listens to the stereophonic signal (S26: YES), the process returns to step S13.

[0076] If the result in step S26 is negative, an end determination is made (S27), and if execution is to start from the placement of a new audio AR object (S27: NO), the process returns to step S11. If the result of the end determination is positive (S27: YES), the program is terminated (S28).

[0077] In addition to the above description, the left microphone array 22A, the right microphone array 23A, the camera 55, and the distance measurement sensor 56 mounted on the audio AR object reproduction device 1 may be used to measure the object generating the sound in the real space 4 coming towards the listener 6, the position of the sound source, the direction of the sound from the sound source to the listener 6, the volume of the sound, etc. This allows for even more accurate detection of the sound in the real space 4. Based on the detection results, the sound image of the audio AR object 5 can be controlled with high precision so that the listener 6 does not hear the sound in the real space 4 due to the audio AR object 5, or so that the listener 6 does not hear the audio AR object 5 due to the sound in the real space 4.

[0078] In the above description, the sound collection and reproduction unit 2 is of a semi-closed type or an open type, but it goes without saying that there is no problem if it is of a closed type.

[0079] As described above, the mixed reality audio AR object playback device and playback method of the first embodiment of the present invention has the characteristic that, when mixing of audio AR objects and real-space sounds occurs, it is possible to separate the audio AR objects and real-space sounds, making them easier to hear.

[0080] [Second embodiment of the present invention] A second embodiment of the present invention will be described with reference to FIGS.

[0081] FIG. 5 is a functional block diagram of an audio AR object reproduction device according to the second embodiment.

[0082] 5 is configured by connecting a sound collection and playback unit 2 and an information processing unit 3A for communication. The information processing unit 3A is configured by adding a microphone array signal addition unit 41, a microphone array signal recognition unit 42, and a registered microphone sound data set 40C to the information processing unit 3 according to the first embodiment. The sound collection and playback unit 2 is a closed-type wireless headphone or wireless earphone.

[0083] The registered microphone sound dataset 40C stores in advance as a dataset sound data (for example, human voice patterns, ambulance sirens, car horns, bicycle bells, etc.; generally, one or more sound data) to be delivered to the listener 6 as sound in the real space 4, and the microphone array signal recognition unit 42 analyzes the microphone array signal (for example, voice analysis processing using FFT analysis and voice recognition processing based on comparison of voice components), and if the microphone array signal is recognized as matching part of the registered sound dataset, the microphone array signal addition unit 41 adds the microphone array signal to the stereophonic audio signal and sends it to the audio collection and playback unit 2. Alternatively, the microphone array signal recognition unit 42 may determine whether the microphone array signal is human voice or not, and add the microphone array signal if it is human voice.

[0084] The sound collection and reproduction unit 2 is a sealed type, and sounds in the real space are difficult to hear directly, but if the sounds match sound data that is considered important and registered, they are added to the stereophonic sound signal and can be heard from the left sound reproduction unit 22B and the right sound reproduction unit 23B.

[0085] FIG. 6 is a diagram showing the hardware configuration of an audio AR object reproduction device 1A according to the second embodiment.

[0086] In the information processing unit 3A in FIG. 6, the registered microphone sound data set 40C is added to the data storage area .

[0087] Next, an audio AR object reproduction process according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a flowchart showing the flow of processing by the audio object reproduction device according to the second embodiment.

[0088] In the flowchart of FIG. 7, steps S40 to S42 and a registered microphone sound data set 40C are added to the flowchart of the audio AR object reproduction program 53B shown in FIG. 4, and step S43 is included instead of step S25.

[0089] If the processor 51 determines that the result in step S18 is affirmative, that is, that the level difference is less than the level difference threshold (S18: YES), the microphone array signal recognition unit 42 performs a voice analysis of the microphone array signal (S40).

[0090] When the processor 51 determines that the microphone array signal matches the registered sound data (S41: YES), the microphone array signal addition unit 41 combines the stereophonic sound signal and the microphone array signal (S42), and sends the combined sound signal from the second communication device 50 to the sound collection and playback unit 2 (S43).

[0091] On the other hand, if the processor 51 determines that the microphone array signal does not match the registered sound data (S41: NO), the process returns to step S19.

[0092] In the above explanation, the stereophonic audio signal and the microphone array signal are combined in step S42, but it is also possible to stop the output of the stereophonic audio signal and output only the microphone array signal, and then resume output of the stereophonic audio signal after the output of the microphone array signal has finished.

[0093] Furthermore, it is also possible to change the method of changing the placement position of the audio AR object depending on whether or not the microphone array signal is recognized as registered sound data as a result of the determination in step S41.

[0094] For example, if the sound data is recognized as registered sound data, the audio AR object 5 is changed in its position to eliminate the sound congestion, and if the sound data is not recognized as registered sound data, the audio AR object 5 is not changed in its position.

[0095] In the above description, the sound collection and reproduction unit 2 is of a sealed type, but it goes without saying that there is no problem if it is of a non-sealed type.

[0096] As described above, the mixed reality audio AR object playback device and playback method of the second embodiment of the present invention have the same features as the first embodiment, and also have the feature that the sounds in the real space 4 heard by the listener 6 can be limited to sounds that can be determined to be important, making it easier to concentrate on listening to the three-dimensional sound of the audio AR object 5.

[0097] [Third embodiment of the present invention] A third embodiment of the present invention will be described with reference to Fig. 8. The audio AR object reproduction devices 1 and 1A shown in the first and second embodiments of the present invention can be applied as the audio AR object reproduction device. This embodiment can be applied when there is movement of the sound source 4A of the sound and the audio AR object 5 in the real space 4.

[0098] 8 is an explanatory diagram showing how the position of an audio AR object is corrected in response to the movement of a sound source in real space. In FIG. 8, only the sound source 4A in real space 4 is depicted as moving, but this can be considered as the relative movement of the sound source 4A in real space 4 and the audio AR object 5.

[0099] The sound source 4A in the real space 4 moves as time passes from T0 to T1 to T2, and so on. At time T3, the directions of the audio AR object 5 and the listener 6 overlap. To avoid this overlap, the placement position of the audio AR object 5 is corrected to placement position 5A.

[0100] The placement correction unit 36 ​​refers to a motion vector to determine the relative movement of the sound in real space with the sound source 4A. The direction and amount of movement from time T2 to T3 are the motion vector, and the direction in which the audio AR object 5 is moved to placement position 5A is set to the opposite direction to the motion vector, and the amount of movement is set to be smaller as the amount of movement of the motion vector increases. Furthermore, the distance from the listener 6 to the position 5 of the audio AR object and placement position 5A is set to be approximately the same. As a result, the amount of movement of the audio AR object 5 can be reduced, and the change in volume before and after placement correction can also be reduced.

[0101] As described above, the third embodiment of the present invention has the feature that the listener 6 can hear a stereophonic sound with little change before and after the position of the audio AR object 5 is corrected.

[0102] [Fourth embodiment of the present invention] A fourth embodiment of the present invention will be described with reference to FIGS.

[0103] FIG. 9 is an external view of a head-mounted display 7 that is an embodiment of an audio AR object playback device.

[0104] The head-mounted display 7 (hereinafter abbreviated as "HMD") shown in FIG. 9 includes a camera 70, a distance measurement sensor 71, a motion sensor 72, an image display unit 73, a left audio reproduction unit 74B, a right audio reproduction unit 75B, a right microphone array 75A, a control unit 81, and a holder 82. The left microphone array 74A is located on the back side of the holder 82 and is hidden in FIG. 9. A user of the HMD 7 wears the holder 82 on the user's head and positions the image display unit 73 of the HMD 7 in front of their eyes. The user of the HMD 7 is also a listener 6 of the stereophonic sound, and therefore, hereinafter, the user of the HMD 7 will be referred to as the listener 6.

[0105] The image display unit 73 may be a semi-transmissive screen or a non-transmissive display. With a semi-transmissive screen, the listener 6 can view the forward background by passing through the semi-transmissive screen, but with a non-transmissive display, the listener 6 can view the forward background by displaying a camera image of the forward background on the non-transmissive display.

[0106] If the image display unit 73 is a semi-transparent screen, the listener 6 directly sees the background in front of him / her through an optical combiner made of a half mirror.

[0107] When the image display unit 73 is a non-transmissive display, it is configured with a liquid crystal panel or the like, and displays real space information and virtual space information of the background in front of the listener 6 as images, and also displays on the screen display contents such as presentation notification information and operation status to the listener 6. The image display unit 73 may also be a three-dimensional image display unit.

[0108] For example, when sound output is started, interrupted, or resumed, the image display unit 73 may display a message to notify the listener 6. This allows the listener to recognize that when sound output is interrupted or resumed, the interruption or resumption is due to a normal control operation and not due to a malfunction.

[0109] Camera 70 is attached so as to capture the background in front of listener 6's line of sight, and ranging sensor 71 measures the distance to the background. Ranging sensor 71 is a sensor that detects the distance to an object, as well as the object's direction and relative speed, by emitting radio waves, light waves, ultrasonic waves, etc. and detecting the waves reflected from the object. Ranging sensor 71 can detect the distance to an object present around listener 6, the object's relative speed, and the object's direction, using listener 6 as a reference point.

[0110] The control unit 81 takes in camera images taken by the camera 70 and distance data measured by the distance sensor 71, and supplies them to an internal memory and CPU. It also has a built-in communication device, and outputs images to be displayed on the image display unit 73 and sounds to be output to the left audio reproduction unit 74B and right audio reproduction unit 75B.

[0111] The control unit 81, camera 70, distance measurement sensor 71, left audio reproduction section 74B, right audio reproduction section 75B, left microphone array 74A, and right microphone array 75A are attached to a holder 82, but this does not have to be as shown in FIG.

[0112] FIG. 10 is a diagram showing the hardware configuration of the head-mounted display 7.

[0113] The HMD 7 shown in Figure 10 includes a camera 70, a distance measurement sensor 71, a motion sensor 72, an image display unit 73, a left microphone array 74A, a right microphone array 75A, a left audio reproduction unit 74B, a right audio reproduction unit 75B, a wireless communication device 76, a processor 77, a RAM 78, and a ROM 79.

[0114] The wireless communication device 76 supports protocols such as mobile communication and wireless LAN, as well as near-field communication. Mobile communication and wireless LAN can be used to connect to an external network and download hybrid object and head-related transfer function data from an external server. The wireless communication device 76 may be configured to connect to the microphone array (74A, 75A) and audio playback units (74B, 75B) via near-field communication. While a representative example of near-field communication is an electronic tag, it is not limited to this. Examples of such a wireless communication include Bluetooth (registered trademark), IrDA (Infrared Data Association), Zigbee (registered trademark), HomeRF (Home Radio Frequency, registered trademark), and wireless LAN (IEEE802.11a, IEEE802.11b, IEEE802.11g). While wireless communication is used here to improve usability, wired communication may also be used.

[0115] The processor 77 includes a CPU and the like.

[0116] The RAM 78 is a volatile memory.

[0117] The ROM 79 is a non-volatile memory medium such as a flash ROM, and includes processing programs such as a basic operation program 79A and an audio AR object playback program 79B, as well as a data storage area 80. The data storage area 80 further includes a hybrid object D / B 40D, a head-related transfer function D / B 40B, and a registered microphone sound data set 40C.

[0118] The processing programs of the basic operation program 79A and the audio AR object reproduction program 79B are expanded in the RAM 78 and executed by the processor 77. The data storage area 80 may also store data necessary for executing the basic operation program 79A and the audio AR object reproduction program 79B.

[0119] The control unit 81 includes a radio 76 , a processor 77 , a RAM 78 , and a ROM 79 .

[0120] The listener 6 wears the HMD 7 on his / her face and captures the real space in front of him / her with the camera 70. At the same time, the distance to the real space is measured with the distance sensor 71. A location in the real space is also selected and the hybrid object is placed there.

[0121] Next, a flow chart showing an example of detailed processing contents in the control unit 81 of FIGS. 9 and 10 will be described with reference to FIGS. 11 and 12. FIG.

[0122] FIG. 11 is a flowchart showing the flow of processing performed by the audio object reproduction device according to the fourth embodiment.

[0123] 11, steps that perform the same processes as those in the flowcharts of the audio AR object playback program 53B shown in FIGS. 4 and 7 are assigned the same numbers, and duplicated explanations will be omitted. In this flowchart, a closed-type audio playback unit is assumed, but this is not limited to this, and an open-type audio playback unit may also be used.

[0124] The flowchart in Fig. 11 differs from the flowchart in Fig. 7 in that the audio AR object D / B 40A is replaced with a hybrid object D / B 40D, the hybrid object is placed in step S50, the position of the hybrid object is corrected in step S51, and the hybrid object is returned to its original position in step S52. The audio AR object in the flowchart in Fig. 7 can be read as the hybrid object.

[0125] FIG. 12 is a flowchart showing another example of the processing flow of the audio object reproduction device according to the fourth embodiment.

[0126] In Fig. 12, the same numbers are assigned to steps that perform the same processes as in the flowchart of Fig. 11, and duplicated explanations will be omitted. In this flowchart, a sealed audio playback unit is assumed, but this is not limited to this and a non-sealed type may also be used.

[0127] The flowchart in Fig. 12 differs from the flowchart in Fig. 11 in steps S53, S54, and S55. A hybrid object is an object in which an audio AR object and an AR object are linked. In the first flowchart in Fig. 11, step S50 places the hybrid object in real space, i.e., places it in a world coordinate system fixed to real space, but in the second flowchart in Fig. 12, step S53 places the hybrid object in the visual space of the HMD 7, i.e., places it in a local coordinate system fixed to the HMD 7.

[0128] The difference between the two is that when the listener 6 moves their face, the former hybrid object moves within the visual field of the listener 6, while the latter hybrid object's position within the visual field is fixed. In other words, the former hybrid object is a hybrid object placed in the coordinate system of real space, and becomes invisible when the listener 6 moves away from that location. On the other hand, because it has the same area as real space, it has the advantage of being able to place a large number of hybrid objects.

[0129] The latter hybrid object is a hybrid object placed in a coordinate system fixed to the HMD 7, and its positional relationship with the image display unit 73 of the HMD 7 is also fixed. Then, when viewed from the viewpoint of the listener 6, the hybrid object placed in the direction in which the display surface of the image display unit 73 exists is displayed on the image display unit 73. If the hybrid object is placed in a coordinate system within the directional range in which the display surface of the image display unit 73 exists, there is an advantage that even if the listener 6 moves while wearing the HMD 7, the image display unit 73 is also fixed to the coordinate system, and therefore the hybrid object can always be displayed and operated.

[0130] Therefore, when the sound of the real space and the sound of the hybrid object are mixed, in the second flowchart of Figure 12, the hybrid object can eliminate the mixing of the sound of the real space and the sound of the hybrid object by having the listener 6 change the direction of his or her face.

[0131] In step S54, the processor 77 waits for a predetermined time to allow the listener 6 to turn his or her face (S54).

[0132] The processor 77 again determines whether the sound source directions overlap (S55), and if the overlap continues (S55: YES), the placement corrector 36 rearranges the hybrid objects (S51).

[0133] If the overlap has been resolved (S55: NO) and the placement position of the hybrid object has been corrected, the hybrid object is returned to its position before correction (S52), or if the placement position of the hybrid object has not been corrected, the hybrid object is left as is (S52).

[0134] In addition to the above explanation, in this embodiment, the hybrid object is fixed in the visual space of the HMD 7 of the listener 6, so it is necessary to prevent the sound of the real space and the sound of the hybrid object from being mixed together due to the movement of the listener 6. The method for avoiding the sound mixing may be as follows.

[0135] In other words, when the movement of the listener 6 brings it closer to the sound of the real space, the movement of the hybrid object's sound is stopped before the hybrid object's sound overlaps with the sound of the real space, and then when it moves away from the sound of the real space and there is no longer any mixing with the sound of the real space, the hybrid object's sound is returned to its original fixed position.

[0136] Also, even if the listener 6 is not moving but there is sound confusion due to the movement of sound in the real space, the position of the audio of the hybrid object may be corrected by horizontal movement (either left or right) or vertical movement.

[0137] Furthermore, the placement position of the AR object of the hybrid object may or may not be moved in synchronization with the correction of the position of the audio of the hybrid object.

[0138] This means that the placement position is moved when it is necessary for the listener 6 to see the real-space object that is the source of the sound in the real space without being visually obstructed by the AR object of the hybrid object, and the placement position is not moved when it is not necessary.

[0139] Alternatively, the display of the AR object of the hybrid object may be stopped before the audio of the hybrid object overlaps with the audio of the real space. Then, when the audio interference is resolved, the AR object of the hybrid object is redisplayed. This allows the listener to see the real-space object that is the source of the audio of the real space without being visually obstructed by the AR object of the hybrid object.

[0140] As described above, even a head-mounted display that handles hybrid objects has the advantage that it is possible to listen to stereophonic sound while avoiding mixing with sounds in the real space.

[0141] The methods for avoiding the mixing of the sound in the real space with the sound of the audio AR object 5 or the hybrid object described in the first, second, third, and fourth embodiments may be as follows. For example, the following methods (1) to (9) may be used.

[0142] (1) In a situation where the sound image of the audio AR object 5 or hybrid object moves vertically in front of the listener 6, the sound image of the audio AR object 5 or hybrid object temporarily moves horizontally (for example, within plus or minus 4 degrees, which is said to be the horizontal perception limit of hearing).

[0143] (2) In a situation where the sound image of the audio of the audio AR object 5 or hybrid object moves horizontally in front of the listener 6, the sound image of the audio of the audio AR object 5 or hybrid object is temporarily moved vertically.

[0144] (3) In a situation where the sound image of the audio of the audio AR object 5 or hybrid object moves in the depth direction relative to the front of the listener 6, the sound image of the audio of the audio AR object 5 or hybrid object is temporarily moved horizontally or vertically.

[0145] (4) The sound image of the audio of the audio AR object 5 or hybrid object is controlled based on the frequency characteristics of the sound in the real world and the frequency characteristics of the sound of the audio AR object 5 or hybrid object. If the frequency characteristics of the sound of the audio AR object 5 or hybrid object do not interfere with the audibility of the sound in the real world, no action is taken.

[0146] (5) When the listener 6 moves their head up, down, forward, or backward, the audio image of the audio AR object 5 or hybrid object does not move. The audio image of the audio AR object 5 or hybrid object also does not move when the head moves within 4 degrees left or right.

[0147] (6) When the listener's body moves up, down, forward, or backward, the audio image of the audio AR object 5 or hybrid object does not move. The audio image of the audio AR object 5 or hybrid object also does not move when the listener's body moves within 4 degrees left or right.

[0148] (7) When the AR object of the hybrid object disappears from view due to head or body movement, the audio output of the hybrid object also stops.

[0149] (8) The output priority of the audio of the audio AR object 5 or hybrid object and the audio of the real space may be determined, and the sound image of the audio of the hybrid object may be controlled according to the priority.

[0150] (9) When there are multiple sounds in the real space, the sound image of the audio AR object 5 or the hybrid object is controlled so as not to interfere with the hearing of each of them.

[0151] Although the embodiments of the present invention have been described above, it goes without saying that the configurations for realizing the techniques of the present invention are not limited to the above-described embodiments, and various modifications are possible. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those having all of the described configurations.

[0152] Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.

[0153] Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

[0154] It should be noted that the components and relative sizes shown in the drawings are simplified and idealized to make the present invention easier to understand, and may have more complex shapes in actual implementation.

[0155] These all fall within the scope of the present invention, and furthermore, the numerical values ​​and messages appearing in the text and figures are merely examples, and the effects of the present invention will not be impaired even if different ones are used.

[0156] Furthermore, the programs described in each processing example may be independent programs, or multiple programs may constitute a single application program.

[0157] The order in which the processes are performed may also be changed.

[0158] Furthermore, some or all of the functions of the invention may be implemented in hardware, for example by designing them as an integrated circuit.

[0159] Alternatively, it may be implemented in software by a microprocessor unit, CPU, or the like interpreting and executing an operating program.

[0160] Furthermore, the scope of software implementation is not limited, and both hardware and software may be used.

[0161] Furthermore, some or all of the functions may be implemented by a server. The server may be any type of server, as long as it can execute the functions in cooperation with other components via communications, such as a local server, cloud server, edge server, or online service. Information such as programs, tables, and files that implement the functions may be stored in a memory, a recording device such as a hard disk or solid-state drive (SSD), or a recording medium such as an IC card, SD card, or DVD, or may be stored in a device on a communications network.

[0162] In addition, the control lines and information lines shown are those that are considered necessary for the explanation, and do not necessarily show all the control lines and information lines in the product. In reality, it can be assumed that almost all components are interconnected. [Explanation of symbols]

[0163] 1: Audio AR object playback device 1A: Audio AR object playback device 2: Audio collection and playback section 3: Information Processing Section 3A: Information Processing Department 4: Real space 4A: Real-world sound sources 5: Audio AR object 5A: Placement position 6: Listener 7: HMD (Head Mounted Display) 20: First communication device 21: Motion sensor 22A: Left microphone array 22B: Left audio playback section 23A: Right microphone array 23B: Right audio playback section 30: Input operation section 31: Communication control section 32: Level detection section 33: Sound source direction detection unit 34: Main control unit 35: Audio AR object placement section 36: Placement correction section 37: Head-related transfer function determination unit 38: Convolution operation unit 40A: Audio AR Object D / B 40B: Head-related transfer function D / B 40C: Registered microphone sound data set 40D: Hybrid Object D / B 41: Microphone array signal addition unit 42: Microphone array signal recognition unit 50: Second communication device 51: Processor 52: RAM 53:ROM 53A: Basic operation program 53B: Audio AR object playback program 54: Data storage area 55: Camera 56: Distance sensor 57: Motion sensor 58: Image display and input operation section 59: Audio section 70: Camera 71: Distance sensor 72: Motion sensor 73: Image display section 74A: Left microphone array 74B: Left audio playback section 75A: Right microphone array 75B: Right audio playback section 76: Radio communication device 77: Processor 78:RAM 79:ROM 79A: Basic operation program 79B: Audio AR object playback program 80: Data storage area 81: Control unit 82: Holder

Claims

1. Multiple microphones and an audio playback device to be worn on the left and right ears of a listener; a processor connected to each of the plurality of microphones and the audio playback device, The processor: Place audio augmented reality objects in real space, acquiring a stereophonic audio signal from the audio augmented reality object, and performing control to output audio based on the stereophonic audio signal from the audio playback device; Detecting the sound source direction of the sound in the real space collected by the plurality of microphones; When a direction of a placement position of the audio augmented reality object is included in a predetermined same direction range with respect to a sound source direction of the sound in the real space, correcting a placement position of the audio augmented reality object so that the placement position of the audio augmented reality object is moved away from the sound source direction of the sound in the real space; performing control to cause the audio playback device to output audio based on the stereophonic audio signal at the corrected placement position; An audio augmented reality object playback device.

2. 2. The audio augmented reality object playback device according to claim 1, The processor: performing a speech recognition process on the real-space sounds collected by the plurality of microphones; synthesizing the sound of the real space and a stereophonic audio signal based on the result of the voice recognition processing, and outputting the synthesized sound to the voice reproduction device; An audio augmented reality object playback device.

3. 2. The audio augmented reality object playback device according to claim 1, The processor: If, after correcting the placement position of the audio augmented reality object, the sound source direction of the sound in the real space and the placement position direction of the audio augmented reality object are out of the same direction range, the placement position of the audio augmented reality object is returned to the state before correction. An audio augmented reality object playback device.

4. 2. The audio augmented reality object playback device according to claim 1, The processor: In response to a relative movement between the audio augmented reality object and a sound source in the real space, correcting a placement position of the audio augmented reality object in a direction opposite to the direction of the relative movement; An audio augmented reality object playback device.

5. The audio augmented reality object playback device according to claim 4, The processor: correcting the placement position of the audio augmented reality object at a distance equal to the distance from the placement position of the audio augmented reality object to the listener before the correction and in a direction opposite to the direction of the relative movement; An audio augmented reality object playback device.

6. 6. The audio augmented reality object playback device according to claim 1, The audio augmented reality object is placed in a world coordinate system fixed to real space. An audio augmented reality object playback device.

7. 6. The audio augmented reality object playback device according to claim 1, the audio augmented reality object is positioned in a local coordinate system fixed to the audio playback device; An audio augmented reality object playback device.

8. 6. The audio augmented reality object playback device according to claim 1, the processor is included in a personal digital assistant or a wearable device, the microphone and the audio playback device are provided in earphones or headphones configured separately from the portable information terminal or the wearable terminal, The mobile information terminal or the wearable terminal is communicatively connected to the earphone or the headphone, respectively. An audio augmented reality object playback device.

9. 6. The audio augmented reality object playback device according to claim 1, The microphone, the audio playback device, and the processor are integrally configured in a head-mounted display. An audio augmented reality object playback device.

10. 1. A method for playing back an audio augmented reality object using a playback device, comprising: a processor of the playback device, placing an audio augmented reality object in a real space; a step of acquiring a stereophonic audio signal from the audio augmented reality object and performing control to output audio based on the stereophonic audio signal from an audio playback device; detecting a sound source direction of the sound in the real space collected by a plurality of microphones; When the direction of the placement position of the audio augmented reality object is included in a predetermined same direction range with respect to the sound source direction of the sound in the real space, correcting the placement position of the audio augmented reality object so that the placement position of the audio augmented reality object is moved away from the sound source direction of the sound in the real space; a step of controlling the audio playback device to output audio based on the stereophonic audio signal at the corrected placement position; 2. A method for playing back an audio augmented reality object, comprising:

Images