Method and system for simulating spatial sound effect in augmented reality

Abstract

The invention provides a method for simulating a spatial sound effect in augmented reality, which is applied to an augmented reality (AR) device, and is characterized by comprising the following steps: in an AR scene, an AR object positioned on an anchor point device sends out a plurality of sound packets; calculating a distance, an angle and a space relationship between the AR device and the anchor point device; and according to the calculated distance, the calculated angle, the calculated spatial relationship, the calculated original sound source and the calculated user volume setting, converting the sound packets into sound signals of 6 degrees of freedom (DOF) through a sound effect positioning algorithm, and transmitting the sound signals to a playing device of the AR device.

Description

Technical Field

[0001] The present invention relates to augmented reality technology, and more particularly to a method and system for simulating spatial sound effects in augmented reality. Background Art

[0002] The establishment of anchor points in existing augmented reality requires a complete scan of the physical anchor points, obtaining the anchor point features to establish the physical anchor point's spatial information (i.e., the model), and then binding the virtual object to the anchor point so that the virtual object can be fully presented on the anchor point. However, during scanning, the image is easily affected by angle, direction, light, surrounding environment, distance, and algorithm, resulting in uncertainty in the anchor point calculation results.

[0003] The above problems have been solved, however, in the prior art, only the relative rotation between the head device and the source device can produce the spatial sound source effect, that is, the spatial effect of 3 degrees of freedom (DOF). In other words, the relationship between the head device and the source device is only a relative 3-axis relationship of a fixed point, and there is no relative 3-axis relationship of a moving point. In other words, when the user approaches or moves away from the source device, there is no change in the sound field, and only the relative rotation between the head device and the source device can produce a change in the sound field. Summary of the invention

[0004] In view of this, the present invention provides a method and system for simulating spatial sound effects in augmented reality, which generates spatial sound source effects by rotating and moving the user's head and changing the relative distance between the user and the source device.

[0005] An embodiment of the present invention provides a method for simulating spatial sound effects in augmented reality, which is applied to an augmented reality (AR) device, and is characterized by comprising: in an AR scene, an AR object located on an anchor device emits a plurality of sound packets; calculating the distance, angle, and spatial relationship between the AR device and the anchor device; and according to the calculated distance, angle, spatial relationship, native sound source, and user volume setting, converting the sound packets into 6-degree-of-freedom (DOF) sound signals through a sound effect localization algorithm and transmitting the sound signals to a playback device of the AR device.

[0006] The embodiment of the present invention also provides a system for simulating spatial sound effects in augmented reality, including an anchor device and an AR device. The anchor device is used to provide an AR object for emitting a plurality of sound packets in an AR scene. The AR device is used to calculate the distance, angle, and spatial relationship between the anchor device and the anchor device, and according to the calculated distance, angle, spatial relationship, native sound source, and user volume setting, the sound packets are converted into 6DOF sound signals through a sound effect positioning algorithm and transmitted to a playback device of the AR device.

[0007] The sound source disclosed by the method and system for simulating spatial sound effects in the embodiments of the present invention can present 6 degrees of freedom (DoF, i.e., head rotation and body movement), and further discloses the reflected sound between the user and the virtual object and the reflected sound between the user and the real object.

[0008] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 A schematic diagram showing a system for simulating spatial sound effects in augmented reality according to an embodiment of the present invention.

[0010] Figure 2 A flow chart showing a method for simulating spatial sound in augmented reality according to an embodiment of the present invention.

[0011] Figure 3 A schematic diagram showing the transmission of a spatial information packet according to an embodiment of the present invention is shown.

[0012] Figure 4 A schematic diagram showing an application scenario of a system for simulating spatial sound effects in augmented reality according to an embodiment of the present invention.

[0013] Figure 5 A top view of an anchor device according to an embodiment of the present invention is shown.

[0014] Figure 6 A side view of an anchor device according to an embodiment of the present invention is shown.

[0015] Figure 7 A schematic diagram of an AR device according to an embodiment of the present invention is shown.

[0017] Figure 8 A flowchart showing the steps of simulating spatial sound effects according to an embodiment of the present invention.

[0018] Fig. 9 A schematic diagram showing an application example of the simulated spatial sound effect according to the first embodiment of the present invention.

[0019] Fig.10 A schematic diagram showing an application example of the simulated spatial sound effect according to the second embodiment of the present invention.

[0020] Main component symbols

[0021]

[0022]

[0023] The following specific implementation manner will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0024] In order to facilitate ordinary technicians in the field to understand and implement the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the embodiments described herein are only used to illustrate and explain the present invention and are not used to limit the present invention.

[0025] Figure 1 A schematic diagram of a system for simulating spatial sound effects in augmented reality according to an embodiment of the present invention is shown. The system includes an anchor device 110 and an augmented reality (AR) device 120. Figure 1 As shown, the anchor device 110 sends a first spatial information packet 111 and a second spatial information packet 112 at intervals of T. The anchor device 110 further includes a transmitter ( Figure 1 Not shown), the first spatial information packet 111 and the second spatial information packet 112 are sent through the first communication technology and the second communication technology. The first spatial information packet 111 and the second spatial information packet 112 both include the anchor point number of the anchor point device 110 and the angle information of the transmitter. The first communication technology and the second communication technology can be a combination of any two of wireless radio frequency signals, ultrasound, infrared, and laser.

[0026] In one embodiment, the transmitter further includes a plurality of sub-transmitters for sending the first spatial information packet and the second spatial information packet via the first communication technology and the second communication technology respectively.

[0027] The first spatial information packet 111 and the second spatial information packet 112 are received by an augmented reality (AR) device 120. The AR device 120 further includes a processing unit ( Figure 1 (not shown) through the processing unit, the spatial relationship between the AR device 120 and the anchor device 110 is obtained according to the packet contents of the first spatial information packet 111 and the second spatial information packet 112, wherein the spatial relationship includes the position, horizontal distance, and vertical height of the AR device 120 relative to the anchor device 110.

[0028] Figure 2 A flow chart showing a method for simulating spatial sound in augmented reality according to an embodiment of the present invention.

[0029] In step S101, the anchor device sends a first spatial information packet via a first communication technology and sends a second spatial information packet via a second communication technology, wherein both the first spatial information packet and the second spatial information packet include angle information, and the angle information includes the azimuth of the transmitter pointing to the horizontal position in the anchor device and the angle between the transmitter and the horizontal plane.

[0030] In step S102 , the AR device receives a first spatial information packet sent by the anchor device via a first communication technology.

[0031] In step S103 , the AR device receives a second spatial information packet sent by the anchor device via a second communication technology.

[0032] In step S104, the processing unit in the AR device obtains the spatial relationship between the AR device and the anchor device according to the first spatial information packet and the second spatial information packet, wherein the spatial relationship includes the position, horizontal distance, and vertical height of the AR device relative to the anchor device. The processing unit determines the position of the AR device relative to the anchor device according to the azimuth in the angle information.

[0033] In step S105, the AR device calculates the distance between the AR device and the anchor device. Since the transmission speeds of the first communication technology and the second communication technology are known, the AR device can calculate the distance between the AR device and the anchor device according to the transmission speed of the first communication technology, the transmission speed of the second communication technology, the time of receiving the first spatial information packet, and the time of receiving the second spatial information packet. For example, the anchor device simultaneously sends the first spatial information packet through radio frequency (RF) and sends the second spatial information packet through ultrasound. At time T0, the AR device receives the first spatial information packet sent by RF, and at time T1, the AR device receives the second spatial information packet sent by ultrasound. Since the propagation speed of RF is approximately equal to the speed of light, and the propagation speed of ultrasound is slower, T1>T0, and the reception time difference between the first spatial information packet and the second spatial information packet is T1-T0. The distance between the AR device and the anchor device is calculated according to the formula of distance = speed x time, the known transmission speed of the first communication technology, the known transmission speed of the second communication technology, and the above-mentioned reception time difference.

[0034] In step S106, the AR device can calculate the horizontal distance and vertical height between the AR device and the anchor device according to the distance calculated in step S105 and the angle between the transmitter and the horizontal plane in the first spatial information packet.

[0035] Figure 3 A schematic diagram showing the transmission of spatial information packets according to an embodiment of the present invention. Figure 3As shown, the anchor device 310 simultaneously sends the first spatial information packet and the second spatial information packet at an interval of time T through the first communication technology 330 and the second communication technology 340. In this embodiment, the first communication technology 330 is radio frequency (RF) and the second communication technology 340 is ultrasound as an example, but it is not limited to these two communication technologies. Since the transmission speed of RF is like the speed of light 3x108m / s, and the transmission speed of ultrasound in air is about 343m / s, the AR device 302 receives the first spatial information packet sent by the anchor device 301 via RF at time T0, and receives the second spatial information packet sent by ultrasound at time T1, T1>T0, according to the above T1-T0 time difference, and the known transmission speed of the first communication technology 330 and the second communication technology 340, the distance between the AR device 302 and the anchor device 301 can be calculated.

[0036] Figure 4 A schematic diagram showing an application scenario of a system for simulating spatial sound effects in augmented reality according to an embodiment of the present invention. Figure 4 As shown, the anchor device 410 is set on the ground, and the transmitters set at various angles periodically send spatial information packets via the first communication technology and the second communication technology. When the user wears the AR device 420 and enters the application range of the anchor device 410, the AR device 420 receives the spatial information packet sent by one of the transmitters 411 of the anchor device 410. The angle between the transmitter 411 and the horizontal plane is 430. Since the anchor device 410 sends the spatial information packet through two communication technologies at the same time, the spatial information packet includes the azimuth pointed by the transmitter 411 and the angle with the horizontal plane. Therefore, the AR device 420 can calculate the distance 440 between the anchor device 410 and the AR device 420 according to the time difference between receiving the spatial information packets sent by the first communication technology and the second communication technology, and the angle 430 between the transmitter 411 and the horizontal plane in the received spatial information packet. After knowing the distance 440 between the anchor device 410 and the AR device 420, the horizontal distance 450 and the vertical height 460 between the anchor device 410 and the AR device 420 can be calculated according to the angle 430 between the transmitter 411 and the horizontal plane. The spatial information packet carries the anchor number and the angle information, wherein the AR device 420 determines which anchor device in the space the received spatial information packet comes from according to the anchor number, and the angle information of the spatial information packet includes the azimuth pointed by the transmitter 411 in the anchor device 410 and the angle 430 with the horizontal plane.

[0037] Figure 5 A top view of an anchor device 500 according to an embodiment of the present invention is shown. Figure 5As shown, the anchor device 500 uses directional communication technology and sets multiple transmitters at different azimuth angles in the anchor device 500 as a criterion for determining the relative position between the AR device and the anchor device 500. According to the characteristics of the directional communication technology, the AR device will only receive spatial information packets from a specific direction. Figure 5 For example, when the AR device is located at an azimuth angle of 180 degrees relative to the anchor device 500, it will only receive the signal from the fourth transmitter 540. The angle information contained in the spatial information packet sent by the fourth transmitter 540 is located at an azimuth angle of 180 degrees to the anchor device and an angle of 0 degrees with the horizontal plane, thereby determining the relative position of the AR device and the anchor device 500. However, due to the directional limitations of directional communication technology, the azimuth angles that can be covered by a single set of transmitters may be limited. Therefore, the anchor device 500 needs to be equipped with multiple sets of transmitters to effectively cover all angles within the entire space range. Figure 5 Taking the angle range covered by each transmitter as 60 degrees as an example, at least six transmitters are required to cover all azimuths, including the first transmitter 510 pointing to an azimuth of 0 degrees covering a range of 330 degrees to 30 degrees, the second transmitter 520 pointing to an azimuth of 60 degrees covering a range of 30 degrees to 90 degrees, the third transmitter 530 pointing to an azimuth of 120 degrees covering a range of 90 degrees to 150 degrees, the fourth transmitter 540 pointing to an azimuth of 180 degrees covering a range of 150 degrees to 210 degrees, and the fourth transmitter 540 pointing to an azimuth of 180 degrees covering a range of 210 degrees to 330 degrees. The range covered by the fifth transmitter 550 with an azimuth of 240 degrees is 210 degrees to 270 degrees, and the range covered by the sixth transmitter 560 with an azimuth of 300 degrees is 270 degrees to 330 degrees. Therefore, no matter where the AR device is located in the anchor device 500, the AR device can receive the spatial information packet sent from the anchor device 500, and then determine from which transmitter the received information is sent according to the angle information in the spatial information packet, thereby determining at which position of the anchor device 500 the AR device is located.

[0038] Figure 6 The three-dimensional side view of the anchor device 600 described in the embodiment of the present invention is shown. In different embodiments, the shape of the anchor device 600 can be selected according to the display angle required by the application site. The anchor device 600 can be hemispherical, spherical, etc., but it is not limited to this. It can be determined according to the actual application site. Here, a spherical anchor device 600 is used as an example for description. Figure 6As shown, the anchor device 600 is provided with a plurality of transmitters with different azimuths, wherein the first transmitter 610 points to an azimuth of 0 degrees and an angle of 0 degrees with the horizontal plane, the second transmitter 620 points to an azimuth of 60 degrees and an angle of 0 degrees with the horizontal plane, the third transmitter 630 points to an azimuth of 60 degrees and an angle of 60 degrees with the horizontal plane, the fourth transmitter 640 points to an azimuth of 0 degrees and an angle of 90 degrees with the horizontal plane, the fifth transmitter 650 points to an azimuth of 300 degrees and an angle of 60 degrees with the horizontal plane, the sixth transmitter 660 points to an azimuth of 300 degrees and an angle of 0 degrees with the horizontal plane, the seventh transmitter 670 points to an azimuth of 300 degrees and an angle of -60 degrees with the horizontal plane, the eighth transmitter 680 points to an azimuth of 0 degrees and an angle of -90 degrees with the horizontal plane, and the ninth transmitter 690 points to an azimuth of 60 degrees and an angle of -60 degrees with the horizontal plane.

[0039] Figure 7 A schematic diagram of an AR device according to an embodiment of the present invention is shown. Figure 7 As shown, since the present invention uses directional communication technology to identify the relative position between the anchor device and the AR device 700, multiple receivers are set at multiple positions on the AR device 700, such as Figure 7 The white dot in the figure is the location where the receiver 710 is set. Figure 7 The configuration in the figure is merely a schematic diagram, which expresses that the AR device 700 needs to set up receivers 710 at various locations so that the AR device 700 can receive spatial information packets from various directions, wherein the AR device may be any form of device: a head-mounted device, smart glasses, a smart phone, a tablet computer, etc.

[0040] Figure 8 A flowchart showing the steps of simulating spatial sound effects according to an embodiment of the present invention.

[0041] Step S201: In an AR scene, a 4D object located at an anchor point emits a plurality of sound packets.

[0042] In step S202 , the AR device calculates the distance, angle and spatial relationship between the AR device and the anchor point according to the aforementioned algorithm.

[0043] In step S203, based on the calculated distance, angle and spatial relationship as well as the native sound source and the user volume setting, the sound packets are converted into 6DoF sound signals through a sound localization algorithm (e.g., Head Related Transfer Functions (HRTF), also known as ATF (Anatomical Transfer Function)) and transmitted to a playback device of the AR device, such as headphones.

[0044] See also Figure 1, the system for simulating spatial sound effects in augmented reality according to an embodiment of the present invention comprises the anchor device 110 and the augmented reality (AR) device 120. The anchor device 110 provides an AR object (not shown) for emitting a plurality of sound packets in the AR scene. The AR device 120 calculates the distance, angle, and spatial relationship between the anchor device 110, and according to the calculated distance, angle, spatial relationship, native sound source, and user volume setting, converts the sound packets into 6DOF sound signals through a sound effect localization algorithm and transmits them to a playback device (not shown) of the AR device 120.

[0045] Fig. 9 A schematic diagram showing an application example of the simulated spatial sound effect according to the first embodiment of the present invention.

[0046] The arrival time of the sound at the user's right ear is distance A / 343m, and the arrival time of the sound at the left ear is ((distance b+distance B) / 343m)*inverted waveform. The sound level at the right ear is sound pressure level (Sound Pressure Level)-20log*distance A / reference distance (user-defined), and the sound level at the left ear is right ear volume-20log*(distance b+distance B) / distance A.

[0047] Fig.10 A schematic diagram showing an application example of the simulated spatial sound effect according to the second embodiment of the present invention.

[0048] Since the user's headband device can analyze the direction of the current anchor point, the corresponding spatial information will be transmitted to the user at the other end in the semi-virtual world (live broadcast or conference), allowing the head device at the other end to render the corresponding space (video + audio). The speaker can also turn around in this relative space (provided by the anchor point) to interact with the audience.

[0049] The sound source disclosed by the method and system for simulating spatial sound effects in the embodiment of the present invention can present 6 degrees of freedom (DoF, i.e., head rotation and body movement), and further discloses the reflected sound between the user and the virtual object and the reflected sound between the user and the real object. In addition, combined with the interaction between the near / remote and AR / anchor points, the interaction between the real world and the semi-virtual world can be presented, and the video and audio have corresponding directions, so that the near-end and remote users are in a situation like a science fiction movie.

[0050] In summary, the present invention meets the requirements of invention patents and a patent application is filed in accordance with the law. The above is only a preferred embodiment of the present invention, and the scope of the present invention is not limited to the above embodiment. Any equivalent modifications or changes made by people familiar with the art of this case according to the spirit of the present invention should be included in the scope of the following patent application.

Claims

1. A method for simulating spatial sound effects in augmented reality, applied to an augmented reality (AR) device, characterized in that: include: In the AR scene, the AR object located on the anchor device emits a plurality of sound packets; Calculating the distance, angle and spatial relationship between the AR device and the anchor device; as well as According to the calculated distance, angle, spatial relationship, native sound source and user volume setting, the sound packets are converted into 6-degree-of-freedom (DOF) sound signals through a sound localization algorithm and transmitted to a playback device of the AR device.

2. The method for simulating spatial sound in augmented reality as claimed in claim 1, characterized in that: The sound localization algorithm is Head Related Transfer Functions (HRTF).

3. A system for simulating spatial sound effects in augmented reality, characterized in that: include: An anchor device, used to provide an AR object for emitting a plurality of sound packets in an AR scene; as well as The augmented reality (AR) device is used to calculate the distance, angle and spatial relationship between the device and the anchor point, and according to the calculated distance, angle, spatial relationship, native sound source and user volume setting, convert the sound packets into 6DOF sound signals through a sound localization algorithm and transmit them to the playback device of the AR device.

4. The system for simulating spatial sound in augmented reality as claimed in claim 3, characterized in that: The sound localization algorithm is the head-related transfer function (HRTF).

Images