A device and method for presenting spatial audio content
By capturing the listener's visual focus position and combining it with spatial audio algorithms, the problems of inaccurate sound image positioning and insufficient immersion in existing technologies have been solved, enabling immersive audio presentation in various environments, especially improving the interactive effects of hearing and vision in games and VR.
Patent Information
- Application Number
- CN202310634369.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2043-05-31
AI Technical Summary
Existing technologies, especially channel-based playback systems, cannot achieve accurate sound image localization and immersion in asymmetrical sound field environments in spatial audio presentation. Furthermore, object-based playback technologies rely on the sound source and have little attention effect, making it impossible to provide a consistent immersive experience in various environments.
The system uses a visual attention point determination unit to capture the listener's visual focus position. It then executes a spatial audio algorithm by increasing the gain or using the visual focus position as the sound source position. Combined with a high-order continuous activation function, the spatial audio algorithm performs fade-in and fade-out processing to enhance the immersive experience of the audio.
It improves the accuracy of sound image positioning and immersion in asymmetrical sound field environments, enhances the interaction between hearing and vision, and provides a better immersive experience, especially in games and VR scenarios.
Smart Images

Figure CN116866817B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of acoustics and relates to audio algorithms, particularly to the presentation of spatial audio content. Background Technology
[0002] Spatial audio (or 3D audio) technology has always been a hot research topic in the multimedia field. Since the late 1980s, the realization and development of microphone array acquisition technology has promoted the development of physical sound field reconstruction technology. Among them, the most famous are the sound reproduction technology based on spherical harmonic decomposition and the wave field synthesis (WFS) technology. Both of these technologies achieve sound field reconstruction by reconstructing the physical sound field, that is, to pursue the consistency between the sound pressure in the reconstructed space and the sound pressure in the original field.
[0003] In the late 1990s, researchers discovered that by considering the perceptual characteristics of the auditory system, it was not necessary to perfectly replicate the original sound field during sound field reconstruction to achieve a good sense of immersion and localization. Based on this, research was conducted on perception-based virtual sound scene reconstruction technology. Compared to physical sound field reconstruction technology, perception-based virtual sound scene reconstruction technology significantly reduces the requirements for playback environment and equipment, making it easier to apply in practice. Therefore, it has received widespread attention from academia and industry. Among the most widely used techniques are amplitude panning (AP) and binaural reconstruction technology based on the head related transfer function (HRTF). Amplitude panning, which originated in 1961, is a virtual sound image control technology that controls the position of the sound image perceived by the human ear by adjusting the amplitude of the signal distributed to the speakers. Using amplitude panning technology, a virtual sound image in a fan-shaped area in front of the listening point can be reconstructed by modulating the amplitude of the signals from the left and right channels of a stereo device, possessing a certain ability to express the position of the sound image and render the environment. The binaural reconstruction technology based on HRTF uses experimentally measured HRTF to simulate the process of a spatial sound image signal propagating to both ears. It can realize the perceptual reconstruction of virtual sound images during playback using headphones with only two channels, providing technical support for binaural 3D audio in mobile environments.
[0004] Amplitude shifting is a highly efficient sound image control technique that adjusts the amplitude of the signal distributed to the speakers to control the position of the sound image perceived by the human ear. In stereo systems, it modulates the signal distribution between the two speakers using the angle between the sound source and the area directly in front of the center point.
[0005] To address the reconstruction of sound source direction in three-dimensional space, VILLE PULKKI of Helsinki University of Technology in Finland proposed Vector-based amplitude panning (VBAP) in 1997. VBAP uses unit vectors from the positions of two or three speakers to synthesize a unit vector of a virtual sound source, thus reconstructing the sound source direction. Figure 1 Assuming the sound source and three speakers are located on the same sphere, and considering the unit vectors of the three speakers as basis vectors, the direction of the virtual sound source is obtained by a linear combination of them.
[0006] Let the unit vectors of the three loudspeakers be l1, l2, and l3, and the signal gains of each loudspeaker be g1, g2, and g3, respectively. Let the unit vector of the sound source be l0. Then l0 can be expressed as:
[0007] l0 = g1l1 + g2l2 + g3l3
[0008] Therefore, by solving for the gain, we can obtain:
[0009]
[0010] When the loudspeakers are on the same spherical surface, the VBAP model is simple, computationally efficient, and provides relatively accurate directional recovery for sounds below 500–600 Hz. By combining sound energy attenuation principles or controlling sound time delay, VBAP technology can be further extended to aspherical surfaces.
[0011] The technology and systems for perceptual reconstruction of virtual sound images have achieved great success in commercial applications. Two-channel stereo systems can control the position of the sound image by adjusting the gain of the signals from the two speakers, creating the auditory perception that the sound image moves within a fan-shaped area in front of the listener's point. Multi-channel surround sound systems extend the range of sound image movement from a fan-shaped area to a circular area; typical multi-channel surround sound systems include Dolby surround sound and DTS surround sound systems. Dolby surround sound systems, in particular, have gained widespread use in theaters and home environments due to their excellent sound rendering effects. The 5.1 multi-channel surround sound system developed by Dolby Laboratories has become standard in home theater systems; the Dolby 7.1 surround sound system (Dolby Surround 7.1) enhances the sense of immersion by adding left and right rear surround channels. At the 2009 Consumer Electronics Show, Dolby showcased its new Dolby Pro Logic technology, upgrading the 7.1 channel system to 9.1 channels by adding two higher-positioned channels in front, providing a certain level of perceived sound image height.
[0012] In a multi-channel surround sound system, the speakers are placed on the same horizontal plane, making it impossible to produce sound images in the air beyond that plane. To extend the listener's perception of sound images from the horizontal plane to true three-dimensional space, audio technology research has advanced to the three-dimensional audio level, using multi-layered speaker setups to provide listeners with a three-dimensional audio experience. In 2005, Japan's NHK Laboratories officially proposed a 22.2-channel multi-channel prototype system, which was included in Japan's three-dimensional audio standard for next-generation ultra-high-definition television. Figure 2 The system's speaker array is divided into three layers to achieve a three-dimensional immersive effect. With the combined effect of the three layers of speakers, a three-dimensional sound image is reconstructed around the listener, overcoming the shortcomings of surround sound in terms of height perception. The system layout is shown in the figure, with 10 speakers in the middle layer, 9 in the upper layer, and 5 in front of the screen in the lower layer (2 of which are low-frequency speakers).
[0013] The above describes the development process of 3D audio. Current 3D audio based on amplitude shifting is typically produced in the recording studio based on the recording engineer's experience. Due to the pre-production of channel information (such as in typical two-channel stereo), inconsistencies or insufficient spatial sense often occur at the playback end.
[0014] - For example, for stereo systems, the distance between the left and right speakers and the number of speakers are not the same for different products;
[0015] - For example, some products have only one speaker and do not have the multiple speakers required for amplitude translation.
[0016] Furthermore, due to the pre-production nature of the creative process, the number of channels also determines the optimal effect of the speaker system:
[0017] - If the audio source is 7.1 channel, it can achieve the best playback effect in a 7.1 channel system;
[0018] - If the audio source is stereo, placing it in a 7.1 channel system, aside from increasing the sense of immersion due to the increased number of channels, does not create a more effective virtual sound image. Essentially, this is because although a multi-channel system is connected, the signals from multiple speakers are still based on two-way signals (simply put, multi-channel audio extended based on two channels cannot achieve precise control over each channel; mathematically speaking, it's an effort to increase the system's rank dimension).
[0019] The sound reproduction method described above, where the number of audio channels is fixed during transmission, is called channel-based sound reproduction. Its disadvantages are quite obvious: to achieve the best reproduction effect, the number of audio channels and the reproduction environment must be strictly matched; for example, Dolby Cinema's sound effects require special production.
[0020] With the increasing demand for immersive experiences, especially with the rise of automotive audio, the number of in-car speakers has increased dramatically. The original channel-based systems have posed significant challenges to existing channel-based systems in terms of tuning and sound image localization (as mentioned above, because music producers and playback end users cannot achieve consistency: car speakers vary, and the car itself is an asymmetrical sound field). Companies like Dolby and DTS began using object-based audio systems, which transmit the source audio information and coordinates of sound objects within the system. Real-time rendering calculations are then performed locally. This approach allows artists to create audio without considering the speaker system layout, and broadcasters can adapt the audio to the actual speaker arrangement and the sound objects' coordinates locally. Therefore, it offers better playback consistency compared to channel-based playback systems. Object-based audio systems are widely used in live concerts (broadcasts) and live streams. During live broadcasts, artists cannot create multi-channel audio for broadcast in real time. Object-based broadcasting allows for a better unification of the playback listening environment, resulting in better playback quality.
[0021] Sound object-based playback requires recording all information from each audio object, and the bandwidth requirements increase with the number of sound objects being played. In contrast, stereo and other channel-based playback technologies transmit the final channel data, resulting in a relatively constant data volume. Furthermore, most current audio source files still rely on channel-based playback, and many television and music programs use channel-based audio sources. Object-based programming is still relatively rare. While some music companies are promoting immersive sound based on sound objects and actively remastering albums, overall resources remain scarce.
[0022] Current sound object-based sound reproduction technology can create better three-dimensional sound images and provide a better immersive experience, but it also has the following drawbacks:
[0023] - It heavily relies on the audio source and cannot be used without the corresponding audio source;
[0024] - The attention effect is not significant. Normally, judging sound and image requires focused attention. This is why people don't usually pay particular attention to the location of different sound images when listening to music, especially when the music is used as background music. Only audiophiles specifically evaluate playback systems pay attention to the quality of sound and image reproduction. When a picture is present, and people's attention is on the picture, the location of sound and image is usually rather vague, and not requiring particularly precise reconstruction accuracy. Current sound reconstruction algorithms cannot reproduce this attention mechanism. Summary of the Invention
[0025] The purpose of this invention is to provide a spatial audio content presentation device, which solves the problem of how to present sound materials in a spatial audio format to enhance immersion.
[0026] To achieve the above objectives, the technical solution adopted by the present invention is as follows:
[0027] This invention provides a spatial audio content presentation device for presenting sound materials, comprising:
[0028] A visual attention point determination unit is used to obtain the position of the visual focus of the listener's gaze;
[0029] The processing unit is signal-connected to the visual attention point determination unit to receive the visual focus position information. The processing unit amplifies the sound sources near the visual focus position or executes a spatial audio algorithm as the sound source position.
[0030] Preferably, the sound material is audio based on a sound object, stereo audio, or mono audio, and the processing unit amplifies the sound source near the visual focus position.
[0031] Furthermore, the processing unit also records an activation function related to Euclidean distance. The activation function is a high-order continuous function, and its output is a global mask of the overall spatial audio algorithm output, which serves to fade in and out of the spatial algorithm effect.
[0032] Furthermore, when the processing unit does not detect the visual focus position, the processing unit does not apply the gain to any sound source.
[0033] Furthermore, the visual attention point determination unit is also used to obtain visual direction information, and the processing unit is used to gain the sound source that is closest to the listener.
[0034] Preferably, the sound material is single-channel audio, and the processing unit executes the spatial audio algorithm using the visual focus position as the sound source position.
[0035] A method for presenting spatial audio content is also provided, which is used to present sound materials, and includes:
[0036] S1. Obtain the visual focus position of the listener's gaze;
[0037] S2. Amplify the sound source near the visual focal point or execute a spatial audio algorithm with the visual focal point as the sound source location.
[0038] Preferably, the sound material is audio or stereo audio based on a sound object, and further includes S11 between S1 and S2: calculating the Euclidean distance between the visual focus position and the sound source, and using an activation function related to the Euclidean distance to make the output of the activation function a global mask of the overall spatial audio algorithm output, thereby achieving a fade-in / fade-out effect on the spatial algorithm effect. The activation function is a high-order continuous one.
[0039] Preferably, step S1 further includes acquiring visual direction information, and step S2 involves multiple objects near the visual focus, with gain processing applied to the sound source that is close to the listener in the visual direction.
[0040] Preferably, the sound material is single-channel audio, and in step S2, the spatial audio algorithm is executed using the visual focus position as the sound source position.
[0041] Due to the application of the above technical solution, the present invention has the following advantages compared with the prior art:
[0042] The spatial audio content presentation apparatus and method of the present invention utilize a visual attention point determination unit to obtain the visual focus position of the listener's gaze, amplify the sound source near the visual focus position to attract the listener's attention, or use the visual focus position as the sound source position to execute a spatial audio algorithm so that the presented audio appears to be emitted from the visual focus position. All of these methods increase the listener's sense of immersion. Attached Figure Description
[0043] The following sections will describe some specific embodiments of the invention in detail by way of example and not limitation, with reference to the accompanying drawings. The same reference numerals in the drawings denote the same or similar parts or portions. Those skilled in the art should understand that these drawings are not necessarily drawn to scale. In the drawings:
[0044] Figure 1 This is a schematic diagram showing how three speakers reproduce the sound source.
[0045] Figure 2 It is a schematic diagram of reconstructing a three-dimensional sound image around the listener using three layers of loudspeakers;
[0046] Figure 3 This refers to the typical audio transmission process when playing content based on objects.
[0047] Figure 4 This is a schematic diagram illustrating the method of selecting an object sound source closer to the observer for gain in this invention;
[0048] Figure 5 This is a schematic diagram of the spatial audio content presentation device of the present invention;
[0049] The reference numerals in the attached figures are explained as follows:
[0050] 1. Visual attention point identification unit;
[0051] 2. Processing unit;
[0052] 3. Speaker. Detailed Implementation
[0053] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0054] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.
[0055] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0056] like Figure 5 The spatial audio content presentation device of the present invention includes a visual attention point determination unit 1, a processing unit 2, and a speaker 3. The processing unit 2 provides the processed driving signal to the speaker 3 to present the spatial audio content.
[0057] The following examples illustrate the applications of obtaining the visual focus position of the listener's gaze, mainly divided into object-based and channel-based audio material scenarios. The visual attention point determination unit 1 uses an eye-tracking device such as an eye tracker, or other devices for obtaining visual attention points.
[0058] In the first scenario, the sound material is based on sound objects, such as in games or similar scenarios.
[0059] When using object-based playback, the typical audio transmission process can be... Figure 3 As shown.
[0060] The above process is described as follows:
[0061] 1. In object-based transmission, in order to save transmission bandwidth as much as possible, current object-based audio transmission is usually based on the compression algorithms of each company.
[0062] 2. Decode the compressed data according to the decoding algorithm corresponding to the encoding to obtain the actual sound source waveform and the corresponding position information;
[0063] 3. Calculate the required drive signals for each speaker based on the spatial audio algorithm, such as VBAP or WFS.
[0064] In the above scenario, both the location information and data of the sound source are present, creating a relatively ideal listening environment. Through visual capture, we only supplement the approach where the attention effect is not significant: by tracking the visual focus, the specific implementation can be... Figure 4 This means that since vision is a projection of the observation direction, while the location information of sound is usually three-dimensional, only the projection direction is calculated when calculating the focus. In this case, the object closer to the observer is selected, such as... Figure 4 As shown.
[0065] The above process is described as follows:
[0066] 1. Obtain the position of visual focus using an eye tracker;
[0067] 2. Calculate the Euclidean distance between the visual focal point and the projection of the audio object;
[0068] a. When no visual focus is found, no gain processing is applied to any sound source object;
[0069] b. When the distance between the visual focus and the sound source is less than a threshold, the audio signal is given a certain gain.
[0070] i. When there are multiple objects in the visual focus, the sound source that is closer to the listener in the visual direction is given a gain.
[0071] The processing unit 2 also records an activation function related to the Euclidean distance (the position of the visual focus and the sound source). The activation function is a high-order continuous function, and the output of the activation function is a global mask of the overall spatial audio algorithm output, which plays a role in the fade-in and fade-out effect of the spatial algorithm.
[0072] The second scenario involves audio material based on audio channels, such as in the case of a home television set.
[0073] When the source material is based on audio channels, the location of the sound focus can be obtained through visual focus, and this location can roughly represent the location of the actual object. Using this method, in single-channel audio scenarios such as games, the location of objects can be equivalently obtained through visual focus, allowing the audio to be combined with visual focus information as sound source location information for use in spatial audio algorithms.
[0074] When the audio is stereo, the left and right channel signals can be amplified in a way that approximates the sound object to enhance the user's interactive experience.
[0075] When the audio is mono, it can be converted into multi-channel audio through spatial audio algorithms.
[0076] This invention addresses two shortcomings mentioned in the background art by introducing visual focus capture. Vision and hearing interact with each other. This "interaction" means that hearing and vision don't necessarily interfere with each other; they interact and complement each other, ultimately aiming to make the listener's perception more accurate. This invention uses a visual attention point determination unit to capture the position of the sound image, compensating for the lack of positional information in subsequent audio. Furthermore, visual capture also provides a subjective observation of attention; when the user lacks visual focus, rendering of the multidimensional sound signal can be stopped, reducing system computing power and energy consumption.
[0077] Introducing visual focus information enhances the interactivity of audio, making it more immersive in interactive scenarios such as games and VR.
[0078] For channel-based stereo signals or object-based signals, visual focus can increase the visual focus's control over the sound source, thereby increasing the immersive experience of games or VR.
[0079] If the aforementioned channel-based signal is single-channel, then the visual focus can be used as the sound source location information, and a three-dimensional stereo algorithm can be used to enhance immersion.
[0080] In summary, this invention obtains the location of the listener's visual attention point through visual capture, and uses this location information to improve the presentation of various sound materials to enhance the sense of immersion.
[0081] The above embodiments are only for illustrating the technical concept and features of the present invention. Their purpose is to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be used to limit the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A method for presenting spatial audio content, employing a spatial audio content presentation device for presenting sound material, characterized in that the spatial audio content presentation device comprises: an eye tracker for obtaining a visual focus of attention position of a listener; a processing unit (2) connected to the eye tracker for receiving the visual focus of attention position information, the processing unit (2) applying a gain to sound sources within a distance smaller than a threshold value from the visual focus of attention position; the method for presenting spatial audio content comprises: S1, obtaining a visual focus of attention position of a listener; S2, applying a gain to sound sources within a distance smaller than a threshold value from the visual focus of attention position; when the sound material is object-based audio, binaural audio, the processing unit (2) applies the gain to sound sources within a distance from the visual focus of attention position, and when no visual focus of attention position is detected, the processing unit (2) does not apply the gain to all sound sources.
2. The method for presenting spatial audio content according to claim 1, wherein S1 further comprises obtaining visual direction information, and there are multiple objects within the visual focus of attention, the gain is applied to the sound source closest to the listener in the visual direction.
Citation Information
Patent Citations
Selective special audio communication
CN103036691A
