Content-based in-vehicle ambisonics effect mixing rendering method and system, and automobile

Abstract

The application particularly relates to a content-based in-vehicle ambisonics effect mixing rendering method and system and a car, the method comprising the following steps: acquiring an audio object, inputting the audio object into an audio processing system; decomposing a time-domain audio signal into a plurality of audio segments, each audio segment comprising metadata information of the audio object in the audio segment; selecting a certain item of information in the metadata information of each audio segment according to the characteristics of the audio segment, judging according to the certain item of information, selecting a rendering method according to a judgment result, and rendering the audio; acquiring a HOA-based gain gHOAn(t) and a VBAP-based gain gVBAPn(t) of each audio channel for a plurality of audio channels played by a plurality of loudspeakers to be mixed; calculating a mixing gain of each loudspeaker according to a formula; and synthesizing audio signals formed by different rendering methods according to the mixing gain, and outputting the audio signals to the loudspeakers through a power amplifier to emit sound, so that the problem of inaccurate sound image positioning and non-smooth and non-fluent sound image movement of the existing ambisonics technology in a car can be solved.

Description

Technical Field

[0001] This invention relates to the field of audio processing technology, specifically to a content-based in-vehicle panoramic sound effect hybrid rendering method, system, and automobile. Background Technology

[0002] Dolby Atmos technology enables 3D spatial sound effects, allowing sound images to be positioned and moved in space, reproducing the original sound field and giving viewers an immersive experience. The most typical application of this technology is Dolby Atmos, an audio playback standard developed by Dolby Laboratories. Suitable for large cinemas and private theaters, it uses unconventional stereo speaker placement and a Dolby Atmos decoder (amplifier) ​​to reproduce realistic 3D sound effects, resulting in more precise sound image positioning and a stronger sense of presence. Unlike traditional 5.1 and 7.1 channel systems, Dolby Atmos not only places speaker units around the audience but also installs several speaker units on the theater ceiling. Furthermore, unlike previous systems where a single audio signal controlled one side of the theater to produce the same sound, Dolby Atmos allows multiple speakers on one side to emit different sounds, creating a more realistic sound effect that progresses from far to near. Combined with the ceiling-mounted speakers, this achieves a surround sound field, revealing more sound details and enhancing the audience's viewing experience.

[0003] In commercial cinemas, Dolby Atmos supports up to 128 lossless sound elements simultaneously contained in a single mix, and up to 64 independent speaker outputs. Dolby Atmos is based on Layered Amplitude Panning, which can be considered as multiple 2D VBAP layers divided according to speaker height. Rendering is based on VBAP within the same layer, while different layers use crossfade-in and crossfade-out techniques.

[0004] Existing immersive sound technology is suitable for ideal spaces such as large commercial cinemas or family homes. However, in cars, the following problems exist: the interior space of a car is small, much smaller than that of a home or cinema, affecting the number and placement of speakers; the interior space of a car is complex and irregular, due to the spatial division of parts such as seats, resulting in a complex shape, which easily leads to multiple reflections and absorption of sound, affecting the immersive sound effect; the limited interior space of a car, due to the constraints of windows, sunroofs, etc., makes it difficult to arrange a large number of speakers like in a cinema, thus affecting the immersive sound effect.

[0005] Due to the aforementioned issues and the spatial audio rendering methods used in large spaces such as cinemas, current panoramic sound technologies, such as Dolby Atmos, cannot achieve precise sound image positioning and smooth sound image movement in the confined space of a car, thus failing to achieve the ideal panoramic sound effect. Summary of the Invention

[0006] Therefore, the technical problem to be solved by the present invention is to overcome the above-mentioned defects, thereby providing a content-based in-vehicle panoramic sound effect hybrid rendering method, system and automobile.

[0007] A content-based panoramic sound blending rendering method includes the following steps:

[0008] S1: Obtain the audio object and input the audio object into the audio processing system;

[0009] S2: The audio processing system decomposes the time-domain audio signal of an audio object into several audio segments at specific time intervals, and each audio segment includes the metadata information of that audio object;

[0010] S3: Based on the characteristics of each audio segment, select one item from the metadata information of each audio segment for judgment, and select either the HOA rendering method or the VBAP rendering method according to the judgment result, and then render the audio.

[0011] S4: For the multiple audio streams played by multiple speakers to be mixed, obtain the gain gHOAn(t) based on HOA and the gain gVBAPn(t) based on VBAP for each audio stream, and calculate the mixing gain of each speaker according to the formula.

[0012] S5: Based on the mixing gain of each speaker, the multi-channel audio rendered by the HOA rendering method or VBAP rendering method in step S3 is synthesized and output to the speakers through a power amplifier to produce sound.

[0013] Furthermore, the metadata information of the audio in step S2 includes the audio image distance, horizontal angle, and vertical angle.

[0014] Furthermore, in step S3, when judging the sound-image distance, the judgment criteria are as follows: if the sound-image distance between two adjacent segments changes by more than 0.5m, the HOA rendering method is selected; if the sound-image distance between two adjacent segments changes by less than or equal to 0.5m, the VBAP rendering method is selected.

[0015] Furthermore, in step S3, when judging the horizontal angle, the judgment criteria are as follows: if the change in the horizontal angle between two adjacent segments exceeds 15°, then the HOA rendering method is selected; if the change in the horizontal angle between two adjacent segments is less than or equal to 15°, then the VBAP rendering method is selected.

[0016] Furthermore, in step S3, when judging the elevation angle, the judgment criteria are as follows: if the elevation angle change between two adjacent segments exceeds 15°, the HOA rendering method is selected; if the elevation angle change between two adjacent segments is less than or equal to 15°, the VBAP rendering method is selected.

[0017] Furthermore, the formula in step S4 is:

[0018] gmn(t)=wn(t)gHOAn(t)+(1-wn(t))gVBAPn(t);

[0019] Where gmn(t) represents the mixing gain of the audio corresponding to the nth speaker, wn(t) represents the mixing weight of each audio channel, gHOAn(t) represents the HOA gain of the audio corresponding to the nth speaker, gVBAPn(t) represents the VBAP gain of the audio corresponding to the nth speaker, and t represents time.

[0020] Furthermore, the mixing weights of each audio channel need to determine whether the audio source is in a moving state, and adaptively select different mixing weight configuration methods based on the determination result; if the audio source is stationary, the mixing weights wn(t) of each audio channel are 0; if the audio source is in a moving state, the mixing weights wn(t) of each audio channel are 0.8.

[0021] Furthermore, the specific time interval in step S2 is 100ms.

[0022] An in-vehicle panoramic sound system based on any of the above-described content-based panoramic sound hybrid rendering methods includes an audio processing system and a speaker system.

[0023] The audio processing system includes an audio input module, a calculation module, a rendering module, an equalization module, an output module, and a power module. The audio input module, calculation module, rendering module, equalization module, and output module are connected in sequence and are all connected to the power module.

[0024] The speaker system is connected to the power module. The speaker system is arranged in four layers vertically and four rows horizontally. The first layer vertically is installed on the left and right doors, close to the vehicle floor, and this layer contains woofers. The second layer vertically is installed on the left and right doors at knee height, and this layer contains midrange speakers. The third layer vertically is installed in the center of the dashboard, the left and right A-pillars, and the upper part of the left and right doors at chest height, and this layer contains high-end speakers and full-range speakers. The fourth layer vertically is installed on the roof at head height, and this layer contains full-range speakers. The subwoofer is located between the first and second layers.

[0025] The first column in the horizontal direction is the center of the dashboard and the A-pillar; the second column is the front of the front door and located on the side of the first row of passengers; the third column is the front of the rear door and the B-pillar, located between the first and second rows of passengers; the fourth column is the C-pillar and located behind the headrests of the rear seats.

[0026] An automobile, the automobile including the above-described in-vehicle panoramic sound system.

[0027] The technical solution of this invention has the following advantages:

[0028] 1. The method of the present invention, by identifying and analyzing the audio content in advance, divides the audio into "moving sound images" and "stationary sound images", and adopts different audio rendering methods for different types, so as to achieve both accurate positioning of sound images and smooth and continuous movement of sound images, thus achieving an ideal panoramic sound effect in the vehicle.

[0029] 2. The panoramic sound system of the present invention makes full use of the effective space inside the vehicle, and arranges a surround speaker system in the horizontal direction around the passengers inside the vehicle. Compared with traditional vehicles, it adds ceiling speakers in the vertical direction, which is conducive to creating a sense of spatial audio.

[0030] 3. The panoramic sound system of the present invention, according to the arrangement position and the timbre characteristics of different frequency speakers, has matched high, medium, low and full-range speakers, which is conducive to simultaneously reproducing the sound field and timbre of the vehicle with high precision, ensuring panoramic sound effect, the sound image range of the vehicle can reach 360° in the horizontal direction and 180° in the vertical direction, and the sound image positioning angle accuracy can reach 25°. Attached Figure Description

[0031] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0032] Figure 1 A schematic diagram of the Cartesian coordinate system;

[0033] Figure 2 A flowchart of the method involved in this invention;

[0034] Figure 3 This is a structural diagram of the audio processing system involved in the present invention;

[0035] Figure 4 This is a layout diagram of the in-vehicle speaker system according to the present invention;

[0036] Figure 5 This demonstrates the accuracy of the in-vehicle audio-visual positioning angle. Detailed Implementation

[0037] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0038] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0039] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection of two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0040] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0041] Currently, there are two mainstream algorithms for panoramic sound rendering: VBAP and HOA.

[0042] VBAP (Vector-Based Amplitude Panning) is a three-dimensional generalization of discrete-amplitude signal feeding in the horizontal or specific sagittal planes. This method achieves localized audio translation through three adjacent speakers without affecting the interaural time difference (ITD) at low frequencies or the frequency spectrum at high frequencies, thus providing more precise sound localization in three-dimensional space. A drawback of VBAP is that it can produce jerky, disjointed spatial sound image movement, making it suitable for precise control of static virtual sound images.

[0043] HOA: High Order Ambisonics (HOA) sound reproduction systems employ spatial spherical harmonic function expansion to approximate the ideal sound field step by step. It records the spatial information of the original sound field and reproduces a three-dimensional sound field that is identical to or as similar as possible to the original. The accuracy of the reproduced sound field can be adjusted by selecting the number of spherical harmonic expansion orders. Due to global translation performed by all speakers, the sense of direction is more blurred and more susceptible to room reflections; limited by the order used, it may result in a lack of high-frequency cues, thus affecting the listener's accuracy in positioning; HOA creates smoother and more fluid sound image movement and is suitable for controlling moving virtual sound images.

[0044] Please see Figure 2 This invention is a content-based panoramic sound hybrid rendering method. The principle is to calculate and analyze the metadata information of an audio object to determine whether it is static audio, moving audio, or omnidirectional environmental audio. If it is static audio, it is rendered using VBAP. If it is moving audio, the rendering method (VBAP or HOA) is determined by comparing the speed of movement with a predefined threshold. The specific steps are as follows:

[0045] S1: Acquire audio objects and input them into the audio processing system. An audio object is a set of audio samples associated with metadata, used to reproduce sound based on the spatial location and other attributes indicated by the metadata. This location may or may not be associated with a single speaker. The object metadata defines the audio object's location, size, shape, gain, and lifetime in three-dimensional space. The audio object's location metadata uses a Cartesian coordinate system to place the audio object. This system uses three orthogonal axes (x, y, z) to locate a point in space relative to a selected origin. The x-coordinate represents the left-right dimension, the y-coordinate represents the front-back dimension, and the z-coordinate represents the up-down dimension, such as... Figure 1 As shown.

[0046] S2: The audio processing system decomposes the time-domain audio signal of an audio object into several audio segments at specific time intervals, and each audio segment includes the metadata information of that audio object;

[0047] S3: Based on the characteristics of each audio segment, select one item from the metadata information of each audio segment for judgment, and select either the HOA rendering method or the VBAP rendering method according to the judgment result, and then render the audio.

[0048] S4: For the multiple audio streams played by multiple speakers to be mixed, obtain the gain gHOAn(t) based on HOA and the gain gVBAPn(t) based on VBAP for each audio stream, and calculate the mixing gain of each speaker according to the formula.

[0049] S5: Based on the mixing gain of each speaker, the multi-channel audio rendered by the HOA rendering method or VBAP rendering method in step S3 is synthesized and output to the speakers through a power amplifier to produce sound.

[0050] In this embodiment, the metadata information of the audio in step S2 includes the audio image distance, horizontal angle, and vertical angle.

[0051] In step S3, when judging the sound-image distance, the judgment criteria are as follows: if the sound-image distance between two adjacent segments changes by more than 0.5m, it is judged as "moving sound-image" and the HOA rendering method is automatically selected; if the sound-image distance between two adjacent segments changes by less than or equal to 0.5m, it is judged as "stationary sound-image" and the VBAP rendering method is selected.

[0052] In step S3, when judging the horizontal angle, the judgment criteria are as follows: if the change in the horizontal angle between two adjacent segments exceeds 15°, it is judged as "moving sound image (such as a car driving by, a bird flying by)", and the HOA rendering method is automatically selected; if the change in the horizontal angle between two adjacent segments is less than or equal to 15°, it is judged as "static sound image (such as the sound of drums from a band)".

[0053] In step S3, when judging the elevation angle, the judgment criteria are as follows: if the elevation angle of two adjacent segments changes by more than 15°, it is judged as "moving sound image" and the HOA rendering method is automatically selected; if the elevation angle of two adjacent segments changes by less than or equal to 15°, it is judged as "static sound image" and the VBAP rendering method is selected.

[0054] In this embodiment, the formula in step S4 is:

[0055] gmn(t)=wn(t)gHOAn(t)+(1-wn(t))gVBAPn(t);

[0056] Where gmn(t) represents the mixing gain of the audio corresponding to the nth speaker, wn(t) represents the mixing weight of each audio channel, gHOAn(t) represents the HOA gain of the audio corresponding to the nth speaker, gVBAPn(t) represents the VBAP gain of the audio corresponding to the nth speaker, and t represents time.

[0057] The mixing weights configured for each audio channel need to determine whether the sound source is in a moving state, and select different mixing weight configuration methods based on the determination result. If the sound source is stationary, the mixing weight wn(t) of each audio channel is 0; if the sound source is in a moving state, the mixing weight wn(t) of each audio channel is 0.8. Therefore, this method adaptively selects the optimal processing method based on the audio content to render the audio. This method can more accurately locate the sound source position while maintaining smooth sound movement, thereby meeting the needs of immersive audio production and playback.

[0058] In this embodiment, the specific time interval in step S2 is 100ms.

[0059] Please see Figure 3 and Figure 4 The present invention also includes an in-vehicle panoramic sound system based on the content-based panoramic sound hybrid rendering method described above, comprising an audio processing system and a speaker system.

[0060] The audio processing system includes an audio input module, a calculation module, a rendering module, an equalization module, an output module, and a power module. The audio input module, calculation module, rendering module, equalization module, and output module are connected in sequence and are all connected to the power module.

[0061] The panoramic sound audio enters the calculation module through the audio input module. The calculation module has audio decomposition and comparison functions. First, the calculation module decomposes the time-domain audio signal into several segments at 100ms intervals, each segment containing information about the audio's time, sound image distance, horizontal angle, and elevation angle. Then, it determines the positional change of a certain item in the audio object's metadata, selects a rendering method, and renders the audio. It judges the change in sound image distance, horizontal angle, or elevation angle of each segment. Taking the horizontal angle as an example, if the change in horizontal angle between two adjacent segments exceeds 15°, it is judged as a "moving sound image (such as a car driving by, a bird flying by)," and the HOA rendering method is automatically selected. Conversely, if the change in horizontal angle between two adjacent segments is less than or equal to 15°, it is judged as a "stationary sound image (such as the sound of drums from a band)," and the VBAP rendering method is selected. Finally, the audio rendering module renders the audio and outputs it to the equalization module. The equalization module calculates the playback weight of each speaker according to different audio rendering methods, performs equalization processing on the audio signal, and outputs the signal to the speaker system through the output module. The power supply module supplies power to all the above modules.

[0062] The speaker system is connected to the power module. The speaker system is arranged in four layers vertically and four rows horizontally. The first layer vertically is installed on the left and right doors, close to the vehicle floor, and this layer contains woofers. The second layer vertically is installed on the left and right doors at a height equivalent to the driver's knees, and this layer contains midrange speakers. The third layer vertically is installed in the center of the front dashboard, the left and right A-pillars, and the upper part of the left and right doors at a height equivalent to the driver's chest, and this layer contains high-end speakers and full-range speakers. The fourth layer vertically is installed on the vehicle roof at a height close to the driver's head, and this layer contains full-range speakers. The subwoofer is located between the first and second layers.

[0063] The first column in the horizontal direction is the center of the dashboard and the A-pillar; the second column is the front of the front door and located on the side of the first row of passengers; the third column is the front of the rear door and the B-pillar, located between the first and second rows of passengers; the fourth column is the C-pillar and located behind the headrests of the rear seats.

[0064] The present invention also includes an automobile that includes the above-described in-vehicle panoramic sound system.

[0065] This invention's content-based panoramic sound rendering method, through pre-identification and analysis of audio content, categorizes audio into "moving sound images" and "stationary sound images," and employs different audio rendering methods for each type. This achieves both precise sound image positioning and smooth, continuous sound image movement, resulting in an ideal panoramic sound effect within the vehicle. The in-vehicle sound image range reaches 360° horizontally and 180° vertically, with a sound image positioning angle accuracy of 25°. See [link to relevant documentation]. Figure 5 . Figure 5 The horizontal axis represents the actual playback angle of the sound image, and the vertical axis represents the angle of the sound image perceived by the listener. Each colored dot represents the perception of the sound image angle by an ideal observer. Points closer to the blue diagonal line in the graph indicate a more accurate sound image playback by the immersive sound system. As can be seen from the graph, most points are concentrated between the two red lines, i.e., within ±25°, which demonstrates the effectiveness of the method and system described in this invention.

[0066] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A content-based panoramic sound hybrid rendering method, characterized in that, Includes the following steps: S1: Obtain the audio object and input the audio object into the audio processing system; S2: The audio processing system decomposes the time-domain audio signal of an audio object into several audio segments at specific time intervals, and each audio segment includes the metadata information of that audio object; S3: Based on the characteristics of each audio segment, select one item from the metadata information of each audio segment for judgment, and select either the HOA rendering method or the VBAP rendering method according to the judgment result, and then render the audio. S4: For the multiple audio streams played by multiple speakers to be mixed, obtain the gain gHOAn(t) based on HOA and the gain gVBAPn(t) based on VBAP for each audio stream, and calculate the mixing gain of each speaker according to the formula. S5: Based on the mixing gain of each speaker, the multi-channel audio rendered by the HOA rendering method or VBAP rendering method in step S3 is synthesized and output to the speakers through the power amplifier to produce sound. The metadata information of the audio in step S2 includes the audio image distance, horizontal angle, and vertical angle; In step S3, when judging the sound-image distance, the judgment criteria are as follows: if the sound-image distance between two adjacent segments changes by more than 0.5m, the HOA rendering method is selected; if the sound-image distance between two adjacent segments changes by less than or equal to 0.5m, the VBAP rendering method is selected. The specific time interval in step S2 is 100ms; In step S3, when judging the horizontal angle, the judgment criteria are as follows: if the change in the horizontal angle between two adjacent segments exceeds 15°, the HOA rendering method is selected; if the change in the horizontal angle between two adjacent segments is less than or equal to 15°, the VBAP rendering method is selected. In step S3, when judging the elevation angle, the judgment criteria are as follows: if the elevation angle of two adjacent segments changes by more than 15°, the HOA rendering method is selected; if the elevation angle of two adjacent segments changes by less than or equal to 15°, the VBAP rendering method is selected.

2. The method according to claim 1, characterized in that, The formula in step S4 is: gmn(t)=wn(t)gHOAn(t)+(1-wn(t))gVBAPn(t); Where gmn(t) represents the mixing gain of the audio corresponding to the nth speaker, wn(t) represents the mixing weight of each audio channel, gHOAn(t) represents the HOA gain of the audio corresponding to the nth speaker, gVBAPn(t) represents the VBAP gain of the audio corresponding to the nth speaker, and t represents time.

3. The method according to claim 2, characterized in that, The mixing weights for each audio channel need to be determined based on whether the audio source is in a moving state, and different mixing weight configuration methods are selected according to the determination result; if the audio source is stationary, the mixing weight wn(t) for each audio channel is 0; if the audio source is in a moving state, the mixing weight wn(t) for each audio channel is 0.

8.

4. An in-vehicle panoramic sound system based on the content-based panoramic sound hybrid rendering method according to any one of claims 1-3, characterized in that, Including audio processing systems and speaker systems; The audio processing system includes an audio input module, a calculation module, a rendering module, an equalization module, an output module, and a power module. The audio input module, calculation module, rendering module, equalization module, and output module are connected in sequence and are all connected to the power module. The speaker system is connected to the power module. The speaker system is arranged in four layers vertically and four rows horizontally. The first layer vertically is installed on the left and right doors, close to the vehicle floor, and this layer contains woofers. The second layer vertically is installed on the left and right doors at knee height, and this layer contains midrange speakers. The third layer vertically is installed in the center of the dashboard, the left and right A-pillars, and the upper part of the left and right doors at chest height, and this layer contains high-end speakers and full-range speakers. The fourth layer vertically is installed on the roof at head height, and this layer contains full-range speakers. The subwoofer is located between the first and second layers. The first column in the horizontal direction is the center of the dashboard and the A-pillar; the second column is the front of the front door and located on the side of the first row of passengers. The third column is located at the front of the rear door and the B-pillar, between the first and second rows of passengers; the fourth column is located at the C-pillar, behind the headrests of the rear seats.

5. A car, characterized in that, The vehicle includes the in-vehicle surround sound system as described in claim 4.

Images