An audio processing method, apparatus, storage medium, and electronic device

Abstract

A kind of audio processing method, device, storage medium and electronic equipment are disclosed, method includes: determining frequency band based on the audio signal of any audio track;Based on the position of different loudspeakers and the position of target object, determine the first gain coefficient of different loudspeakers under VBAP algorithm and the second gain coefficient under DBAP algorithm;Based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the second gain coefficient, determine the target gain coefficient of current frequency band;Based on the total frequency band corresponding to any audio track, obtain fusion gain coefficient using target gain coefficient, fusion gain coefficient includes the sub-coefficient corresponding to different loudspeakers;Based on sub-coefficient gain audio signal, use the different loudspeakers corresponding to sub-coefficient, output gained audio signal. In this way, the sound image positioning accuracy of full frequency band can be considered in the audio processing process in vehicle scene, and the listening experience of user is improved.

Description

Technical Field

[0001] This disclosure relates to the field of audio processing technology, and in particular to an audio processing method, apparatus, storage medium and electronic device. Background Technology

[0002] In in-vehicle scenarios, the channel configuration of audio systems has gradually evolved from four channels to complex channels such as seven and nine channels in order to achieve a panoramic sound solution. This requires the audio system to decompose stereo music into multiple independent audio tracks (such as vocals, piano, guitar, drums, etc.) and render them reasonably into complex channels (such as seven or nine channels) to provide users with an immersive music experience.

[0003] In related technologies, when the channel configuration of an audio system is complex, it is difficult to achieve full-frequency sound image positioning accuracy, which affects the user's listening experience. Summary of the Invention

[0004] To address the aforementioned technical issues, this disclosure provides an audio processing method, apparatus, storage medium, and electronic device. In vehicle-mounted scenarios, audio processing can be performed by fusing vector-based amplitude panning (VBAP) and distance-based amplitude panning (DBAP) algorithms to ensure sound image positioning accuracy across the entire frequency band and improve the user's listening experience.

[0005] A first aspect of this disclosure provides an audio processing method, the method comprising: Based on the audio signal of any audio track, determine at least one frequency band.

[0006] Based on the positions of different speakers and the target object, the first gain coefficient of different speakers under the VBAP algorithm is determined, and the second gain coefficient of different speakers under the DBAP algorithm is determined.

[0007] The target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient.

[0008] Based on all frequency bands corresponding to any audio track, the target gain coefficient is used to obtain the fusion gain coefficient, which includes sub-coefficients corresponding to different speakers.

[0009] The audio signal is amplified based on sub-coefficients, and the amplified audio signal is output using different loudspeakers corresponding to the sub-coefficients.

[0010] A second aspect of this disclosure provides an audio processing apparatus, the apparatus comprising: The first determining module is used to determine at least one frequency band based on the audio signal of any audio track.

[0011] The second determining module is used to determine the first gain coefficient of different speakers under the VBAP algorithm and the second gain coefficient of different speakers under the DBAP algorithm based on the positions of different speakers and the position of the target object.

[0012] The third determining module is used to determine the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient.

[0013] The fourth determining module is used to obtain a fusion gain coefficient based on the target gain coefficient and all frequency bands corresponding to any audio track. The fusion gain coefficient includes sub-coefficients corresponding to different speakers.

[0014] The output module is used to gain the audio signal based on sub-coefficients and output the gained audio signal using different speakers corresponding to the sub-coefficients.

[0015] A third aspect of this disclosure provides a computer-readable storage medium storing a computer program for performing the audio processing method proposed in the first aspect embodiment.

[0016] A fourth aspect of this disclosure provides an electronic device including a processor; a memory for storing processor-executable instructions; the processor being configured to read the executable instructions from the memory and execute the instructions to implement the audio processing method proposed in the first aspect embodiment.

[0017] This disclosure provides an audio processing method, apparatus, storage medium, and electronic device that determines at least one frequency band based on the audio signal of any audio track. By dividing the audio signal into frequency bands, sound field localization at different frequencies can be processed more precisely. Furthermore, based on the positions of different speakers and the target object, a first gain coefficient for different speakers under the VBAP algorithm and a second gain coefficient for different speakers under the DBAP algorithm are determined. This allows subsequent calculations to leverage the high localization accuracy of the first gain coefficient of the VBAP algorithm and the robustness of the second gain coefficient of the DBAP algorithm in irregular speaker layouts, preserving the original characteristics of both algorithms and achieving complementary advantages during the fusion process. Further, based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. Thus, by assigning different weights to different frequency bands, the low-frequency band emphasizes the DBAP algorithm to enhance sound field stability and robustness, while the high-frequency band emphasizes the VBAP algorithm to improve sound image localization accuracy, achieving segmented fusion optimization in the frequency domain. Furthermore, based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained by fusing the target gain coefficients. This not only effectively avoids abrupt gain changes caused by frequency band weight switching or positional variations, but also, through the obtained fused gain coefficient, combines the characteristics of both algorithms to improve the sound image localization accuracy across the entire frequency band. Further, the audio signal is amplified based on sub-coefficients, and the amplified audio signal is output using different speakers corresponding to the sub-coefficients. In this way, the amplified audio signal has high sound image localization accuracy across the entire frequency band, improving the user's listening experience. Attached Figure Description

[0018] Figure 1 This is a diagram of an in-vehicle system architecture provided in an exemplary embodiment of the present disclosure; Figure 2 This is a schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure; Figure 3 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure; Figure 4 This is a flowchart illustrating step S200 provided in an exemplary embodiment of this disclosure; Figure 5 This is another schematic diagram of step S200 provided in an exemplary embodiment of this disclosure; Figure 6 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of this disclosure; Figure 7 This is a flowchart illustrating step S250 provided in an exemplary embodiment of this disclosure; Figure 8 This is another flowchart illustrating step S200 provided in an exemplary embodiment of this disclosure; Figure 9 This is a flowchart illustrating step S293 provided in an exemplary embodiment of this disclosure; Figure 10 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure; Figure 11 This is a flowchart illustrating step S300 provided in an exemplary embodiment of this disclosure; Figure 12 This is a flowchart illustrating step S500 provided in an exemplary embodiment of this disclosure; Figure 13 This is a schematic diagram of the structure of an audio processing apparatus provided in an exemplary embodiment of the present disclosure; Figure 14 This is another structural schematic diagram of an audio processing apparatus provided in an exemplary embodiment of the present disclosure; Figure 15 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0019] To explain this disclosure, exemplary embodiments of the disclosure will now be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the disclosure, and not all of them. It should be understood that the disclosure is not limited to exemplary embodiments.

[0020] It should be noted that, unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this disclosure.

[0021] Application Overview Audio systems typically use either vector-based amplitude panning (VBAP) or distance-based amplitude panning (DBAP) algorithms for audio rendering.

[0022] The VBAP algorithm is a technique that linearly combines the direction vectors of different speakers to obtain the amplitude gain of each speaker, and then performs amplitude weighting on the audio signal based on the amplitude gain to obtain the direction vector of the virtual sound source. In this way, the VBAP algorithm can control the human ear's perception of the sound source location based on the interaural level difference (ILD) between different speakers reaching the human ear.

[0023] The DBAP algorithm is a technique that calculates the attenuation of the spatial distance between a virtual sound source and different speakers to obtain the amplitude gain of each speaker, and then performs amplitude weighting on the audio signal based on the amplitude gain. In this way, the DBAP algorithm can control the human ear's perception of the sound source's location based on the sound level attenuation characteristics of different speakers reaching the human ear.

[0024] However, the VBAP algorithm suffers from unstable sound imaging and insufficient surround sound when processing low-frequency audio signals, while the DBAP algorithm suffers from blurred sound imaging and lack of directionality when processing high-frequency audio signals. Therefore, using either the VBAP or DBAP algorithm alone makes it difficult to achieve accurate sound imaging across the entire frequency range, thus affecting the user's listening experience.

[0025] To address the aforementioned technical problems, this disclosure provides an audio processing method, apparatus, storage medium, and electronic device that determines at least one frequency band based on the audio signal of any audio track. This frequency band division of the audio signal allows for more precise processing of sound field localization at different frequencies. Furthermore, based on the positions of different speakers and the target object, a first gain coefficient for each speaker under the VBAP algorithm and a second gain coefficient for each speaker under the DBAP algorithm are determined. This leverages the high localization accuracy of the first gain coefficient of the VBAP algorithm and the robustness of the second gain coefficient of the DBAP algorithm in irregular speaker layouts for subsequent calculations, preserving the original characteristics of both algorithms and achieving complementary advantages during the fusion process. Further, based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. By assigning different weights to different frequency bands, the low-frequency band emphasizes the DBAP algorithm to enhance sound field stability and robustness, while the high-frequency band emphasizes the VBAP algorithm to improve sound image localization accuracy, achieving segmented fusion optimization in the frequency domain. Furthermore, based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained by fusing the target gain coefficients. This not only effectively avoids abrupt gain changes caused by frequency band weight switching or positional variations, but also, through the obtained fused gain coefficient, combines the characteristics of both algorithms to improve the sound image localization accuracy across the entire frequency band. Further, the audio signal is amplified based on sub-coefficients, and the amplified audio signal is output using different speakers corresponding to the sub-coefficients. In this way, the amplified audio signal has high sound image localization accuracy across the entire frequency band, improving the user's listening experience.

[0026] Exemplary System Figure 1 This is a diagram of an in-vehicle system architecture provided in an exemplary embodiment of this disclosure.

[0027] like Figure 1 As shown, the vehicle system 100 in this disclosure may include: a processor 110, a memory 120, and an audio device 130.

[0028] The processor 110 is used to execute program instructions to achieve coordinated control and data processing of various modules (devices) in the vehicle system 100. The processor 110 is electrically connected to the memory 120 and the audio device 130 respectively to perform instruction processing and data processing.

[0029] The memory 120 is used to store program instructions that can be executed by the processor 110. The processor 110 can load and execute the program instructions in the memory 120 to implement the audio processing method.

[0030] The memory 120 is also used to store data, such as data related to audio processing methods, for use by the processor 110. The memory 120 is also used to store the operating system, applications, and various other data resources, providing an operating environment and data support for other functions of the vehicle.

[0031] In some embodiments, the memory 120 may include volatile memory, such as dynamic random access memory (DRAM), static random access memory (SRAM), etc.; it may also include non-volatile memory (NVM), such as read-only memory (ROM), flash memory, etc.

[0032] The audio device 130 is used to process, transmit, receive or play audio signals. It can independently perform specific audio-related functions and can be electrically connected to the processor 110 or the memory 120 through a standardized interface.

[0033] Processor 110 is configured to execute instructions stored in memory 120 to perform the following operations: Based on the audio signal of any audio track, at least one frequency band is determined. Based on the positions of different speakers and the target object, a first gain coefficient for each speaker under the vector-based amplitude shift (VBAP) algorithm and a second gain coefficient for each speaker under the distance-based amplitude shift (DBAP) algorithm are determined. Based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. Based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained using the target gain coefficient, where the fused gain coefficient includes sub-coefficients corresponding to different speakers. The audio signal is amplified based on the sub-coefficients, and the amplified audio signal is output using the different speakers corresponding to the sub-coefficients.

[0034] The vehicle-mounted system 100 shown in this embodiment executes executable instructions in memory to execute an audio signal based on any audio track and determine at least one frequency band. This allows for more precise processing of sound field localization at different frequencies by dividing the audio signal into frequency bands. Furthermore, based on the positions of different speakers and the target object, a first gain coefficient for each speaker under the VBAP algorithm and a second gain coefficient for each speaker under the DBAP algorithm are determined. This allows subsequent calculations based on the high localization accuracy of the first gain coefficient of the VBAP algorithm and the robustness of the second gain coefficient of the DBAP algorithm in irregular speaker layouts, preserving the original characteristics of both algorithms and achieving complementary advantages during the fusion process. Further, based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. Thus, by assigning different weights to different frequency bands, the low-frequency band emphasizes the DBAP algorithm to enhance sound field stability and robustness, while the high-frequency band emphasizes the VBAP algorithm to improve sound image localization accuracy, achieving segmented fusion optimization in the frequency domain. Furthermore, based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained by fusing the target gain coefficients. This not only effectively avoids abrupt gain changes caused by frequency band weight switching or positional variations, but also, through the obtained fused gain coefficient, combines the characteristics of both algorithms to improve the sound image localization accuracy across the entire frequency band. Further, the audio signal is amplified based on sub-coefficients, and the amplified audio signal is output using different speakers corresponding to the sub-coefficients. In this way, the amplified audio signal has high sound image localization accuracy across the entire frequency band, improving the user's listening experience.

[0035] Exemplary methods Figure 2 This is a schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure.

[0036] The embodiments disclosed herein can be applied to electronic devices equipped with multi-channel speakers (e.g., vehicle audio systems).

[0037] like Figure 2 As shown, in one implementation, the audio processing method illustrated in this disclosure includes steps S100-S500.

[0038] Step S100: Determine at least one frequency band based on the audio signal of any audio track.

[0039] In this context, an audio track refers to an audio signal of a single instrument or voice part separated from mixed stereo music, such as a vocal track, piano track, guitar track, drum track, etc. This disclosure does not limit the specific type of audio track.

[0040] A frequency band refers to the different frequency ranges divided in the frequency domain of an audio signal. Different frequency bands have significantly different perceptual characteristics for the human ear. The frequency band can be divided uniformly or non-uniformly according to the perceptual characteristics of the human ear (such as Mel scale or Bark scale). The embodiments of this disclosure do not limit the specific method of frequency band division.

[0041] In one implementation, the audio track separation technology can be used to extract the audio from the original multi-channel audio signal, or it can be a mono or stereo audio track recorded separately by the user. Audio track separation technologies include, for example, source separation algorithms based on deep learning (such as U-Net or Transformer architecture separation models), or algorithms based on nonnegative matrix factorization. This disclosure does not limit the specific implementation of the audio track separation technology.

[0042] The following embodiments of this disclosure are all illustrated using piano tracks as examples.

[0043] In one implementation, the electronic device can use a 256th-order Fast Fourier Transform (FFT) to divide the audio signal of the piano track into multiple frequency bands based on the characteristics of human hearing or the requirements of sound field rendering.

[0044] It should be noted that FFT is an algorithm that efficiently converts time-domain signals into frequency-domain signals.

[0045] In one implementation, the electronic device can divide the spectral data into multiple frequency bands based on subband decomposition technology. This disclosure does not limit the number of frequency bands to be divided; for example, it can be divided into three bands: low-frequency, mid-frequency, and high-frequency, or it can be divided into more bands, such as five or seven, to accommodate more refined sound field control requirements.

[0046] The following embodiments of this disclosure are all illustrated by way of dividing the frequency bands into three frequency bands.

[0047] By using this method, the electronic device divides the audio signal of the current audio track into multiple frequency bands, providing a basis for subsequent use of differentiated rendering strategies for different frequency bands.

[0048] Step S200: Based on the positions of different speakers and the target object, determine the first gain coefficient of different speakers under the VBAP algorithm, and determine the second gain coefficient of different speakers under the DBAP algorithm.

[0049] The first gain coefficient is the gain of different loudspeakers obtained by directional vector decomposition using the VBAP algorithm, and is used to reflect the directionality of the sound image.

[0050] The second gain coefficient is the gain of different loudspeakers obtained through the distance attenuation model of the DBAP algorithm, used to reflect the sense of distance and immersion of the sound source.

[0051] Using this method, the electronic device obtains the gain coefficients of VBAP and DBAP, providing basic data for subsequent fusion processing.

[0052] Step S300: Determine the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient.

[0053] Electronic devices can generate corresponding target gain coefficients in different frequency bands by setting different weights for the first and second gain coefficients respectively. This can improve the accuracy of sound image localization across the entire frequency band.

[0054] An example is given of an electronic device that performs frequency band division based on a 256-level FFT.

[0055] For example, in the low frequency band, , .

[0056] In the mid-frequency band, , .

[0057] High frequency band , .

[0058] in, The first weight is used to indicate the proportion of the first gain coefficient in the fusion process. The second weight indicates the proportion of the second gain coefficient in the fusion process. n is the frequency band index, used to identify different frequency band ranges.

[0059] Thus, in the low-frequency band, because the human ear has a weaker sense of direction for low-frequency sound sources but is more sensitive to a sense of envelopment, a lower first weight and a higher second weight are used to make low-frequency rendering focus more on the diffusion and envelopment provided by the DBAP algorithm. In the mid-frequency band, the VBAP and DBAP algorithms are balanced by setting the first and second weights equally. In the high-frequency band, a higher first weight and a lower second weight are used to make the VBAP algorithm dominate the high-frequency sound image localization, thereby leveraging the VBAP algorithm's advantage in directionality to improve the accuracy of high-frequency sound images.

[0060] It should be noted that the specific data for the first and second weights described above are for illustrative purposes only and can be adjusted according to the specific rendering scenario and rendering target in practical applications. This disclosure does not limit the specific setting method of the first and second weights.

[0061] In one implementation, the electronic device sets the target gain coefficient for the current frequency band based on Formula 1.

[0062] Formula 1: .

[0063] in, The first gain coefficient, Here, is the second gain coefficient, i is the speaker index, and b is the frequency band index. This represents the target gain coefficient for the current frequency band.

[0064] By employing this approach and assigning differentiated weights to different frequency bands, the low-frequency band can focus more on the sound image diffusion and sense of immersion brought by the DBAP algorithm, while the high-frequency band can focus more on the precise sound image localization brought by the VBAP algorithm, thereby achieving a balance between sound image stability and sense of immersion across the entire frequency band.

[0065] Step S400: Based on all frequency bands corresponding to any audio track, a fusion gain coefficient is obtained by fusing using the target gain coefficient. The fusion gain coefficient includes sub-coefficients corresponding to different speakers.

[0066] In one implementation, the electronic device can perform inter-frame smoothing on the target gain coefficients and use the target gain coefficients after inter-frame smoothing to fuse them to obtain fused gain coefficients.

[0067] Inter-frame smoothing is a technique that smoothly interpolates the gain coefficients between adjacent audio frames. This technique can prevent sudden changes in gain coefficients caused by changes in track position or weight, thereby avoiding problems such as popping sounds or unpleasant listening experience.

[0068] For example, inter-frame smoothing can be achieved using a first-order low-pass filter. That is, the electronic device can weight and superimpose the target gain coefficient of the previous frame and the target gain coefficient of the current frame in a proportional manner to obtain the target gain coefficient of the current frame after inter-frame smoothing.

[0069] In one implementation, the electronic device obtains the target gain coefficient of the current frame after inter-frame smoothing based on Equation 2.

[0070] Formula 2: .

[0071] ,τ∈[10ms,50ms].

[0072] in, The target gain coefficient for the current frame. λ is the target gain coefficient of the previous frame, λ is the smoothing factor, T is the frame interval time, and τ is the time constant.

[0073] It should be noted that the ranges of values ​​for the smoothing factor λ and the time constant τ mentioned above are for illustrative purposes only. In practical applications, they can be flexibly adjusted according to specific rendering scenarios and auditory requirements. This disclosure does not limit these aspects.

[0074] Further based on the aforementioned example, the electronic device can use the target gain coefficient after inter-frame smoothing to fuse the low-frequency band, mid-frequency band, and high-frequency band corresponding to the piano track to obtain the fused gain coefficient.

[0075] By adopting this approach, electronic devices can effectively avoid abrupt gain changes caused by frequency band switching, thereby ensuring the continuity and naturalness of the audio signal in the time domain.

[0076] Step S500: Gain the audio signal based on the sub-coefficients, and output the gained audio signal using the different speakers corresponding to the sub-coefficients.

[0077] In one implementation, the electronic device multiplies the audio signal by the sub-coefficients corresponding to different speakers to obtain the audio signal with the corresponding gain for each speaker, and then outputs it through the corresponding speaker.

[0078] By employing this approach, electronic devices can achieve precise rendering of sound fields across different frequency bands. Specifically, the low-frequency band utilizes the DBAP algorithm to achieve more uniform sound field coverage, the high-frequency band uses the VBAP algorithm to achieve more accurate sound image localization, and the mid-frequency band balances sound field uniformity and localization accuracy through weighted fusion. In this way, the electronic device can dynamically adjust its rendering strategy based on the frequency band characteristics of the audio track, thereby significantly enhancing the immersiveness and realism of the listening experience while maintaining the naturalness of the sound field.

[0079] The technical solution disclosed in this embodiment determines at least one frequency band based on the audio signal of any audio track. This frequency band division of the audio signal allows for more precise processing of sound field localization at different frequencies. Furthermore, based on the positions of different speakers and the target object, a first gain coefficient for each speaker under the VBAP algorithm and a second gain coefficient for each speaker under the DBAP algorithm are determined. This allows subsequent calculations based on the high localization accuracy of the first gain coefficient of the VBAP algorithm and the robustness of the second gain coefficient of the DBAP algorithm in irregular speaker layouts, preserving the original characteristics of both algorithms and achieving complementary advantages during the fusion process. Further, based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. By assigning different weights to different frequency bands, the low-frequency band emphasizes the DBAP algorithm to enhance sound field stability and robustness, while the high-frequency band emphasizes the VBAP algorithm to improve sound image localization accuracy, achieving segmented fusion optimization in the frequency domain. Finally, based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained by fusing using the target gain coefficient. This approach not only effectively avoids abrupt gain changes caused by frequency band weight switching or positional shifts, but also, through the obtained fused gain coefficients, combines the characteristics of both algorithms to improve the sound image localization accuracy across the entire frequency band. Furthermore, by amplifying the audio signal based on the sub-coefficients and utilizing different speakers corresponding to those sub-coefficients, the amplified audio signal is output. Thus, the amplified audio signal exhibits high sound image localization accuracy across the entire frequency band, enhancing the user's listening experience.

[0080] Figure 3 This is a flowchart illustrating step S100 provided in an exemplary embodiment of this disclosure.

[0081] like Figure 3 As shown, in one implementation, step S100, determining at least one frequency band based on the audio signal of any audio track, includes steps S110-S120.

[0082] Step S110: Based on windowing and frame-segmentation processing and fast Fourier transform processing of the audio signal of any audio track, spectrum data is obtained.

[0083] Windowing framing is a processing method that divides a continuous audio signal into multiple short frames and applies a window function (such as the Hanning window function, the Hamming window function, etc.) to each frame.

[0084] Windowing is used to reduce spectral leakage at frame boundaries, making subsequent spectral analysis more accurate.

[0085] Window functions can affect spectral resolution and sidelobe suppression. The Hanning window function strikes a good balance between sidelobe suppression and spectral resolution. The Hamming window function is similar to the Hanning window function in terms of main lobe width and sidelobe suppression, and is also suitable for short-time spectral analysis of audio signals.

[0086] The embodiments disclosed herein do not limit the specific processing method of windowing and framing.

[0087] Spectral data is the frequency domain representation of an audio signal, which includes multiple bands, each corresponding to a frequency range. The bands can also record the energy magnitude (amplitude) and phase information of each frequency component.

[0088] An example is given of a case where an electronic device processes the audio signal of any audio track based on a 256th-order FFT to obtain spectral data.

[0089] For example, an electronic device can divide an audio signal into frames with a frame length of 256 sampling points, and set a 50% overlap between adjacent audio frames (i.e., a frame shift of 128 sampling points). After applying a Hanning window function to each frame signal, FFT processing is performed. Since FFT is an algorithm that efficiently converts time-domain signals into frequency-domain signals, it can transform audio frames from time-domain waveforms into a spectral representation, thereby obtaining the amplitude and phase information of each frequency component, i.e., spectral data.

[0090] By employing this approach, the electronic device preprocesses the audio signal, eliminating interference caused by time-domain non-stationarity and truncation effects. This provides high-fidelity, high-precision spectral data for subsequent processes, thus laying the data foundation for high-quality frequency band division and spatial audio-visual rendering.

[0091] Step S120: Based on the frequency band division of the spectrum data, at least one frequency band is determined.

[0092] Further based on the aforementioned example, when an electronic device is divided into three frequency bands, the electronic device can divide the frequency range corresponding to the first 4 FFTs (approximately 0-281.25Hz) into the low frequency band, the frequency range corresponding to the 5th to 23rd FFTs (approximately 281.25-2062.5Hz) into the mid frequency band, and the frequency range corresponding to the 24th and above FFTs (approximately 2062.5Hz and above) into the high frequency band.

[0093] The embodiments disclosed herein do not limit the specific method of dividing the frequency band.

[0094] By using this method, the electronic device decouples the audio signal by dividing the spectrum data into frequency bands. This allows the electronic device to determine an appropriate spatial rendering strategy for each frequency band in subsequent processes, thereby improving the user's listening experience.

[0095] The technical solution shown in this disclosure involves an electronic device preprocessing the audio signal of the current audio track to obtain spectrum data, and then dividing the spectrum data into multiple frequency bands, providing a basis for adopting differentiated rendering strategies for different frequency bands.

[0096] Figure 4 This is a flowchart illustrating step S200 provided in an exemplary embodiment of this disclosure.

[0097] like Figure 4 As shown, in one implementation, step S200 involves determining the first gain coefficient of different speakers under the VBAP algorithm based on the positions of different speakers and the target object, and determining the second gain coefficient of different speakers under the DBAP algorithm, including steps S210-S230.

[0098] Step S210: Determine the spatial coordinate system based on the position of the target object.

[0099] In one implementation, the electronic device establishes a three-dimensional spatial coordinate system based on the position of the target object (e.g., a person listening to a sound) as the origin.

[0100] Step S220: Based on the positions of different speakers in the world coordinate system, determine the positions of different speakers in the spatial coordinate system.

[0101] In one implementation, the electronic device determines the position coordinates of different speakers in a three-dimensional spatial coordinate system based on the positions of different speakers in the world coordinate system. The position coordinates are used to characterize the position of the speakers.

[0102] Step S230: Based on the positions of different speakers in the spatial coordinate system, determine the first gain coefficient of different speakers under the VBAP algorithm, and determine the second gain coefficient of different speakers under the DBAP algorithm.

[0103] In one implementation, the electronic device calculates the azimuth and pitch angles of different position coordinates relative to the origin based on the positions of different speakers in the three-dimensional spatial coordinate system, and determines the first gain coefficient of different speakers in the direction of the target object through direction vector decomposition based on the VBAP algorithm.

[0104] In one implementation, the electronic device calculates the distance between the different position coordinates of the different speakers and the origin based on the positions of the different speakers in the three-dimensional spatial coordinate system, and calculates the second gain coefficient of the different speakers by combining the distance attenuation model of the DBAP algorithm.

[0105] Using this method, the electronic device obtains the gain coefficients of VBAP and DBAP, providing basic data for subsequent fusion processing.

[0106] Figure 5 This is another flowchart illustrating step S200 provided in an exemplary embodiment of this disclosure.

[0107] like Figure 5 As shown, in one implementation, step S200 involves determining the first gain coefficient of different speakers under the VBAP algorithm based on the positions of different speakers and the target object, and determining the second gain coefficient of different speakers under the DBAP algorithm, including steps S240-S270.

[0108] Step S240: Determine the location of the virtual sound source based on the positions of different speakers and the position of the target object.

[0109] The target object can be a person listening to the sound. The location of the target object can be determined by in-vehicle seat sensors, a face detection module, or other sensing methods, or it can be a preset standard listening position (e.g., the center of the driver's seat). This disclosure does not limit the specific method by which the electronic device determines the location of the target object.

[0110] The positions of different speakers can be determined based on their spatial coordinates in the three-dimensional coordinate system inside the vehicle.

[0111] The virtual sound source location refers to the acoustic focal coordinates of a non-existent sound source, synthesized in physical space using multi-speaker arrays or binaural rendering technology. Specifically, the virtual sound source location can be determined based on the target object's azimuth (i.e., the angle of the virtual sound source on the horizontal plane (e.g., 0° directly in front, 90° to the right)), elevation (i.e., the height angle of the virtual sound source in the vertical direction (e.g., above the head, below the ground)), and distance (i.e., the straight-line physical distance between the virtual sound source and the center point of the listener's ears).

[0112] For example, an electronic device can establish a three-dimensional spatial coordinate system with the location of the target object as the origin, and determine the location of the virtual sound source based on the azimuth, elevation, and distance within this coordinate system.

[0113] By employing this method, the electronic device determines the location of the virtual sound source, providing a clear geometric reference point for subsequent gain coefficient calculations.

[0114] Step S250: Based on the virtual sound source location, determine the target speaker and non-target speaker among all speakers.

[0115] In one implementation, after obtaining the location of the virtual sound source, the electronic device can first perform a speaker layout preprocessing operation to obtain a speaker vector set. This speaker vector set includes the direction vectors corresponding to different speakers.

[0116] The electronic device selects target loudspeakers based on the virtual sound source location, that is, a subset of loudspeakers that have the best geometric matching relationship with the virtual sound source location in space. This facilitates subsequent precise gain allocation based on this subset of loudspeakers.

[0117] In this process, once the target speaker is identified, the electronic device can obtain the direction vector corresponding to the target speaker.

[0118] Furthermore, the electronic device can identify all speakers except the target speaker as non-target speakers.

[0119] In this case, the electronic device can obtain the direction vector corresponding to the non-target speaker when it is determined that the non-target speaker is not the target speaker.

[0120] The present invention does not impose any restrictions on the selection rules and number of target loudspeakers, which can be determined according to the actual spatial layout and algorithm requirements.

[0121] By using this method, electronic devices can quickly identify the effective loudspeakers participating in the current acoustic image localization calculation, reducing unnecessary computation and improving algorithm efficiency.

[0122] Step S260: Determine the first initial gain coefficient of the target loudspeaker based on the direction vector of the target loudspeaker.

[0123] In one implementation, the electronic device determines the first initial gain coefficient of the target loudspeaker based on Equation 3.

[0124] Formula 3: g = p × inv (L_base).

[0125] Where g is the first initial gain coefficient, L_base is the direction vector of the target loudspeaker, p is the virtual sound source position, and inv is the inverse matrix.

[0126] By using this method, the electronic device maps the desired coordinates of the virtual sound source onto the physical speaker array, establishing a precise physical spatial mapping relationship and achieving accurate positioning of the virtual sound source.

[0127] Step S270: If the first initial gain coefficient of the target loudspeaker is greater than or equal to 0, the first gain coefficient of different loudspeakers under the VBAP algorithm is determined based on normalizing the first initial gain coefficient of the target loudspeaker and setting the second initial gain coefficient of the non-target loudspeakers to 0. The first gain coefficient includes the normalized first initial gain coefficient and the second initial gain coefficient set to 0.

[0128] In one implementation, the electronic device normalizes the first initial gain coefficient of the target loudspeaker based on Formula 4 to obtain the normalized first initial gain coefficient.

[0129] Formula 4: g_norm = g / sqrt(sum(g)) 2 )).

[0130] Where g_norm is the first initial gain coefficient after normalization, g is the first initial gain coefficient, sqrt is the square root operation, and sum is the summation operation.

[0131] By employing this method, the electronic device normalizes the first initial gain coefficient, which is greater than or equal to 0, thus preventing the speaker from outputting in reverse phase and improving the accuracy and stability of sound image localization. Furthermore, the electronic device directly sets the second initial gain coefficient of non-target speakers to 0, preventing interference from non-target speakers on sound image localization and improving the purity and accuracy of sound image localization.

[0132] The technical solution illustrated in this disclosure uses the VBAP algorithm to linearly combine the direction vectors of different speakers, accurately reconstructing the directionality of the virtual sound source and ensuring the accuracy of sound image localization in the high-frequency band. This provides an accurate directional gain reference for subsequent frequency band fusion algorithms. Simultaneously, by normalizing the gain coefficients of non-target speakers and clearing them to zero, the energy consistency of the audio output is effectively guaranteed, avoiding volume anomalies caused by the superposition of multiple speakers.

[0133] Figure 6 This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of this disclosure.

[0134] like Figure 6 As shown, in one implementation, after determining the initial gain coefficient of the target loudspeaker based on the direction vector of the target loudspeaker in step S260, step S280 is also included.

[0135] Step S280: If the first initial gain coefficient of the target loudspeaker is less than 0, the target loudspeaker and non-target loudspeakers among all loudspeakers are re-determined based on the virtual sound source position.

[0136] It's important to note that in the VBAP algorithm, when the virtual sound source is located outside the area formed by the target speaker (e.g., due to irregular geometry of the speaker array or offset of the virtual sound source), the initial gain coefficient of the target speaker may be negative. A negative gain coefficient physically indicates that the speaker needs to output inverted phase. This can lead to errors in sound image localization, degraded sound quality, and negative effects such as phase cancellation, impacting the user's listening experience.

[0137] In one implementation, when the electronic device detects that the first initial gain coefficient of the target loudspeaker is less than 0, it triggers a target loudspeaker reselection process, that is, it re-determines a new combination of target loudspeakers based on the virtual sound source location and recalculates the first initial gain coefficient until the first initial gain coefficient of all target loudspeakers is not less than 0.

[0138] For example, if the location of the target speaker forms a first triangular region, and the location of the virtual sound source shifts outside the first triangular region, the electronic device can reselect the target speaker from at least one adjacent second triangular region. For instance, if the electronic device initially selects the front left speaker (FL) and the front center speaker (FC) as target speakers, but the value of FL in the calculated first initial gain coefficient is negative, then the electronic device can switch to the front center speaker (FC) and the front right speaker (FR) as new target speakers, recalculate the first initial gain coefficient, until the first initial gain coefficient of all target speakers is non-negative.

[0139] The embodiments disclosed herein effectively ensure the physical correctness of the VBAP algorithm and the stability of sound image localization at any virtual sound source location by detecting the negative gain coefficient and reselecting the target loudspeaker. This avoids sound degradation, sound image shift, or phase cancellation problems caused by a negative gain coefficient, and improves the robustness of sound image rendering of the in-vehicle multi-channel system in complex acoustic scenarios.

[0140] Figure 7 This is a flowchart illustrating step S250 provided in an exemplary embodiment of this disclosure.

[0141] like Figure 7 As shown, in one implementation, step S250, based on the virtual sound source location, determines the target speaker and non-target speaker among all speakers, including steps S251-S252.

[0142] Step S251: Based on the virtual sound source location, the target loudspeaker among all loudspeakers is determined using a triangulation algorithm.

[0143] The triangulation algorithm refers to the algorithm that projects the direction vectors of all speakers in the speaker array onto a unit sphere and triangulates these direction vectors to divide the sphere into multiple triangular regions. Each triangular region is composed of the direction vectors of two or three neighboring speakers.

[0144] Triangulation algorithms include the Delaunay triangulation algorithm, which guarantees that there are no other speaker points in each triangular region, thus achieving geometrically optimal speaker grouping.

[0145] Other types of calculation methods can also be used for triangulation algorithms. This disclosure does not limit the specific calculation method of the triangulation algorithm.

[0146] Based on the triangulation algorithm, the electronic device can determine the triangular region into which the direction vector of the virtual sound source falls, and identify the vertex speaker of the triangular region as the target speaker.

[0147] For example, when the audio system of an electronic device is configured with seven channels (including FL, FC, FR, rear left speaker (RL), rear right speaker (RR), left surround speaker (SL), and right surround speaker (SR)), the electronic device can perform Delaunay triangulation on the azimuth-elevation plane to generate multiple triangular regions. If the direction vector of the virtual sound source falls into the triangular region formed by FL, FC, and SL, then FL, FC, and SL are the target speakers; if the direction vector of the virtual sound source falls into the triangular region formed by FC and FR, then FC and FR are the target speakers.

[0148] By using this method, the electronic device can obtain the minimum number of speakers and the clearest virtual sound image based on the triangulation algorithm, avoiding sound image blurring and diffusion.

[0149] Step S252: Based on the target loudspeaker, identify the non-target loudspeakers among all loudspeakers.

[0150] The electronic device identifies all speakers other than the target speaker as non-target speakers. The gain coefficient of non-target speakers will be set to 0 in the VBAP algorithm, meaning they will not participate in the sound image rendering of the current virtual sound source location.

[0151] The technical solution illustrated in this disclosure utilizes a triangulation algorithm to accurately determine the target loudspeaker, ensuring the geometric accuracy of the VBAP algorithm and providing a reliable basis for subsequent precise sound image localization. The triangulation algorithm can accurately determine the direction surrounding the virtual sound source, thereby improving the directional sense and localization accuracy of high-frequency sounds.

[0152] Figure 8 This is another flowchart illustrating step S200 provided in an exemplary embodiment of this disclosure.

[0153] like Figure 8 As shown, in one implementation, step S200, based on the positions of different speakers and the position of the target object, determines the first gain coefficient of different speakers under the VBAP algorithm, and determines the second gain coefficient of different speakers under the DBAP algorithm, and also includes steps S290-S293.

[0154] Step S290: Determine the location of the virtual sound source based on the positions of different speakers and the location of the target object.

[0155] For details on the implementation of step S290, please refer to the specific implementation of step S240. This embodiment will not elaborate further on this aspect.

[0156] Step S291: Determine the distance between the location of the virtual sound source and the locations of different speakers.

[0157] In one implementation, after obtaining the location of the virtual sound source, the electronic device can first perform a speaker layout preprocessing operation to obtain a set of speaker coordinates, which includes the coordinates corresponding to different speakers.

[0158] For example, the set of speaker coordinates is L, L = {L1, L2, L3, ..., Ln}. Where L1, L2, L3, ..., Ln are the coordinates of different speakers.

[0159] In one implementation, after the electronic device obtains the location of the virtual sound source, it can further perform a distance attenuation index preset operation.

[0160] The distance attenuation index is a power-law parameter in a mathematical model that describes how sound intensity (or sound pressure) decreases as the propagation distance increases.

[0161] For example, the distance attenuation index is β. β is, for example, 1, used to characterize inverse attenuation. This disclosure does not limit the specific value of the distance attenuation index.

[0162] In one implementation, after the electronic device obtains the location of the virtual sound source, it can further perform a minimum distance threshold preset operation.

[0163] The minimum distance threshold is d_min, which is used to avoid division by zero errors or overgain.

[0164] In one implementation, the electronic device calculates the Euclidean distance between the virtual sound source and different speakers based on Equation 5.

[0165] Euclidean distance is the straight-line distance between two points in three-dimensional space, used to quantify the spatial distance between virtual sound sources and different speakers.

[0166] Formula 5: d_i = ||S-L_i||.

[0167] Where d_i is the Euclidean distance between the virtual sound source and the i-th speaker, S is the coordinate of the virtual sound source, and L_i is the coordinate of the i-th speaker. ||·|| represents the Euclidean norm, used to calculate straight-line distances in three-dimensional space.

[0168] By using this method, the electronic device can obtain the distance between the location of the virtual sound source and the locations of different speakers, which facilitates the subsequent accurate calculation of the initial gain coefficient of different speakers.

[0169] Step S292: Based on the preset distance attenuation index, the preset minimum distance threshold, and the distance, determine the third initial gain coefficient for different speakers.

[0170] In one implementation, the electronic device can calculate the third initial gain coefficient for different speakers based on Equation 6.

[0171] Formula 6: raw_i = 1 / max(d_i, d_min) β .

[0172] Where raw_i is the third initial gain coefficient of the i-th speaker, d_i is the Euclidean distance between the virtual sound source and the i-th speaker, d_min is the preset minimum distance threshold, and β is the distance attenuation exponent.

[0173] It should be noted that in actual applications, electronic devices can adjust parameters according to the acoustic characteristics of the vehicle interior, and the specific values ​​of the parameters are not limited in the embodiments disclosed herein.

[0174] Using this method, the electronic device calculates a third initial gain coefficient based on physical distance relationships, providing an initial energy distribution benchmark that conforms to the laws of acoustic physics for subsequent audio processing. This physically driven energy distribution method helps to ensure that the energy distribution of the final rendered sound field is consistent with the real physical world, providing listeners with a natural and believable spatial auditory experience.

[0175] Step S293: Based on the normalization of the third initial gain coefficient, determine the second gain coefficient of different loudspeakers under the DBAP algorithm. The second gain coefficient includes the normalized third initial gain coefficient.

[0176] The electronic equipment normalizes the third initial gain coefficient of all speakers to obtain the second gain coefficient, which can effectively prevent gain overflow caused by individual speakers being too close together, thereby improving the stability and balance of the overall sound field energy.

[0177] By adopting this method, the electronic device normalizes the third initial gain coefficient, thus avoiding the speaker from outputting in reverse phase, which helps to improve the accuracy and stability of sound image localization.

[0178] The technical solution illustrated in this disclosure uses the DBAP algorithm to calculate gain based on the distance relationship between the virtual sound source and each speaker. This results in speakers closer to the virtual sound source receiving greater gain, while speakers farther away receive less gain, thus creating a natural sound image diffusion effect. The DBAP algorithm does not rely on the regularity of the speaker array, therefore it can be applied to speaker arrays with arbitrary layouts, providing a good sense of immersion and surround sound for low-frequency audio, effectively compensating for the deficiency of the VBAP algorithm in low-frequency sound image surround sound.

[0179] Figure 9 This is a flowchart illustrating step S293 provided in an exemplary embodiment of this disclosure.

[0180] like Figure 9 As shown, in one implementation, step S293, based on normalizing the third initial gain coefficient, determines the second gain coefficient of different loudspeakers under the DBAP algorithm, including step S2931.

[0181] Step S2931: Based on energy normalization or amplitude normalization of the third initial gain coefficient, determine the second gain coefficient of different loudspeakers under the DBAP algorithm.

[0182] Energy normalization refers to normalizing the sum of squares of the gain coefficients to make it equal to 1, ensuring that the total output acoustic energy remains constant in a multi-speaker system.

[0183] Amplitude normalization refers to normalizing the sum of the absolute values ​​of the gain coefficients to make them equal to 1, ensuring that when multiple speakers are emitting sound simultaneously, the total output energy will fluctuate with the number of active speakers.

[0184] In one implementation, the electronic device performs energy normalization on the third initial gain coefficient based on Equation 7 to determine the second gain coefficient for different speakers under the DBAP algorithm.

[0185] Formula 7: gain_i = raw_i / sqrt(Σraw_j 2 ).

[0186] Where gain_i represents the second gain coefficient of the i-th speaker, raw_i represents the third initial gain coefficient of the i-th speaker, and raw_j represents the third initial gain coefficient of all speakers traversed. sqrt represents the square root operation of the summation result within the parentheses.

[0187] This normalization method can ensure that the total energy of the output signal of each speaker remains constant, which is suitable for scenarios with high requirements for energy conservation (such as vehicle scenarios that are sensitive to volume stability).

[0188] In one implementation, the electronic device performs amplitude normalization on the third initial gain coefficient based on Equation 8 or Equation 9 to determine the second gain coefficient for different speakers under the DBAP algorithm.

[0189] Formula 8: gain_i = raw_i / max(raw_j).

[0190] Formula 9: gain_i = raw_i / sum(raw_j).

[0191] Where gain_i represents the second gain coefficient of the i-th speaker, raw_i represents the third initial gain coefficient of the i-th speaker, and raw_j represents the third initial gain coefficient of all speakers traversed. max(raw_j) represents the maximum value among the third initial gain coefficients of all speakers, and sum(raw_j) represents the sum of the absolute values ​​of the third initial gain coefficients of all speakers.

[0192] The technical solutions illustrated in this disclosure provide two different normalization strategies for electronic devices to adapt to diverse acoustic environment requirements. In this way, while ensuring sound image diffusion, the electronic device can maintain the consistency of audio output energy or amplitude, further enhancing the user's listening experience.

[0193] Figure 10This is another schematic flowchart of an audio processing method provided in an exemplary embodiment of the present disclosure.

[0194] like Figure 10 As shown, in one implementation, before step S300, which determines the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient in any frequency band and the second weight of the second gain coefficient, steps S600-S700 are also included.

[0195] Step S600: Determine the location of the virtual sound source based on the positions of different speakers and the position of the target object.

[0196] For details on the implementation of step S600, please refer to the specific implementation of step S240. This embodiment will not elaborate further on this aspect.

[0197] It should be noted that electronic devices may also execute step S600 as a sub-step of S200. This disclosure does not limit this aspect.

[0198] Step S700: Based on the positions of different speakers, the position of the target object, and the position of the virtual sound source, determine the first weight of the first gain coefficient and the second weight of the second gain coefficient in any frequency band, wherein the sum of the first weight and the second weight is 1, and the first weights in different frequency bands may be the same or different.

[0199] In one implementation, the electronic device can dynamically adjust the first weight and the second weight based on the degree of deviation of the target object from the geometric center of the speaker array (determined based on the positions of different speakers).

[0200] For example, when the target object is located at the geometric center of the speaker array, the electronic device can accurately reconstruct the direction of the virtual sound source based on the direction vector of the VBAP algorithm. In this case, the electronic device can set a larger first weight for the first gain coefficient. When the target object is off-center from the geometric center of the speaker array, the positioning accuracy of the virtual sound source based on the VBAP algorithm decreases. In this case, the electronic device can set a larger second weight for the second gain coefficient to utilize the distance attenuation characteristics of the DBAP algorithm to provide a more natural sound image transition, thereby compensating for the sound image distortion caused by the positional offset and improving the listening experience of the target object that is off-center from the geometric center of the speaker array.

[0201] The technical solution illustrated in this disclosure achieves a combination of accurate high-frequency sound image localization and enhanced low-frequency surround sound by assigning different weights to the VBAP and DBAP algorithms in different frequency bands. This solves the technical problem that a single algorithm cannot simultaneously achieve accurate sound image localization across the entire frequency band. Furthermore, through the adjustability of the weight parameters and dynamic compensation for the offset position of the target object, the electronic device can flexibly adapt to various in-vehicle music rendering scenarios and listening positions, significantly improving the immersive listening experience for passengers in different seats.

[0202] Figure 11 This is a flowchart illustrating step S300 provided in an exemplary embodiment of this disclosure.

[0203] like Figure 11 As shown, in one implementation, step S300 involves determining the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, including steps S310-S330.

[0204] Step S310: When the frequency band is a high frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is greater than the second weight.

[0205] By adopting the implementation method of this step, in the high-frequency band, the first weight is set to be greater than the second weight, which enhances the VBAP algorithm's sound image localization capability in the high-frequency band, ensures the clarity and directionality of high-frequency sound images, and thus improves the accuracy of the target object's perception of the virtual sound source location.

[0206] Step S320: When the frequency band is the intermediate frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is equal to the second weight.

[0207] Using this approach, in the mid-frequency band, the first weight and the second weight are set to be equal, enabling the VBAP algorithm and the DBAP algorithm to work together in the mid-frequency band. This balances the accuracy of sound image localization with the naturalness of sound field envelopment, achieving a balanced integration of mid-frequency sound image and atmosphere.

[0208] Step S330: When the frequency band is a low frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is less than the second weight.

[0209] By adopting the implementation method of this step, in the low frequency band, the first weight is set to be less than the second weight, which strengthens the distance attenuation characteristics of the DBAP algorithm in the low frequency band, enhances the sense of immersion and naturalness of the low frequency sound image, and thus improves the target object's perception of the low frequency sound field atmosphere.

[0210] The technical solution disclosed herein, through the above-mentioned frequency band weight configuration, enables the electronic device to achieve complementary advantages of the two algorithms, VBAP and DBAP, so as to achieve the optimal balance between sound image localization and sound field envelopment in high frequency, mid frequency and low frequency bands.

[0211] Figure 12 This is a flowchart illustrating step S500 provided in an exemplary embodiment of this disclosure.

[0212] like Figure 12 As shown, in one implementation, step S500 involves gaining the audio signal based on sub-coefficients and outputting the gained audio signal using different speakers corresponding to the sub-coefficients, including steps S510-S530.

[0213] Step S510: Based on the time-domain restoration processing of the audio signal, the time-domain restored audio signal is obtained.

[0214] Time-domain restoration is a process of converting an audio signal from its frequency domain representation back to its time domain representation.

[0215] In one implementation, time-domain restoration can be achieved through the inverse fast fourier transform (IFFT), that is, the spectral data after frequency domain gain fusion processing is inversely transformed into a time-domain waveform signal. Furthermore, since there is overlap between adjacent frames when the electronic device processes the audio signal in frames during the aforementioned process, the electronic device can use methods such as overlap-add (OLA) or overlap-save to stitch together the time-domain signals of different frames to obtain a complete time-domain restored audio signal, ensuring the continuity and seamless stitching of the time-domain restoration.

[0216] By using this method, the electronic device completes a closed-loop conversion from the frequency domain to the time domain, enabling the spatial audio rendering results to be correctly reproduced by the actual speaker array.

[0217] Step S520: Gain the audio signal after time-domain restoration based on the sub-coefficients to obtain the gained audio signal corresponding to different speakers.

[0218] In one implementation, the electronic device multiplies the time-domain restored audio signal by the sub-coefficients corresponding to different speakers in the fusion gain coefficient to obtain the audio signal with different gain corresponding to different speakers.

[0219] The sub-coefficient represents the relative amplitude weight of the current speaker in the current audio track space rendering. The larger the sub-coefficient, the stronger the sound output by the speaker. In this way, the sound output by the speaker can form a corresponding directional sound image localization in the perception of the target object.

[0220] It should be noted that, as can be seen from step S400 in the aforementioned embodiment, the sub-coefficients have undergone inter-frame smoothing. Therefore, during the temporal gain process, the gain change between adjacent frames is smooth and will not produce popping sounds or auditory discomfort caused by sudden gain changes.

[0221] By employing this method, the electronic device applies sub-coefficients to the time-domain restored audio signal, precisely modulating the signal amplitudes output by different speakers to their respective spatial sound field components. In this way, when multiple speakers work together, their sound waves are superimposed at the target object's location to synthesize the desired virtual sound image. This accurately maps the algorithm-level spatial coordinate information into a perceptible physical sound field, while maintaining sound image positioning accuracy across the entire frequency range, thus enhancing the user's listening experience.

[0222] Step S530: Based on different speakers, output the enhanced audio signal.

[0223] In one implementation, the electronic device outputs the gained audio signals corresponding to different speakers to the corresponding speakers for playback, thereby realizing the spatial positioning and rendering of the audio track in the multi-channel speaker array.

[0224] Using this method, in scenarios where multiple audio tracks are rendered simultaneously, the electronic device processes the audio signals of different tracks (such as vocals, piano, guitar, and drums), and the signals output to the corresponding speakers are superimposed and mixed at the speakers, ultimately presenting a multi-channel music effect with a sense of spatial layering, thus enhancing the user's listening experience.

[0225] This disclosure illustrates a technical solution whereby an electronic device first performs time-domain restoration processing and then applies the smoothed sub-coefficients to different speakers for gain output. This achieves the entire process from frequency-domain fusion gain calculation to time-domain multi-channel audio output, ensuring the integrity of audio processing and output sound quality. This enables users to obtain accurate sound image positioning across the entire frequency band and a high-quality immersive listening experience in the in-car multi-channel audio system.

[0226] The solutions disclosed herein are not limited to the embodiments mentioned above.

[0227] Exemplary device The audio processing method provided by the embodiments of this disclosure has been described above. It is understood that, in order to implement the various functions of the audio processing method, the audio processing apparatus may include corresponding hardware and software for implementing the hardware functions.

[0228] Those skilled in the art will readily recognize that the steps of the audio processing method described in conjunction with the embodiments of this disclosure can be implemented in hardware or in a combination of software-driven hardware. Whether a function is implemented in hardware or software-driven hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0229] Figure 13 This is a schematic diagram of the structure of an audio processing apparatus provided in an exemplary embodiment of the present disclosure.

[0230] like Figure 13 As shown, in one implementation, the audio processing device 1300 includes: The first determining module 1310 is used to determine at least one frequency band based on the audio signal of any audio track.

[0231] The second determining module 1320 is used to determine the first gain coefficient of different speakers under the vector-based amplitude translation VBAP algorithm based on the positions of different speakers and the position of the target object, and to determine the second gain coefficient of different speakers under the distance-based amplitude translation DBAP algorithm.

[0232] The third determining module 1330 is used to determine the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient.

[0233] The fourth determining module 1340 is used to obtain a fusion gain coefficient based on the target gain coefficient and all frequency bands corresponding to any audio track. The fusion gain coefficient includes sub-coefficients corresponding to different loudspeakers.

[0234] The output module 1350 is used to gain the audio signal based on sub-coefficients and output the gained audio signal using different speakers corresponding to the sub-coefficients.

[0235] This disclosure provides an audio processing apparatus that determines at least one frequency band based on the audio signal of any audio track. By dividing the audio signal into frequency bands, it enables more precise processing of sound field localization at different frequencies. Furthermore, based on the positions of different speakers and the target object, it determines a first gain coefficient for different speakers under the VBAP algorithm and a second gain coefficient for different speakers under the DBAP algorithm. This allows subsequent calculations to retain the original characteristics of both algorithms by leveraging the high localization accuracy of the first gain coefficient of the VBAP algorithm and the robustness of the second gain coefficient of the DBAP algorithm in irregular speaker layouts, thus achieving complementary advantages during the fusion process. Further, based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient, a target gain coefficient for the current frequency band is determined. By assigning different weights to different frequency bands, the low-frequency band emphasizes the DBAP algorithm to enhance sound field stability and robustness, while the high-frequency band emphasizes the VBAP algorithm to improve sound image localization accuracy, achieving segmented fusion optimization in the frequency domain. Finally, based on all frequency bands corresponding to any audio track, a fused gain coefficient is obtained by fusing using the target gain coefficient. This approach not only effectively avoids abrupt gain changes caused by frequency band weight switching or positional shifts, but also, through the obtained fused gain coefficients, combines the characteristics of both algorithms to improve the sound image localization accuracy across the entire frequency band. Furthermore, by amplifying the audio signal based on the sub-coefficients and utilizing different speakers corresponding to those sub-coefficients, the amplified audio signal is output. Thus, the amplified audio signal exhibits high sound image localization accuracy across the entire frequency band, enhancing the user's listening experience.

[0236] Figure 14 This is another structural schematic diagram of an audio processing apparatus provided in an exemplary embodiment of the present disclosure.

[0237] like Figure 14 As shown, in one implementation, the first determining module 1310 is further configured to obtain spectrum data based on windowing and frame-segmentation processing and fast Fourier transform processing of the audio signal of any audio track; and to determine at least one frequency band based on frequency band division of the spectrum data.

[0238] In one implementation, the second determining module 1320 is further configured to: determine a spatial coordinate system based on the position of the target object; determine the positions of different speakers in the spatial coordinate system based on the positions of different speakers in the world coordinate system; determine the first gain coefficient of different speakers under the VBAP algorithm based on the positions of different speakers in the spatial coordinate system; and determine the second gain coefficient of different speakers under the DBAP algorithm.

[0239] In one implementation, the second determining module 1320 is further configured to determine the virtual sound source position based on the positions of different speakers and the position of the target object; determine the target speaker and non-target speakers among all speakers based on the virtual sound source position; determine the first initial gain coefficient of the target speaker based on the direction vector of the target speaker; and, if the first initial gain coefficient of the target speaker is greater than or equal to 0, determine the first gain coefficient of different speakers under the VBAP algorithm by normalizing the first initial gain coefficient of the target speaker and setting the second initial gain coefficient of the non-target speaker to 0. The first gain coefficient includes the normalized first initial gain coefficient and the second initial gain coefficient set to 0.

[0240] In one implementation, the audio processing device 1300 further includes a fifth determining module 1360.

[0241] The fifth determination module 1360 is used to redetermine the target speaker and non-target speakers among all speakers based on the virtual sound source position when the initial gain coefficient of the target speaker is less than 0.

[0242] In one implementation, the second determining module 1320 is further configured to determine the target loudspeaker among all loudspeakers based on the virtual sound source location using a triangulation algorithm; and to determine the non-target loudspeakers among all loudspeakers based on the target loudspeaker.

[0243] In one implementation, the second determining module 1320 is further configured to determine the virtual sound source position based on the positions of different speakers and the position of the target object; determine the distance between the position of the virtual sound source and the positions of different speakers; determine the third initial gain coefficient of different speakers based on a preset distance attenuation index, a preset minimum distance threshold, and the distance; and determine the second gain coefficient of different speakers under the DBAP algorithm based on normalizing the third initial gain coefficient, wherein the second gain coefficient includes the normalized third initial gain coefficient.

[0244] In one implementation, the second determining module 1320 is further configured to determine the second gain coefficient of different loudspeakers under the DBAP algorithm based on energy normalization or amplitude normalization of the third initial gain coefficient.

[0245] In one implementation, the audio processing device 1300 further includes a sixth determining module 1370 and a seventh determining module 1380.

[0246] The sixth determining module 1370 is used to determine the location of the virtual sound source based on the positions of different speakers and the position of the target object.

[0247] The seventh determining module 1380 is used to determine the first weight of the first gain coefficient and the second weight of the second gain coefficient in any frequency band based on the positions of different speakers, the position of the target object and the position of the virtual sound source, wherein the sum of the first weight and the second weight is 1, and the first weight in different frequency bands may be the same or different.

[0248] In one implementation, the third determining module 1330 is configured to: determine the target gain coefficient of the current frequency band based on a first weight of a first gain coefficient and a second weight of a second gain coefficient when the frequency band is a high frequency band, wherein the first weight is greater than the second weight; determine the target gain coefficient of the current frequency band based on a first weight of a first gain coefficient and a second weight of a second gain coefficient when the frequency band is a mid frequency band, wherein the first weight is equal to the second weight; and determine the target gain coefficient of the current frequency band based on a first weight of a first gain coefficient and a second weight of a second gain coefficient when the frequency band is a low frequency band, wherein the first weight is less than the second weight.

[0249] In one implementation, the output module 1350 is further configured to: obtain a time-domain restored audio signal based on time-domain restoration processing of the audio signal; gain the time-domain restored audio signal based on sub-coefficients to obtain a gained audio signal corresponding to different speakers; and output the gained audio signal based on different speakers.

[0250] The beneficial technical effects corresponding to the exemplary embodiments of this device can be found in the corresponding beneficial technical effects of the exemplary method section above, and will not be repeated here.

[0251] Exemplary electronic devices Figure 15 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure.

[0252] like Figure 15 As shown, the electronic device 1500 includes at least one processor 1510 and a memory 1520.

[0253] The processor 1510 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 1500 to perform desired functions.

[0254] The memory 1520 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 1510 may execute one or more computer program instructions to implement the audio processing methods and / or other desired functions of the various embodiments of this disclosure described above.

[0255] In one example, the electronic device 1500 may also include an input device 1530 and an output device 1540, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0256] The input device 1530 may also include, for example, a keyboard, a mouse, etc.

[0257] The output device 1540 can output various information to the outside, including, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0258] Of course, for the sake of simplicity, Figure 15 Only some of the components of the electronic device 1500 relevant to this disclosure are shown, omitting components such as buses, input / output interfaces, etc. In addition, the electronic device 1500 may include any other suitable components depending on the specific application.

[0259] Exemplary computer program products and computer-readable storage media In addition to the methods and apparatus described above, embodiments of this disclosure may also provide a computer program product, including computer program instructions that, when executed by a processor, cause the processor to perform the steps of the audio processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0260] Computer program products can be written in any combination of one or more programming languages ​​to perform the operations of embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on a user's computing device, partially on a user's computing device, as a standalone software package, partially on a user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0261] Furthermore, embodiments of this disclosure may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps of the audio processing methods of the various embodiments of this disclosure described in the "Exemplary Methods" section above.

[0262] Computer-readable storage media may take the form of any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may include, but is not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0263] The basic principles of this disclosure have been described above with reference to specific embodiments. However, the advantages, benefits, and effects mentioned in this disclosure are merely examples and not limitations, and should not be considered as essential features of each embodiment of this disclosure. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the scope of this disclosure to the necessity of employing the aforementioned specific details for implementation.

[0264] Various modifications and variations can be made to this disclosure without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. An audio processing method, the method comprising: Based on the audio signal of any audio track, determine at least one frequency band; Based on the positions of different speakers and the target object, the first gain coefficient of the different speakers under the vector-based amplitude translation VBAP algorithm is determined, and the second gain coefficient of the different speakers under the distance-based amplitude translation DBAP algorithm is determined. Based on the first weight of the first gain coefficient corresponding to any of the frequency bands, and the second weight of the corresponding second gain coefficient, the target gain coefficient of the current frequency band is determined; Based on all the frequency bands corresponding to any of the audio tracks, a fusion gain coefficient is obtained by fusing the target gain coefficient, wherein the fusion gain coefficient includes sub-coefficients corresponding to different speakers; The audio signal is amplified based on the sub-coefficients, and the amplified audio signal is output using the different speakers corresponding to the sub-coefficients.

2. The audio processing method according to claim 1, wherein, The determination of at least one frequency band based on the audio signal of any audio track includes: Based on windowing and frame-segmentation processing and fast Fourier transform processing of the audio signal of any of the aforementioned audio tracks, spectral data is obtained. Based on the frequency band division of the spectrum data, at least one frequency band is determined.

3. The audio processing method according to claim 1, wherein, The determination of the first gain coefficient of different speakers under the vector-based amplitude translation (VBAP) algorithm and the second gain coefficient of different speakers under the distance-based amplitude translation (DBAP) algorithm, based on the positions of different speakers and the target object, includes: Based on the location of the target object, determine the spatial coordinate system; Based on the different positions of the speakers in the world coordinate system, determine the different positions of the speakers in the spatial coordinate system; Based on the positions of the different speakers in the spatial coordinate system, the first gain coefficient of the different speakers under the VBAP algorithm is determined, and the second gain coefficient of the different speakers under the DBAP algorithm is determined.

4. The audio processing method according to claim 1, wherein, The determination of the first gain coefficient of different loudspeakers under the vector-based amplitude translation VBAP algorithm based on the positions of different loudspeakers and the target object includes: The location of the virtual sound source is determined based on the positions of the different speakers and the position of the target object; Based on the location of the virtual sound source, the target speaker and non-target speaker among all the speakers are determined; Based on the direction vector of the target loudspeaker, the first initial gain coefficient of the target loudspeaker is determined; When the first initial gain coefficient of the target loudspeaker is greater than or equal to 0, the first gain coefficient of different loudspeakers under the VBAP algorithm is determined by normalizing the first initial gain coefficient of the target loudspeaker and setting the second initial gain coefficient of the non-target loudspeaker to 0. The first gain coefficient includes the normalized first initial gain coefficient and the second initial gain coefficient set to 0.

5. The audio processing method according to claim 4, wherein, After determining the first initial gain coefficient of the target loudspeaker based on its direction vector, the method further includes: If the first initial gain coefficient of the target loudspeaker is less than 0, the target loudspeaker and the non-target loudspeaker among all the loudspeakers are re-determined based on the virtual sound source location.

6. The audio processing method according to claim 4, wherein, The step of determining the target speaker and non-target speaker among all the speakers based on the virtual sound source location includes: Based on the location of the virtual sound source, the target loudspeaker among all the loudspeakers is determined using a triangulation algorithm; Based on the target loudspeaker, identify the non-target loudspeakers among all the loudspeakers.

7. The audio processing method according to claim 1, wherein, Determining the second gain coefficient for different loudspeakers under the distance-based amplitude shift DBAP algorithm includes: The location of the virtual sound source is determined based on the positions of the different speakers and the position of the target object; Determine the distance between the location of the virtual sound source and the locations of the different speakers; Based on a preset distance attenuation index, a preset minimum distance threshold, and the distance, a third initial gain coefficient for different speakers is determined; Based on the normalization of the third initial gain coefficient, the second gain coefficient of different loudspeakers under the DBAP algorithm is determined, and the second gain coefficient includes the normalized third initial gain coefficient.

8. The audio processing method according to claim 7, wherein, The step of determining the second gain coefficient of different loudspeakers under the DBAP algorithm based on normalizing the third initial gain coefficient includes: The second gain coefficient of different loudspeakers under the DBAP algorithm is determined by performing energy normalization or amplitude normalization on the third initial gain coefficient.

9. The audio processing method according to claim 1, wherein, Before determining the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient in any of the frequency bands and the second weight of the second gain coefficient, the method further includes: The location of the virtual sound source is determined based on the positions of the different speakers and the position of the target object; Based on the different positions of the speakers, the position of the target object, and the position of the virtual sound source, determine the first weight of the first gain coefficient in any frequency band, and the second weight of the second gain coefficient, wherein the sum of the first weight and the second weight is 1, and the first weight in different frequency bands may be the same or different.

10. The audio processing method according to claim 1, wherein, Determining the target gain coefficient of the current frequency band based on a first weight of the first gain coefficient in any of the frequency bands and a second weight of the second gain coefficient includes: When the frequency band is a high-frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is greater than the second weight; When the frequency band is a mid-frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is equal to the second weight; When the frequency band is a low frequency band, the target gain coefficient of the current frequency band is determined based on the first weight of the first gain coefficient and the second weight of the second gain coefficient, wherein the first weight is less than the second weight.

11. The audio processing method according to claim 1, wherein, The step of gaining the audio signal based on the sub-coefficients and outputting the gained audio signal using different speakers corresponding to the sub-coefficients includes: The audio signal is obtained by performing time-domain restoration processing on the audio signal; Based on the sub-coefficients, the audio signal after time-domain restoration is amplified to obtain the amplified audio signal corresponding to different speakers; Based on the different speakers, the audio signal is output with increased gain.

12. An audio processing apparatus, the apparatus comprising: The first determining module is used to determine at least one frequency band based on the audio signal of any audio track; The second determining module is used to determine, based on the positions of different speakers and the position of the target object, the first gain coefficient of the different speakers under the vector-based amplitude translation VBAP algorithm, and the second gain coefficient of the different speakers under the distance-based amplitude translation DBAP algorithm; The third determining module is used to determine the target gain coefficient of the current frequency band based on the first weight of the first gain coefficient corresponding to any frequency band and the second weight of the corresponding second gain coefficient. The fourth determining module is used to obtain a fusion gain coefficient based on all the frequency bands corresponding to any of the audio tracks and using the target gain coefficient, wherein the fusion gain coefficient includes sub-coefficients corresponding to different speakers; An output module is used to gain the audio signal based on the sub-coefficients and output the gained audio signal using the different speakers corresponding to the sub-coefficients.

13. A computer-readable storage medium storing a computer program for performing the audio processing method according to any one of claims 1-11.

14. An electronic device, the electronic device comprising: processor; Memory used to store the processor's executable instructions; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the audio processing method according to any one of claims 1-11.

Images