Audio rendering method and device, equipment and storage medium

Abstract

The invention discloses an audio rendering method and device, equipment and a storage medium, and relates to the technical field of audio signal processing, and the method comprises the steps: obtaining a target 2D audio, and separating a target stereo from the target 2D audio; performing sound source object type analysis on the target stereo to determine whether a target sound source object corresponding to the target stereo is a point sound source, if the target sound source object is the point sound source, converting the target stereo into a monaural audio signal, generating first rendering metadata based on the monaural audio signal, and generating second rendering metadata based on the first rendering metadata; rendering the monaural audio signal based on the first rendering metadata; and if the sound source is not the point sound source, generating second rendering metadata based on the left channel signal and the right channel signal of the target stereo, and rendering the left channel signal and the right channel signal based on the second rendering metadata. By judging the type of the sound source, the problems of sound field information loss and sound quality degradation are solved.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, and in particular to an audio rendering method, apparatus, device, and storage medium. Background Technology

[0002] The object-based VBAP (Vector Base Amplitude Panning, a 3D audio localization technique) method processes audio based on audio object track segmentation. These audio object tracks originate from the audio production domain or master tape, and different audio object tracks are independent and unrelated. For 2D audio to 3D audio conversion, VBAP input consists of multiple correlated AI (Artificial Intelligence) separated stereo objects. These stereo objects can be point sound sources or multi-point sound sources with a sound field width.

[0003] Existing technology converts stereo objects into mono signals through time-domain averaging, obtains point source audio objects, and then feeds them to the VBAP renderer for rendering. This method suffers from high-frequency component attenuation or even cancellation when rendering stereo audio from multiple point sources, severely impacting the rendered sound quality. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an audio rendering method, apparatus, device, and storage medium capable of determining the type of sound source and solving the problems of sound field information loss and sound quality degradation. The specific solution is as follows:

[0005] Firstly, this application provides an audio rendering method, including:

[0006] The target 2D audio is acquired, and the target stereo is separated from the target 2D audio using a target sound source separation model; wherein the target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture, and the target stereo is an audio signal with independent acoustic characteristics;

[0007] The target stereo is analyzed for sound source object type to determine whether the target sound source object corresponding to the target stereo is a point sound source. If the target sound source object corresponding to the target stereo is a point sound source, the target stereo is converted into a mono audio signal. Based on the mono audio signal, the corresponding first rendering metadata is generated, and based on the first rendering metadata, the mono audio signal is rendered at the speaker end.

[0008] If the target sound source object corresponding to the target stereo is not a point sound source, then the corresponding second rendering metadata is generated based on the left channel signal and the right channel signal of the target stereo, and the left channel signal and the right channel signal are rendered at the speaker end based on the second rendering metadata.

[0009] Optionally, the sound source object type analysis of the target stereo includes:

[0010] Determine the first energy value corresponding to the left channel signal and the second energy value corresponding to the right channel signal of the target stereo, and determine the channel level difference between the left channel signal and the right channel signal based on the first energy value and the second energy value;

[0011] Obtain the first frequency domain signal corresponding to the left channel signal and the second frequency domain signal corresponding to the right channel signal, and determine the frequency domain amplitude difference between the left channel signal and the right channel signal based on the first frequency domain signal and the second frequency domain signal;

[0012] Obtain the first phase spectrum corresponding to the left channel signal and the second phase spectrum corresponding to the right channel signal, and determine the phase difference between the left channel signal and the right channel signal based on the first phase spectrum and the second phase spectrum;

[0013] The channel difference between the left channel signal and the right channel signal is determined based on the channel level difference, the frequency domain amplitude difference and the phase difference. The number of target sound source objects is identified based on the direction of arrival spectrum technology to obtain the corresponding identification results. The sound source object type analysis of the target stereo is performed based on the channel difference and the identification results.

[0014] Optionally, identifying the number of target sound source objects based on the direction of arrival spectrum technology includes:

[0015] The DOA value corresponding to each frequency point of the left channel signal and the right channel signal is calculated based on the direction of arrival spectrum technique, and a target histogram is constructed based on each DOA value;

[0016] The number of spectral peaks in the target histogram is detected, and the number of target sound source objects is identified based on the corresponding detection results.

[0017] Optionally, converting the target stereo sound into a mono audio signal includes:

[0018] Calculate the energy proportion of the first frequency domain signal and the second frequency domain signal at each frequency point, and perform weighted fusion of the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the energy proportion to obtain the mono amplitude spectrum;

[0019] The mono spectrum is obtained by using the phase spectrum of any channel signal and the amplitude spectrum of the mono channel, and the mono audio signal is obtained based on the mono spectrum.

[0020] Optionally, converting the target stereo sound into a mono audio signal includes:

[0021] Calculate the complex correlation coefficients of the first frequency domain signal and the second frequency domain signal at each frequency point to obtain the channel correlation between the left channel signal and the right channel signal;

[0022] Based on the channel correlation, the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal are weighted and fused to obtain the mono channel amplitude spectrum;

[0023] The mono spectrum is obtained by using the phase spectrum of any channel signal and the amplitude spectrum of the mono channel, and the mono audio signal is obtained based on the mono spectrum.

[0024] Optionally, generating the corresponding first rendering metadata based on the mono audio signal includes:

[0025] The mono audio signal is analyzed to determine the location information of the target sound source object corresponding to the mono audio signal, and the first rendering metadata is generated based on the location information.

[0026] Optionally, the audio rendering method further includes:

[0027] The ambient sound in the target 2D audio is rendered using a target environment rendering algorithm, and the target low-frequency component in the target stereo is low-pass filtered to render the target 2D audio; wherein, the target low-frequency component is an audio component with a frequency not greater than a preset frequency threshold.

[0028] Secondly, this application provides an audio rendering apparatus, comprising:

[0029] An audio separation module is used to acquire target 2D audio and separate target stereo sound from the target 2D audio using a target sound source separation model; wherein, the target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture, and the target stereo sound is an audio signal with independent acoustic characteristics;

[0030] The sound source object type analysis module is used to perform sound source object type analysis on the target stereo to determine whether the target sound source object corresponding to the target stereo is a point sound source. If the target sound source object corresponding to the target stereo is a point sound source, the target stereo is converted into a mono audio signal, and corresponding first rendering metadata is generated based on the mono audio signal. The mono audio signal is then rendered at the speaker end based on the first rendering metadata.

[0031] The signal rendering module is used to generate corresponding second rendering metadata based on the left channel signal and the right channel signal of the target stereo if the target sound source object corresponding to the target stereo is not a point sound source, and to render the left channel signal and the right channel signal at the speaker end based on the second rendering metadata.

[0032] Thirdly, this application provides an electronic device, comprising:

[0033] Memory, used to store computer programs;

[0034] A processor for executing the computer program to implement the aforementioned audio rendering method.

[0035] Fourthly, this application provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the aforementioned audio rendering method.

[0036] This application first acquires target 2D audio and then uses a target sound source separation model to separate target stereo sound from the target 2D audio. The target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture. The target stereo sound is an audio signal with independent acoustic characteristics. Then, a sound source object type analysis is performed on the target stereo sound to determine whether the target sound source object corresponding to the target stereo sound is a point sound source. If the target sound source object corresponding to the target stereo sound is a point sound source, the target stereo sound is converted into a mono audio signal. Based on the mono audio signal, corresponding first rendering metadata is generated, and based on the first rendering metadata, the mono audio signal is rendered at the speaker end. Finally, if the target sound source object corresponding to the target stereo sound is not a point sound source, corresponding second rendering metadata is generated based on the left and right channel signals of the target stereo sound, and based on the second rendering metadata, the left and right channel signals are rendered at the speaker end. Therefore, this application achieves adaptive processing of sound sources with different spatial characteristics by distinguishing between point sound sources and non-point sound sources through sound source object type analysis; avoids the high-frequency attenuation problem caused by traditional time-domain averaging by performing stereo-to-mono conversion on point sound sources and generating first rendering metadata; achieves complete preservation of the original sound field width and spatial distribution information by preserving the stereo signal of non-point sound sources and generating second rendering metadata for the left and right channels respectively; and solves the problems of sound field information loss and sound quality degradation by combining different types of audio signals and their corresponding rendering metadata for spatial rendering. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0038] Figure 1 This is a flowchart of an audio rendering method disclosed in this application;

[0039] Figure 2 This is a schematic diagram of an audio rendering process disclosed in this application;

[0040] Figure 3 This is a schematic diagram of the VBAP principle disclosed in this application;

[0041] Figure 4 This is a schematic diagram of a rendering area division disclosed in this application;

[0042] Figure 5This application discloses a flowchart of a single-point sound source audio rendering process.

[0043] Figure 6 This application discloses a flowchart of a multi-point sound source audio rendering process.

[0044] Figure 7 This application discloses a specific audio rendering flowchart;

[0045] Figure 8 This is a flowchart of a sound source quantity identification method disclosed in this application;

[0046] Figure 9 This is a schematic diagram of the structure of an audio rendering device disclosed in this application;

[0047] Figure 10 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0048] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0049] Current audio rendering methods suffer from high-frequency component attenuation or even cancellation when rendering stereo sound from multiple sound sources, severely impacting the rendered sound quality. To address this, this application provides an audio rendering method that solves the problems of sound field information loss and sound quality degradation by identifying the sound source type.

[0050] See Figure 1 As shown, this embodiment of the invention discloses an audio rendering method, including:

[0051] Step S11: Obtain the target 2D audio and use the target sound source separation model to separate the target stereo sound from the target 2D audio; wherein, the target sound source separation model is a model built based on temporal convolutional network and Transformer architecture, and the target stereo sound is an audio signal with independent acoustic characteristics.

[0052] This embodiment achieves 2D audio to 3D audio transcoding through a collaborative architecture of "sound source separation - coordinate mapping - 3D rendering". The specific solution is as follows: Figure 2 As shown, it includes:

[0053] 1. Multiple input sources: Receives 2D audio input (stereo / mono);

[0054] 2. Sound source separation and object extraction: A sound source separation model based on Temporal Convolutional Network (TCN) + Transformer is adopted to separate N independent sound source objects ("vocals, guitar, drums" in music) from 2D audio and output independent audio tracks for each object.

[0055] 3. Generate object rendering metadata based on user requirements and rendering preprocessing: User requirements include the placement of instruments (drums, bass, piano) and human voices separated from the sound source;

[0056] 4. 3D rendering:

[0057] Perform "object rendering" on the separated object audio and assign it to the corresponding speaker channel according to the mapped 3D coordinates;

[0058] Select environment rendering algorithm for non-object components;

[0059] The low-frequency components of each object's sound are extracted by low-pass filtering (cutoff frequency 80Hz) and summarized into the LFE channel.

[0060] The core technical details of this embodiment are as follows:

[0061] (1) Input module: Supported input formats: 2D audio, such as stereo PCM (Pulse Code Modulation, an encoding format), MP3, AAC (Advanced Audio Coding); mono WAV (Waveform Audio Format, an audio format);

[0062] (2) Sound source separation model: Model selection: End-to-end separation model based on temporal convolutional network (TCN) + Transformer, which supports the separation of 1-8 sound source objects;

[0063] Separation parameters: sampling rate 48kHz, frame length 2048 points, frame shift 512 points, separation signal-to-noise ratio (SDR) not less than 6dB;

[0064] Output: Independent PCM audio tracks for each object (16-bit depth, 48kHz sampling rate).

[0065] (3) Generate object rendering metadata based on user requirements and rendering preprocessing: User requirements include the placement of instruments (drums, bass, piano) and human voices separated from the sound source.

[0066] (4) Object rendering preprocessing: including stereo to mono conversion, point source and multi-point source detection, and rendering metadata generation;

[0067] (5) Object Rendering: Speaker rendering is performed on object audio tracks 1-N, mapping them onto a three-dimensional sound space. The principle is to use vector synthesis to calculate the gain distribution of the generated sound source signal across multiple nearest speakers. The calculation formula can be expressed as:

[0068] ;

[0069] Where g is the gain distribution vector and p is the Cartesian coordinate of the sound source. This is the coordinate matrix of the speakers within the rendering area.

[0070] Furthermore, the principle of VBAP in this embodiment is as follows: Figure 3 As shown, the audio channel rendering area division structure is as follows: Figure 4 As shown.

[0071] This embodiment first requires preparing the stereo file to be processed, and then performing AI audio separation on the stereo file to extract the left and right channel signals corresponding to object 1 (i.e., the target stereo). In other words, the target stereo in this application includes the left channel signal and the right channel signal.

[0072] Step S12: Analyze the source object type of the target stereo sound to determine whether the target source object corresponding to the target stereo sound is a point source. If the target source object corresponding to the target stereo sound is a point source, convert the target stereo sound into a mono audio signal, generate corresponding first rendering metadata based on the mono audio signal, and render the mono audio signal at the speaker end based on the first rendering metadata.

[0073] In this embodiment, the process of analyzing the sound source object type of the target stereo sound may specifically include: determining the first energy value corresponding to the left channel signal and the second energy value corresponding to the right channel signal of the target stereo sound, and determining the channel level difference between the left channel signal and the right channel signal based on the first energy value and the second energy value; acquiring the first frequency domain signal corresponding to the left channel signal and the second frequency domain signal corresponding to the right channel signal, and determining the frequency domain amplitude difference between the left channel signal and the right channel signal based on the first frequency domain signal and the second frequency domain signal; acquiring the first phase spectrum corresponding to the left channel signal and the second phase spectrum corresponding to the right channel signal, and determining the phase difference between the left channel signal and the right channel signal based on the first phase spectrum and the second phase spectrum; determining the channel difference degree between the left channel signal and the right channel signal based on the channel level difference, the frequency domain amplitude difference, and the phase difference; identifying the number of target sound source objects based on the direction of arrival spectrum technology to obtain the corresponding identification results; and performing sound source object type analysis on the target stereo sound based on the channel difference degree and the identification results.

[0074] Specifically, object type detection is performed on the separated left and right channels of Object 1. This is achieved through two core dimensions: "channel difference measurement" and "multi-source recognition."

[0075] I. Quantitative Detection Method for Left and Right Channel Differences:

[0076] The difference between the left and right channels is the core characteristic of stereo properties, reflecting the spatial distribution of the sound source (such as leftward, rightward, and surround sound). It is quantified through three dimensions: amplitude difference, phase difference, and correlation, and is suitable for evaluating the spatial positioning accuracy of separated audio.

[0077] (a) Amplitude difference detection (energy distribution imbalance):

[0078] 1. Overall Amplitude Difference (ILD (Inter-Channel Level Difference)):

[0079] Calculate the energy ratio of the left and right channels of the entire signal to reflect the overall bias:

[0080] Let the left channel signal be The right channel signal is The total energy is respectively (i.e., the first energy value) (i.e., the second energy value), where N is the signal length.

[0081] The formula for the overall amplitude difference is:

[0082] ;

[0083] in Avoid having a denominator of zero.

[0084] Interpretation of Results:

[0085] The left channel has significantly stronger energy (sound source is shifted to the left).

[0086] The right channel has significantly stronger energy (sound source is shifted to the right).

[0087] Balanced energy on both sides (sound source centered or surrounds).

[0088] 2. Frequency domain amplitude difference (frequency band-dependent energy bias):

[0089] The left and right amplitude differences of different frequency components may vary (e.g., low frequencies are biased to the left, high frequencies to the right). After frame segmentation and windowing (frame length 512 points, Hamming window), an FFT (Fast Fourier Transform) is performed to obtain the frequency domain signal. (i.e., the first frequency domain signal) (i.e., the second frequency domain signal), k is the frequency point, and the amplitude difference at each frequency point is:

[0090] ;

[0091] Calculate the standard deviation of the amplitude difference across the entire frequency band:

[0092] ;

[0093] Where K is the number of FFT points, This represents the mean ILD.

[0094] Interpretation of Results: This indicates a large amplitude difference between frequency bands (complex spatial distribution), while the opposite indicates a uniform distribution.

[0095] (ii) Phase difference detection (phase shift caused by time difference):

[0096] Phase difference is caused by the time difference of arrival (ITD) of the left and right channels, reflecting the horizontal direction angle of the sound source. The calculation method is as follows:

[0097] 1. After frame segmentation, perform FFT on the left and right channels to obtain the phase spectra corresponding to the left and right channels respectively. (i.e., the first phase spectrum) and (i.e., the second phase spectrum);

[0098] 2. Correct phase folding (eliminate jumps) and calculate the phase difference at each frequency point: ; where unwarp() is the phase unwinding function, ensuring that the phase difference is within ; Within the range.

[0099] 3. Calculate the absolute mean of the phase difference: ;

[0100] Interpretation of Results:

[0101] Small phase difference (sound source is close to the center line);

[0102] : Moderate phase difference (sound source deviates to the left or right);

[0103] Large phase difference (sound source is close to the left and right extreme positions).

[0104] (III) Vocal tract correlation detection (consistency of left and right signals):

[0105] The lower the correlation, the greater the difference in content between the left and right channels (which may include multiple sound sources or spatial effects). The calculation method is as follows:

[0106] 1. Time-domain correlation coefficient (overall consistency): ;

[0107] 2. Frequency domain correlation coefficient (frequency band consistency):

[0108] ;

[0109] in, The right channel conjugate is used to calculate the average value across the entire frequency band after taking the absolute value. .

[0110] Interpretation of Results:

[0111] and : The left and right heights are consistent (single sound source centered);

[0112] or (Single sound source offset or slight spatial effect);

[0113] or (Multiple sound sources or strong spatial effects).

[0114] In addition, in this embodiment, the process of identifying the number of target sound source objects based on the direction of arrival spectrum technology may specifically include: calculating the DOA (Direction of Arrival) value corresponding to each frequency point of the left and right channel signals based on the direction of arrival spectrum technology, and constructing a target histogram based on each DOA value; detecting the number of spectral peaks in the target histogram, and identifying the number of target sound source objects based on the corresponding detection results. Specific judgment indicators are shown in Table 1.

[0115] Table 1 Sound Source Identification Table

[0116]

[0117] Furthermore, in one specific embodiment, the process of converting the target stereo signal into a mono audio signal includes: calculating the energy ratio of the first frequency domain signal and the second frequency domain signal at each frequency point, and weightedly fusing the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the energy ratio to obtain a mono amplitude spectrum; obtaining a mono spectrum using the phase spectrum of any channel signal and the mono amplitude spectrum, and obtaining a mono audio signal based on the mono spectrum.

[0118] Specifically, when there is a significant energy difference between the left and right channels (such as stronger vocals in the left channel and stronger accompaniment in the right channel), the channel with stronger energy is allowed to dominate the synthesis at the corresponding frequency based on the weighting of the energy proportion of each frequency point, thereby improving signal clarity.

[0119] 1. Processing steps:

[0120] Frame segmentation, windowing, and FFT transformation are performed to obtain... and ;

[0121] Energy Calculation: Calculate the energy of the left and right channels at each frequency point. and Total Energy (To avoid the denominator being 0, a minimum value can be added.) ).

[0122] Weighted synthesis: Amplitude spectrum weighted by energy percentage:

[0123] The phase spectrum retains the left channel (or right channel).

[0124] Inverse Transform and Superposition: Reconstructing a Mono Frequency Domain Signal Perform an IFFT to obtain the time-domain frame. The final frames are superimposed to obtain the complete mono signal. .

[0125] 2. Core Formula:

[0126] Energy percentage weighting:

[0127] ;

[0128] Synthetic amplitude spectrum: ;

[0129] In another specific implementation, converting the target stereo signal into a mono audio signal includes: calculating the complex correlation coefficients of the first frequency domain signal and the second frequency domain signal at each frequency point to obtain the channel correlation between the left channel signal and the right channel signal; performing weighted fusion of the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the channel correlation to obtain the mono amplitude spectrum; obtaining the mono spectrum using the phase spectrum of any channel signal and the mono amplitude spectrum, and obtaining the mono audio signal based on the mono spectrum.

[0130] Specifically, this method introduces the "channel correlation" index. At frequencies with high correlation (such as human voices, where the difference between the left and right channels is small), equal amplitude averaging is used, while at frequencies with low correlation (such as stereo surround sound effects, where the difference between the left and right channels is large), energy weighting is used, taking into account both signal consistency and detail preservation.

[0131] Frame segmentation and windowing, FFT transformation: The method is consistent with the aforementioned approach, yielding... and ;

[0132] Correlation calculation: Calculate the complex correlation coefficient of the left and right channels at each frequency point:

[0133] ;

[0134] Take absolute value (Range 0~1, the closer to 1, the higher the correlation), where, for . conjugate.

[0135] Hybrid weighted synthesis: The amplitude spectrum is weighted by correlation, as shown in the formula:

[0136] ;

[0137] The phase spectrum retains the left channel and is then inversely transformed and superimposed.

[0138] Core formula:

[0139] ;

[0140] for . conjugate.

[0141] Synthetic amplitude spectrum:

[0142] ;

[0143] In the third specific implementation, when the left and right channel signals are equally important, the arithmetic mean of the amplitude spectra of the left and right channels is taken in the frequency domain, and the phase spectrum retains either channel (usually the left channel with stronger energy) to ensure that the phase of the synthesized signal is stable.

[0144] Processing steps:

[0145] Frame-by-frame windowing: for the stereo left channel and right channel The signal is divided into frames, with a frame length of N, and weighted using a Hanning window w(n) to obtain the windowed signal.

[0146] ;

[0147] Where i is the frame index.

[0148] Frequency domain transformation: Perform FFT on the windowed signal of each frame to obtain the left channel frequency domain signal. and right channel frequency domain signal , where k = 0, 1, ..., N-1 are frequency points.

[0149] Amplitude synthesis: Calculate the synthesized amplitude spectrum:

[0150] ;

[0151] Phase spectrum: ;

[0152] Finally, the inverse transformation and superposition process is performed, consistent with the steps described above.

[0153] Core formula:

[0154] Synthetic amplitude spectrum: ;

[0155] Mono frequency domain signal:

[0156] ;

[0157] Furthermore, in this embodiment, generating corresponding first rendering metadata based on the mono audio signal includes: analyzing the mono audio signal to determine the location information of the target sound source object corresponding to the mono audio signal, and generating the first rendering metadata based on the location information.

[0158] That is, based on the stereo signal of Object 1 separated by AI audio, the motion trajectory and spatial location information are analyzed to generate rendering metadata for Object 1.

[0159] Step S13: If the target sound source object corresponding to the target stereo is not a point sound source, then generate corresponding second rendering metadata based on the left channel signal and the right channel signal of the target stereo respectively, and render the left channel signal and the right channel signal at the speaker end based on the second rendering metadata.

[0160] In this embodiment, if the target sound source object corresponding to the target stereo is not a point sound source, then the trajectory / position detection of object 1 is performed on the left channel and the right channel of object 1, and rendering metadata of the left channel and the right channel of object 1 are generated respectively.

[0161] The object trajectory / position detection scheme employs a forward DOA estimation module based on "phase difference-amplitude difference fusion":

[0162] 1. Core Principle:

[0163] After AI separation, a single audio object (such as a drum) in the stereo left / right channels will have phase difference (IPD, Inter-Channel Phase Difference) and amplitude difference (ILD) due to the distance difference between the sound source and the left / right microphones (or the "virtual microphones" simulated during AI separation).

[0164] In the front sound field, the angle of the sound source (The angle with the front is negative on the left and positive on the right, ranging from -90° to +90°) The smaller the angle (closer to the front), the smaller the distance difference, and the closer IPD and ILD are to 0; the larger the angle (closer to the left and right front), the larger the absolute values ​​of IPD and ILD.

[0165] This module reduces the impact of noise interference on individual features by fusing the calculation results of IPD and ILD, thereby improving the accuracy of front position estimation (especially for mid-to-high frequency audio objects, such as pianos and snare drums).

[0166] 2. Module implementation process:

[0167] Assuming the input is a stereo pair of single audio objects separated by AI. (Left channel) (Right channel), sampling rate The microphone spacing is d (if not defined during AI separation, it can be set to a typical value of 0.15m, which matches the distance between human ears), and the sound wave velocity is c = 343m / s (at room temperature). The specific steps include:

[0168] Phase difference (IPD) calculation:

[0169] The frequency domain phases of the left and right channels are respectively , The phase difference needs to be corrected to Within the range, the formula is ;

[0170] Based on the geometric relationship of the front sound field, the relationship between phase difference and angle is as follows:

[0171] ;

[0172] In the formula, Let N be the frequency of the k-th point, and N be the number of FFT points. This is a fixed phase shift (which can be eliminated by calibration using a reference sound source directly in front).

[0173] Amplitude difference calculation:

[0174] The amplitude difference reflects the difference in energy attenuation between the left and right channels (in the front sound field, if the sound source is biased to the left, the amplitude of the left channel is greater, and vice versa), the formula is:

[0175] ;

[0176] in, Avoid using zero denominators; the unit is dB.

[0177] 3. Forward angle estimation:

[0178] Preliminary angle estimation at a single frequency point:

[0179] For each frequency point k, the angle is first calculated using IPD (to eliminate...). back):

[0180] ;

[0181] Then, an angle mapping is established using ILD (pre-trained with sound source samples at known angles). Lookup table, such as the 30-degree angle to the left front. 30 degrees to the right front )get .

[0182] Fusion and weighting:

[0183] Give higher weight to frequencies where the energy of an audio object is concentrated (such as the 100Hz band of a drum):

[0184] ;

[0185] The integration perspective is: ;

[0186] Forward constraint correction:

[0187] If the fusion angle exceeds the front range Correct it to the nearest boundary value to ensure that the output conforms to the "forward position" definition.

[0188] 4. Output specific location representation:

[0189] The output is "horizontal angle + confidence level", for example: "drum: 15 degrees left in front (92%)", where the confidence level is calculated from the angular consistency of the energy concentration frequency band (the smaller the angular standard deviation, the higher the confidence level).

[0190] Finally, the left channel of Object 1, the right channel of Object 1, the rendering metadata of the left channel of Object 1, and the rendering metadata of the right channel of Object 1 are input into the speaker rendering module to complete the final spatial rendering process at the speaker end.

[0191] It should be noted that the audio rendering method in this embodiment further includes: rendering the ambient sound in the target 2D audio using a target environment rendering algorithm, and performing low-pass filtering on the target low-frequency components in the target stereo to render the target 2D audio; wherein, the target low-frequency components are audio components with a frequency not greater than a preset frequency threshold.

[0192] That is, select an environment rendering algorithm for non-object components; extract the low-frequency components of each object's sound through low-pass filtering (cutoff frequency 80Hz) and summarize them into the LFE channel.

[0193] It is understood that, in this embodiment, the method for processing point sound sources is as follows: Figure 5 As shown, the processing method for non-point sound sources is as follows: Figure 6 As shown, the handling method for cases where it is unknown whether the sound source is a point source is as follows: Figure 7 As shown, specifically, if the object type detection result is "point source", the stereo is converted to mono, and then the speaker rendering stage begins; if the object type detection result is not "point source", the left channel of object 1 and the right channel of object 1 are directly input into the speaker for rendering.

[0194] Simultaneously, the left and right channels of Object 1 are input into the object trajectory / position detection module. If the object type detection result is "point source," single-source object rendering metadata is generated. If the object type detection result is "multi-point source," stereo object rendering metadata is generated. Finally, all required input signals (stereo-to-mono signal, left channel of Object 1, right channel of Object 1, rendering metadata) enter the speaker rendering stage to complete the final speaker-side rendering processing.

[0195] Therefore, this application achieves adaptive processing of sound sources with different spatial characteristics by distinguishing between point sound sources and non-point sound sources through sound source object type analysis; avoids the high-frequency attenuation problem caused by traditional time-domain averaging by performing stereo-to-mono conversion on point sound sources and generating first rendering metadata; achieves complete preservation of the original sound field width and spatial distribution information by preserving the stereo signal of non-point sound sources and generating second rendering metadata for the left and right channels respectively; and solves the problems of sound field information loss and sound quality degradation by combining different types of audio signals and their corresponding rendering metadata for spatial rendering.

[0196] As can be seen from the previous embodiment, this application describes the overall process of audio rendering. To make the technical solution in this application more complete, the process of identifying the number of target sound sources will be described next. See [link to relevant documentation]. Figure 8 As shown, this application provides a method for identifying the number of sound sources, including:

[0197] Step S21: Calculate the DOA value corresponding to each frequency point of the left and right channel signals based on the direction of arrival spectrum technique, and construct a target histogram based on the DOA values.

[0198] It should be noted that the "single content" separated by AI may contain multiple point sound sources (such as the separated "drum group" which contains multiple point sound sources such as bass drum and snare drum). The core of the judgment is to identify independent sound sources in different directions through spatial feature clustering and directional spectral peak detection.

[0199] (I) Multi-source identification based on DOA spectral peaks:

[0200] The steps to determine the number of sound sources using the number of spectral peaks estimated by DOA are as follows:

[0201] 1. Calculate the DOA at each frequency point (based on phase difference, assuming microphone spacing d = 0.15m and sound velocity 343 m / s), then

[0202] ;

[0203] in, For frequency, The angle between the front and the back, the range .

[0204] 2. Construct a DOA histogram (i.e., target histogram), with an angle interval of 1 degree, and calculate the rated energy percentage for each angle:

[0205] ;

[0206] Step S22: Detect the number of spectral peaks in the target histogram and identify the number of target sound source objects based on the corresponding detection results.

[0207] This embodiment requires detecting spectral peaks in the histogram (local maximum values ​​exceeding a threshold), and the number of spectral peaks represents the number of sound sources.

[0208] Threshold setting: 30% of the maximum value of the region histogram is the threshold peak, and independent peaks exceeding the threshold are regarded as a point source;

[0209] Interpretation of results: Two or more independent peaks indicate multiple point sources; one peak indicates only one source (or densely distributed sources).

[0210] In addition, this embodiment is based on multi-source verification using blind source separation (BSS):

[0211] If there is ambiguity (peak overlap) in the DOA spectrum peak detection, further decomposition is performed using blind source separation:

[0212] 1. Convert stereo signal Consider it as a mixed signal, assumed to be composed of sound sources at various points M:

[0213] ;

[0214] Where A is a 2×M mixing matrix, For the m-th sound source, It is noise.

[0215] 2. Use the FastlCA algorithm to estimate the separation matrix W and separate the candidate sound sources. ;

[0216] 3. Calculate the energy percentage of each candidate sound source. ;

[0217] like If the number of sound sources is not less than 2, then it is determined that there are multiple point sound sources.

[0218] Therefore, this application achieves adaptive processing of sound sources with different spatial characteristics by distinguishing between point sound sources and non-point sound sources through sound source object type analysis; avoids the high-frequency attenuation problem caused by traditional time-domain averaging by performing stereo-to-mono conversion on point sound sources and generating first rendering metadata; achieves complete preservation of the original sound field width and spatial distribution information by preserving the stereo signal of non-point sound sources and generating second rendering metadata for the left and right channels respectively; and solves the problems of sound field information loss and sound quality degradation by combining different types of audio signals and their corresponding rendering metadata for spatial rendering.

[0219] See Figure 9 As shown, an embodiment of the present invention discloses an audio rendering apparatus, comprising:

[0220] The audio separation module 11 is used to acquire target 2D audio and separate target stereo sound from the target 2D audio using a target sound source separation model; wherein, the target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture, and the target stereo sound is an audio signal with independent acoustic characteristics;

[0221] The sound source object type analysis module 12 is used to perform sound source object type analysis on the target stereo to determine whether the target sound source object corresponding to the target stereo is a point sound source. If the target sound source object corresponding to the target stereo is a point sound source, the target stereo is converted into a mono audio signal, and corresponding first rendering metadata is generated based on the mono audio signal. The mono audio signal is then rendered at the speaker end based on the first rendering metadata.

[0222] The signal rendering module 13 is used to generate corresponding second rendering metadata based on the left channel signal and the right channel signal of the target stereo if the target sound source object corresponding to the target stereo is not a point sound source, and to render the left channel signal and the right channel signal at the speaker end based on the second rendering metadata.

[0223] In some specific embodiments, the sound source object type analysis module 12 may specifically include:

[0224] The level difference determination unit is used to determine a first energy value corresponding to the left channel signal and a second energy value corresponding to the right channel signal of the target stereo, and to determine the channel level difference between the left channel signal and the right channel signal based on the first energy value and the second energy value;

[0225] The frequency domain amplitude difference determination unit is used to acquire a first frequency domain signal corresponding to the left channel signal and a second frequency domain signal corresponding to the right channel signal, and determine the frequency domain amplitude difference between the left channel signal and the right channel signal based on the first frequency domain signal and the second frequency domain signal;

[0226] A phase difference determination unit is used to acquire a first phase spectrum corresponding to the left channel signal and a second phase spectrum corresponding to the right channel signal, and determine the phase difference between the left channel signal and the right channel signal based on the first phase spectrum and the second phase spectrum;

[0227] The sound source object type analysis submodule is used to determine the channel difference degree between the left channel signal and the right channel signal based on the channel level difference, the frequency domain amplitude difference and the phase difference, identify the number of target sound source objects based on the direction of arrival spectrum technology to obtain the corresponding identification results, and perform sound source object type analysis on the target stereo based on the channel difference degree and the identification results.

[0228] In some specific embodiments, the sound source object type analysis submodule may specifically include:

[0229] The histogram construction unit is used to calculate the DOA value corresponding to each frequency point of the left channel signal and the right channel signal based on the direction of arrival spectrum technique, and construct a target histogram based on each DOA value;

[0230] The quantity recognition unit is used to detect the number of spectral peaks in the target histogram and to identify the number of target sound source objects based on the corresponding detection results.

[0231] In some specific embodiments, the sound source object type analysis module 12 may specifically include:

[0232] The first mono amplitude spectrum acquisition unit is used to calculate the energy ratio of the first frequency domain signal and the second frequency domain signal at each frequency point, and to perform weighted fusion of the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the energy ratio to obtain the mono amplitude spectrum.

[0233] The first mono audio signal acquisition unit is used to acquire a mono spectrum using the phase spectrum of any channel signal and the mono amplitude spectrum, and to acquire the mono audio signal based on the mono spectrum.

[0234] In some specific embodiments, the sound source object type analysis module 12 may specifically include:

[0235] The channel correlation acquisition unit is used to calculate the complex correlation coefficient of the first frequency domain signal and the second frequency domain signal at each frequency point to obtain the channel correlation between the left channel signal and the right channel signal.

[0236] The second mono amplitude spectrum acquisition unit is used to perform weighted fusion of the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the channel correlation to obtain the mono amplitude spectrum.

[0237] The second mono audio signal acquisition unit is used to acquire a mono spectrum using the phase spectrum of any channel signal and the mono amplitude spectrum, and to acquire the mono audio signal based on the mono spectrum.

[0238] In some specific embodiments, the sound source object type analysis module 12 may specifically include:

[0239] The first rendering metadata generation unit is used to analyze the mono audio signal to determine the location information of the target sound source object corresponding to the mono audio signal, and generate the first rendering metadata based on the location information.

[0240] In some specific embodiments, the audio rendering device further includes:

[0241] An ambient sound rendering module is used to render the ambient sound in the target 2D audio using a target environment rendering algorithm, and to perform low-pass filtering on the target low-frequency components in the target stereo to render the target 2D audio; wherein, the target low-frequency components are audio components with a frequency not greater than a preset frequency threshold.

[0242] Furthermore, embodiments of this application also disclose an electronic device, Figure 10 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0243] Figure 10This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the audio rendering method disclosed in any of the foregoing embodiments. Furthermore, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0244] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0245] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0246] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the audio rendering method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include a computer program capable of performing other specific tasks.

[0247] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed audio rendering method. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0248] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0249] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0250] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0251] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0252] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An audio rendering method, characterized in that, include: The target 2D audio is acquired, and the target stereo is separated from the target 2D audio using a target sound source separation model; wherein the target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture, and the target stereo is an audio signal with independent acoustic characteristics; The target stereo is analyzed for sound source object type to determine whether the target sound source object corresponding to the target stereo is a point sound source. If the target sound source object corresponding to the target stereo is a point sound source, the target stereo is converted into a mono audio signal. Based on the mono audio signal, the corresponding first rendering metadata is generated, and based on the first rendering metadata, the mono audio signal is rendered at the speaker end. If the target sound source object corresponding to the target stereo is not a point sound source, then the corresponding second rendering metadata is generated based on the left channel signal and the right channel signal of the target stereo, and the left channel signal and the right channel signal are rendered at the speaker end based on the second rendering metadata.

2. The audio rendering method according to claim 1, characterized in that, The analysis of the sound source object type for the target stereo includes: Determine the first energy value corresponding to the left channel signal and the second energy value corresponding to the right channel signal of the target stereo, and determine the channel level difference between the left channel signal and the right channel signal based on the first energy value and the second energy value; Obtain the first frequency domain signal corresponding to the left channel signal and the second frequency domain signal corresponding to the right channel signal, and determine the frequency domain amplitude difference between the left channel signal and the right channel signal based on the first frequency domain signal and the second frequency domain signal; Obtain the first phase spectrum corresponding to the left channel signal and the second phase spectrum corresponding to the right channel signal, and determine the phase difference between the left channel signal and the right channel signal based on the first phase spectrum and the second phase spectrum; The channel difference between the left channel signal and the right channel signal is determined based on the channel level difference, the frequency domain amplitude difference and the phase difference. The number of target sound source objects is identified based on the direction of arrival spectrum technology to obtain the corresponding identification results. The sound source object type analysis of the target stereo is performed based on the channel difference and the identification results.

3. The audio rendering method according to claim 2, characterized in that, The identification of the number of target sound source objects based on the direction of arrival spectrum technology includes: The DOA value corresponding to each frequency point of the left channel signal and the right channel signal is calculated based on the direction of arrival spectrum technique, and a target histogram is constructed based on each DOA value; The number of spectral peaks in the target histogram is detected, and the number of target sound source objects is identified based on the corresponding detection results.

4. The audio rendering method according to claim 2, characterized in that, The step of converting the target stereo sound into a mono audio signal includes: Calculate the energy proportion of the first frequency domain signal and the second frequency domain signal at each frequency point, and perform weighted fusion of the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal based on the energy proportion to obtain the mono amplitude spectrum; The mono spectrum is obtained by using the phase spectrum of any channel signal and the amplitude spectrum of the mono channel, and the mono audio signal is obtained based on the mono spectrum.

5. The audio rendering method according to claim 2, characterized in that, The step of converting the target stereo sound into a mono audio signal includes: Calculate the complex correlation coefficients of the first frequency domain signal and the second frequency domain signal at each frequency point to obtain the channel correlation between the left channel signal and the right channel signal; Based on the channel correlation, the first amplitude spectrum corresponding to the left channel signal and the second amplitude spectrum corresponding to the right channel signal are weighted and fused to obtain the mono channel amplitude spectrum; The mono spectrum is obtained by using the phase spectrum of any channel signal and the amplitude spectrum of the mono channel, and the mono audio signal is obtained based on the mono spectrum.

6. The audio rendering method according to claim 1, characterized in that, The generation of corresponding first rendering metadata based on the mono audio signal includes: The mono audio signal is analyzed to determine the location information of the target sound source object corresponding to the mono audio signal, and the first rendering metadata is generated based on the location information.

7. The audio rendering method according to any one of claims 1 to 6, characterized in that, Also includes: The ambient sound in the target 2D audio is rendered using a target environment rendering algorithm, and the target low-frequency component in the target stereo is low-pass filtered to render the target 2D audio; wherein, the target low-frequency component is an audio component with a frequency not greater than a preset frequency threshold.

8. An audio rendering apparatus, characterized in that, include: An audio separation module is used to acquire target 2D audio and separate target stereo sound from the target 2D audio using a target sound source separation model; wherein, the target sound source separation model is a model built based on a temporal convolutional network and a Transformer architecture, and the target stereo sound is an audio signal with independent acoustic characteristics; The sound source object type analysis module is used to perform sound source object type analysis on the target stereo to determine whether the target sound source object corresponding to the target stereo is a point sound source. If the target sound source object corresponding to the target stereo is a point sound source, the target stereo is converted into a mono audio signal, and corresponding first rendering metadata is generated based on the mono audio signal. The mono audio signal is then rendered at the speaker end based on the first rendering metadata. The signal rendering module is used to generate corresponding second rendering metadata based on the left channel signal and the right channel signal of the target stereo if the target sound source object corresponding to the target stereo is not a point sound source, and to render the left channel signal and the right channel signal at the speaker end based on the second rendering metadata.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the audio rendering method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the audio rendering method as described in any one of claims 1 to 7.

Images