An audio upmix method, apparatus, device, medium and product

Abstract

The application discloses an audio up-mix method, device, equipment, medium and product, comprising: performing short-time Fourier transform on an input two-channel stereo audio signal to obtain a time-frequency domain signal; extracting a spatial state parameter for each time-frequency processing unit; creating a unified spatial radiation physical model, in which the time-frequency domain signal is decomposed into a directional direct state component and a diffuse ambient state component; determining three-dimensional target sound field coordinates based on the spatial state parameter and a target playback layout; based on the three-dimensional target sound field coordinates and the target playback layout, generating a gain distribution of direct sound at the target playback layout and a distribution of ambient sound at the target playback layout respectively, constructing a target multi-channel sound field, and then constructing a multi-channel mapping optimization target function to determine a multi-channel allocation weight matrix; and performing multi-channel rendering on the time-frequency domain signal to synthesize a multi-channel time domain audio signal through inverse short-time Fourier transform. In this way, a high-quality multi-channel audio signal can be generated.

Description

Technical Field

[0001] This application relates to the field of audio processing technology, and in particular to an audio upmixing method, apparatus, device, medium and product. Background Technology

[0002] In audio playback scenarios such as immersive home theaters, object-based digital audio workstations, and television and car audio systems, a core requirement is to smoothly and naturally convert traditional two-channel stereo into immersive multi-channel audio. Existing stereo upmixing solutions typically remain at the level of simple signal distribution, treating feature extraction, target direction mapping, and demapping matrix as three isolated steps. This fragmentation leads to highly disjointed physical modeling in the time and frequency domains, resulting in an unbalanced sound field and affecting sound quality.

[0003] Therefore, how to improve the sound quality of multi-channel audio signals after upmixing is a technical problem that needs to be solved by those skilled in the art. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide an audio upmixing method, apparatus, device, medium, and product capable of improving the sound quality of multi-channel audio signals. The specific solution is as follows: Firstly, this application provides an audio upmixing method, including: A short-time Fourier transform is performed on the input two-channel stereo audio signal to obtain the time-frequency domain signal; For each time-frequency processing unit of the time-frequency domain signal, spatial state parameters including lateral offset, channel correlation, and extended fit are extracted; A unified space radiation physics model is created, wherein the time-frequency domain signal is decomposed into a directional direct state component and a diffuse environment state component in the unified space radiation physics model. The three-dimensional target sound field coordinates are determined based on the spatial state parameters and the target playback layout. Based on the three-dimensional target sound field coordinates and the target playback layout, the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout are generated respectively, and combined to construct the target multi-channel sound field. Based on the unified spatial radiation physics model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed, and the multi-channel allocation weight matrix of each time-frequency processing unit is determined based on the multi-channel mapping optimization objective function. The time-frequency domain signal is rendered in multiple channels based on the multi-channel weighting matrix, and then synthesized into a multi-channel time-domain audio signal through short-time inverse Fourier transform.

[0005] Optionally, the lateral offset is obtained based on a difference measure between the left and right channels and is used to characterize the degree of lateral shift of the sound image; The channel correlation is obtained based on the similarity measure between the left and right channels and is used to characterize the degree to which the sound image tends to concentrate or diffuse. The extension adaptation degree is determined based on the frequency of the current time-frequency processing unit and the channel correlation, and is used to characterize the degree of adaptation of the time-frequency processing unit to the extension of the high-frequency channel.

[0006] Optionally, the covariance matrix of the current time-frequency processing unit is decomposed into eigenvalues ​​to determine the directional direct state component in the direction of the largest eigenvalue and the diffuse environment state component in the direction of the second largest eigenvalue.

[0007] Optionally, in the unified spatial radiation physics model, the radiation weights of the directional direct state component in the left and right channels, and the radiation weights of the diffuse environment state component in the left and right channels are determined based on the spatial state parameters.

[0008] Optionally, determining the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout includes: Based on the spatial state parameters and the three-dimensional physical boundary constraints of the target replay layout, the horizontal projection and vertical projection are generated. By embedding the lateral and vertical projections into a unit sphere using direction cosine mapping, the three-dimensional target sound field coordinates are obtained.

[0009] Optionally, the multi-channel mapping optimization objective function includes a reconstruction error term, a time continuity penalty term, and a frequency continuity penalty term; The reconstruction error term is used to measure the deviation between the target multi-channel sound field and the mapped multi-channel output; the temporal continuity penalty term is used to penalize the change of the multi-channel weighting matrix between adjacent time frames; and the frequency continuity penalty term is used to penalize the change of the multi-channel weighting matrix between adjacent frequency bands.

[0010] Secondly, this application provides an audio upmixing device, comprising: The signal transformation module is used to perform short-time Fourier transform on the input two-channel stereo audio signal to obtain a time-frequency domain signal; The parameter extraction module is used to extract spatial state parameters, including lateral offset, channel correlation and extended fit, for each time-frequency processing unit of the time-frequency domain signal. The model creation module is used to create a unified space radiation physics model, wherein the time-frequency domain signal is decomposed into a directional direct state component and a diffuse environment state component in the unified space radiation physics model. The coordinate determination module is used to determine the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout. The sound field construction module is used to generate the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout based on the three-dimensional target sound field coordinates and the target playback layout, and combine them to construct the target multi-channel sound field. The weight determination module is used to construct a multi-channel mapping optimization objective function based on a unified spatial radiation physics model and the target multi-channel sound field, and to determine the multi-channel allocation weight matrix for each time-frequency processing unit based on the multi-channel mapping optimization objective function. The time-domain signal determination module is used to perform multi-channel rendering of the time-frequency domain signal based on the multi-channel weighting matrix, and synthesize a multi-channel time-domain audio signal through short-time inverse Fourier transform.

[0011] Thirdly, this application provides an electronic device, including a memory and a processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to implement the aforementioned audio upmixing method.

[0012] Fourthly, this application provides a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the aforementioned audio upmixing method.

[0013] Fifthly, this application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the aforementioned audio upmixing method.

[0014] As can be seen from the above scheme, the present invention provides an audio upmixing method, including: performing a short-time Fourier transform on the input two-channel stereo audio signal to obtain a time-frequency domain signal; extracting spatial state parameters, including lateral offset, channel correlation, and extended fit, for each time-frequency processing unit of the time-frequency domain signal; creating a unified spatial radiation physics model, wherein the unified spatial radiation physics model decomposes the time-frequency domain signal into directional direct state components and diffuse environment state components; and determining the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout. Based on the three-dimensional target sound field coordinates and the target playback layout, the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout are generated respectively, and combined to construct the target multi-channel sound field; a multi-channel mapping optimization objective function is constructed based on the unified spatial radiation physical model and the target multi-channel sound field, and the multi-channel allocation weight matrix of each time-frequency processing unit is determined based on the multi-channel mapping optimization objective function; the time-frequency domain signal is multi-channel rendered based on the multi-channel allocation weight matrix, and multi-channel time-domain audio signal is synthesized by short-time Fourier inverse transform.

[0015] As can be seen, the beneficial effects of this application are as follows: This application extracts the spatial state parameters of the time-frequency domain signal and creates a unified spatial radiation physical model, decomposes the time-frequency domain signal into directional direct state components and diffuse environmental state components, determines the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout, and then constructs the target multi-channel sound field. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed, and the multi-channel allocation weight matrix is ​​determined. In this way, a unified spatial radiation physical model is created, integrating spatial state parameter extraction, signal decomposition, sound field construction, and multi-channel allocation matrix into the same optimization solution framework, eliminating the sound field imbalance caused by process fragmentation. Under the unified model, the signal is decomposed into directional direct state components and diffuse environmental state components, and gain distributions are generated separately, taking into account both the clarity and stability of the main sound image and the seamless immersion of the surround sound field. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed. The multi-channel allocation weight matrix of each time-frequency processing unit is determined through global optimization, and time continuity and frequency continuity constraints are introduced to suppress sound image drift and improve the sound quality of multi-channel audio signals.

[0016] Correspondingly, the audio upmixing device, apparatus, readable storage medium, and product provided in this application also have the aforementioned technical effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of an audio upmixing method provided in this application embodiment; Figure 2 This is a schematic diagram of an audio upmixing device provided in an embodiment of this application; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] In modern audio playback scenarios (such as Dolby Atmos home theaters, object-based digital audio workstations (DAWs), TV soundbars, and in-vehicle 3D audio systems), a core requirement is the smooth and natural conversion of massive amounts of traditional two-channel stereo audio into immersive multi-channel audio. Existing stereo upmixing solutions typically remain at the level of simple signal distribution, making it difficult to simultaneously achieve a clear and stable main sound image and a seamless surround sound field. In practical applications, engineering problems that degrade the listening experience, such as sound image drift and ambient sound separation noise, are extremely prone to occur.

[0021] Existing stereo upmixing technologies face several intractable technical bottlenecks in both industry (such as various surround sound plugins) and academia (such as algorithms based on blind source separation): First, traditional upmixing processes are independent of each other, leading to an unbalanced sound field. Many existing algorithms treat feature extraction, target direction mapping, and demapping matrices as three isolated steps. This fragmentation results in highly inconsistent physical modeling in the time and frequency domains. For example, when processing a complex symphony, traditional algorithms might leave the fundamental frequency of the violin in the left front speaker, but incorrectly map its high-frequency overtones to the left surround or top speaker, causing the timbre of a single instrument to be torn apart in space. Second, the extraction of ambient sound is prone to comb filtering effects. When extracting and assigning diffuse sounds (such as reverberation, cheers, and other ambient sounds) from stereo to the surround and height channels, without a spatially constrained decorrelation strategy, these signals are highly susceptible to destructive interference with the front main channel once they are mixed in the room. The auditory manifestations are a hollow sound, a collapsed midrange, and a thinned soundstage—the so-called phase-flip disaster. Furthermore, the sound pressure levels from the multi-channel output are excessive. When a two-channel signal is simultaneously mapped to 11 or more speakers (e.g., 7.1.4), the superposition of sound pressure levels leads to an increase in overall output energy. Without strict energy conservation constraints, this severely disrupts the dynamic balance set by the original mixing engineer.

[0022] Therefore, this application constructs a unified framework for solving constrained physical models, transforming sound field mapping into an optimization problem with a penalty term. This framework not only mathematically guarantees high-precision sound image localization but also perfectly adapts to modern real-time audio industry standards through a streaming framework, enabling the generation of high-quality multi-channel audio signals.

[0023] See Figure 1 As shown in the figure, this application discloses an audio upmixing method, including: Step S11: Perform a short-time Fourier transform on the input two-channel stereo audio signal to obtain the time-frequency domain signal.

[0024] This application embodiment can receive two-channel stereo audio signals, perform short-time Fourier transform, and obtain time-frequency domain signals. The signal is represented as a two-dimensional matrix in the time-frequency domain, where the time axis represents continuous analysis frames and the frequency axis represents discrete frequency bins.

[0025] Step S12: For each time-frequency processing unit of the time-frequency domain signal, extract spatial state parameters including lateral offset, channel correlation and extended fit.

[0026] For time-frequency domain signals, the signal unit corresponding to the intersection of a specific time frame and a specific frequency segment is called a time-frequency processing unit, denoted as i=(m,k), where m is the time frame index and k is the frequency segment index. For each time-frequency processing unit, the covariance matrix is ​​calculated and the spatial state parameters are extracted.

[0027] In this embodiment, the lateral offset is obtained based on the difference measure between the left and right channels and is used to characterize the degree of lateral offset of the sound image; the channel correlation is obtained based on the similarity measure between the left and right channels and is used to characterize the degree to which the sound image tends to concentrate or spread; the extension adaptation is determined based on the frequency of the current time-frequency processing unit and the channel correlation and is used to characterize the degree of adaptation of the time-frequency processing unit to the height channel.

[0028] The spatial state parameter can be expressed as: Including lateral offset Channel relevance Expand compatibility The extraction process for each parameter will be explained below.

[0029] Based on inter-channel difference measurement. By comparing the energy distribution, amplitude difference, or perceived loudness difference between the left and right channels within the current analysis unit, a metric is constructed to characterize the lateral shift of the sound image, determining whether the sound image is shifted to the left, right, or center. Values ​​can range from -1 to 1, where -1 indicates the sound source is completely shifted to the left, 0 indicates the sound source is centered, and 1 indicates the sound source is completely shifted to the right.

[0030] Based on inter-channel similarity metrics. This measures the front-to-back and height expansion. Higher correlation between two channels indicates a weaker expansion tendency (sound image more concentrated in the front field); lower correlation indicates a stronger expansion tendency (sound image more easily diffuses to the sides, back, and height). It is determined by evaluating frequency band characteristics (e.g., high frequencies tend to contribute to a sense of height) and phase consistency. The higher the similarity... The larger the value, the higher the channel correlation. This means that different frequency bands are assigned different weights when calculating channel correlation. The frequency band weights are set according to the sensitivity of human spatial perception: higher frequencies have a larger weight, and lower frequencies have a smaller weight. Phase consistency reflects the degree of synchronization of the waveform changes in the left and right channels. The more consistent the phases of the left and right channels, the more synchronized the waveform fluctuations, and the higher the similarity; the more chaotic the phases of the left and right channels, the less synchronized the waveform changes, and the lower the similarity. For each frequency point covered by each time-frequency processing unit, the similarity of the left and right channels at that frequency point can be obtained first by evaluating phase consistency; then, the corresponding weight can be assigned to that frequency point by evaluating the frequency band characteristics, with higher frequencies having a larger weight than lower frequencies; the similarity of each frequency point is multiplied by its corresponding weight and summed, then normalized to obtain the channel correlation. Therefore, the channel correlation reflects both the overall level of synchronization of the left and right channel waveforms and highlights the contribution of the high-frequency band to human spatial perception, making the channel correlation more accurately characterize the concentration or diffusion trend of the sound image.

[0031] This is a joint mapping between frequency and attributes. This metric is used to evaluate whether the current audio unit is suitable for three-dimensional expansion into surround and height channels. It is based on preset frequency response preferences (e.g., high-frequency components are generally more likely to stimulate height perception in the human ear). For example, based on... The mapping is obtained by nonlinearly mapping the frequency band; the larger the value, the better. The mapping relationship determined based on the preset frequency response preference can be implemented as follows: = f(k)·g( ), where f(k) is the preset mapping weight for frequency band index k (the value is larger for high-frequency bands), g( (Regarding channel relevance) A monotonically increasing mapping function; The higher the value, the more suitable the current time-frequency processing unit is for extending to surround and height channels. Higher frequencies have a greater weight than lower frequencies.

[0032] Step S13: Create a unified space radiation physics model, wherein the time-frequency domain signal is decomposed into directional direct state components and diffuse environment state components in the unified space radiation physics model.

[0033] This embodiment can perform eigenvalue decomposition on the covariance matrix of the current time-frequency processing unit to determine the directional direct state component in the direction of the largest eigenvalue and the diffuse environment state component in the direction of the second largest eigenvalue.

[0034] Furthermore, in the unified spatial radiation physics model, the radiation weights of the directional direct state component on the left and right channels, and the radiation weights of the diffuse environment state component on the left and right channels are determined based on the spatial state parameters.

[0035] This embodiment can introduce a two-state space radiation variable into the unified space radiation physics model: . For directional direct-path state components, the main body responsible for acoustic-image localization. To diffuse the environmental state components, which are responsible for the sense of sound field enclosure. The input is the time-frequency domain dual-channel observation vector, which is the instantaneous frequency domain signal. In the unified space radiation physics model, it is expressed as: ,in, The adaptive dual-channel observation model matrix is ​​driven by spatial state parameters. This is the model residual error term. As model error, characterization and The result is an upmixer and a multi-channel downmixer obtained after reverse mixing. The error between the input value x and the actual input value. Subsequently, since optimization is performed directly in the multi-channel domain, this is transformed into an error term for multi-channel optimization. L represents the left channel signal, and R represents the right channel signal.

[0036] Among them, the adaptive dual-channel observation model matrix It is a set of spatial state parameters The driving 2×2 complex matrix, whose elements represent the two-state-space radiation variables. The coupling relationship between the left and right channel observation signals after spatial radiation; its specific form can be parameterized as follows: =[[α1( , ), β1( , )],[α2( , ), β2( , )]], where column α corresponds to the direct component In the radiation weights of the left and right channels, column β corresponds to the environmental component. The radiation weights in the left and right channels; the specific values ​​of α and β are determined by... It is obtained through a preset continuous mapping function, such as a mapping function based on lateral bias. Equal power translation law and correlation-based , The decorrelation ratio is combined. That is, the matrix column corresponding to the direct state component is mapped by the lateral offset through the equal power translation law to determine its radiation weight in the left and right channels; the matrix column corresponding to the environmental state component is mapped by the channel correlation and extended fit through the decorrelation ratio to determine its radiation weight in the left and right channels.

[0037] Because the system is underdetermined, minimum mean square error estimation from statistical signal processing is introduced to find the optimal solution. It is assumed that the direct state and the environment state are statistically orthogonal (uncorrelated), i.e. Based on the known covariance matrix of the i-th time-frequency processing unit... The covariance matrix can be subjected to dimensionality reduction algorithms such as eigenvalue decomposition or principal component analysis to obtain the corresponding eigenvectors. , According to the principles of spatial acoustics, the direction of the maximum eigenvalue (principal component) represents the directional direct state with the most concentrated energy and the highest correlation. The second largest eigenvalue (residual component) represents the non-directional diffusion environment state. .

[0038] Step S14: Determine the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout.

[0039] Determining the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout includes: generating lateral and vertical projections based on the three-dimensional physical boundary constraints of the spatial state parameters and the target playback layout; embedding the lateral and vertical projections into a unit sphere through direction cosine mapping to obtain the three-dimensional target sound field coordinates.

[0040] This application embodiment is based on spatial state parameters. Generate two intermediate projections: ; ; in, It represents the lateral projection amount, mainly reflecting the horizontal distribution. It represents the vertical projection, which mainly reflects the projection in the height direction; and This is a preset (based on empirical values) mapping scaling factor; It is a hyperbolic tangent function used to provide smooth boundary compression and prevent out-of-bounds distortion; The constraint function, namely Clip clamping, prevents excessively large values ​​and is used to combine centralized constraints and target layout constraints to suppress unreasonable height expansion. Here, L represents the target playback layout (e.g., as in 7.1.4, containing L, R, C, LFE, Ls, Rs, Lrs, Rrs, Ltf, Rtf, Ltr, Rtr, 12 channels). Then, the two intermediate projection quantities are embedded onto the unit sphere through direction cosine mapping to obtain the three-dimensional target sound field coordinates. : , , .

[0041] x, y, and z represent the three dimensions in a predefined three-dimensional coordinate system. The three-dimensional mapping in this application is not three independent sets of discrete control quantities, but rather a unified spatial radiation physics model as described above. A continuous three-dimensional coordinate embedding process driven by spatial state parameters.

[0042] Step S15: Based on the three-dimensional target sound field coordinates and the target playback layout, generate the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout, and combine them to construct the target multi-channel sound field.

[0043] Among them, based on the three-dimensional target sound field coordinates Construct a target multi-channel sound field by matching the target playback layout L. The target multi-channel sound field is as follows: ; in, This represents the gain distribution of the direct sound in the target playback layout, i.e., based on the directional coordinates. exist The calculation of directional energy allocation on each target channel can specifically employ projection functions such as Vector Magnitude Shift (VBAP) to generate normalized shift gain. This ensures stability in the front end; g represents gain. This represents the distribution of ambient sound in the target playback layout (responsible for spreading the sense of immersion to the sides, rear, and top), that is, the calculation of distributing ambient components to the surround, rear, and height channels according to preset diffusion weights, which can be expressed as: = w_a⊙(1- w_a is the preset environment channel activation weight vector for each channel in the target playback layout L (surround / height channels have larger weights, and front / center channels have smaller weights), and ⊙ represents element-wise multiplication. For example, the surround / height channel weights can be greater than the front / center channel weights.

[0044] Step S16: Construct a multi-channel mapping optimization objective function based on the unified spatial radiation physics model and the target multi-channel sound field, and determine the multi-channel allocation weight matrix for each time-frequency processing unit based on the multi-channel mapping optimization objective function.

[0045] The multichannel mapping optimization objective function includes a reconstruction error term, a temporal continuity penalty term, and a frequency continuity penalty term. The reconstruction error term measures the deviation between the target multichannel sound field and the mapped multichannel output. The temporal continuity penalty term penalizes the change in the multichannel weighting matrix between adjacent time frames. The frequency continuity penalty term penalizes the change in the multichannel weighting matrix between adjacent frequency bands.

[0046] In order to determine the theoretical target multi-channel sound field as a real-time output distribution, this application constructs a multi-channel mapping relationship with dual time-frequency constraints. The optimal objective functional is the multichannel mapping optimization objective function. Assign weight matrices to the multi-channel audio.

[0047] ; Among them, the first item To reconstruct the error term, the frequency band weight matrix is ​​used. Ensure the fitting accuracy of important frequency bands. Frequency band weight matrix. It is a diagonal matrix, and its diagonal elements are... (k) Set according to the preset frequency response preference curve. You can refer to A-weighting or equal loudness curves to give greater weight to the mid-to-high frequency bands that are sensitive to human ears (such as 1kHz–4kHz) and less weight to the extremely low and extremely high frequencies.

[0048] Second item This is a penalty for time continuity. Assign weight matrices to the multi-channel mapping matrix between the current frame and the previous frame. The difference between the multi-channel weighting matrices corresponding to adjacent time frames in the same frequency band k. The larger the value, the smoother the sound image movement, effectively suppressing transient jumps. This is the time smoothing coefficient (preset as a positive real number). is the frequency smoothing coefficient (preset positive real number), and the two are used to balance reconstruction accuracy and smoothness; ‖·‖2² represents taking the L2 norm of the matrix or vector (i.e., the sum of squares of the elements of the matrix). Indicates the independent variable The value that minimizes the objective function within the parentheses is the solution for the optimal multi-channel weighting matrix. The first-order difference in the time direction is defined as: =M{(m,k)}-M{(m-1,k)}.

[0049] Third item This is a frequency continuity penalty term. The mapping difference between adjacent frequency bands is the first-order difference between the mapping matrices of the current frequency band k and the previous adjacent frequency band k-1 within the same time frame m: =M{(m,k)}-M{(m,k-1)}, which forces the fundamental frequency and high-frequency overtones of the instrument to be mapped to the same physical space direction, thus solving the frequency tearing problem.

[0050] Step S17: Perform multi-channel rendering on the time-frequency domain signal based on the multi-channel weighting matrix, and synthesize a multi-channel time-domain audio signal through short-time Fourier inverse transform.

[0051] The embodiments of this application can perform rendering under the weighted multi-channel weighted matrix, and perform ambient sound phase decorrelation extension and other post-processing behaviors. The processing object is the time-frequency domain signal. After processing, it is synthesized into the time domain through short-time Fourier inverse transform, and finally outputs a high-fidelity, immersive multi-channel audio stream.

[0052] In other words, this application provides a spatial reconstruction framework that abandons the traditional discrete matrix lookup table method and the crude frequency band splitting method. It introduces a unified spatial radiation physical model, which encapsulates the complex spatial feature evaluation, the physical entity constraints of the target playback layout, and the multi-channel mapping relationship under the same optimization solution framework, and realizes an engineering adaptive upmixing scheme that takes into account the absolute stability of the subject, the continuity of the sense of spatial enclosure, and the psychological acoustic safety.

[0053] This application constructs a unified spatial physical model, integrating spatial constraint modeling, three-dimensional target sound field construction, and multi-channel mapping matrix solving into a continuous optimization spatial reconstruction process. This can include: First, performing spatial feature evaluation based on time-frequency domain signal characteristics: The input stereo signal is converted to the time-frequency domain using Short-Time Fourier Transform (STFT). For each time-frequency processing unit, spatial state parameters including lateral offset, channel correlation, and extended fit are extracted. Specifically, a two-channel audio stream can be received, buffered, overlapped, and framed before STFT transformation. For each time-frequency processing unit (the two-channel stereo signal, after STFT transformation, is represented as a two-dimensional matrix in the time-frequency domain, where the time axis represents continuous analysis frames and the frequency axis represents discrete frequency points; the signal unit corresponding to the intersection of a specific time frame and a specific frequency band is called a time-frequency processing unit), its covariance matrix is ​​calculated, and spatial state parameters are extracted. Secondly, a unified spatial radiation physical model is constructed, introducing a dual-state spatial radiation variable that differs from traditional mono point sources (deconstructing it into a highly directional direct state and a highly diffuse environmental state). Combined with the three-dimensional physical boundary constraints of the target speaker layout (such as Dolby Atmos 7.1.4), an ideal target sound field model is constructed. Next, a multi-channel mapping optimization objective function containing spatiotemporal parameters is established, and the optimal multi-channel allocation weight matrix for each time-frequency processing unit in the current frame is determined directly through numerical optimization. Finally, under this allocation weight, rendering, ambient sound phase decorrelation extension, and other post-processing behaviors are performed, with the processing object being the time-frequency domain signal. After processing, it is synthesized into the time domain through short-time inverse Fourier transform (ISTFT), ultimately outputting a high-fidelity, immersive multi-channel audio stream.

[0054] In this way, unified physical modeling and mapping optimization integrate the dual-state space radiation variables, multi-speaker physical layout constraints, and multi-channel allocation matrices into the optimization equation with regularization penalty terms, thus establishing a continuous physical mapping from the two-dimensional plane to three-dimensional space. This ensures sound robustness by introducing time and frequency continuity penalty terms into the optimization objective function, improving the stability of vocal lead singing and sustained string instruments. Physical modeling decomposition: instead of using a rigid two-point sound source model, it introduces... It aligns with the modeling of human binaural hearing psychology, enabling precise anchoring of dialogue positions; it possesses engineered streaming deployment capabilities, allowing it to run seamlessly in real-time with extremely low latency through corresponding models, either in the form of audio plugins (VST3 (Virtual Studio Technology 3) / AU (Audio Units) / AAX (Avid Audio eXtension)) or in the form of automotive DSP (Digital Signal Processor) firmware, with extremely low computational power consumption.

[0055] As can be seen, the embodiments of this application extract the spatial state parameters of the time-frequency domain signal and create a unified spatial radiation physical model. The time-frequency domain signal is decomposed into directional direct state components and diffuse environmental state components. Based on the spatial state parameters and the target playback layout, the three-dimensional target sound field coordinates are determined, and then the target multi-channel sound field is constructed. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed, and the multi-channel allocation weight matrix is ​​determined. In this way, a unified spatial radiation physical model is created, integrating spatial state parameter extraction, signal decomposition, sound field construction, and multi-channel allocation matrix into the same optimization solution framework, eliminating the sound field imbalance caused by process fragmentation. Under the unified model, the signal is decomposed into directional direct state components and diffuse environmental state components, and gain distributions are generated separately, taking into account both the clarity and stability of the main sound image and the seamless immersion of the surround sound field. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed. The multi-channel allocation weight matrix of each time-frequency processing unit is determined through global optimization, and time continuity and frequency continuity constraints are introduced to suppress sound image drift and improve the sound quality of multi-channel audio signals.

[0056] Further, see Figure 2 As shown in the figure, this application embodiment provides an audio upmixing device, including: The signal transformation module 11 is used to perform a short-time Fourier transform on the input two-channel stereo audio signal to obtain a time-frequency domain signal. The parameter extraction module 12 is used to extract spatial state parameters, including lateral offset, channel correlation and extended fit, for each time-frequency processing unit of the time-frequency domain signal. Model creation module 13 is used to create a unified space radiation physics model, wherein the time-frequency domain signal is decomposed into directional direct state components and diffuse environment state components in the unified space radiation physics model. The coordinate determination module 14 is used to determine the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout. The sound field construction module 15 is used to generate the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout based on the three-dimensional target sound field coordinates and the target playback layout, and combine them to construct the target multi-channel sound field. The weight determination module 16 is used to construct a multi-channel mapping optimization objective function based on the unified spatial radiation physics model and the target multi-channel sound field, and to determine the multi-channel allocation weight matrix of each time-frequency processing unit based on the multi-channel mapping optimization objective function. The time-domain signal determination module 17 is used to perform multi-channel rendering of the time-frequency domain signal based on the multi-channel weighting matrix, and synthesize a multi-channel time-domain audio signal through short-time inverse Fourier transform.

[0057] As can be seen, the embodiments of this application extract the spatial state parameters of the time-frequency domain signal and create a unified spatial radiation physical model. The time-frequency domain signal is decomposed into directional direct state components and diffuse environmental state components. Based on the spatial state parameters and the target playback layout, the three-dimensional target sound field coordinates are determined, and then the target multi-channel sound field is constructed. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed, and the multi-channel allocation weight matrix is ​​determined. In this way, a unified spatial radiation physical model is created, integrating spatial state parameter extraction, signal decomposition, sound field construction, and multi-channel allocation matrix into the same optimization solution framework, eliminating the sound field imbalance caused by process fragmentation. Under the unified model, the signal is decomposed into directional direct state components and diffuse environmental state components, and gain distributions are generated separately, taking into account both the clarity and stability of the main sound image and the seamless immersion of the surround sound field. Based on the unified spatial radiation physical model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed. The multi-channel allocation weight matrix of each time-frequency processing unit is determined through global optimization, and time continuity and frequency continuity constraints are introduced to suppress sound image drift and improve the sound quality of multi-channel audio signals.

[0058] See Figure 3 As shown in the figure, this application discloses an electronic device 20, including a processor 21 and a memory 22; wherein, the memory 22 is used to store a computer program; the processor 21 is used to execute the computer program, the audio upmixing method disclosed in the foregoing embodiment.

[0059] For details regarding the audio upmixing method described above, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0060] Furthermore, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, and the storage method can be temporary storage or permanent storage.

[0061] In addition, the electronic device 20 also includes a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26; wherein, the power supply 23 is used to provide operating voltage for the various hardware devices on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0062] Furthermore, embodiments of this application also disclose a computer-readable storage medium for storing a computer program, wherein the computer program, when executed by a processor, implements the audio upmixing method disclosed in the foregoing embodiments.

[0063] For details regarding the audio upmixing method described above, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0064] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implement the audio upmixing method disclosed in the foregoing embodiments.

[0065] For details regarding the audio upmixing method described above, please refer to the relevant content disclosed in the foregoing embodiments, which will not be repeated here.

[0066] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0067] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0068] The above provides a detailed description of an audio upmixing method, apparatus, device, medium, and product provided by this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An audio upmix method, characterized by, include: A short-time Fourier transform is performed on the input two-channel stereo audio signal to obtain the time-frequency domain signal; For each time-frequency processing unit of the time-frequency domain signal, spatial state parameters including lateral offset, channel correlation, and extended fit are extracted; A unified space radiation physics model is created, wherein the time-frequency domain signal is decomposed into a directional direct state component and a diffuse environment state component in the unified space radiation physics model. The three-dimensional target sound field coordinates are determined based on the spatial state parameters and the target playback layout. Based on the three-dimensional target sound field coordinates and the target playback layout, the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout are generated respectively, and combined to construct the target multi-channel sound field. Based on the unified spatial radiation physics model and the target multi-channel sound field, a multi-channel mapping optimization objective function is constructed, and the multi-channel allocation weight matrix of each time-frequency processing unit is determined based on the multi-channel mapping optimization objective function. The time-frequency domain signal is rendered in multiple channels based on the multi-channel weighting matrix, and then synthesized into a multi-channel time-domain audio signal through short-time inverse Fourier transform.

2. The audio upmixing method according to claim 1, characterized in that, The lateral offset is obtained based on the difference between the left and right channels and is used to characterize the degree of lateral shift of the sound image. The channel correlation is obtained based on the similarity measure between the left and right channels and is used to characterize the degree to which the sound image tends to concentrate or diffuse. The extension adaptation degree is determined based on the frequency of the current time-frequency processing unit and the channel correlation, and is used to characterize the degree of adaptation of the time-frequency processing unit to the extension of the high-frequency channel.

3. The audio upmixing method according to claim 1, characterized in that, Eigenvalue decomposition is performed on the covariance matrix of the current time-frequency processing unit to determine the directional direct state component in the direction of the largest eigenvalue and the diffuse environment state component in the direction of the second largest eigenvalue.

4. The audio upmixing method according to claim 1, characterized in that, In the unified spatial radiation physics model, the radiation weights of the directional direct state component and the diffuse environment state component in the left and right channels are determined based on the spatial state parameters.

5. The audio upmixing method according to claim 1, characterized in that, Determining the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout includes: Based on the spatial state parameters and the three-dimensional physical boundary constraints of the target replay layout, the horizontal projection and vertical projection are generated. By embedding the lateral and vertical projections into a unit sphere using direction cosine mapping, the three-dimensional target sound field coordinates are obtained.

6. The audio upmixing method according to claim 1, characterized in that, The multichannel mapping optimization objective function includes a reconstruction error term, a time continuity penalty term, and a frequency continuity penalty term; The reconstruction error term is used to measure the deviation between the target multi-channel sound field and the mapped multi-channel output; the temporal continuity penalty term is used to penalize the change of the multi-channel weighting matrix between adjacent time frames; and the frequency continuity penalty term is used to penalize the change of the multi-channel weighting matrix between adjacent frequency bands.

7. An audio upmixing device, characterized in that, include: The signal transformation module is used to perform short-time Fourier transform on the input two-channel stereo audio signal to obtain a time-frequency domain signal; The parameter extraction module is used to extract spatial state parameters, including lateral offset, channel correlation and extended fit, for each time-frequency processing unit of the time-frequency domain signal. The model creation module is used to create a unified space radiation physics model, wherein the time-frequency domain signal is decomposed into a directional direct state component and a diffuse environment state component in the unified space radiation physics model. The coordinate determination module is used to determine the three-dimensional target sound field coordinates based on the spatial state parameters and the target playback layout. The sound field construction module is used to generate the gain distribution of direct sound in the target playback layout and the distribution of ambient sound in the target playback layout based on the three-dimensional target sound field coordinates and the target playback layout, and combine them to construct the target multi-channel sound field. The weight determination module is used to construct a multi-channel mapping optimization objective function based on a unified spatial radiation physics model and the target multi-channel sound field, and to determine the multi-channel allocation weight matrix for each time-frequency processing unit based on the multi-channel mapping optimization objective function. The time-domain signal determination module is used to perform multi-channel rendering of the time-frequency domain signal based on the multi-channel weighting matrix, and synthesize a multi-channel time-domain audio signal through short-time inverse Fourier transform.

8. An electronic device, characterized in that, Includes memory and processor, wherein: The memory is used to store computer programs; The processor is configured to execute the computer program to implement the audio upmixing method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, Used to store a computer program, wherein the computer program, when executed by a processor, implements the audio upmixing method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instruction is executed by the processor, it implements the audio upmixing method as described in any one of claims 1 to 6.

Images