Audio signal rendering method, device and electronic device

Abstract

The present disclosure relates to a method, device and electronic device for rendering an audio signal, and relates to the field of audio signal processing. The rendering method comprises: estimating the reverberation time of an audio signal at each of a plurality of time points; and rendering the audio signal according to the reverberation time of the audio signal.

Description

Technical Field

[0001] The present disclosure relates to the technical field of audio signal processing, and in particular to an audio signal rendering method, an audio signal rendering device, a chip, a computer program, an electronic device, a computer program product, and a non-transitory computer-readable storage medium. Background Art

[0002] Reverberation refers to the acoustic phenomenon that sound continues to exist after the sound source stops making sound. Reverberation is caused by the slow propagation speed of sound waves in the air, and the fact that sound waves are blocked and reflected by walls or surrounding obstacles during propagation.

[0003] In order to objectively evaluate reverberation, the ISO 3382-1 standard defines a series of objective evaluation indicators for the unit impulse response of a house. As one of the objective evaluation indicators, the reverberation decay time, also known as the reverberation time, is an important indicator for measuring the reverberation of a room. The reverberation time calculates the time required for the house reverberation to drop by 60dB by selecting different reverberation decay ranges. Summary of the invention

[0004] According to some embodiments of the present disclosure, a method for estimating reverberation time is provided, comprising: constructing a model of an objective function according to differences between an attenuation curve of an audio signal and a parameter-containing function of a fitting curve of the attenuation curve at multiple historical time points, and weights corresponding to the multiple historical time points, wherein the weight corresponding to a later time point is smaller than the weight corresponding to a previous time point; solving the objective function with parameters of the parameter-containing function of the fitting curve as variables and minimizing the model of the objective function as the goal, and determining the fitting curve of the attenuation curve; and estimating the reverberation time of the audio signal according to the fitting curve.

[0005] According to other embodiments of the present disclosure, a method for rendering an audio signal is provided, comprising: determining the reverberation time of the audio signal using the estimation method in any one of the above embodiments; and rendering the audio signal according to the reverberation time of the audio signal.

[0006] According to some further embodiments of the present disclosure, a method for rendering an audio signal is provided, comprising: estimating a reverberation time of the audio signal at each of a plurality of time points; and rendering the audio signal according to the reverberation time of the audio signal.

[0007] According to some further embodiments of the present disclosure, a reverberation time estimation device is provided, comprising: a construction unit, configured to construct a model of an objective function according to differences between an attenuation curve of an audio signal and a parameter-containing function of a fitting curve of a function of the attenuation curve at multiple historical time points, and weights corresponding to the multiple historical time points, wherein the weights vary with time; a determination unit, configured to solve the objective function with the parameters of the parameter-containing function of the fitting curve as variables and with the goal of minimizing the model of the objective function, and to determine the fitting curve of the attenuation curve; and an estimation unit, configured to estimate the reverberation time of the audio signal according to the fitting curve.

[0008] According to some further embodiments of the present disclosure, there is provided a rendering device for an audio signal, comprising: the reverberation time estimation device of any one of the embodiments; and a rendering unit, configured to perform rendering processing on the audio signal according to the reverberation time of the audio signal.

[0009] According to some further embodiments of the present disclosure, a rendering device for an audio signal is provided, comprising: an estimating device for estimating the reverberation time of the audio signal at each of a plurality of time points; and a rendering unit for rendering the audio signal according to the reverberation time of the audio signal.

[0010] According to some further embodiments of the present disclosure, a chip is provided, comprising: at least one processor and an interface, wherein the interface is used to provide computer-executable instructions to the at least one processor, and the at least one processor is used to execute the computer-executable instructions to implement the reverberation time estimation method or the audio signal rendering method of any of the above embodiments.

[0011] According to some further embodiments of the present disclosure, a computer program is provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method for estimating the reverberation time or the method for rendering an audio signal of any one of the above embodiments.

[0012] According to some further embodiments of the present disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, wherein the processor is configured to execute the reverberation time estimation method or the audio signal rendering method of any one of the above embodiments based on instructions stored in the memory device.

[0013] According to some further embodiments of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored, and when the program is executed by a processor, the method for estimating the reverberation time or the method for rendering an audio signal of any of the above embodiments is implemented.

[0014] According to some further embodiments of the present disclosure, a computer program product is provided, comprising instructions, which, when executed by a processor, implement the method for estimating the reverberation time or the method for rendering an audio signal according to any one of the embodiments described in the present disclosure.

[0015] According to some further embodiments of the present disclosure, a computer program is provided, comprising instructions, which, when executed by a processor, implement the method for estimating the reverberation time or the method for rendering an audio signal according to any one of the embodiments described in the present disclosure.

[0016] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present disclosure and constitute a part of the present application. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure. In the drawings:

[0018] Figure 1 A schematic diagram illustrating some embodiments of an audio signal processing process;

[0019] Figure 2 Schematic diagrams of some embodiments showing different stages of sound wave propagation;

[0020] Figure 3a to 3e A schematic diagram showing some embodiments of RIR curves;

[0021] Figure 4a A flowchart showing some embodiments of a method for estimating reverberation time according to the present disclosure;

[0022] Figure 4b A flowchart showing some embodiments of a method for rendering an audio signal according to the present disclosure;

[0023] Figure 4c A block diagram showing some embodiments of a reverberation time estimation apparatus according to the present disclosure;

[0024] Figure 4d A block diagram showing some embodiments of an apparatus for rendering an audio signal according to the present disclosure;

[0025] Figure 4e A block diagram showing some embodiments of a rendering system according to the present disclosure;

[0026] Figure 5 A block diagram showing some embodiments of the electronic device of the present disclosure;

[0027] Figure 6 A block diagram showing some other embodiments of the electronic device of the present disclosure;

[0028] Figure 7 A block diagram showing some embodiments of the chip of the present disclosure. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is by no means intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0030] Unless otherwise specifically stated, the relative arrangement of the parts and steps described in these embodiments, numerical expressions and numerical values ​​do not limit the scope of the present disclosure. At the same time, it should be understood that, for ease of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship. The technology, methods and equipment known to ordinary technicians in the relevant field may not be discussed in detail, but in appropriate cases, the technology, methods and equipment should be regarded as a part of the authorization specification. In all examples shown and discussed here, any specific value should be interpreted as being merely exemplary, rather than as a limitation. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further discussed in subsequent drawings.

[0031] Figure 1 Schematic diagram illustrating some embodiments of an audio signal processing process.

[0032] like Figure 1 As shown, on the production side, based on the audio data and audio source data, the audio track interface and the general audio metadata (such as ADM extension, etc.) are used for authorization and metadata marking. For example, standardization processing can also be performed.

[0033] In some embodiments, the processing result of the production side is subjected to spatial audio encoding and decoding processing to obtain a compressed result.

[0034] On the consumer side, metadata recovery and rendering are performed based on the processing results (or compression results) on the production side using the audio track interface and general audio metadata (such as ADM extension, etc.); the processing results are subjected to audio rendering and then input into the audio device.

[0035] In some embodiments, the input to the audio processing may include scene information and metadata, target-based audio signals, FOA (First-Order Ambisonics), HOA (Higher-Order Ambisonics), stereo, surround sound, etc.; the input to the audio processing includes stereo audio output, etc.

[0036] Figure 2 Schematic diagram of some embodiments showing different stages of sound wave propagation.

[0037] like Figure 2 As shown in the figure, the propagation of sound waves in the environment and reaching people can be divided into three stages: direct path, early reflections, and late reverberation.

[0038] Take a simplified room impulse response as an example. In the first stage, when the sound source is excited, the signal is transmitted from the sound source to the listener in a straight line. This process will cause a delay of T0. This path is called the direct path. The direct path can provide the listener with information about the direction of the sound.

[0039] The direct path is followed by the early reflection stage, which originates from reflections from nearby objects and walls. This part of the reverberation gives the listener information about the geometry and materials of the space. This part has a higher density of response due to the multiple reflection paths.

[0040] After more reflections, the energy of the signal continues to decay, forming the tail of the reverberation, which is called the late reverberation. This part has Gaussian statistical characteristics, and its power spectrum also carries information such as the size of the environment and the absorption rate of the material.

[0041] Whether in audio signal processing, music production and mixing, or in immersive applications such as virtual reality and augmented reality, reverberation is an important part of the audio experience. There are many technical routes to show the reverberation effect.

[0042] In some embodiments, the most direct way is to record the impulse response of the house unit in the real scene, and then convolve it with the audio signal to reproduce the reverberation. The recording method can achieve a more realistic effect, but due to the fixed scene, there is no room for flexible adjustment in the later stage.

[0043] In some embodiments, reverberation can also be artificially generated by an algorithm. Methods for artificially generating reverberation include parameterized reverberation and reverberation based on acoustic modeling.

[0044] For example, the parametric reverberation generation method may be the FDN (Feedback Delay Networks) method. Parametric reverberation usually has good real-time performance and low computing power requirements, but requires manual input of reverberation related parameters, such as reverberation time, direct sound intensity ratio, etc. Such parameters usually cannot be directly obtained from the scene and need to be manually selected and adjusted to achieve an effect that matches the target scene.

[0045] For example, reverberation based on acoustic modeling is more accurate, and the impulse response of the house units in the scene can be calculated through scene information. Moreover, reverberation based on acoustic modeling is highly flexible and can reproduce reverberation at any location in any scene.

[0046] However, the disadvantage of acoustic modeling is the computational overhead, which usually requires more calculations to achieve good results. The reverberation of acoustic modeling has been greatly optimized during the development process, and with the improvement of hardware computing power, it has gradually met the requirements of real-time processing.

[0047] For example, in an environment where computing resources are relatively scarce, the RIR (Room Impulse Response) can be pre-calculated through acoustic modeling, and the parameters required for parametric reverberation can be obtained through the room unit impulse response, so as to calculate the reverberation in real-time applications.

[0048] In some embodiments, in order to obtain a more realistic and immersive auditory experience, acoustic modeling (room acoustics modeling, environmental acoustics modeling, etc.) may be performed on the environment.

[0049] Acoustic modeling can be applied in the field of architecture. For example, in the design of concert halls, cinemas and performance venues, acoustic modeling before construction can ensure that the building has good acoustic characteristics to achieve good auditory effects; in other scenarios such as classrooms, subway stations and other public places, acoustic modeling will also be used to perform certain auditory designs to ensure that the acoustic conditions of the environment can meet the design expectations.

[0050] With the development of virtual reality, games and immersive applications, in addition to the need for acoustic modeling in the construction of real scenes, digital applications also require environmental acoustic modeling. For example, in different scenes of the game, it is hoped that the user will be presented with sounds that match the current scene, which requires environmental acoustic modeling of the game scene.

[0051] In some embodiments, in order to adapt to different occasions, environmental acoustic modeling has evolved into several different frameworks. In principle, there are two main categories: wave-based modeling, which is based on the wave characteristics of sound waves and the analytical solution of the wave equation is obtained for modeling; geometric acoustic modeling (GA): based on the geometric properties of the environment, the sound is assumed to be a ray for estimation and modeling.

[0052] For example, wave acoustics modeling provides the most accurate results because it adheres to the physics of sound waves. The computational complexity of this approach is usually very high.

[0053] For example, although geometric acoustic modeling is not as accurate as wave acoustic modeling, it is much faster. In geometric acoustic modeling, the wave characteristics of sound are ignored and the behavior of sound propagation in the air is assumed to be the same as the propagation of rays. This assumption is applicable to high-frequency sounds, but because the propagation of low-frequency sounds is dominated by wave characteristics, it will introduce estimation errors for low-frequency sounds.

[0054] In some embodiments, the RIR can be obtained by calculation through the acoustic modeling method. In this way, the acoustic modeling is not restricted by the physical space, which increases the flexibility of the application. In addition, the acoustic modeling method also avoids some troubles caused by physical measurement, such as the influence of environmental noise, and the need for multiple measurements at different positions and directions.

[0055] In some embodiments, the method of geometric acoustic modeling is derived from the acoustic rendering equation:

[0056]

[0057] G in the equation is the point set on the sphere around point x'. l(x',Ω) is the time-dependent acoustic radiance from point x' to the direction of Ω. l0(x',Ω) is the sound energy emitted from point x', and R is the bidirectional reflectance distribution function (BRDF). It is an operator for the sound energy reflected from point x to point x' to the direction of Ω, which determines the type of reflection and describes the acoustic material of the plane.

[0058] In some embodiments, the geometric acoustic modeling method may be an image source method, a ray tracing method, etc. The image source method can only find the path of mirror emission. The ray tracing method overcomes this problem and can find the path of any reflection property, including diffuse reflection.

[0059] For example, the main idea of ​​ray tracing is to shoot rays from the sound source, reflect them through the scene, and find a feasible path from the sound source to the listener.

[0060] For each outgoing ray, first, the ray is randomly or according to a pre-set distribution to be emitted in a direction. If the sound source is directional, the energy carried by the ray is weighted according to the direction of the outgoing ray.

[0061] Then, the ray propagates in its direction. When it hits the scene, it will be reflected. According to the acoustic material of the scene position where it hits, the ray will have a new outgoing direction and continue to propagate in the new outgoing direction.

[0062] When a ray encounters a listener during propagation, the path is recorded. As the ray propagates and reflects, the propagation of the ray can be terminated when a certain condition is met.

[0063] For example, there may be two conditions for terminating ray propagation.

[0064] One condition is that each time a ray is reflected, the material of the scene absorbs part of the ray's energy; during the propagation of the ray, as the distance increases, the propagation medium (such as air) also absorbs the ray's energy; when the energy carried by the ray continues to decay and reaches a certain threshold, the ray stops propagating.

[0065] Another condition is "Russian Roulette". In this condition, the ray has a certain probability of being terminated at each reflection. This probability is determined by the absorptivity of the material, but since materials often have different absorptivity for sounds in different frequency bands, this condition is less used in acoustic ray tracing applications.

[0066] In addition, since the importance of the first part of the reverberation is usually higher than that of the last part, and considering the amount of calculation, in actual applications, the maximum number of reflections of the ray can also be set. When the number of reflections of the ray and the scene exceeds the set value, the reflection of the ray will stop.

[0067] When a certain number of rays are emitted, several paths from the sound source to the listener can be obtained. For each path, the energy carried by the ray along the path can be known. Based on the length of the path and the propagation speed of sound in the medium, the time t required for the path to propagate can be calculated, and thus an energy response E can be obtained. n (t). The RIR of the sound source in this scene to the listener can be expressed as:

[0068]

[0069] a p is the weight value related to the total number of emitted rays, t is the time, E n (t) is the response energy intensity of the path, n is the sequence number of the nth path, and N is the total number of paths. In the computer calculation process, p(t) can be a discrete value.

[0070] In some embodiments, according to different attenuation ranges, the attenuation time can be divided into EDT (early decay time), T 20 、T 30 and T 60 , these four indicators all belong to reverberation time.

[0071] EDT represents the time required for the reverberation to decay from 0 dB to -10 dB, and is the calculated time required for the reverberation to decay by 60 dB. 20 、T 30 They represent the time it takes for the reverberation to decay from -5dB to -25dB and -35dB, respectively, and the time required to decay by 60dB. 60 Indicates the time required for the reverberation to decay from 0dB to -60dB.

[0072] These indicators have a high correlation within the same house, but they also show great differences in certain house attributes.

[0073] In some embodiments, other objective indicators of reverberation also include: sound strength, clarity measures, spatial impression, etc. As shown in Table 1:

[0074]

[0075] Reverberation time is an important indicator for measuring the auditory perception of reverberation in a house, and is also a necessary parameter for generating reverberation through artificial reverberation methods. In real-time applications, in order to save real-time computing resources, the reverberation result obtained by geometric acoustic modeling can be used in the preprocessing stage to calculate the duration of house reverberation, and this parameter can be used to calculate artificial reverberation.

[0076] In some embodiments, the reverberation in the house can be calculated by combining the image sound source method with the ray tracing method. For direct paths and low-order early reflections, the image sound source method can be used to find the path of the sound from the person to the image sound source; the remaining energy intensity of the path is calculated based on the energy of the sound source, the length of the path, the energy absorbed by the path after reflection from the wall, and the energy absorbed by the air; the time position of the response generated by the path is obtained through the length of the path and the propagation speed of the sound in the air.

[0077] In addition, since air and walls have different absorption rates for sounds in different frequency bands, the obtained results will be saved separately for each frequency band.

[0078] For the late reverberation caused by more reflections and scattering, rays can be generated evenly in all directions from the listener's position; when the ray encounters an obstacle or wall, the next ray is emitted from the intersection point according to its material properties; when the ray intersects the sound source, a path from the listener to the sound source can be obtained, and then perhaps the time and intensity of the response generated by this path can be obtained.

[0079] When the ray is reflected by an obstacle for a certain depth of times, or the energy of the ray is lower than a certain threshold, the path can be stopped. The results of all paths are combined to finally obtain a time-energy scatter plot, which is the obtained RIR.

[0080] Compared with the house unit impulse response obtained from actual measurements, calculating the house reverberation time from the results of geometric acoustic simulation has several advantages: the time point when the reverberation starts can be accurately obtained; there is no need to perform post-processing operations such as filtering on the obtained house unit impulse response; the simulated RIR does not contain noise.

[0081] The results obtained by the calculation method of the above embodiment also have the following advantages: it is convenient to confirm the time point when the reverberation starts. In the RIR obtained from the acoustic simulation results, the first point in the time domain is the time point when the reverberation starts; for different frequency bands, RIR is calculated separately. To calculate the reverberation time of a certain frequency band, it is only necessary to calculate from the RIR of the frequency band, and no frequency division filtering operation is required; the calculated RIR all comes from the response brought by the path from the sound source to the listener, and there is no background noise problem.

[0082] In some embodiments, an attenuation curve (attenuation curve) is first calculated based on the RIR. The attenuation curve E(t) is a graphical representation of the sound pressure value of the house changing with time after the sound source stops, which can be obtained by Schroeder's backwards integration:

[0083]

[0084] p(τ) is the RIR, which represents the change of the sound pressure at the measurement point over time. t is time, and dτ is the differential of time. In actual computer applications, E(t) is represented by discrete values.

[0085] In the actual response, RIR has a finite length and cannot be integrated to positive infinity. Therefore, in theory, some energy will be lost due to this truncation. Therefore, some compensation can be made to correct the lost energy. One way is to add a constant C to the attenuation curve.

[0086]

[0087] Figure 3a Schematic diagram showing some embodiments of RIR curves.

[0088] like Figure 3a As shown, the three curves are respectively the RIR curve, the uncompensated attenuation curve, and the compensated attenuation curve.

[0089] After obtaining the decay curve, the linear fitting method can be used to fit a part of the decay curve to obtain the reverberation time. 20 , select the part of the attenuation curve where it drops from 5dB to 25dB from the steady state; for T 30 Then select the part of the attenuation curve where the value drops from 5dB to 35dB from the steady state; for T 60 Then select the part of the attenuation curve that drops 60dB from the steady state. Calculate the slope of the straight line used for fitting as the attenuation rate d, in dB per second, and the corresponding reverberation time is 60 / d.

[0090] Specifically, for the obtained attenuation curve E(t), we hope to find f(x) = a + bx to minimize the target R 2 =∑ i (E(t i )-f(t i )) 2 That is R 2 =(a,b)=Σ i (E(t i )-(a+bt i )) 2The minimum value can be obtained. From this, the desired conditions can be obtained:

[0091]

[0092]

[0093] And further get the equation:

[0094]

[0095]

[0096] Where n is the total number of energy points in the attenuation curve, from which we can calculate

[0097]

[0098]

[0099] Where cov is the covariance, σ is the variance, Take the average values ​​of E and t respectively.

[0100]

[0101]

[0102] After obtaining the linear fitting result of the attenuation curve, a is the desired slope, i.e., the attenuation rate. Then, the reverberation time value can be obtained. Finally, the reverberation time is estimated by E(t) to be RT = -60 / b.

[0103] In geometric acoustic modeling using ray tracing as a simulation method, the number of times the ray bounces in the scene is often limited due to computational considerations, that is, the number of times the ray is reflected in the scene is truncated.

[0104] When the reverberation time of the scene where the user is located is so long that the used path depth is insufficient to cover the entire reverberation time, the truncation of the path depth causes some of the actual energy to be discarded, which in turn causes the energy of the RIR to decay faster at the tail, presenting a form similar to exponential decay.

[0105] Figure 3b , 3c Schematic diagram showing some embodiments of RIR curves.

[0106] like Figure 3b As shown in the figure, for a reverberation curve with sufficient depth, the energy (dB) of RIR decays linearly and can be accurately estimated by linear fitting.

[0107] like Figure 3cAs shown in the figure, for a reverberation curve with insufficient depth, the energy (dB) of RIR should decay linearly. However, the lack of depth will cause a part of the energy to be lost, which accelerates the decay. It is impossible to accurately estimate the corresponding decay curve through linear fitting. From the decay graph, it can be seen that truncation leads to two problems:

[0108] 1. At a time earlier than the reverberation time, the decay curve has no energy

[0109] 2. The slope at the tail of the decay curve will be greater than the slope of the previous interruption, which makes the decay curve look like a curve with nonlinear characteristics.

[0110] Figure 3d , 3e Schematic diagram showing some embodiments of RIR curves.

[0111] like Figure 3d , 3e As shown in Figure 1, when the path depth is truncated, this RIR shape will cause the reverberation time estimated using the traditional linear fitting method to be too small. In a real-time reverberation system, setting an inaccurate reverberation value for the artificial reverberation method will also affect the immersion of the playback system.

[0112] In some embodiments, in order to solve the technical problems caused by the above-mentioned path depth truncation, the linear fitting method of the reverberation time in is improved. Using the improved method, the estimated reverberation time can be compensated from the decay curve of energy loss.

[0113] The attenuation curve E′(t) obtained by ray tracing as a simulation method is expected to be fitted by finding f(x)=a+bx. At the same time, due to the possibility of depth truncation, E′(t) is not necessarily an accurate attenuation curve. It is expected that the slope of the fitted curve can match the ideal attenuation curve E(t) without depth truncation. At the same time, due to the characteristics of depth truncation, it can be considered that if energy loss caused by depth truncation occurs, the error of the latter part of E′(t) will be greater than that of the former part, and the former part is more credible than the latter part.

[0114] In some embodiments, a method is proposed to use a weighted minimization objective in the time domain to fit a straight line to the attenuation curve, thereby obtaining the reverberation time.

[0115] In order to solve the problem that E′(t) is not necessarily accurate, the minimization target R of the linear fitting is 2 =∑ i (E′(t i )-f(t i )) 2 Based on the fitting target E′(t i ) contribution is weighted,

[0116]

[0117] E'(t) is the attenuation curve of RIR calculation obtained by simulation calculation, f(x) = a + bx is the straight line used for fitting, which is the weighted value that changes with time. It is hoped to find the values ​​of a and b, so as to find the value that can make The smallest straight line f(x).

[0118] So we can get

[0119]

[0120]

[0121] And further get the equation

[0122]

[0123]

[0124] From this we can calculate

[0125]

[0126] Where mean is the mean. Finally, the reverberation time is estimated by E′(t) to be RT=-60 / b.

[0127] In some embodiments, for the minimization objective, the attenuation curve may be weighted instead of the square of the difference between the attenuation curve and the fitted line:

[0128]

[0129] Alternatively, use the standard deviation instead of the variance as the minimization objective. For example:

[0130] R new =∑ i k(t i )E′(t i )-f(x i ))

[0131] R new =∑ i (k(t i )E′(t i )-f(x i ))

[0132] In some embodiments, one specific solution for selecting the weight k(t) is to make the weight satisfy that the weight decreases as time increases.

[0133] This design takes into account that the further back the attenuation curve is, the less accurate it is and the lower the weight it should have.

[0134] By making the weight k(t) satisfy the condition that the weight k(t) decreases with the increase of time, the true reverberation time can be estimated more accurately when the energy attenuation curve obtained by the acoustic simulation is affected by the path depth truncation. When it is not affected, the estimated original reverberation time can obtain a consistent estimated time.

[0135] Considering that the energy of reverberation decreases over time, one can use, for example,

[0136] k(t)=a(E′(t)-min(E′(t))) b / (mean(E′(t))+min(E′(t))) c

[0137] a, b, c are user-defined coefficients, which can be constants or coefficients obtained based on specific parameters. In the present invention, the coefficients can be added or subtracted before any term in the formula, or the offset can be added or subtracted on any term.

[0138] In some embodiments, weights independent of E'(t) may also be used, for example:

[0139] k(t)=ae -t , e is the natural logarithm, a is the free weight value, or

[0140] k(t)=mt+n, where m and n are freely selected coefficients.

[0141] The selection of different weights k(t) will affect the effect of reverberation time compensation, so it can be selected according to the characteristics of the audio signal.

[0142] In ray tracing based rendering engines, a method to correct for errors in reverberation length estimation caused by insufficient ray path depth.

[0143] 1. Use a weighted minimization objective in the time domain to fit the straight line and the attenuation curve to obtain the reverberation time.

[0144] 2. Regarding the selection of weights, one specific solution is to make the weights satisfy the requirement that the weights decrease as time goes by.

[0145] Figure 4a A flow chart showing some embodiments of a method for estimating reverberation time according to the present disclosure.

[0146] like Figure 4aAs shown, in step 410, a model of the objective function is constructed based on the differences between the attenuation curve of the audio signal and the parameter-containing function of the fitting curve of the attenuation curve at multiple historical time points, and the weights corresponding to the multiple historical time points. The weights vary with time.

[0147] For example, the weight corresponding to the later time point is less than the weight corresponding to the earlier time point. For example, the attenuation curve is determined according to the RIR of the audio signal.

[0148] In some embodiments, the weights corresponding to the multiple historical time points are used to perform a weighted sum of the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points. The model of the objective function is constructed based on the weighted sum of the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points.

[0149] For example, using the weights corresponding to multiple historical time points, the variances or standard deviations of the parameter-containing functions of the attenuation curve and its fitting curve at multiple historical time points are weighted summed.

[0150] In some embodiments, the attenuation curve is weighted at multiple historical time points using weights corresponding to multiple historical time points; a model of the objective function is constructed based on the difference between the weighted result of the attenuation curve and the parameter function of the fitting curve at multiple historical time points.

[0151] For example, the differences between the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points are summed to construct a model of the objective function.

[0152] For example, a model of the objective function is constructed according to the weighted result of the attenuation curve and the variance or standard deviation of the parameter-containing function of the fitting curve at multiple historical time points.

[0153] For example, the variance or standard deviation of the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points is summed to construct a model of the objective function.

[0154] In some embodiments, weights corresponding to multiple historical time points are determined based on statistical characteristics of the function of the attenuation curve; and a model of the objective function is constructed based on the weights corresponding to the multiple historical time points.

[0155] For example, the weights of the multiple historical time points are determined according to the minimum value and the average value of the function of the attenuation curve and the values ​​of the function of the attenuation curve at the multiple historical time points.

[0156] For example, the weights of multiple historical time points are determined based on the difference between the value of the function of the attenuation curve at multiple historical time points and the minimum value of the function of the attenuation curve, and the sum of the minimum value of the function of the attenuation curve and the average value of the function of the attenuation curve. The weights of multiple historical time points are positively correlated with the difference and negatively correlated with the sum.

[0157] For example, the weights of multiple historical time points are determined according to the ratio of the difference value to the sum value at multiple historical time points.

[0158] In some embodiments, the weights corresponding to the multiple historical time points are independent of the characteristics of the decay curve. For example, the weights of the multiple historical time points are determined according to an exponential function or a linear function that decreases over time; and the model of the objective function is constructed according to the weights of the multiple historical time points.

[0159] In some embodiments, weights corresponding to multiple historical time points are determined based on the characteristics of the sound signal; and a model of the objective function is constructed based on the weights corresponding to the multiple historical time points.

[0160] In step 420, the objective function is solved with the parameters of the function containing parameters of the fitting curve as variables and the model minimizing the objective function as the goal, and the fitting curve of the attenuation curve is determined.

[0161] In some embodiments, a first extreme value equation is determined based on a partial derivative of the objective function with respect to the slope coefficient of a linear function; a second extreme value equation is determined based on a partial derivative of the objective function with respect to the intercept coefficient of the linear function; and the first extreme value equation and the second extreme value equation are solved to determine the slope coefficient of the fitting curve.

[0162] In step 430, the reverberation time of the audio signal is estimated according to the fitting curve.

[0163] In some embodiments, the reverberation time is determined according to the slope coefficient of the linear function. For example, the reverberation time is proportional to the inverse of the slope coefficient of the linear function.

[0164] In some embodiments, the reverberation time is determined according to the slope coefficient of the linear function and a preset reverberation decay energy value. For example, the reverberation time is determined according to the ratio of the preset reverberation decay energy value to the slope coefficient. The preset reverberation decay energy value may be 60 dB.

[0165] Figure 4b A flow chart showing some embodiments of a method for rendering an audio signal according to the present disclosure.

[0166] like Figure 4b As shown, in step 510, the reverberation time of the audio signal is estimated at each of the multiple time points. For example, the reverberation time of the audio signal is determined using the estimation method in any of the above embodiments.

[0167] In step 520, the audio signal is rendered according to the reverberation time of the audio signal.

[0168] In some embodiments, reverberation of the audio signal is generated according to the reverberation time, and the reverberation is added to the bitstream of the audio signal. For example, the reverberation is generated according to at least one of the type of the acoustic environment model or the estimated late reverberation gain.

[0169] For example, the acoustic environment model includes physical reverberation, artificial reverberation, and sampled reverberation, etc. Sampled reverberation includes concert hall sampled reverberation, recording studio sampled reverberation, etc.

[0170] In some embodiments, various parameters of reverberation may be estimated through AcousticEnv(), and the reverberation may be added to the bitstream of the audio signal.

[0171] For example, AcousticEnv() is an extended static metadata acoustic environment, and the metadata decoding syntax is as follows.

[0172]

[0173] b_earlyReflectionGain 1 includes bits, which are used to indicate whether the earlyReflectionGain field exists in AcousticEnv(), 0 indicates that it does not exist, and 1 indicates that it exists; b_lateReverbGain includes 1 bit, which indicates whether the lateReverbGain field exists in AcousticEnv(), 0 indicates that it does not exist, and 1 indicates that it exists; reverbType includes 2 bits, which indicates the acoustic environment model type, 0 represents "Physical", 1 represents "Artificial", 2 represents "Sample", and 3 represents "Extended Type"; earlyReflectionGain includes 7 bits, which indicates early reflection gain; lateReverbGain includes 7 bits, which indicates late reverberation gain; lowFreqProFlag includes 1 bit, which indicates low-frequency separation processing. 0 means no reverberation processing for low frequencies to maintain clarity; convolutionReverbType includes 5 bits, indicating the sampling reverberation type, {0,1,2...N}, for example, 0 indicates concert hall sampling reverberation, and 1 indicates recording studio sampling reverberation; numSurface includes 3 bits, indicating the number of surface() contained in acousticEnv(), with a value of {0,1,2,3,4,5}; Surface() is a metadata decoding interface for walls of the same material.

[0174] In some embodiments, the Figure 4e In an embodiment of the rendering system, an audio signal is rendered.

[0175] Figure 4e A block diagram of some embodiments of a rendering system according to the present disclosure is shown.

[0176] like Figure 4e As shown, the audio rendering system includes a rendering metadata system and a core rendering system.

[0177] The metadata system contains control information that describes audio content and rendering technology, such as whether the audio payload input is single-channel, dual-channel, multi-channel, object, or sound field HOA, as well as the dynamic sound source and listener position information, and the rendered acoustic environment information (such as house shape, size, wall material, etc.).

[0178] The core rendering system performs rendering for corresponding playback devices and environments based on different audio signal representations and the corresponding Metadata parsed from the metadata system.

[0179] Figure 4c A block diagram showing some embodiments of a reverberation time estimation apparatus according to the present disclosure.

[0180] like Figure 4c As shown, the reverberation time estimation device 6 includes: a construction unit 61, which is used to construct a model of the objective function according to the differences between the attenuation curve of the audio signal and the parameter-containing function of its fitting curve at multiple historical time points, and the weights corresponding to the multiple historical time points, wherein the weights vary with time; a determination unit 62, which is used to solve the objective function with the parameters of the parameter-containing function of the fitting curve as variables and the model of the objective function as the goal to minimize the objective function, and determine the fitting curve of the attenuation curve; an estimation unit 63, which is used to estimate the reverberation time of the audio signal according to the fitting curve.

[0181] For example, the weight corresponding to the later time point is less than the weight corresponding to the earlier time point. For example, the attenuation curve is determined according to the RIR of the audio signal.

[0182] In some embodiments, the construction unit 61 uses weights corresponding to multiple historical time points to perform weighted summation of the differences between the parameter functions of the attenuation curve and its fitting curve at multiple historical time points; and constructs a model of the objective function based on the weighted sum of the differences between the parameter functions of the attenuation curve and its fitting curve at multiple historical time points.

[0183] For example, using the weights corresponding to multiple historical time points, the variances or standard deviations of the parameter-containing functions of the attenuation curve and its fitting curve at multiple historical time points are weighted summed.

[0184] In some embodiments, the construction unit 61 uses weights corresponding to multiple historical time points to perform weighted processing on the attenuation curve at multiple historical time points; and constructs a model of the objective function based on the differences between the weighted results of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points.

[0185] For example, the construction unit 61 sums the differences between the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points to construct a model of the objective function.

[0186] For example, the construction unit 61 constructs a model of the objective function according to the weighted result of the attenuation curve and the variance or standard deviation of the parameter-containing function of the fitting curve at multiple historical time points.

[0187] For example, the construction unit 61 sums the variance or standard deviation of the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points to construct a model of the objective function.

[0188] In some embodiments, the construction unit 61 determines weights corresponding to multiple historical time points according to the statistical characteristics of the function of the attenuation curve; and constructs a model of the objective function according to the weights corresponding to the multiple historical time points.

[0189] For example, the construction unit 61 determines the weights of the multiple historical time points according to the minimum value and the average value of the function of the attenuation curve and the values ​​of the function of the attenuation curve at the multiple historical time points.

[0190] For example, the construction unit 61 determines the weights of multiple historical time points based on the difference between the value of the function of the attenuation curve at multiple historical time points and the minimum value of the function of the attenuation curve, and the sum of the minimum value of the function of the attenuation curve and the average value of the function of the attenuation curve. The weights of the multiple historical time points are positively correlated with the difference and negatively correlated with the sum.

[0191] For example, the construction unit 61 determines the weights of the multiple historical time points according to the ratio of the difference value to the sum value at the multiple historical time points.

[0192] In some embodiments, the weights corresponding to the multiple historical time points are independent of the characteristics of the attenuation curve. For example, the construction unit 61 determines the weights of the multiple historical time points according to an exponential function or a linear function that decreases over time; and constructs a model of the objective function according to the weights of the multiple historical time points.

[0193] In some embodiments, the construction unit 61 determines weights corresponding to multiple historical time points according to the characteristics of the sound signal; and constructs a model of the objective function according to the weights corresponding to the multiple historical time points.

[0194] In some embodiments, the determination unit 62 determines a first extreme value equation based on a partial derivative of the objective function with respect to the slope coefficient of the linear function; determines a second extreme value equation based on a partial derivative of the objective function with respect to the intercept coefficient of the linear function; and solves the first extreme value equation and the second extreme value equation to determine the slope coefficient of the fitting curve.

[0195] In some embodiments, the estimation unit 63 determines the reverberation time according to the slope coefficient of the linear function. For example, the reverberation time is proportional to the inverse of the slope coefficient of the linear function.

[0196] In some embodiments, the estimation unit 63 determines the reverberation time according to the slope coefficient of the linear function and the preset reverberation decay energy value. For example, the estimation unit 63 determines the reverberation time according to the ratio of the preset reverberation decay energy value to the slope coefficient. The preset reverberation decay energy value may be 60 dB.

[0197] Figure 4d A block diagram showing some embodiments of an apparatus for rendering an audio signal according to the present disclosure.

[0198] like Figure 4d As shown, the audio signal rendering device 7 includes: a reverberation time estimation device 71 in any of the above embodiments, used to determine the reverberation time of the audio signal using the reverberation time estimation method in any of the above embodiments; and a rendering unit 72, used to render the audio signal according to the reverberation time of the audio signal.

[0199] In some embodiments, the rendering unit 72 generates reverberation of the audio signal according to the reverberation time and adds the reverberation to the bitstream of the audio signal. For example, the rendering unit 72 generates the reverberation according to at least one of the type of the acoustic environment model or the estimated late reverberation gain.

[0200] Figure 5 A block diagram showing some embodiments of the electronic device of the present disclosure.

[0201] like Figure 5 As shown, the electronic device 5 of this embodiment includes: a memory 51 and a processor 52 coupled to the memory 51, and the processor 52 is configured to execute the reverberation time estimation method or the audio signal rendering method in any embodiment of the present disclosure based on the instructions stored in the memory 51.

[0202] The memory 51 may include, for example, a system memory, a fixed non-volatile storage medium, etc. The system memory may store, for example, an operating system, an application program, a boot loader, a database, and other programs.

[0203] Reference below Figure 6, which shows a schematic diagram of the structure of an electronic device suitable for implementing the embodiments of the present disclosure. The electronic device in the embodiments of the present disclosure may include, but is not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0204] Figure 6 A block diagram showing some other embodiments of the electronic device of the present disclosure.

[0205] like Figure 6 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603. In RAM 603, various programs and data required for the operation of the electronic device are also stored. The processing device 601, ROM 602, and RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0206] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0207] According to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0208] In some embodiments, a chip is also provided, including: at least one processor and an interface, the interface being used to provide computer execution instructions to the at least one processor, and the at least one processor being used to execute the computer execution instructions to implement the reverberation time estimation method or audio signal rendering method of any of the above embodiments.

[0209] Figure 7 A block diagram showing some embodiments of the chip of the present disclosure.

[0210] like Figure 7 As shown, the processor 70 of the chip is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the processor 70 is the operation circuit, and the controller 704 controls the operation circuit 703 to extract data from the memory (weight memory or input memory) and perform operations.

[0211] In some embodiments, the operation circuit 703 includes a plurality of processing units (Process Engine, PE) therein. In some embodiments, the operation circuit 703 is a two-dimensional systolic array. The operation circuit 703 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some embodiments, the operation circuit 703 is a general-purpose matrix processor.

[0212] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 702 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 701 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 708.

[0213] The vector calculation unit 707 can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc.

[0214] In some embodiments, the vector calculation unit 707 can store the processed output vector to the unified buffer 706. For example, the vector calculation unit 707 can apply a nonlinear function to the output of the operation circuit 703, such as a vector of accumulated values, to generate an activation value. In some embodiments, the vector calculation unit 707 generates a normalized value, a merged value, or both. In some embodiments, the processed output vector can be used as an activation input to the operation circuit 703, for example, for use in a subsequent layer in a neural network.

[0215] The unified memory 706 is used to store input data and output data.

[0216] The memory unit access controller 705 (Direct Memory Access Controller, DMAC) moves the input data in the external memory to the input memory 701 and / or the unified memory 706, stores the weight data in the external memory into the weight memory 702, and stores the data in the unified memory 706 into the external memory.

[0217] The bus interface unit (BIU) 510 is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 709 through the bus.

[0218] An instruction fetch buffer 709 connected to the controller 704 and used to store instructions used by the controller 704;

[0219] The controller 704 is used to call the instructions cached in the memory 709 to control the working process of the computing accelerator.

[0220] Generally, the unified memory 706, the input memory 701, the weight memory 702 and the instruction fetch memory 709 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0221] In some embodiments, a computer program product is further provided, comprising: instructions, which, when executed by a processor, cause the processor to execute the method for estimating the reverberation time or the method for rendering an audio signal of any of the above embodiments.

[0222] According to some further embodiments of the present disclosure, a computer program is provided, comprising instructions, which, when executed by a processor, implement the method for estimating the reverberation time or the method for rendering an audio signal according to any one of the embodiments described in the present disclosure.

[0223] According to some further embodiments of the present disclosure, a method for rendering an audio signal is provided, comprising: estimating a reverberation time of an audio signal at each of a plurality of time points; and rendering the audio signal according to the reverberation time of the audio signal.

[0224] In some embodiments, the rendering processing of the audio signal includes: generating a reverberation of the audio signal according to the reverberation time, and the reverberation is added to a code stream of the audio signal.

[0225] In some embodiments, generating the reverberation of the audio signal comprises generating the reverberation according to at least one of a type of an acoustic environment model or an estimated late reverberation gain.

[0226] In some embodiments, estimating the reverberation time of the audio signal includes: constructing a model of an objective function according to an attenuation curve of the audio signal, a parameter-containing function of a fitting curve of the attenuation curve, and weights corresponding to a plurality of historical time points, wherein the weights vary with time; solving the objective function with parameters of the parameter-containing function of the fitting curve as variables and minimizing the model of the objective function as a goal to determine a fitting curve of the attenuation curve; and estimating the reverberation time of the audio signal according to the fitting curve.

[0227] In some embodiments, constructing the model of the objective function includes: constructing the model of the objective function according to the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points, and the weights corresponding to the multiple historical time points.

[0228] In some embodiments, the weight corresponding to the later historical time point is less than the weight corresponding to the earlier historical time point.

[0229] In some embodiments, constructing the model of the objective function includes: using the weights corresponding to the multiple historical time points to perform weighted summation of the differences between the attenuation curve and the parameter function of the fitting curve at the multiple historical time points; constructing the model of the objective function according to the weighted sum of the differences between the attenuation curve and the parameter function of the fitting curve at the multiple historical time points.

[0230] In some embodiments, the weighted summation of the differences between the attenuation curve and the parameter function of the fitting curve at the multiple historical time points includes: using the weights corresponding to the multiple historical time points to weighted sum the variances or standard deviations of the attenuation curve and the parameter function of the fitting curve at the multiple historical time points.

[0231] In some embodiments, the model of constructing the objective function includes: using the weights corresponding to the multiple historical time points to perform weighted processing on the attenuation curve at the multiple historical time points; constructing the model of the objective function based on the difference between the weighted result of the attenuation curve and the parameter function of the fitting curve at the multiple historical time points.

[0232] In some embodiments, constructing the model of the objective function includes: summing the differences between the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points to construct the model of the objective function.

[0233] In some embodiments, constructing the model of the objective function includes: constructing the model of the objective function according to the variance or standard deviation of the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at multiple historical time points.

[0234] In some embodiments, constructing the model of the objective function includes: summing the variance or standard deviation of the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points to construct the model of the objective function.

[0235] In some embodiments, constructing the model of the objective function includes: determining the weights corresponding to the multiple historical time points based on the statistical characteristics of the parameter-containing function of the attenuation curve; and constructing the model of the objective function based on the weights corresponding to the multiple historical time points.

[0236] In some embodiments, determining the weights of the multiple historical time points includes: determining the weights of the multiple historical time points based on the minimum value and the average value of the parameter-containing function of the attenuation curve, and the values ​​of the parameter-containing function of the attenuation curve at the multiple historical time points.

[0237] In some embodiments, determining the weights of the multiple historical time points includes: determining the weights of the multiple historical time points based on the difference between the value of the parameter function of the attenuation curve at the multiple historical time points and the minimum value of the parameter function of the attenuation curve, and the sum of the minimum value of the parameter function of the attenuation curve and the average value of the parameter function of the attenuation curve, the weights of the multiple historical time points are positively correlated with the difference, and negatively correlated with the sum.

[0238] In some embodiments, determining the weights of the multiple historical time points includes: determining the weights of the multiple historical time points according to a ratio of the difference value to the sum value at the multiple historical time points.

[0239] In some embodiments, the weights corresponding to the multiple historical time points are independent of the characteristics of the attenuation curve.

[0240] In some embodiments, constructing the model of the objective function includes: determining weights corresponding to the multiple historical time points according to the characteristics of the sound signal; and constructing the model of the objective function according to the weights corresponding to the multiple historical time points.

[0241] In some embodiments, constructing the model of the objective function includes: determining the weights of the multiple historical time points according to an exponential function or a linear function that decreases over time; and constructing the model of the objective function according to the weights of the multiple historical time points.

[0242] In some embodiments, the parameter-containing function of the fitting curve is a linear function with time as a variable, and estimating the reverberation time of the audio signal according to the fitting curve includes: determining the reverberation time according to a slope coefficient of the linear function.

[0243] According to some further embodiments of the present disclosure, there is provided a rendering device for an audio signal, comprising: an estimating device for estimating the reverberation time of the audio signal at each of a plurality of time points; and a rendering unit for performing rendering processing on the audio signal according to the reverberation time of the audio signal.

[0244] In some embodiments, the estimation device includes: a construction unit, used to construct a model of the objective function based on the attenuation curve of the audio signal, a parameter-containing function of a fitting curve of the attenuation curve, and weights corresponding to multiple historical time points, wherein the weights vary with time; a determination unit, used to solve the objective function with the parameters of the parameter-containing function of the fitting curve as variables and with the goal of minimizing the model of the objective function, so as to determine the fitting curve of the attenuation curve; and an estimation unit, used to estimate the reverberation time of the audio signal based on the fitting curve.

[0245] Those skilled in the art will appreciate that the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. When implemented using software, the above embodiments may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When a computer instruction or computer program is loaded or executed on a computer, a process or function according to an embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. Moreover, the present disclosure may take the form of a computer program product implemented on one or more computer-usable non-transient storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0246] Although some specific embodiments of the present disclosure have been described in detail by way of example, it should be understood by those skilled in the art that the above examples are for illustration only and are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that the above embodiments may be modified without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.

Claims

1. A method for rendering an audio signal, comprising: At each time point in a plurality of time points, estimating a reverberation time of the audio signal; Rendering the audio signal according to the reverberation time of the audio signal, The rendering of the audio signal includes: generating a reverberation of the audio signal according to the reverberation time, wherein the reverberation is added to a bit stream of the audio signal, Generating the reverberation of the audio signal comprises: In the case where an early reflection gain and a late reverberation gain exist in an extended static metadata acoustic environment, the reverberation is generated according to the early reflection gain, the late reverberation gain and the type of the acoustic environment model, and the type of the acoustic environment model includes physical reverberation, artificial reverberation and sampled reverberation.

2. The rendering method according to claim 1, wherein: The generating the reverberation comprises: The reverberation is generated according to the estimated late reverberation gain.

3. The rendering method according to claim 1, wherein: The estimating the reverberation time of the audio signal comprises: Constructing a model of an objective function according to the attenuation curve of the audio signal, a parameter-containing function of a fitting curve of the attenuation curve, and weights corresponding to a plurality of historical time points, wherein the weights vary with time; Taking the parameters of the parameter-containing function of the fitting curve as variables and minimizing the model of the objective function as the goal, solving the objective function to determine the fitting curve of the attenuation curve; The reverberation time of the audio signal is estimated according to the fitting curve.

4. The rendering method according to claim 3, wherein: The model for constructing the objective function includes: The model of the objective function is constructed according to the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points, and the weights corresponding to the multiple historical time points.

5. The rendering method according to claim 3, wherein: The weight corresponding to the later historical time point is smaller than the weight corresponding to the earlier historical time point.

6. The rendering method according to claim 4, wherein: The model for constructing the objective function includes: Using the weights corresponding to the multiple historical time points, weighted summation is performed on the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points; The model of the objective function is constructed according to the weighted sum of the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points.

7. The rendering method according to claim 6, wherein: The weighted summing of the differences between the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points includes: The weights corresponding to the multiple historical time points are used to perform a weighted summation of the variances or standard deviations of the parameter-containing functions of the attenuation curve and the fitting curve at the multiple historical time points.

8. The rendering method according to claim 3, wherein: The model for constructing the objective function includes: Using the weights corresponding to the multiple historical time points, weighted processing is performed on the attenuation curve at the multiple historical time points; The model of the objective function is constructed according to the difference between the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points.

9. The rendering method according to claim 8, wherein: The model for constructing the objective function includes: The differences between the weighted result of the attenuation curve and the parameter-containing function of the fitting curve at the multiple historical time points are summed to construct a model of the objective function.

10. The rendering method according to claim 8, wherein: The model for constructing the objective function includes: The model of the objective function is constructed according to the weighted result of the attenuation curve and the variance or standard deviation of the parameter-containing function of the fitting curve at multiple historical time points.

11. The rendering method according to claim 10, wherein: The model for constructing the objective function includes: The weighted result of the attenuation curve and the variance or standard deviation of the parameter-containing function of the fitting curve at the multiple historical time points are summed to construct a model of the objective function.

12. The rendering method according to claim 3, wherein: The model for constructing the objective function includes: Determining weights corresponding to the plurality of historical time points according to statistical characteristics of the parameter-containing function of the attenuation curve; A model of the objective function is constructed according to the weights corresponding to the multiple historical time points.

13. The rendering method according to claim 12, wherein: Determining the weights of the multiple historical time points includes: The weights of the multiple historical time points are determined according to the minimum value and the average value of the parameter-containing function of the attenuation curve and the values ​​of the parameter-containing function of the attenuation curve at the multiple historical time points.

14. The rendering method according to claim 13, wherein: Determining the weights of the multiple historical time points includes: The weights of the multiple historical time points are determined based on the difference between the value of the parameter function of the attenuation curve at the multiple historical time points and the minimum value of the parameter function of the attenuation curve, and the sum of the minimum value of the parameter function of the attenuation curve and the average value of the parameter function of the attenuation curve, and the weights of the multiple historical time points are positively correlated with the difference and negatively correlated with the sum.

15. The rendering method according to claim 14, wherein: Determining the weights of the multiple historical time points includes: The weights of the multiple historical time points are determined according to the ratio of the difference value to the sum value at the multiple historical time points.

16. The rendering method according to claim 3, wherein: The weights corresponding to the multiple historical time points are independent of the characteristics of the attenuation curve.

17. The rendering method according to claim 3, wherein: The model for constructing the objective function includes: Determining weights corresponding to the multiple historical time points according to characteristics of the sound signal; A model of the objective function is constructed according to the weights corresponding to the multiple historical time points.

18. The rendering method according to claim 16, wherein: The model for constructing the objective function includes: Determining the weights of the multiple historical time points according to an exponential function or a linear function that decreases over time; A model of the objective function is constructed according to the weights of the multiple historical time points.

19. The rendering method according to any one of claims 3 to 18, wherein: The parameter-containing function of the fitting curve is a linear function with time as a variable, and estimating the reverberation time of the audio signal according to the fitting curve includes: The reverberation time is determined according to the slope coefficient of the linear function.

20. The rendering method according to claim 19, wherein: The reverberation time is proportional to the inverse of the slope coefficient of the linear function.

21. The rendering method according to claim 19, wherein: Determining the reverberation time according to the slope coefficient of the linear function comprises: The reverberation time is determined according to the slope coefficient of the linear function and a preset reverberation decay energy value.

22. The rendering method according to claim 21, wherein: Determining the reverberation time according to the slope coefficient and a preset reverberation attenuation energy value includes: The reverberation time is determined according to the ratio of the preset reverberation decay energy value to the slope coefficient.

23. The rendering method according to claim 22, wherein: The preset reverberation attenuation energy value is 60dB.

24. The rendering method according to claim 19, wherein: The determining of the fitting curve of the attenuation curve comprises: Determining a first extreme value equation according to a partial derivative of the objective function with respect to the slope coefficient of the linear function; Determining a second extreme value equation according to a partial derivative of the objective function with respect to the intercept coefficient of the linear function; Solve the first extreme value equation and the second extreme value equation to determine the slope coefficient of the parameter-containing function of the fitting curve.

25. The rendering method according to any one of claims 3 to 18, wherein: The attenuation curve is determined according to a unit impulse response RIR of the audio signal.

26. A device for rendering an audio signal, comprising: An estimating device for estimating the reverberation time of the audio signal at each of a plurality of time points; A rendering unit, configured to render the audio signal according to the reverberation time of the audio signal, The rendering unit generates a reverberation of the audio signal according to the reverberation time, and the reverberation is added to a bitstream of the audio signal. In the case where an early reflection gain and a late reverberation gain exist in an extended static metadata acoustic environment, the rendering unit generates the reverberation according to the early reflection gain, the late reverberation gain and a type of an acoustic environment model, and the types of the acoustic environment models include physical reverberation, artificial reverberation and sampled reverberation.

27. The rendering device according to claim 26, wherein: The estimation device comprises: A construction unit, configured to construct a model of an objective function according to an attenuation curve of an audio signal, a parameter-containing function of a fitting curve of the attenuation curve, and weights corresponding to a plurality of historical time points, wherein the weights vary with time; A determination unit, configured to solve the objective function with the parameters of the parameter-containing function of the fitting curve as variables and the model minimizing the objective function as a target, so as to determine the fitting curve of the attenuation curve; An estimating unit is used to estimate the reverberation time of the audio signal according to the fitting curve.

28. A chip, comprising: At least one processor and an interface, wherein the interface is used to provide computer-executable instructions to the at least one processor, and the at least one processor is used to execute the computer-executable instructions to implement the audio signal rendering method according to any one of claims 1 to 25.

29. An electronic device comprising: Memory; and A processor coupled to the memory, wherein the processor is configured to execute the audio signal rendering method according to any one of claims 1 to 25 based on instructions stored in the memory device.

30. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein when the program is executed by a processor, the method for rendering an audio signal according to any one of claims 1 to 25 is implemented.

31. A computer program product comprising instructions, which, when executed by a processor, cause the processor to perform the method for rendering an audio signal according to any one of claims 1-25.

Images