How to generate channel- and object-based audio from channel-based audio
The audio processing method addresses the challenge of converting channel-based audio to object-based audio by using a control and adaptive post-processing module to reduce artifacts, enhancing the quality of audio playback.
Patent Information
- Application Number
- JP2024524745
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2022-01-18
- Filing Date
- 2022-10-14
- Publication Date
- 2024-11-13
AI Technical Summary
Existing audio processing systems face challenges in converting traditional channel-based audio content, such as 5.1 or 7.1 surround sound, into channel- and object-based formats, often introducing artifacts and requiring manual evaluation, which is inefficient and prone to inaccuracies.
An audio processing method and apparatus that includes a control module for evaluating object extraction and rendering, an adaptive post-processing module, and a modification module to automatically convert channel-based audio into object-based audio, reducing artifacts by analyzing and adjusting audio objects and bed channels using signal processing techniques.
The method effectively reduces audio artifacts in converted audio content by automatically evaluating and modifying audio objects and bed channels, improving the quality of channel- and object-based audio playback.
Smart Images

Figure 2024541930000045 
Figure 2024541930000046 
Figure 2024541930000047
Abstract
Description
[Technical field]
[0001]
[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to the following prior applications: ES Patent Application No. P202130998, filed October 25, 2021, and U.S. Provisional Application No. 63 / 298673, filed January 12, 2022, all of which are incorporated by reference in their entireties.
[0002] This disclosure relates to audio processing, and more particularly to generating object-based audio from channel-based audio. [Background technology]
[0003]
[0003] Unless otherwise indicated herein, the approaches described in this section are not prior art to the claims of this application, nor are they admitted to be prior art by inclusion in this section.
[0004]
[0004] In recent years, in the multimedia industry, three-dimensional (3D) movie and television contents have become increasingly popular in cinemas and at home. Following these developments, several audio playback systems have also been proposed. Traditional multi-channel systems such as stereo audio (e.g., 2-channel, 5.1-channel surround sound, 7.1-channel surround sound) have been extended to create a more immersive sound field.
[0005]
[0005] An example of a next-generation audio system is a format that contains both audio channels, called bed channels, and audio objects. An audio object is an individual audio element that exists for a defined period of time and has metadata such as spatial information that describes the position, speed, and size of the audio object. A bed channel is an audio channel that is meant to be played back at a predefined, fixed speaker position. During transmission, the object and bed channels are sent separately and are then used by the playback system to adaptively recreate the artistic intent based on the specific placement of playback speakers in the playback environment; the generation of audio output based on the placement of speakers is sometimes called rendering. Summary of the Invention
[0006]
[0006] One problem with existing audio processing systems is that the majority of existing audio content is channel-based, such as 5.1, 7.1, or stereo. To convert the traditional channel-based content to channel and object-based formats, it is necessary to develop automated techniques or tools to extract object and bed channels from the traditional mix. Furthermore, it is also desirable for the automated rendering tool to further modify or upmix the extracted audio object and bed channels to improve the playback of the traditional content. Furthermore, there may be artifacts and inaccurate estimations introduced in the automatic object extraction and ambience upmixing process, and therefore it is also desirable to detect these issues in an automated manner to improve the quality of the final output content. The embodiments aim to evaluate the statistics of the extracted audio object and bed channels to identify discontinuities, and adjust the extracted audio object and bed channels as necessary to reduce the discontinuities. This automated evaluation and adjustment is an improvement over conventional methods that may require extensive manual evaluation and manipulation by an audio engineer.
[0007]
[0007] An embodiment uses audio signal processing techniques to automatically convert any multi-channel audio content (e.g., 5.1, 7.1, etc.) from a channel-based format to a channel-and-object-based format. To improve the quality of the channel-and-object-based audio content, the system implements three modules: (1) a control module for verifying and evaluating the results of the object extraction and rendering module; (2) an adaptive post-processing module for obtaining post-processing parameters based on the results of the control module; and (3) a modification module for modifying the extracted channel-and-object-based audio content based on the obtained post-processing parameters.
[0008]
[0008] According to an embodiment, a computer-implemented audio processing method includes receiving a channel-based audio signal, generating a reference audio signal based on the channel-based audio signal, and generating a plurality of audio objects and a plurality of bed channels based on the channel-based audio signal. The method further includes generating a rendered audio signal based on the plurality of audio objects and the plurality of bed channels. The method further includes generating a detection score based on a plurality of partial loudnesses of the plurality of signals. The plurality of signals includes the reference audio signal, the plurality of audio objects, the plurality of bed channels, the rendered audio signal, and the channel-based audio signal. The detection score indicates audio artifacts in one or more of the plurality of audio objects and the plurality of bed channels. The method further includes generating a plurality of parameters based on the detection score. The method further includes generating a plurality of modified audio objects and a plurality of modified bed channels based on the channel-based audio signal, the plurality of audio objects, the plurality of bed channels, and the plurality of parameters.
[0009]
[0009] As a result, the modified audio objects and the modified bed channel have reduced audio artifacts compared to the unmodified audio objects and the unmodified bed channel.
[0010]
[0010] According to another embodiment, an apparatus includes one or more speakers and a processor. The processor is configured to control the apparatus to perform one or more methods described herein. The apparatus may further include similar details to one or more of the methods described herein.
[0011]
[0011] According to another embodiment, a non-transitory computer-readable medium stores a computer program which, when executed by a processor, controls an apparatus to perform processes including one or more of the methods described herein.
[0012] The following detailed description and accompanying drawings provide a further understanding of the nature and advantages of various implementations. [Brief description of the drawings]
[0013] [Figure 1] FIG. 1 is a block diagram of an audio content generator 100. [Diagram 2]
[0014] FIG. 2 is a flow chart of a method 200 of audio processing. [Figure 3A]
[0015] FIG. 3A is a diagram showing the mapping between channel numbers and regions. [Figure 3B] FIG. 3B is a diagram showing the mapping between channel numbers and regions. [Figure 4]
[0016] FIG. 4 is a device architecture 400 for implementing the features and processes described herein, according to an embodiment. [Diagram 5]
[0017] FIG. 5 is a flow chart of an audio processing method 500 . DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0014]
[0018] The present application describes techniques related to audio processing. In the following description, for the purpose of explanation, numerous examples and specific details are set forth to provide a thorough understanding of the present disclosure. However, it will be apparent to those skilled in the art that the present application, as defined by the claims, may include some or all of the features in these examples, either alone or in combination with other features described below, and may further include modifications and equivalents of the features and concepts described herein.
[0015]
[0019] In the following description, various methods, processes, and procedures are detailed. Although certain steps may be described in a particular order, such order is primarily for convenience and clarity. Certain steps may be repeated multiple times, may occur before or after other steps even if those steps are otherwise described in a different order, and may occur in parallel with other steps. The only time a second step is required to follow a first step is when the first step must be completed before the second step can begin. Such situations will be specifically pointed out if they are not clear from the context.
[0016]
[0020] In this specification, the terms "and", "or", and "and / or" are used. Such terms should be understood to have an inclusive meaning. For example, "A and B" may mean at least: "both A and B", "at least both A and B". As another example, "A or B" may mean at least: "at least A", "at least B", "both A and B", "at least both A and B". As another example, "A and / or B" may mean at least: "A and B", "A or B". Where an exclusive OR is intended, such will be specifically noted, for example, "either A or B", "at most one of A and B", etc.
[0017]
[0021] This specification describes various processing functions in terms of structures such as blocks, elements, components, circuits, etc. Generally, these structures may be implemented by a processor controlled by one or more computer programs.
[0018]
[0022] FIG. 1 is a block diagram of an audio content generator 100. The audio content generator 100 generally converts an input channel-based audio signal 130 into an output audio signal 150 including audio objects, e.g., a channel-and-object-based audio signal (also referred to as a modified audio signal 150). The channel-based audio signal 130 generally corresponds to a multi-channel audio signal, such as a stereo signal, e.g., a 2-channel, a 5.1-channel surround signal, a 7.1-channel surround signal, etc. The channel-based audio signal 130 generally includes a number of audio samples, e.g., each channel has a number of samples. The audio samples may be arranged in blocks. As described in more detail herein, the audio content generator 100 operates in blocks, where each block has a duration between 0.20 seconds and 0.30 seconds. According to a particular embodiment, the block size is 0.25 seconds; this value provides reasonable results for the listener (or viewer) and can be adjusted as desired. The channel-based audio signal 130 may have a sample rate of 48 kHz, in which case a 0.25 second block size results in approximately 12,000 samples per block. The output audio signal 150, also referred to as the modified audio signal 150, generally results from transforming and modifying the channel-based audio signal 130, as described in more detail herein.
[0019]
[0023] The components of the audio content generator 100 may be implemented by one or more processors controlled by one or more computer programs. The audio content generator 100 includes a bed generator 102, an object extractor 104, a metadata estimator 106, a renderer 108, a bed generator 110, a renderer 112, a controller 114, an adaptive post-processor 116, and a signal modifier 118. The audio content generator 100 may include other components that are not detailed herein for the sake of clarity.
[0020]
[0024] The bed generator 102 receives a channel-based audio signal 130 and performs bed generation to generate one or more bed channels 132 based on the channel-based audio signal 130. Typically, the bed channels include audio signal components represented in a channel-based format, each of the bed channels corresponding to a sound reproduction at a predefined fixed position. The bed channels may be directional audio signals, also called direct signals. A bed channel for the diffuse audio signals, also called the diffuse signal The bed channel 132 may include a bed channel for direct audio signals. The direct signal corresponds to audio that is to be perceived as coming from a defined position or a defined direction. The diffuse signal corresponds to audio that is not to be perceived as coming from a defined direction, for example, to represent relatively complex audio textures such as background or ambience in the sound field for efficient authoring and distribution. In particular, the bed channel 132 corresponds to a diffuse signal generated based on the channel-based audio signal 130. The bed channel 132 may include one or more height channels.
[0021]
[0025] The object extractor 104 receives the channel-based audio signal 130 and performs audio object extraction to generate one or more audio objects 134 based on the channel-based audio signal 130. Each audio object 134 corresponds to audio data and metadata, where the metadata indicates information such as object position, object size, object speed, etc.; the output system uses the metadata to output the audio data according to a specific speaker arrangement at the output end. This can be contrasted with the bed channels 132, i.e., having each bed channel specifically associated with one or more speakers. The metadata will be described in more detail with reference to the metadata estimator 106.
[0022]
[0026] The object extractor 104 may include a signal decomposer configured to decompose the channel-based audio signal 130 into a directional audio signal and a diffuse audio signal. In these embodiments, the object extractor 104 may be configured to extract the audio object from the directional audio signal. In some embodiments, the signal decomposer may include a component decomposer and a probability calculator. The component decomposer is configured to perform a signal component decomposition on the channel-based audio signal 130. The probability calculator is configured to calculate a probability for diffuseness by analyzing the decomposed signal components.
[0023]
[0027] Alternatively or additionally, the object extractor 104 may include a spectral composer and a temporal composer. The spectral composer is configured to perform a spectral construction for each frame in the channel-based audio signal 130 to identify and aggregate channels that contain the same audio object. A frame is a vector of a predetermined number of consecutive samples (typically several hundred) for each channel in the signal at a given time. The temporal composer is configured to perform a temporal synthesis of the identified and aggregated channels over a set of frames to form an audio object along time. For example, the spectral composer may include, for each of the set of frames, a frequency divider configured to divide a frequency range into a set of sub-bands. Thus, the spectral composer may be configured to identify and aggregate channels that contain the same audio object based on a similarity of at least one of an envelope and a spectral shape among the set of sub-bands.
[0024]
[0028] The metadata estimator 106 receives the audio objects 134 and performs metadata estimation to generate metadata 136 based on the audio objects 134. The metadata 136 typically includes a timestamp and a location, which may be given as an (x,y,z) coordinate. The metadata estimator 106 may perform the metadata estimation using panning-law inverting. To estimate the “x” location of a given audio object, the metadata estimator 106 may calculate the arctangent of the left-to-right energy ratio of the given audio object. To estimate the “y” location, the metadata estimator 106 may calculate the arctangent of the back-to-front energy ratio of the given audio object. To estimate the "z" position, the metadata estimator 106 may utilize the estimates of the "x" and "y" positions to compute a predetermined function z=f(x,y); in one embodiment, f is a dome function that evaluates as z=1 when x and y are in the center of the speaker layout and as z=0 when x and y are on the border of the speaker layout.
[0025]
[0029] The renderer 108 receives the bed channel 132, the audio objects 134, and the metadata 136, performs rendering, and generates a rendered audio signal 138 based on the bed channel 132, the audio objects 134, and the metadata 136. The rendered audio signal 138 is a channel-based audio signal including one or more of a 5.1 channel signal, a 7.1 channel signal, a 5.1.4 channel signal, a 7.1.4 channel signal, etc. The rendered audio signal 138 may include two channel-based audio signals, one of which omits ceiling channels. For example, the rendered audio signal 138 may include a 5.1.4 channel signal and a 5.1 channel signal, a 7.1.4 channel signal and a 7.1 channel signal, etc.
[0026]
[0030] The bed generator 110 receives the channel-based audio signal 130, performs bed generation, and generates one or more reference bed channels 140. The reference bed channels 140 include bed channels for both the direct signal and the diffuse signal. In contrast, the bed channel 132 includes only the diffuse signal. The bed generator 110 may be similar to the bed generator 102 in other respects.
[0027]
[0031] The renderer 112 receives the reference bed channel 140, performs rendering, and generates a reference audio signal 142 based on the reference bed channel 140. The reference audio signal 142 is a channel-based audio signal, including one or more of a 5.1 channel signal, a 7.1 channel signal, a 5.1.4 channel signal, a 7.1.4 channel signal, etc. In general, the reference audio signal has a format similar to that used for the rendered audio signal 138; for example, if the rendered audio signal 138 is a 5.1.4 channel signal and a 5.1 channel signal, the reference audio signal is a 5.1.4 channel signal. In comparison with the rendered audio signal 138, the reference audio signal 142 is also rendered based on the channel-based audio signal 130; however, the reference audio signal 142 is rendered based on the bed channel, not on audio objects or metadata. The renderer 112 may be similar to the renderer 108 in other respects.
[0028]
[0032] The controller 114 receives the channel-based audio signal 130, the bed channel 132, the audio object 134, the metadata 136, the rendered audio signal 138, and the reference audio signal 142, calculates a number of signal metrics, and generates a detection score 144 based on the channel-based audio signal 130, the bed channel 132, the audio object 134, the metadata 136, the rendered audio signal 138, and the reference audio signal 142. The signal metrics may be calculated based on partial loudnesses of the signals. The detection score 144 indicates audio artifacts in one or more of the audio objects and the bed channel. For example, the bed channel 132 may have audio artifacts resulting from a particular operation of the bed generator 102; the audio object 134 may have audio artifacts resulting from a particular operation of the object extractor 104; or both the bed channel 132 and the audio object 134 may have audio artifacts. Further details of the controller 114 are described with reference to FIG.
[0029]
[0033] 2 is a flow chart of an audio processing method 200. Method 200 can be performed by controller 114 (see FIG. 1) as implemented by one or more processors capable of executing one or more computer programs. As described in relation to FIG. 1, controller 114 receives four inputs. The first input is the audio object 134, the bed channel 132, and the metadata 136, which are the outputs of the preceding components. obj,i where i ∈ [1,...,I] is the object index and I is the number of objects. bed,jwhere j∈[1,...,B] is the bed channel index and B is the number of bed channels. The metadata can be written as m i where i ∈ [1,...,I] is the object index. The second input is a channel-based audio signal 130, which is in It can be written as follows. The third input is the rendered audio signal 138, which may include a rendered signal with a ceiling channel, e.g., 5.1.4 or 7.1.4, and a rendered signal without a ceiling channel, e.g., 5.1 or 7.1, which are represented by X out and X out,f It is possible to write it as follows: The fourth input is a reference audio signal 142, which may be 5.1.4 or 7.1.4, ref It is possible to write it as follows. Note that X in ,X out ,X out,f ,X ref are written in bold in the original text. In general, the controller 114 uses the reference audio signal 142 to detect the quality of the rendered audio signal 138 .
[0030]
[0034] As mentioned above, the audio content generator 100 (see FIG. 1) processes the channel-based audio signal 130 block-by-block sequentially. The block length L may be set as L=0.25s. However, the block length can be changed to any desired value.
[0031]
[0035] At 202, a reference audio signal 142 (X ref ), audio object 134 (x obj,i ), bed channel 132 (x bed,j), and the rendered audio signal 138 (X out and X out,f ), and the channel-based audio signal 130 (X in Calculate the number of partial loudnesses (denoted as x ref,ktc ,x obj,ikt ,x bed,jkt ,x out,ktc ,x out,f,ktc ,x in,ktc where k∈[1,...,K] is the frequency band index, K is the total number of frequency bands, t is the current block index, c∈[1,...,C] is the channel index, and C is the total number of channels. Loudness is calculated based on the psychological acoustics of the human ear, and the evaluation of loudness information is correlated with the evaluation of sound quality.
[0032]
[0036] In 204, the ratio of the energy of the object to the energy of the bed channel and the energy of the object, r, is calculated according to Eq. (1). t Calculate:
[0033]
number
[0037] In formula (1), x ^ obj,t is the energy of the audio object 134 and can be calculated according to equation (2):
[0034]
number
[0038] In formula (1), x ^ bed,t is the energy of the bed channel 132 and can be calculated according to equation (3):
[0035]
number
[0039] In equations (2) and (3), the variables t,i,C,k,K,j,B,k,K are as described above in relation to 202. The energy of the audio object 134 calculated in equation (2) may be smoothed over time according to equation (4):
[0036]
number
[0040] The energy of the bed channel 132 calculated in equation (3) may be smoothed over time according to equation (5):
[0037]
number
[0041] In equations (4) and (5), μ is a smoothing parameter, which may be set to 0.7; this value may be adjusted as needed, for example to a range of 0.6 and 0.8. For example, a user of the audio content generator 100 (see FIG. 1) may listen to the modified audio signal 150, evaluate it, adjust the smoothing parameter, and continue the iterative evaluation until the smoothing parameter produces an acceptable result. ^ obj,0 and x ^ bed,0 is initialized to zero.
[0038]
[0042] In other words, the ratio r t is the ratio between the first energy and the second energy, where the first energy is the energy of the audio object 134 and the second energy is the sum of the energy of the audio object 134 and the energy of the bed channel 132. The ratio is calculated to determine the contribution of each object to the total energy.
[0039]
[0043] At 206, the average position of each audio object 134 in block t is calculated based on the metadata 136. First, the metadata m i,pwhere p is the time sample in block t, and (t-1)*L≦p≦t*L. Secondly, the average position m i,t is obtained according to equation (6).
[0040]
number
[0044] In equation (6), L is the block length as mentioned above. Third, the average position in block t is smoothed using the positions in the previous blocks according to equation (7).
[0041]
number
[0045] In equation (7), μ m is a smoothing parameter. The smoothing parameter is adjustable and is generally in the range of 0.5 to 1.0; a typical value for the smoothing parameter is 0.7. For example, a user of the audio content generator 100 (see FIG. 1) can listen to the modified audio signal 150, evaluate it, adjust the smoothing parameter, and continue iterative evaluation until the smoothing parameter produces an acceptable result. In the first block, m i,0 is set to zero. In other words, the average position of the audio object 134 is calculated to check for potential discontinuities between blocks of a given object.
[0042]
[0046] In 208, boost score based on partial loudness ij Calculate the number of partial loudnesses, Partial loudness of a channel-based audio signal 130 x in,ktc , Partial loudness x of the reference audio signal 142 ref,ktc , Partial loudness x of audio object 134 obj,ikt , Partial loudness of the rendered audio signal 138 xout,ktc and x out,f,ktc Includes. Final boost score boost score is calculated based on selecting two or more boost scores; according to an embodiment, the two largest boost scores are added together to calculate a final boost score. score The full details of calculating is detailed in the next eight steps.
[0043]
[0047] First of all the band sums of the partial loudness, i.e. the signal energy, are calculated according to equations (8.1, 8.2, 8.3, 8.4, 8.5):
[0044]
number
[0048] Second, the proportion of each channel in the total loudness is calculated according to equations (9.1, 9.2, 9.3, 9.4):
[0045]
number
[0049] Third, the difference between each partial loudness and the previous block is calculated according to formulas (10.1, 10.2, 10.3):
[0046]
number
[0050] In other words, d out,tc is the partial loudness of the current block of rendered audio 138 x out,tc,r and the partial loudness x of the previous block of the rendered audio 138 ^ out,(t-1)c,r Similarly, d ref,tc is the partial loudness x of the current block of the reference audio 142 ref,tc,r and the partial loudness x of the previous block of the reference audio 142 ^ ref,(t-1)c,rNote that the partial loudnesses of the previous block are written with a caret (^) to indicate that they have been smoothed; see 220 below.
[0047]
[0051] Fourth, the position of each block (m i,t ) and the previous block position (m ^ i,t-1 ) and the difference d m,t is calculated according to equation (11):
[0048]
number
[0052] In equation (11), position m i,t may be calculated in 206. Note that the positions of the previous blocks are noted with a caret (^) to indicate that they have been smoothed; see 220 below.
[0049]
[0053] Fifth, the index f of the object whose energy ratio exceeds the threshold θ is calculated according to the process in TABLE 1: TABLE 1
[0050] [Table 1]
[0054] In other words, in line 1, the energy ratio is calculated. In line 2, if the energy ratio exceeds a threshold θ, then the object i is added to the index; if not, then the object is not added to the index. In this way, quiet objects, e.g., those whose energy ratio does not exceed the threshold, are not indexed. The threshold can be adjusted as desired; a typical range for the threshold is between 0.0 and 0.5, with a typical value that works well being 0.2. For example, a user of the audio content generator 100 (see FIG. 1 ) can listen to the modified audio signal 150, evaluate it, adjust the threshold, and continue iterative evaluation until the threshold produces acceptable results.
[0051]
[0055] Sixth, the rendered audio signal 138(x out,tc,r ) and reference audio signal 142(x ref,tc,r ) and the loudness ratio of the rendered audio signal 138 (x our,f,tc,r ) and channel-based audio signals 130(x in,tc,r ) and the loudness ratio difference between them is calculated according to formulas (12.1, 12.2):
[0052]
number
[0056] In other words, the loudness difference d tc,r and d tc,f,r is used to detect whether there are energy changes in corresponding channels between the rendered audio signal 138 and the reference audio signal 142, and between the rendered audio signal 138 and the channel-based audio signal 130.
[0053]
[0057] Seventh, the weight score ij ,Correlation score,cor ij ,Diff score ijThese calculations include seven sub-steps. In sub-step 1, d ti,r <0.0, i≦5 is the case of C=9, or i≦7 is the case of C=11 (d ti,r <0.0 and also i≦5 if C=0 or i≦7 if Find an index i such that C=11), where C is the total number of channels as discussed in 202. This means that for renderers 5.1 to 5.1.4, and 5.1 or 7.1 to 7.1.4, we only consider those channels that have energy reduction in the horizontal plane channels.
[0054]
[0058] In step 2, d tj,r >0.0. This is used to find which channels have an energy increase.
[0055]
[0059] In sub-step 3, it is checked whether channel indexes i and j are in the same spatial region. The mapping shown in Figures 3A-3B is used to make this determination. Figure 3A shows the mapping between channel numbers and regions for 5.1.4, which has 9 channels, and Figure 3B shows the mapping between channel numbers and regions for 7.1.4, which has 11 channels.
[0056]
[0060] For C=9, using FIG. 3A, if i=1 and j=3, 4, 6, 8, then i, j are in the same region. If i=2 and j=3,5,7,9 then i,j are in the same region. If i=3 and j=1,2,6,7 then i,j are in the same region. If i=4 and j=6,8 then i,j are in the same region. If i=5 and j=7,9 then i,j are in the same region.
[0061] For C=11, using FIG. 3B, if i=1 and j=3, 4, 6, 8, 10, then i, j are in the same region. If i=2 and j=3,5,7,9,11, then i,j are in the same region. If i=3 and j=1,2,8,9 then i,j are in the same region. If i=4,6 and j=6,8,10, then i,j are in the same region. If i=5,7 and j=7,9,11 then i,j are in the same region.
[0062] If channel indices i and j are in the same spatial region, the weight score w ij If so, proceed to sub-step 4; otherwise, proceed to sub-step 1 again.
[0057]
[0063] Weight score w ij indicates the degree of energy change in channel j between adjacent blocks. The weight score wij can be calculated according to equation (13):
[0058]
number
[0064] In other words, the weight score is the loudness delta (d out,tj , see equation (10.1)) and the loudness difference of the reference audio 142 (d ref,tj , (see equation (10.2)).
[0059]
[0065] In sub-step 4, the weight score is set to w if any of the conditions (Conditions 1-6) in TABLE 2 are met. ij Updated to set =0: TABLE 2
[0060] [Table 2]
[0066] These parameters θ1 to θ17 is a threshold value. In general, the threshold value is set to a value such that if any of the conditions in TABLE 2 are met, a given weight score is set to zero. In such cases, the probability that the extracted object has artifacts is small, so the weight score is set to 0 to make the final score small as well. For example, in the case of condition 4, d m,f If r is small, the object is continuous and there are no artifacts. t If is large, most of the content in the input will be extracted into objects and there will be no artifacts. An example of the default value is: θ1= -0.1, θ2= 0.1, θ3= -0.1, θ4= 0.1, θ5= -0.15, θ6= 0.15, θ7= -0.25, θ8= 0.25, θ9= 0.5, θ 10 = -0.35, θ 11 = 0.35, θ 12 = -0.02, θ 13 = 0.02, θ 14 = -0.01, θ 15 = 0.01, θ 16 = 0.7, θ 17 = 0.1,
[0067] In step 5, the partial loudness x of channel i in block t-1 is calculated. out,k(t-1)i and the partial loudness x of channel j in block t out,ktj Correlation between ijwhich is used to check whether the content energy in channel i and the content energy in channel j are correlated for the rendered audio 138.
[0061]
[0068] In sub-step 6, the difference score diff between the loudness ratios of channel i and channel j is calculated according to the following two steps: ij First, we calculate the position weight parameter p ij The position weight parameter pij can be calculated according to the process in TABLE 3: TABLE 3
[0062] [Table 3]
[0069] In other words, the process of TABLE 3 is used to increase the positional weight when channels i and j are in the front (see Figures 3A-3B), because the front channels are more important for listening.
[0063]
[0070] Second, the difference score diff according to Eq. (14) ij Calculate:
[0064]
number
[0071] In equation (14), the function f1 is p ij ,d tj,r ,d ti,r An example of f1 is given by equation (15):
[0065]
number
[0072] In other words, the difference score is the difference between the loudness ratio differences for the channels (see equation (12.1)) with respect to the position weight parameter p ij The difference score indicates the degree of energy boost in channel j.
[0066]
[0073] In sub-step 7, the boost score for the current i,j pair is calculated as score ij is calculated using equation (16):
[0067]
number
[0074] In equation (16), the function f2 is ij ,w ij ,diff ij An example of f2 is given by equation (17):
[0068]
number
[0075] In other words, the boost score is a function of the correlation of partial loudness between channels. ij , see sub-step 5 above) and the degree of energy change in the channel between adjacent blocks (weight score w ij ,see Eq. (13)) and the difference score of the loudness ratio of the channel (diff ij , see sub-step 6 above). Thus, the final boost score will be high if the degree of energy boost in channel j is high, if the contents in channels i and j are highly correlated, and if the contents in channel j change quickly between neighboring blocks.
[0069]
[0076] Eighth, the two highest difference scores diff ij Final boost score using boost score For example, the maximum difference score is the boost score AThe next largest difference score is the boost score B If it is a component of the final boost score score can be calculated according to equation (18):
[0070]
number
[0077] At 210, the rendered audio 138 (d out,tc ) and reference audio 142(d ref,tc The deviation metric between the partial loudness of The deviation metrics are calculated using the standard deviation metric. out,t ,std ref,t ,std dif,t ,std r,t Includes std out,t For all channels, d out,tc The standard deviation of is calculated. ref,t For all channels, d ref,tc The standard deviation of is calculated. out,t and std ref,t deviation difference std dif,t can be calculated according to equation (19):
[0071]
number
[0078] In other words, the deviation difference is the difference between the standard deviation of the partial loudness of the rendered audio 138 and the standard deviation of the partial loudness of the reference audio 142 .
[0072]
[0079] std out,t and std ref,t Deviation ratio std r,t can be calculated according to equation (20):
[0073]
number
[0080] In other words, the deviation ratio is the minimum of the threshold parameter and the ratio, which is the ratio between the standard deviation of the partial loudness of the rendered audio 138 and the standard deviation of the partial loudness of the reference audio 142. The threshold parameter ratio_threshold acts as an upper limit for the deviation ratio. A typical value for the threshold parameter is 8; this value is equal to or less than the std out,t / std ref,t If is large enough, std r,t , the ratio std out,t / std ref,t or increased to make it more sensitive to std r,t , the ratio std out,t / std ref,t can be reduced to make it robust to outliers. For example, the ratio std out,t / std ref,t If R is large but no artifacts are present, the threshold parameter ratio_threshold should be reduced.
[0074]
[0081] In 212, the continuity score of block t, con score is calculated according to equation (21):
[0075]
number
[0082] In equation (21), the function f3 is std dif,t ,std r,t ,boost score An example of f3 is given by equation (22):
[0076]
number
[0083] In other words, the continuity score is in the range of 0 and 1 due to the hyperbolic tangent function applied to positive numbers, and also increases upon increasing one or more components of the combination, such as the deviation delta, deviation ratio, and final boost score.
[0077]
[0084] In 214, the weight of the object energy obj score is calculated according to equation (23):
[0078]
number
[0085] In equation (23), f4 is the energy ratio r t (see equation (1)). An example of f4 is given by equation (24):
[0079]
number
[0086] In other words, the weight of the object energy obj score is in the range of 1 to about 1.25 due to the hyperbolic tangent function being applied to the squared values with a minimum value of zero, and the energy ratio r t increases as θ increases above 0.5. In short, a greater weight of object energy results in objects with more energy.
[0080]
[0087] In 216, the loudness weight of the rendered audio signal 138 is score First, the total loudness of the rendered audio signal 138, x out,t is calculated according to equation (25):
[0081]
number
[0088] In other words, total loudness x out,t is the partial loudness (x out,tc , see also equation (8.2)) over all channels C.
[0082]
[0089] Second, loudness weightingscore is calculated according to equation (26):
[0083]
number
[0090] In equation (26), the function f5 is the total loudness x out,t An example of f5 is given by equation (27):
[0084]
number
[0091] In other words, loudness weighting score is in the range of 0 to 1 due to the hyperbolic tangent function applied to positive numbers, and the total loudness x out,t increases as m increases. Thus, larger loudness weight scores result in larger values of loudness in the rendered audio signal 138.
[0085]
[0092] In 218, the detection score for the block is t is calculated according to equation (28):
[0086]
number
[0093] In other words, the detection score t is the continuity score con score (see also equation (21)), and the object energy weight obj score (see also equation (23)) and the loudness weight loud score (see also equation (26)). An example of f6 is given by equation (29):
[0087]
number
[0094] In other words, the detection score t is the continuity score, conscore and the object energy weight obj score and loudness weight loud score In general, the detection score increases as one or more of its components increases.
[0088]
[0095] At 220, the ratio of the total loudness of the rendered audio signal 138 (x out,tc,r ) and the ratio of the total loudness of the reference audio signal 142 (x ref,tc,r ) and the energy (x obj,it ) and the position (m i,t ) are smoothed, respectively. The smoothed ratio of the total loudness of the rendered audio signal 138 is x ^ out,tc,r and can be calculated according to formula (30.1):
[0089]
number
[0096] In equation (30.1), the ratio of the total loudness of the rendered audio signal 138 (x out,tc,r ) can be calculated according to formula (9.1).
[0090]
[0097] The smoothed ratio of the total loudness of the reference audio signal 142 is x ^ ref,tc,r and can be calculated according to formula (30.2):
[0091]
number
[0098] In equation (30.2), the ratio of the total loudness of the reference audio signal 142 (x ref,tc,r ) can be calculated according to formula (9.2).
[0092]
[0099] The smoothed ratio of each of the audio objects 142 is x ^ obj,it and can be calculated according to equation (30.3):
[0093]
number
[0100] In equation (30.3), the energy (x obj,it ) can be calculated according to formula (8.1).
[0094]
[0101] The smoothed position of each of the audio objects 142 is m ^ i,t and can be calculated according to equation (30.4):
[0095]
number
[0102] In equation (30.4), the position (m i,t ) can be calculated according to equation (6).
[0096]
[0103] In equations (30.1, 30.2, 30.3, 30.4), the values of each signal in the current block (t) are smoothed according to the smoothing parameter (μ) using the values of the previous block (t-1). The default value of the smoothing parameter is 0.5. The smoothing parameter can be adjusted as desired by a user of the audio content generator 100 (see FIG. 1), for example according to an evaluation of listening to the modified audio signal 150. If the evaluation indicates that the modified audio signal 150 is undesirable, eg, contains discontinuities, the smoothing parameter may be increased. If the evaluation indicates that the modified audio signal 150 is desirable, e.g., does not contain discontinuities, the smoothing parameter can be reduced to make the modified audio signal 150 more responsive to the current results of the bed generation and object extraction.
[0097]
[0104] The smoothed values calculated according to equations (30.1, 30.2, 30.3, 30.4) are used in calculating equations (10.1, 10.2, 10.3, 11) for the next block; see 208 above.
[0098]
[0105] Returning to FIG. 1, the adaptive post-processor 116 receives the detection scores 144, performs averaging and smoothing, and generates parameters 146 based on the detection scores 144. The adaptive post-processor 116 may operate on a block-by-block basis. To perform the averaging, the adaptive post-processor 116 calculates an average detection score for a given block t by averaging the detection scores of K preceding and K succeeding blocks according to the process detailed in TABLE 4.
[0099]
number
[0100] [Table 4]
[0106] In other words, in line 1, the average detection score is initialized to zero. In line 2, the block count i is looped from tK to t+K. In lines 3-4, the weights w are calculated, and the weights are decreased the further away from a given block t towards its predecessor or successor blocks. In line 4, the exponential function may be replaced by another function if desired; in general, the weight w decreases as dis increases. In line 5, the weights are applied to the detection score of each block, and the weighted detection scores are added together to produce the average detection score.
[0101]
[0107] In the process of TABLE 4, the parameter K is an adjustable value that can range from 1 to 15. Increasing K corresponds to increasing the continuity detection threshold and decreasing K corresponds to decreasing the discontinuity detection threshold. Values of K that work well can be 5 and 10. The adaptive post-processor 116 may start with a value of 5 and the user can evaluate the results to produce modified audio 150; if the results are not acceptable, the user can adjust K to 10 and evaluate the results.
[0102]
[0108] In essence, the adaptive post-processor 116 performs averaging to look at (i.e., consider) one or more blocks and identify discontinuities based on the detection score 144 .
[0103]
[0109] To perform the smoothing, the adaptive post-processor 116 may adjust the average detection score according to the process detailed in TABLE 5: TABLE 5
[0104] [Table 5]
[0110] Parameter a f and a l(Ael) is the smoothing parameter; their sum is 1.0. f Values for a can be in the range of 0.60 and 0.80; a value of 0.70 can work well. l The values for (AEL) may be in the range of 0.20 and 0.40; a value of 0.30 may work well. The user can evaluate the results of generating modified audio 150; if the results are unacceptable, the user can adjust the smoothing parameters and evaluate the results.
[0105]
[0111] In other words, in lines 1-2, if the average detection score of the current block is greater than or equal to the average detection score of the previous block, the average detection score of the current block is adjusted slightly towards the average detection score of the previous block, e.g., decreased. In lines 3-4, if the average detection score of the current block is less than the average detection score of the previous block, the average detection score of the current block is adjusted slightly towards the average detection score of the previous block, e.g., increased.
[0106]
[0112] In essence, the adaptive post-processor 116 performs smoothing to reduce the change in detection score between successive blocks and to reduce the alarm rate threshold, at the expense of increasing the false alarm rate, making the system more sensitive to discontinuity detection.
[0107]
[0113] The signal modifier 118 receives the channel based audio signal 130, the bed channel 132, the audio object 134, and the parameters 146, performs signal modification, and generates a modified audio signal 150 based on the channel based audio signal 130, the bed channel 132, the audio object 134, and the parameters 146. The modified audio signal 150 includes a modified audio object and a modified bed channel. The modified audio object corresponds to the audio object 134 modified according to the parameters 146. The modified bed channel corresponds to the bed channel 132 modified according to the parameters 146. The modified audio signal 150 may also include metadata 136. The signal modifier 118 may modify the input as follows:
[0108]
[0114] First, the signal modifier 118 adjusts the mixing parameter wetdry t is calculated according to equation (31):
[0109]
number
[0115] Average detection score score ^ t is calculated by the adaptive post-processor 116 described above. In other words, the mixing parameter wetdry t acts as a crossfade or mix between the original input, e.g., the channel-based audio signal 130, and the extracted signals, e.g., the audio object 134 and the bed channel 132. The mixing parameters range from 0 (e.g., bypass) to 1 (e.g., applying the full contribution of the extracted audio object 134 and the bed channel 132).
[0110]
[0116] The signal modifier 118 modifies the extracted audio object 134 according to equation (32):
[0111]
number
[0117] The signal modifier 118 modifies the bed channel 132 differently depending on which channel is being modified. For left, right and center channels (j=L,R,C), the signal modifier 118 performs the modification of the bed channel 132 according to equation (33.1):
[0112]
number
[0118] For the left and left rear surround channels (j=Lss, Lrs) the signal modifier 118 performs a modification of the bed channel 132 according to equation (33.2):
[0113]
number
[0119] For the right and right rear surround channels (j=Rss, Rrs) the signal modifier 118 performs a modification of the bed channel 132 according to equation (33.3):
[0114]
number
[0120] In other words, the signal modifier 118 cross-fades the extracted signal, e.g. the bed channel 132 or the audio object 134, with the original signal, e.g. the channel-based audio signal 130, using the mixing parameters to generate a modified audio signal 150.
[0115]
[0121] 4 is a device architecture 400 for implementing the features and processes described herein, according to an embodiment. The architecture 400 may be implemented in any electronic device, including, but not limited to, desktop computers, consumer audio / visual (AV) devices, wireless broadcast devices, mobile devices, such as smartphones, tablet computers, laptop computers, wearable devices, and the like. In the illustrated exemplary embodiment, the architecture 400 is for a laptop computer and includes a processor 401, a peripherals interface 402, an audio subsystem 403, a speaker 404, a microphone 405, sensors 406 (e.g., accelerometer, gyro, barometer, magnetometer, camera, and the like), a position processor 407 (e.g., GNSS receiver, and the like), a wireless communication subsystem 408 (e.g., Wi-Fi, Bluetooth, cellular, and the like), and an I / O subsystem 409 (including touch controller 410 and other input controllers 411, touch surface 412, and other input / control devices 413). Other architectures having more or fewer components may also be used to implement the disclosed embodiments.
[0116]
[0122] Memory interface 414 is coupled to processor 401, peripherals interface 402, and memory 415 (e.g., Flash, RAM, ROM, etc.). Memory 415 stores computer program instructions and data, including, but not limited to, operating system instructions 416, communications instructions 417, GUI instructions 418, sensor processing instructions 419, telephony instructions 420, electronic messaging instructions 421, web browsing instructions 422, audio processing instructions 423, GNSS / navigation instructions 424, and applications / data 425. Audio processing instructions 423 include instructions for performing the audio processing described herein.
[0117]
[0123] According to an embodiment, the architecture 400 may correspond to a PC or laptop computer that an audio engineer might use to generate a modified audio signal 150 from a channel-based audio signal 130 (see FIG. 1).
[0118]
[0124] Figure 5 is a flow chart of a method 500 of audio processing. The method 500 may be performed by a device (e.g., a laptop computer, a mobile phone, etc.) that includes the components of the architecture 400 of Figure 4 to perform functions such as the audio content generator 100 (see Figure 1) by executing one or more computer programs.
[0119]
[0125] A channel-based audio signal is received at 502. For example, the audio content generator 100 (see FIG. 1) may receive the channel-based audio signal 130, for example from storage in the memory 415 (see FIG. 4).
[0120]
[0126] A reference audio signal is generated based on the channel-based audio signal at 504. For example, the renderer 112 (see FIG. 1) may generate the reference audio signal 142 based on the channel-based audio signal 130.
[0121]
[0127] At 506, an audio object and a bed channel are generated based on the channel-based audio signal. For example, based on the channel-based audio signal 130, the bed generator 102 (see FIG. 1) can generate the bed channel 132 and the object generator 104 can generate the audio object 134.
[0122]
[0128] At 508, a rendered audio signal is generated based on the audio objects and the bed channel. For example, the renderer 108 (see FIG. 1) can generate the rendered audio signal 138 based on the audio objects 134 and the bed channel 132. The renderer 108 can also use the metadata 136 when generating the rendered audio signal 138.
[0123]
[0129] At 510, a detection score is generated based on partial loudness of a number of signals, where the number of signals includes a reference audio signal, an audio object, a bed channel, a rendered audio signal, and a channel-based audio signal. The detection score is indicative of audio artifacts in one or more of the plurality of audio objects and the plurality of bed channels. For example, the controller 114 (see FIG. 1 ) may generate the detection score 144 based on partial loudness of the reference audio signal 142, the audio object 134, the bed channel 132, the rendered audio signal 138, and the channel-based audio signal 130. When generating the detection score 144, the controller 114 may perform one or more sub-steps, including one or more of the steps shown in the method 200 of FIG. 2 .
[0124]
[0130] At 512, parameters are generated based on the detection scores. For example, the adaptive post-processing processor 116 (see FIG. 1) can generate the parameters 146 based on the detection scores 144. The adaptive post-processing processor 116 operates on a block-by-block basis and may include an adjustable threshold for looking at blocks before and after the current block (i.e., defining the range of considerations) when generating the parameters.
[0125]
[0131] At 514, a modified audio object and a modified bed channel are generated based on the channel-based audio signal, the audio object, the bed channel, and the parameters. For example, the signal modifier 118 (see FIG. 1) can generate a modified audio signal 150 including, for example, a modified audio object and a modified bed channel based on the channel-based audio signal 130, the audio object 134, the bed channel 132, and the parameters 146. The signal modifier 118 may include mixing parameters that act as a cross-fade between the original input (e.g., the channel-based audio signal 130) and the extracted signal (e.g., the audio object 134 and the bed channel 132).
[0126]
[0132] The modified audio signal 150 can then be stored in the device's memory, such as in a solid state memory, transmitted to another device, for example for cloud storage, rendered into an audio presentation, and output as sound, for example using one or more speakers.
[0127]
[0133] Method 500 may include additional steps corresponding to other functions of audio content generator 100, etc., as described herein.
[0128]
[0134] Implementation details
[0135] The embodiments may be implemented in hardware, executable modules stored on a computer-readable medium, or a combination of both, such as programmable logic arrays. Unless otherwise specified, the steps performed by the embodiments are not necessarily related in nature to any particular computer or other apparatus, although they may be in a particular embodiment. In particular, various general-purpose machines may be used with programs written in accordance with the teachings herein, or it may be more convenient to construct more specialized apparatus, such as integrated circuits, to perform the required method steps. Thus, the embodiments may be implemented in one or more computer programs operating on one or more programmable computer systems, each including at least one processor, at least one data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device or port, and at least one output device or port. The program code is applied to the input data to perform the functions described herein and to generate output information. The output information is applied to one or more output devices, in a known manner.
[0129]
[0136] Each such computer program is preferably stored or downloaded onto a general-purpose or dedicated programmable computer-readable storage medium or device (e.g., solid state memory or medium, magnetic or optical medium, etc.) to configure and operate a computer when the storage medium or device is read by a computer system to execute the procedures described herein. The system of the present invention may be envisioned as being embodied as a computer-readable storage medium configured with a computer program, in which case the storage medium so configured causes the computer system to operate in a specific, predetermined manner to perform the functions described herein. Software itself, but intangible or transitory signals, are excluded insofar as they are within the scope of non-patentable subject matter.
[0130]
[0137] Aspects of the system described herein may be implemented in any suitable computer-based sound processing network environment for processing digital or digitized audio files. Part of an adaptive audio system may include one or more networks including any desired number of individual machines, including one or more routers (not shown) that function to buffer and route data transmitted between the computers. Such networks may be constructed in a variety of different network protocols and may be the Internet, a wide area network (WAN), a local area network (LAN), or any combination thereof.
[0131]
[0138] One or more of the components, blocks, processes, or other functional elements may be implemented via a computer program that controls the execution of a processor-based computing device of the system. It should also be noted that the various functions disclosed herein may be described using any number of combinations of hardware, firmware, and / or as data and / or instructions embodied in various machine-readable or computer-readable media in terms of their operations, register transfers, logical components, and / or other characteristics. Computer-readable media in which such formatted data and / or instructions may be embodied include, but are not limited to, physical, non-transitory, non-volatile storage media in various forms, such as optical, magnetic, or semiconductor storage media.
[0132]
[0139] The above description illustrates various embodiments of the present disclosure, along with examples of how aspects of the disclosure may be implemented. The above examples and embodiments should not be considered as being the only embodiments, but are presented to illustrate the flexibility and advantages of the present disclosure as defined by the following claims. Based on the above disclosure and claims, other arrangements, embodiments, implementations and equivalents will be apparent to those skilled in the art and may be used without departing from the spirit and scope of the disclosure as defined by the claims.
[0133]
[0140] Various aspects of the present invention can be understood with reference to the enumerated example embodiments (EEEs) set forth below.
[0134] (EEE1) 1. A computer implemented method for audio processing comprising: receiving a channel-based audio signal; generating a reference audio signal based on the channel-based audio signal; generating a plurality of audio objects and a plurality of bed channels based on the channel-based audio signal; generating a rendered audio signal based on the plurality of audio objects and the plurality of bed channels; generating a detection score based on a plurality of partial loudnesses of a plurality of signals, the plurality of signals including a reference audio signal, a plurality of audio objects, a plurality of bed channels, a rendered audio signal, and a channel-based audio signal, the detection score being indicative of audio artifacts in one or more of the plurality of audio objects and the plurality of bed channels; generating a plurality of parameters based on the detection scores; and The method includes generating a plurality of modified audio objects and a plurality of modified bed channels based on the channel-based audio signal, the plurality of audio objects, the plurality of bed channels, and the plurality of parameters.
[0135] (EEE2) The computer implemented method of EEE1 further includes outputting, via one or more speakers, a representation of the plurality of modified audio objects and the plurality of modified bed channels as sound.
[0136] (EEE3) In the computer-implemented method of any one of EEE1-2, the channel-based audio signal includes a plurality of blocks, a given block of the plurality of blocks includes a plurality of samples, and a detection score is generated for each block for the plurality of blocks.
[0137] (EEE4) 4. The computer implemented method of any one of claims EEE1-3, wherein generating a detection score comprises calculating a plurality of partial loudnesses; The plurality of partial loudnesses include a partial loudness of a reference audio signal, a partial loudness of a plurality of audio objects, a partial loudness of a plurality of bed channels, a partial loudness of a rendered audio signal, and a partial loudness of a channel-based audio signal.
[0138] (EEE5) The computer implemented method of any one of EEE1-4, wherein generating the detection score comprises calculating a ratio between the first energy and the second energy; The first energy is the energy of the multiple audio objects, the second energy is the sum of the energies of the multiple audio objects and the energies of the multiple bed channels, A detection score is generated based on a ratio of the first energy to the second energy.
[0139] (EEE6) 6. The computer implemented method of any one of claims 8 to 10, wherein generating a detection score comprises calculating an average position for each of a plurality of audio objects; A detection score is generated based on the average position for each of the multiple audio objects.
[0140] (EEE7) The computer implemented method of any one of EEE1-6, wherein generating a detection score comprises: calculating a plurality of boost scores based on a plurality of partial loudnesses, the plurality of partial loudnesses including a partial loudness of a channel-based audio signal, a partial loudness of a reference audio signal, a partial loudness of a plurality of audio objects, and a partial loudness of a rendered audio signal; and calculating a final boost score based on a sum of a largest of the plurality of boost scores and a next largest of the plurality of boost scores; A detection score is generated based on the final boost score.
[0141] (EEE8) A computer implemented method of claim 8, wherein a given boost score of the plurality of boost scores comprises a product of a first value, a second value, and a third value; The first value is the correlation of partial loudness between multiple channels of a given signal, The second value is a measure of the energy change in multiple channels of a given signal between adjacent blocks; The third value is a difference score between the loudness ratios of the channels of a given signal.
[0142] (EEE9) The computer implemented method of any one of EEE1-8, wherein generating a detection score comprises calculating a plurality of deviation metrics between the partial loudness of the rendered audio signal and the partial loudness of a reference audio signal; The plurality of deviation metrics include a deviation differential and a deviation ratio; the deviation difference is the difference between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; The detection score is generated based on a number of deviation metrics.
[0143] (EEE10) In the computer implemented method of EEE9, a detection score is generated based on a hyperbolic tangent function applied to the product of the deviation difference and the deviation ratio.
[0144] (EEE11)
[0023] The computer implemented method of any one of claims 8 to 10, wherein generating the detection score comprises calculating a continuity score based on the deviation difference, the deviation ratio, and the boost score; the deviation difference is the difference between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the boost score is based on the partial loudness of the channel-based audio signal, the partial loudness of the reference audio signal, the partial loudness of the multiple audio objects, and the partial loudness of the rendered audio signal; A detection score is generated based on the continuity score.
[0145] (EEE12) In the computer implemented method of EEE11, a detection score is generated based on a hyperbolic tangent function applied to the sum of a first value and a second value, the first value being the product of the deviation difference and the deviation ratio, and the second value being a continuity score.
[0146] (EEE13) The computer implemented method of any one of EEE1-12, wherein generating the detection score comprises calculating a weight for the object energy based on a ratio of the first energy and the second energy; The first energy is the energy of multiple audio objects, The second energy is the sum of the energies of the audio objects and the bed channels. A detection score is generated based on the object energy weights.
[0147] (EEE14) In the EEE13 computer implemented method, a detection score is generated based on a hyperbolic tangent function applied to object energy weights.
[0148] (EEE15) 5. The computer implemented method of claim 1, wherein generating a detection score comprises calculating a loudness weight of the partial loudness of the rendered audio signal; The loudness weight increases as the partial loudness of the rendered audio signal increases. A detection score is generated based on the loudness weighting.
[0149] (EEE16) The computer implemented method of any one of EEE1-15, wherein generating a detection score comprises: calculating a continuity score based on the deviation difference, the deviation ratio, and the boost score; calculating an object energy weight based on a ratio between a first energy and a second energy, the first energy being an energy of a plurality of audio objects, and the second energy being a sum of an energy of the plurality of audio objects and an energy of a plurality of bed channels; and calculating a loudness weight of a partial loudness of the rendered audio signal, the loudness weight increasing as the partial loudness of the rendered audio signal increases; the deviation difference being the difference between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the boost score is based on the partial loudness of the channel-based audio signal, the partial loudness of the reference audio signal, the partial loudness of the multiple audio objects, and the partial loudness of the rendered audio signal; The detection score is generated based on the continuity score, the object energy weight and the loudness weight.
[0150] (EEE17) The computer implemented method of any one of EEE1-16, wherein generating the detection score comprises smoothing a ratio of a total loudness of the rendered audio signal, a ratio of a total loudness of the reference audio signal, an energy of each of the plurality of audio objects, and a position of each of the plurality of audio objects; wherein the detection score is generated based on a smoothed ratio of a total loudness of the rendered audio signal, a smoothed ratio of a total loudness of the reference audio signal, a smoothed energy of each of the plurality of audio objects, and a smoothed position of each of the plurality of audio objects.
[0151] (EEE18) A non-transitory computer readable storage medium stores a computer program which, when executed by a processor, controls an apparatus to perform processes including the methods of any one of EEE1-17.
[0152] (EEE19) The audio processing apparatus includes a processor, the processor being configured to control the apparatus to perform processes including the methods of any one of claims EEE1-17.
[0153] (EEE20) The apparatus of EEE19 further includes one or more speakers configured to output a representation of the plurality of modified audio objects and the plurality of modified bed channels. [Prior art documents] [Patent documents]
[0154] [Patent Document 1] U.S. Patent No. 9,756,445 [Patent Document 2] U.S. Pat. No. 9,794,718 [Patent Document 3] U.S. Pat. No. 9,165,558 [Patent Document 4] U.S. Pat. No. 10,275,685 [Patent Document 5] U.S. Patent No. 6,167,404 [Patent Document 6] US Patent Application Publication No. 2020 / 0322743 [Patent Document 7] US Patent Application Publication No. 2017 / 0098452 [Patent Document 8] US Patent Application Publication No. 2020 / 0126570 [Non-patent literature]
[0155] [Non-Patent Document 1] Philip Coleman, Andreas Franck, Jon Francombe, Qingju Liu, Teofilo de Campos, Richard J. Hughes, Dylan Menzies, Marcos F. Simon Galvez, Yan Tang, James Woodcock, Philip JB Jackson, Frank Melchior, Chris Pike, Filippo M. Fazi, Trevor J. Cox and Adrian Hilton, An Audio-Visual System for Object-Based Audio: From Recording to Listening, in IEEE Transactions on Multimedia (Volume: 20, Issue: 8, Aug. 2018), DOI: 10.1109 / TMM.2018.2794780. [Non-Patent Document 2] Benjamin Guy Shirley, Improving Television Sound for People with Hearing Impairments, PhD Thesis, University of Salford (2013), DOI: 10.13140 / 2.1.3823.4881. [Non-Patent Document 3] Joao Martins, Object-Based Audio and Sound Reproduction (April 26, 2018), available at <audioxpress.com / article / object-based-audio-and-sound-reproduction>.
Claims
1. 1. A computer implemented method for audio processing comprising: receiving a channel-based audio signal; generating a reference audio signal based on the channel-based audio signal; generating a plurality of audio objects and a plurality of bed channels based on the channel-based audio signal; generating a rendered audio signal based on the plurality of audio objects and the plurality of bed channels; generating a detection score based on a plurality of partial loudnesses of a plurality of signals, the plurality of signals including the reference audio signal, the plurality of audio objects, the plurality of bed channels, the rendered audio signal, and the channel-based audio signal, the detection score being indicative of audio artifacts in one or more of the plurality of audio objects and the plurality of bed channels; generating a plurality of parameters based on the detection scores; and generating a plurality of modified audio objects and a plurality of modified bed channels based on the channel-based audio signal, the plurality of audio objects, the plurality of bed channels, and the plurality of parameters; A computer-implemented method comprising:
2. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises calculating the plurality of partial loudnesses; The method of claim 1, wherein the multiple partial loudnesses include a partial loudness of the reference audio signal, a partial loudness of the multiple audio objects, a partial loudness of the multiple bed channels, a partial loudness of the rendered audio signal, and a partial loudness of the channel-based audio signal.
3. 2. The computer implemented method of claim 1, wherein generating the detection score comprises calculating a ratio between a first energy and a second energy; the first energy is an energy of the plurality of audio objects, and the second energy is a sum of an energy of the plurality of audio objects and an energy of the plurality of bed channels; The method of claim 1, wherein the detection score is generated based on a ratio of the first energy and the second energy.
4. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises calculating an average position for each of the plurality of audio objects; The method of claim 1, wherein the detection score is generated based on an average position for each of the plurality of audio objects.
5. 2. The computer implemented method of claim 1, wherein generating the detection score comprises: calculating a plurality of boost scores based on the plurality of partial loudnesses, the plurality of partial loudnesses including a partial loudness of the channel-based audio signal, a partial loudness of the reference audio signal, a partial loudness of the plurality of audio objects, and a partial loudness of the rendered audio signal; and calculating a final boost score based on the sum of the largest of the plurality of boost scores and the next largest of the plurality of boost scores; wherein the detection score is generated based on the final boost score.
6. 6. The computer-implemented method of claim 5, wherein a given boost score of the plurality of boost scores comprises a product of a first value, a second value, and a third value; the first value being a correlation of partial loudness between multiple channels of a given signal; the second value being a measure of the energy variation in multiple channels of a given signal between adjacent blocks; The method of claim 1, wherein the third value is a difference score between a plurality of loudness ratios of a plurality of channels of the given signal.
7. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises calculating a plurality of deviation metrics between the partial loudness of the rendered audio signal and the partial loudness of the reference audio signal; the plurality of deviation metrics include a deviation difference and a deviation ratio; the deviation difference being the difference between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between a standard deviation of the partial loudness of the rendered audio signal and a standard deviation of the partial loudness of the reference audio signal; The method of claim 1, wherein the detection score is generated based on the plurality of deviation metrics.
8. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises calculating a continuity score based on a deviation difference, a deviation ratio, and a boost score; the deviation difference being the difference between the standard deviation of the partial loudness of the rendered audio signal and the standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between a standard deviation of the partial loudness of the rendered audio signal and a standard deviation of the partial loudness of the reference audio signal; the boost score is based on a partial loudness of the channel-based audio signal, a partial loudness of the reference audio signal, a partial loudness of the plurality of audio objects, and a partial loudness of the rendered audio signal; The method, wherein the detection score is generated based on the continuity score.
9. 9. The computer-implemented method of claim 8, wherein the detection score is generated based on a hyperbolic tangent function applied to a sum of a first value and a second value, the first value being a product of the deviation difference and the deviation ratio, and the second value being the continuity score.
10. 2. The computer implemented method of claim 1, wherein generating the detection score comprises calculating a weight for an object energy based on a ratio of a first energy to a second energy; the first energy being an energy of the plurality of audio objects; the second energy being a sum of energies of the audio objects and energies of the bed channels; The method of claim 1, wherein the detection score is generated based on a weight of the object energy.
11. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises calculating a loudness weight of a partial loudness of the rendered audio signal; the loudness weight increases as the partial loudness of the rendered audio signal increases; The method of claim 1, wherein the detection score is generated based on the loudness weight.
12. 2. The computer implemented method of claim 1, wherein generating the detection score comprises: calculating a continuity score based on the deviation difference, the deviation ratio, and the boost score; calculating an object energy weight based on a ratio between a first energy and a second energy, the first energy being the energy of the plurality of audio objects and the second energy being the sum of the energies of the plurality of audio objects and the energies of the plurality of bed channels; and calculating a loudness weight of a partial loudness of the rendered audio signal, the loudness weight increasing as the partial loudness of the rendered audio signal increases; wherein the deviation difference is a difference between a standard deviation of the partial loudness of the rendered audio signal and a standard deviation of the partial loudness of the reference audio signal; the deviation ratio is based on a ratio between a standard deviation of the partial loudness of the rendered audio signal and a standard deviation of the partial loudness of the reference audio signal; the boost score is based on a partial loudness of the channel-based audio signal, a partial loudness of the reference audio signal, a partial loudness of the plurality of audio objects, and a partial loudness of the rendered audio signal; A method according to claim 1, wherein the detection score is generated based on the continuity score, the object energy weight and the loudness weight.
13. 2. The computer-implemented method of claim 1, wherein generating the detection score comprises smoothing a ratio of a total loudness of the rendered audio signal, a ratio of a total loudness of the reference audio signal, an energy of each of the plurality of audio objects, and a position of each of the plurality of audio objects; wherein the detection score is generated based on a smoothed proportion of a total loudness of the rendered audio signal, a smoothed proportion of a total loudness of the reference audio signal, a smoothed energy of each of the plurality of audio objects, and a smoothed position of each of the plurality of audio objects.
14. A non-transitory computer-readable storage medium storing a computer program, the computer program being configured to, when executed by a processor, control an apparatus to perform processes including the method of any one of claims 1-13.
15. 14. An audio processing device comprising a processor, the processor being configured to control the device to perform processing comprising the method of any one of claims 1-13.
Citation Information
Patent Citations
Projection-based audio object extraction from audio content
US10275685B2
Method and system for audio processing of dialog, music, effect and height objects
US20170098452A1
Methods and systems for rendering object based audio
US20200126570A1
A method converting multichannel audio content into object-based audio content and a method for processing audio content having a spatial position
US20200322743A1
Multimedia plug-in using dynamic objects
US6167404A
