A method for accompaniment gain compensation based on streaming loudness matching

Abstract

The present application belongs to the technical field of digital audio processing, and particularly relates to a accompaniment gain compensation method based on streaming loudness matching. The method comprises: obtaining streaming accompaniment audio frames, extracting loudness difference value, peak safety margin and paragraph matching confidence, and obtaining compensation state parameters; determining prospective compensation trigger results and prior gain envelope according to the compensation state parameters; determining sub-band gain budget and generating candidate gain track according to the compensation state parameters and the prior gain envelope in the case of obtaining the prior gain envelope; determining target gain track according to the candidate gain track, and performing gain compensation and true peak protection on the streaming accompaniment audio frames according to the target gain track, to obtain compensated accompaniment audio stream. The present application can improve the real-time performance, adaptability and output stability of accompaniment loudness compensation, and reduce the risk of peak overrun and distortion.

Description

Technical Field

[0001] This invention relates to the field of digital audio processing technology, and specifically to a method for accompaniment gain compensation based on streaming loudness matching. Background Technology

[0002] Digital audio processing technology is widely used in audio-visual entertainment, online content distribution, in-vehicle multimedia playback, and accompaniment reconstruction. After track splitting, compression, transcoding, or link switching, accompaniment audio often experiences overall energy reduction, local frequency band imbalance, and changes in peak headroom. This directly affects playback continuity, auditory consistency, and the safety of the power amplifier. For systems that require continuous output of accompaniment content, sudden volume drops, inconsistent loudness between segments, inability to reuse compensation strategies for repeated segments, and transient segments easily triggering peak over-limits are typical problems affecting the actual experience and system stability.

[0003] Existing solutions often employ fixed-gain amplification, single-frame loudness correction, or simple limiting methods. These approaches tend to focus only on the static compensation requirements of the current segment, which can easily lead to problems such as insufficient compensation, excessive compensation, frequency band imbalance, peak distortion, and uneven transitions between adjacent segments. They also struggle to balance real-time performance, adaptability, and output stability. Summary of the Invention

[0004] This invention provides a streaming loudness matching-based accompaniment gain compensation method, which addresses at least the problem of how to balance real-time loudness compensation, adaptive segment reuse, and peak safety control in streaming accompaniment processing.

[0005] This invention provides a method for accompaniment gain compensation based on streaming loudness matching, the method comprising: Acquire streaming accompaniment audio frames, extract loudness difference, peak safety margin and segment matching confidence to obtain compensation state parameters; The prospective compensation triggering result is determined based on the compensation state parameters. When the prospective compensation triggering result meets the compensation conditions, the target compensation template is matched from the historical compensation template set based on the paragraph matching confidence to obtain the prior gain envelope. Given the prior gain envelope, the subband gain budget is determined based on the compensation state parameters and the prior gain envelope, and candidate gain trajectories are generated based on the subband gain budget. The target gain trajectory is determined based on the candidate gain trajectory, and gain compensation and true peak protection are performed on the streaming accompaniment audio frames based on the target gain trajectory to obtain the compensated accompaniment audio stream.

[0006] In one possible implementation, loudness difference, peak safety margin, and segment matching confidence are extracted to obtain compensation state parameters, including: determining the loudness difference based on the short-time loudness value of the streaming accompaniment audio frame and the target loudness; determining the peak safety margin based on the difference between the predicted peak and the peak limit threshold of the streaming accompaniment audio frame; determining the segment matching confidence based on the similarity between the streaming accompaniment audio frame and historical audio segments; the compensation state parameters also include transient intensity determined based on the short-time energy change rate of the streaming accompaniment audio frame, and sub-band masking margin determined based on the energy distribution of each sub-band of the streaming accompaniment audio frame.

[0007] In one possible implementation, the look-ahead compensation triggering result is determined based on the compensation state parameters, including: determining the triggering judgment value based on the loudness difference, transient intensity, and paragraph matching confidence; and determining that the look-ahead compensation triggering result satisfies the compensation conditions if the triggering judgment value is greater than the triggering threshold and the peak safety margin is greater than the safety threshold.

[0008] In one possible implementation, when the look-ahead compensation triggering result meets the compensation condition, a priori gain envelope is obtained by matching the target compensation template from the historical compensation template set based on the paragraph matching confidence. This includes: extracting paragraph description features from the streaming accompaniment audio frames; determining the target compensation template based on the matching degree between the paragraph description features and the historical paragraph description features corresponding to each historical compensation template in the historical compensation template set; and reading the gain envelope information corresponding to the target compensation template to obtain the priori gain envelope.

[0009] In one possible implementation, the historical compensation templates in the historical compensation template set include historical segment description features and gain envelope information. The historical segment description features are used to characterize the temporal energy distribution and spectral distribution of historical audio segments.

[0010] In one possible implementation, the subband gain budget is determined based on the compensation state parameters and the prior gain envelope, including: determining the compensation target gain for each subband based on the loudness difference; determining the perception allowable gain for each subband based on the subband masking margin; determining the peak limit gain for each subband based on the peak safety margin; determining the reference gain for each subband based on the prior gain envelope; and determining the minimum value among the compensation target gain, perception allowable gain, peak limit gain, and reference gain as the subband gain budget for the corresponding subband.

[0011] In one possible implementation, generating candidate gain trajectories includes: using the gain change trend corresponding to the prior gain envelope as a reference trajectory, generating multiple candidate gain trajectories with different gain change rates within the range limited by the subband gain budget.

[0012] In one possible implementation, determining the target gain trajectory based on candidate gain trajectories includes: determining the loudness recovery deviation, peak exceedance risk, and gain change rate corresponding to each candidate gain trajectory; determining the trajectory cost of each candidate gain trajectory based on the loudness recovery deviation, peak exceedance risk, and gain change rate; and determining the candidate gain trajectory with the minimum trajectory cost as the target gain trajectory.

[0013] In one possible implementation, gain compensation and true peak protection are performed on the streaming accompaniment audio frame to obtain a compensated accompaniment audio stream, including: adjusting the gain of the sub-band signal according to the target gain trajectory and performing sub-band synthesis to obtain a compensated audio frame; if the predicted peak value of the compensated audio frame is greater than the peak value limit threshold, the compensated audio frame is subjected to amplitude limiting processing to obtain a compensated accompaniment audio stream.

[0014] In one possible implementation, the method further includes: when a target compensation template is matched, determining the loudness restoration result and peak over-limit detection result based on the compensated accompaniment audio stream; correcting the gain envelope information corresponding to the target compensation template based on the loudness restoration result and peak over-limit detection result; and writing the corrected target compensation template back to the historical compensation template set.

[0015] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: By extracting loudness difference, peak safety margin, and segment matching confidence, a joint representation of the current accompaniment compensation requirement is achieved, simultaneously incorporating loudness deviation, peak constraints, and segment correlation into the compensation decision. By determining the look-ahead compensation trigger result based on compensation state parameters and calling historical compensation templates when compensation conditions are met, prior reuse of repeated and similar segments is achieved, reducing fluctuations caused by relying solely on real-time estimation of the current frame. By determining the sub-band gain budget based on compensation state parameters and prior gain envelope, frequency-band constraints on the sub-band compensation amplitude are achieved, ensuring the compensation process balances overall loudness recovery and local frequency band balance. By generating candidate gain trajectories and filtering target gain trajectories, comparative selection of different compensation rhythms is achieved, avoiding abrupt changes and mismatches caused by simply fixing the gain method. By performing true peak protection while performing gain compensation, peak overshoot and distortion are suppressed while increasing accompaniment loudness. Attached Figure Description

[0016] Figure 1 This is a schematic flowchart of the method of the present invention; Figure 2 This is a schematic diagram of short-time loudness comparison in a specific embodiment of the present invention; Figure 3 This is a schematic diagram of look-ahead triggering and paragraph matching in a specific embodiment of the present invention; Figure 4This is a thermal diagram illustrating the subband gain budget in a specific embodiment of the present invention; Figure 5 This is a schematic diagram comparing the envelope before and after template update in a specific embodiment of the present invention. Detailed Implementation

[0017] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and not intended to limit the scope of the invention. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present invention for ease of explanation.

[0018] Streaming loudness matching can be understood as a dynamic loudness control mechanism for continuous audio output. Its focus is not on offline unified correction of the entire audio file, but on online loudness consistency processing involving continuous input, analysis, and adjustment. This type of processing emphasizes generating compensation results synchronized with the playback process in real time, based on the energy distribution, temporal continuity, and output safety boundaries of the current audio segment under finite time delay conditions. This allows the output loudness to gradually converge to the target range while maintaining a natural transition between adjacent segments. For accompaniment audio, streaming loudness matching involves not only overall volume alignment but also the connection relationships between local segments, the use of historical references for repeated segments, and compensation rhythm control under peak constraints. Based on this, this invention proposes an accompaniment gain compensation scheme based on streaming loudness matching, focusing on continuous analysis, look-ahead triggering, template matching, sub-band budget allocation, candidate trajectory screening, and template write-back updates of streaming accompaniment data.

[0019] like Figure 1 As shown, a method for compensating accompaniment gain based on streaming loudness matching is proposed, which includes: Acquire streaming accompaniment audio frames, extract loudness difference, peak safety margin and segment matching confidence to obtain compensation state parameters; The accompaniment data stream obtained from track splitting is continuously framed to form streaming accompaniment audio frames arranged in chronological order. After each streaming accompaniment audio frame enters the analysis window, the audio processing unit calculates the short-time loudness value, predicted peak value, and segment matching confidence, and combines this with the target loudness, peak limit threshold, and historical audio segments to form compensation state parameters. These compensation state parameters characterize the loudness compensation requirement, peak safety space, and historical segment reuse possibility of the current streaming accompaniment audio frame, and are time-aligned with the current streaming accompaniment audio frame. After the compensation state parameters are written to the current analysis window, the process proceeds to determine the lookahead compensation trigger result, generate the prior gain envelope, and calculate the sub-band gain budget. When the current streaming accompaniment audio frame is an empty frame, a silent frame, or a significantly distorted frame, the audio processing unit records the corresponding frame's valid state and imposes value restrictions on the compensation state parameters to prevent abnormal inputs from directly affecting subsequent gain control.

[0020] The loudness difference, peak safety margin, and segment matching confidence are extracted to obtain compensation state parameters, including: determining the loudness difference based on the short-time loudness value of the streaming accompaniment audio frame and the target loudness; determining the peak safety margin based on the difference between the predicted peak value and the peak limit threshold of the streaming accompaniment audio frame; determining the segment matching confidence based on the similarity between the streaming accompaniment audio frame and historical audio segments; the compensation state parameters also include transient intensity determined based on the short-time energy change rate of the streaming accompaniment audio frame, and sub-band masking margin determined based on the energy distribution of each sub-band of the streaming accompaniment audio frame.

[0021] In one embodiment, the loudness difference, peak safety margin, and segment matching confidence are determined through the correspondence between the current streaming accompaniment audio frame and the reference object. The streaming accompaniment audio frames can be formed with a frame length of 20 to 40 milliseconds, and the analysis window can cover the current streaming accompaniment audio frame and several adjacent streaming accompaniment audio frames. The short-time loudness value can be obtained based on the statistical results of the sampled energy within the analysis window, or it can be obtained using a weighted loudness measurement method commonly used in the art. The target loudness can be preset according to the calibration output of the playback device, the user's volume level, and the accompaniment playback mode, or it can be corrected based on the loudness statistical values ​​of multiple analysis windows within the stable segment at the beginning of the song. The loudness difference can be expressed as: in, This is the loudness difference. For target loudness, This represents the short-time loudness value corresponding to the current streaming backing audio frame. The loudness difference is used to output the direction and magnitude of the deviation of the current streaming backing audio frame from the target loudness. When the loudness difference is positive, it indicates that the current streaming backing audio frame is lower than the target loudness, and there is a need for subsequent gain enhancement; when the loudness difference is zero or negative, it indicates that the current streaming backing audio frame is close to or higher than the target loudness, and subsequent gain compensation enters a hold or suppress state.

[0022] The predicted peak value can be determined based on the sampling peak value of the current streaming backing audio frame, the peak value variation trend of adjacent streaming backing audio frames, and the current gain estimate. The peak limit threshold can be set based on the digital full-amplitude upper limit, the digital-to-analog converter safety margin, and the power amplifier distortion tolerance. The peak safety margin can be expressed as: in, For peak safety margin, The peak value limit threshold is... This represents the predicted peak value for the current streaming backing track audio frame. The peak safety margin is used to output the gain space that the current streaming backing track audio frame can still use below the peak limit threshold. The larger the peak safety margin, the more gain the current streaming backing track audio frame still has; when the peak safety margin is close to zero or less than zero, it means that the current streaming backing track audio frame has approached or exceeded the peak limit boundary, and subsequent compensation needs to enter the clipping or gain suppression path.

[0023] Segment matching confidence is used to characterize the similarity between the current streaming accompaniment audio frame and historical audio segments. Historical audio segments are those that have been played and compensated within the same audio stream. The audio processing unit extracts current segment description features from the segment comparison window corresponding to the current streaming accompaniment audio frame and extracts historical segment description features from historical audio segments. Segment description features include temporal energy distribution and spectral distribution. Temporal energy distribution represents the energy fluctuation pattern within a local window, and spectral distribution represents the energy proportion of the main frequency sub-bands. Segment matching confidence can be expressed as: in, Match confidence scores to paragraphs. This represents the similarity in temporal energy distribution between the descriptive features of the current paragraph and the descriptive features of historical paragraphs. The similarity in spectral distribution between the descriptive features of the current paragraph and the descriptive features of historical paragraphs. This is the similarity weighting coefficient. The similarity weighting coefficient can be set based on the rhythmic intensity and frequency band stability of the accompaniment segment, with a value ranging from 0 to 1. Accompaniment segments with a strong rhythm can increase the weight corresponding to the temporal energy distribution, while sustained harmony accompaniment segments can increase the weight corresponding to the spectral distribution. A higher segment matching confidence indicates that the current streaming accompaniment audio frame is closer to historical audio segments, and the higher the reliability of matching the target compensation template from the historical compensation template set.

[0024] The compensation state parameters may also include transient intensity and sub-band masking margin. Transient intensity is determined based on the short-time energy change rate between the current streaming accompaniment audio frame and adjacent streaming accompaniment audio frames, and is used to characterize drum beats, strong onsets, or energy spikes. Sub-band masking margin is determined based on the energy distribution of each sub-band and the energy difference between adjacent sub-bands, and is used to characterize the allowable gain space for each sub-band without causing significant auditory abnormalities. After extraction, the audio processing unit checks the validity of the compensation state parameters; if the current streaming accompaniment audio frame is a silent frame, a discontinuous frame, or a significantly distorted frame, the segment matching confidence is limited to a low value, and the usage range of the peak safety margin and sub-band masking margin is tightened to ensure stable input for subsequent stages.

[0025] The prospective compensation triggering result is determined based on the compensation state parameters. When the prospective compensation triggering result meets the compensation conditions, the target compensation template is matched from the historical compensation template set based on the paragraph matching confidence to obtain the prior gain envelope. After receiving the compensation status parameters, the audio processing unit determines whether the current analysis window should enter the prior compensation path. The compensation status parameters include loudness difference, peak safety margin, and segment matching confidence, and may also incorporate transient intensity. Based on these parameters, the audio processing unit generates a look-ahead compensation trigger result, which indicates whether the current streaming accompaniment audio frame is suitable for calling the historical compensation template set. When the look-ahead compensation trigger result meets the compensation conditions, the audio processing unit retrieves the historical compensation template set based on the segment matching confidence, determines the target compensation template that matches the current streaming accompaniment audio frame, reads the gain envelope information from the target compensation template, and maps the gain envelope information to the current analysis window to generate a prior gain envelope. The prior gain envelope is time-aligned with the current streaming accompaniment audio frame and output to the sub-band gain budget calculation stage. When the look-ahead compensation trigger result does not meet the compensation conditions, the audio processing unit does not call the historical compensation template set, and the current analysis window does not enter the template-driven prior compensation path.

[0026] The look-ahead compensation triggering result is determined based on the compensation state parameters, including: determining the triggering judgment value based on the loudness difference, transient intensity, and paragraph matching confidence; and determining that the look-ahead compensation triggering result meets the compensation conditions when the triggering judgment value is greater than the triggering threshold and the peak safety margin is greater than the safety threshold.

[0027] In one embodiment, the look-ahead compensation triggering result is jointly determined by the trigger determination value and the peak safety margin, ensuring that the compensation path not only has insufficient response loudness but is also constrained by transient changes and the possibility of historical segment reuse. The trigger determination value can be determined according to the following relationship: in, To trigger the judgment value, The normalized loudness difference. The normalized transient intensity, Match confidence scores to paragraphs. Weights for loudness difference For transient intensity weights, Calculate confidence weights for paragraph matching.

[0028] The loudness difference weight, transient intensity weight, and segment matching confidence weight are all non-negative numbers and can be configured according to the accompaniment playback mode, track quality, and the stability of historical templates. When prioritizing rapid loudness recovery, the loudness difference weight can be set higher; when there are obvious drum beats, strong onsets, or short energy changes, the transient intensity weight can be increased; when the current segment has a strong repetition relationship with historical segments, the segment matching confidence weight can be increased.

[0029] The normalized loudness difference is obtained by amplitude limiting mapping of the loudness difference, and its value can be limited to the range of 0 to 1 to prevent extreme loudness differences from directly dominating the trigger decision. The normalized transient intensity is obtained by mapping the short-time energy change rate, and its value can also be limited to the range of 0 to 1. The trigger threshold is used to limit the minimum decision level for entering the prior compensation path, and can be set based on the number of false triggers, compensation response delay, and loudness fluctuation range under different song types in prototype testing. The safety threshold is used to limit the minimum allowable value of the peak safety margin, and can be set based on the peak limit threshold, the digital-to-analog converter safety margin, and the power amplifier distortion tolerance.

[0030] When the current trigger judgment value is greater than the trigger threshold and the peak safety margin is greater than the safety threshold, the look-ahead compensation trigger result meets the compensation conditions. If the trigger judgment value is high but the peak safety margin is insufficient, the audio processing unit will not enter the historical compensation template call path to avoid peak exceeding the limit caused by subsequent gain enhancement. If the trigger judgment value is lower than the trigger threshold, the current analysis window maintains the normal compensation judgment. To reduce path jitter caused by single-frame disturbances, the trigger results of multiple consecutive analysis windows can be stably confirmed, and the prior compensation path is only entered when the compensation conditions are continuously met.

[0031] When the look-ahead compensation trigger result meets the compensation conditions, the prior gain envelope is obtained by matching the target compensation template from the historical compensation template set based on the paragraph matching confidence. This includes: extracting paragraph description features from the streaming accompaniment audio frames; determining the target compensation template based on the matching degree between the paragraph description features and the historical paragraph description features corresponding to each historical compensation template in the historical compensation template set; and reading the gain envelope information corresponding to the target compensation template to obtain the prior gain envelope.

[0032] In one embodiment, the target compensation template is determined by the matching degree between the current segment description features and the historical segment description features. The audio processing unit establishes a segment comparison window based on the current streaming accompaniment audio frame. The segment comparison window covers the current streaming accompaniment audio frame and several adjacent streaming accompaniment audio frames, and its length can be set according to the frame length, buffer delay, and accompaniment beat period. When the segment comparison window is too short, the segment description features may only reflect the energy changes of a single frame; when the segment comparison window is too long, the template retrieval delay increases. Therefore, a window length that can cover the local rhythmic pattern can be selected within the allowable range of real-time processing delay.

[0033] The audio processing unit extracts segment description features from the segment comparison window. These features include temporal energy distribution and spectral distribution. The temporal energy distribution characterizes the peak and valley positions, rise intervals, fall intervals, and duration of energy within a local window; the spectral distribution characterizes the energy proportion of the main frequency sub-bands, the band centroid, and the band extension. Each historical compensation template in the historical compensation template set corresponds to a set of historical segment description features and a set of gain envelope information.

[0034] When matching templates, the degree of matching can be determined according to the following relationship: in, For matching degree, For temporal energy distribution similarity, For spectral distribution similarity, Weights are assigned to the temporal energy distribution. These weights can be set based on the type of accompaniment segment. Segments with a strong rhythm can have their temporal energy distribution weight increased, while sustained harmony segments can have their weight decreased and the influence of the spectral distribution increased accordingly.

[0035] After calculating the matching degree, the audio processing unit selects the historical compensation template with the highest matching degree that meets the template call threshold as the target compensation template. The template call threshold can be set based on the number of historical compensation templates, track quality, mismatch tolerance, and system response latency. When there are many historical compensation templates, the template call threshold can be increased to prevent low-similarity templates from being called; when the track quality is poor, the template call threshold can be increased, and checks on spectral distribution consistency can be added. If the highest matching degree does not meet the template call threshold, the audio processing unit does not call the historical compensation template set to prevent erroneous gain envelopes from entering subsequent budget calculations.

[0036] After the target compensation template is determined, the audio processing unit reads the gain envelope information from the target compensation template and maps the gain envelope information to the current analysis window based on the alignment between the current segment comparison window and the historical segment window, forming a priori gain envelope. If the length of the gain envelope information exceeds the current analysis window, the corresponding interval is truncated; if the length of the gain envelope information is shorter than the current analysis window, it is supplemented by a smooth extension method to ensure that the subsequent subband gain budget receives continuous input.

[0037] The historical compensation template set includes historical segment description features and gain envelope information. The historical segment description features are used to characterize the temporal energy distribution and spectral distribution of historical audio segments.

[0038] In one embodiment, the historical compensation template set is organized according to the segment characteristics of historical audio segments, giving clear boundaries to template retrieval, template reading, and template maintenance. The historical compensation template set can be grouped and stored according to song identifier, accompaniment version identifier, and audio stream session identifier. Historical compensation templates within the same group come from the same accompaniment data stream or the same track version, in order to avoid mixing templates formed under different songs or different track qualities.

[0039] Each historical compensation template includes historical segment description features and gain envelope information. The historical segment description features characterize the temporal energy and spectral distribution of the historical audio segment, and can consist of the energy sequence within the historical segment comparison window, the energy proportion of the main frequency sub-bands, and the results of frequency band centroid changes. The gain envelope information records the gain change trajectory formed by the historical audio segment during the compensation process, and can include the envelope start position, envelope duration interval, envelope control points, and the gain value corresponding to each envelope control point.

[0040] Before writing historical compensation templates, the audio processing unit checks historical audio segments for abnormal silences, obvious pops, track remnants, or peak values ​​exceeding limits. If any of these abnormalities are found, the corresponding segment is not written into the historical compensation template set to prevent abnormal envelopes from entering subsequent retrieval paths. The historical compensation template set can have a capacity limit, which is set based on storage space, retrieval latency, and the number of repeatable segments per song. When the number of historical compensation templates reaches the capacity limit, historical compensation templates that have not been used for a long time, whose matching degree has been consistently below the template call threshold, or whose gain envelope information is unstable can be deleted, while the most recently used historical compensation templates with stable compensation results are retained.

[0041] During template retrieval, the audio processing unit can first perform initial screening based on the spectral distribution, and then calculate the matching degree based on the temporal energy distribution to reduce real-time retrieval overhead. After the target compensation template is invoked, the number of invocations, the most recent invocation time, and the loudness recovery result after the invocation can be recorded to provide a basis for subsequent gain envelope information correction. Through the above organization method, the historical compensation template set can support template reuse for repeated segments and near-repeated segments, and maintain a low retrieval load under streaming processing conditions.

[0042] Given the prior gain envelope, the subband gain budget is determined based on the compensation state parameters and the prior gain envelope, and candidate gain trajectories are generated based on the subband gain budget. After receiving the compensation state parameters and the prior gain envelope, the audio processing unit divides the current streaming accompaniment audio frame into multiple sub-bands and calculates the allowable compensation range within each sub-band. The compensation state parameters provide the loudness compensation requirements, peak safety margin, and sub-band perceived boundaries, while the prior gain envelope provides historical compensation trends or the reference direction of change for the current analysis window. Based on the above data, the audio processing unit determines the sub-band gain budget for each sub-band, which limits the maximum compensation amount that can be performed on each sub-band within the current analysis window. After completing the sub-band gain budget, the audio processing unit constructs multiple candidate gain trajectories within the budget limit, referencing the gain change trend of the prior gain envelope. The candidate gain trajectories are time-aligned with the current streaming accompaniment audio frame and then proceed to the target gain trajectory selection stage.

[0043] The subband gain budget is determined based on the compensation state parameters and the prior gain envelope, including: determining the compensation target gain of each subband based on the loudness difference; determining the perception allowable gain of each subband based on the subband masking margin; determining the peak limit gain of each subband based on the peak safety margin; determining the reference gain of each subband based on the prior gain envelope; and determining the minimum value among the compensation target gain, perception allowable gain, peak limit gain, and reference gain as the subband gain budget for the corresponding subband.

[0044] In one embodiment, the process of forming the sub-band gain budget is refined. The audio processing unit can use a filter bank or short-time frequency domain analysis to divide the current streaming accompaniment audio frame into multiple sub-bands. The number of sub-bands can be set according to the processor load, frequency resolution, and real-time processing latency, for example, from 8 to 32.

[0045] Each sub-band corresponds to a current energy value, an allowed upper limit of gain, and a reference gain value. The loudness difference is used to determine the compensation target gain, which represents the amount of gain that the corresponding sub-band theoretically needs to add to make the current streaming accompaniment audio frame approach the target loudness, without considering other constraints. Since different sub-bands contribute differently to the overall loudness, the compensation target gain can be allocated based on the sub-band energy proportion, so that the main energy sub-bands undertake a larger loudness recovery task, avoiding applying excessively high gain to low-energy ineffective sub-bands.

[0046] Subband masking margin is used to determine the perceptually permissible gain, which represents the upper limit of gain allowed for the corresponding subband without causing significant auditory abrupt changes, frequency band harshness, or localized pressure. The subband masking margin can be determined based on the current subband energy, the energy difference between adjacent subbands, and the spectral stability of the current segment. Subbands with relatively stable energy and sufficient coverage by adjacent frequency bands can be given a larger perceptually permissible gain; subbands with isolated energy, significant abrupt changes, or located in sensitive frequency bands can be given a smaller perceptually permissible gain. Peak safety margin is used to determine the peak limit gain, which represents the upper limit of gain allowed for the corresponding subband without causing the synthesized audio peak to exceed the peak limit threshold. The smaller the peak safety margin, the lower the peak limit gain. Prior gain envelope is used to determine the reference gain, which represents the gain trend boundary given by the historical compensation template or current look-ahead judgment, and is not directly used as the executed gain.

[0047] The subband gain budget can be expressed as: in, For the first The first analysis window Subband gain budget for each subband For the first The first analysis window The compensation target gain of each sub-band For the first The first analysis window Perceptual allowable gain of each sub-band For the first The first analysis window Peak-limited gain of each subband For the first The first analysis window Reference gain of each subband For analysis window number, The sub-items are numbered.

[0048] This expression is used to compress compensation requirements, perceived boundaries, peak safety boundaries, and prior trend boundaries into a single budget result. After budget calculation, the audio processing unit can perform adjacent-band smoothing on the sub-band gain budget to avoid excessive differences between adjacent sub-band budgets. The adjacent-band smoothing range can be set based on the frequency response smoothing capability of the audio output device, the sensitivity of the human ear to frequency band jumps, and the spectral stability of the current song. If the energy of a certain sub-band is lower than the silence judgment threshold, the audio processing unit can limit the sub-band gain budget of that sub-band to a low value to avoid amplifying background noise or track separation residue in invalid frequency bands.

[0049] Generating candidate gain trajectories includes: using the gain change trend corresponding to the prior gain envelope as a reference trajectory, generating multiple candidate gain trajectories with different gain change rates within the range limited by the subband gain budget.

[0050] In one embodiment, the process of generating candidate gain trajectories is refined. After obtaining the subband gain budget for each subband, the audio processing unit maps the prior gain envelope to the current analysis window to obtain reference trajectories for each time step and each subband. The reference trajectory can exhibit forms such as a gentle rise, a short-term hold, a local fall, or stable maintenance, used to represent the desired direction of gain change within the current analysis window. Candidate gain trajectories are generated based on the reference trajectories, but each control point cannot exceed the subband gain budget of the corresponding subband.

[0051] To accommodate the compensation needs of different accompaniment segments, the audio processing unit can be configured with multiple gain change rate levels, such as low, medium, and high. The gain change rate represents the maximum allowable gain change between adjacent control points, and its setting is based on factors including frame length, playback buffer latency, the output device's sensitivity to gain abrupt changes, and the energy change rate of the accompaniment segment. The low gain change rate level is suitable for smooth segments, reducing perceived abrupt changes during compensation; the medium gain change rate level is suitable for typical accompaniment transition segments, balancing response speed and continuity; the high gain change rate level is suitable for segments with short-duration loudness drops and sufficient peak safety margin, but requires more stringent peak checks. During candidate gain trajectory generation, the audio processing unit constructs corresponding trajectories at each gain rate level, centered on a reference trajectory, and performs amplitude clipping and continuity checks on each trajectory.

[0052] Amplitude clipping ensures that the trajectory gain of any sub-band at any given time does not exceed the sub-band gain budget; continuity checking ensures that the gain change between adjacent times does not exceed the allowable rate of gain change for the current gear. If a local spike exists in the prior gain envelope and this spike exceeds the sub-band gain budget, the audio processing unit clips the local spike and sets transition segments before and after the spike to maintain the continuity of the candidate gain trajectories. If the difference between multiple candidate gain trajectories is below the trajectory discrimination threshold, the one with the lower rate of change can be retained to reduce the amount of subsequent filtering calculations. The trajectory discrimination threshold can be set according to the real-time processing load and the target gain trajectory filtering accuracy.

[0053] After generation, each candidate gain trajectory is written to the trajectory buffer and a time correspondence is established with the current streaming accompaniment audio frame. The trajectory buffer can be indexed by analysis window number, sub-band number, and rate of change level. Subsequent filtering stages can directly read the loudness recovery deviation, peak exceedance risk, and gain rate of change corresponding to each candidate gain trajectory. If the gain budget of all sub-bands within the current analysis window is lower than the minimum executable gain threshold, the audio processing unit does not generate high-rate-of-change candidate trajectories, but only retains low-rate-of-change trajectories or keeps the trajectory in place, avoiding unexecutable compensation schemes when the peak safety margin is insufficient. The minimum executable gain threshold can be set based on the gain control precision of the audio processing unit, the output device's sensitivity to minute gain changes, and the quantization error range of the playback link.

[0054] The target gain trajectory is determined based on the candidate gain trajectory, and gain compensation and true peak protection are performed on the streaming accompaniment audio frames based on the target gain trajectory to obtain the compensated accompaniment audio stream. The audio processing unit reads the candidate gain trajectories corresponding to the current analysis window from the trajectory buffer and performs prediction and evaluation on each candidate gain trajectory. The evaluation includes loudness recovery deviation, peak exceedance risk, and gain change rate after executing the candidate gain trajectory. Based on the evaluation results, the audio processing unit determines the trajectory cost of each candidate gain trajectory and identifies the candidate gain trajectory with the lowest trajectory cost as the target gain trajectory. After the target gain trajectory is determined, the audio processing unit adjusts the gain of the sub-band signal according to the target gain trajectory and performs sub-band synthesis to obtain a compensated audio frame. The compensated audio frame enters the true peak protection path. When the predicted peak value exceeds the peak limit threshold, amplitude limiting is performed, and the compensated accompaniment audio stream is output. If the current analysis window has matched the target compensation template, the compensated accompaniment audio stream also enters the template write-back path.

[0055] Determining the target gain trajectory based on candidate gain trajectories includes: determining the loudness recovery deviation, peak exceedance risk, and gain change rate corresponding to each candidate gain trajectory; determining the trajectory cost of each candidate gain trajectory based on the loudness recovery deviation, peak exceedance risk, and gain change rate; and determining the candidate gain trajectory with the minimum trajectory cost as the target gain trajectory.

[0056] In one embodiment, the selection of target gain trajectories is uniformly evaluated using trajectory cost, ensuring that loudness recovery, peak safety, and gain continuity are considered simultaneously in the same selection process. After candidate gain trajectories enter the evaluation window, the audio processing unit temporarily applies each candidate gain trajectory to the sub-band signal corresponding to the current analysis window, forming a predicted compensation result. Loudness recovery deviation is used to characterize the difference between the predicted compensation result and the target loudness; the smaller the difference, the closer the candidate gain trajectory is to the current loudness compensation requirement.

[0057] Peak exceedance risk characterizes the degree to which the predicted peak value of the compensation result approaches or exceeds the peak limit threshold. When the predicted peak value exceeds the peak limit threshold, the corresponding candidate gain trajectory is marked as a high-risk trajectory; when the predicted peak value does not exceed the peak limit threshold but the remaining safety margin is small, the corresponding candidate gain trajectory is marked as a restricted trajectory. Gain change rate characterizes how quickly the gain of the candidate gain trajectory changes between adjacent control points. If the change is too fast, it can easily cause abrupt changes in auditory perception; if the change is too slow, it may not be able to compensate for changes in accompaniment energy in time.

[0058] The trajectory cost can be expressed as: in, For the first Trajectory cost of candidate gain trajectories, The candidate gain trajectory number. For the first Loudness recovery bias corresponding to each candidate gain trajectory For the first Peak exceedance risk corresponding to each candidate gain trajectory For the first The rate of change of gain corresponding to each candidate gain trajectory Weighting for loudness recovery bias, As the peak exceedance risk weight, This is the weight for the rate of change of gain.

[0059] The weights for loudness recovery deviation, peak exceedance risk, and gain change rate can be configured based on the peak margin of the output link, the frequency of accompaniment switching, and the requirements for auditory smoothness. When the peak margin is small, the peak exceedance risk weight is higher than other weights; when the accompaniment mode switching is frequent, the loudness recovery deviation weight can be increased; when the output device is sensitive to gain abrupt changes, the gain change rate weight can be increased.

[0060] After calculating the trajectory cost for all candidate gain trajectories, the audio processing unit determines the candidate gain trajectory with the lowest trajectory cost as the target gain trajectory. If multiple candidate gain trajectories have the same trajectory cost, the candidate gain trajectory with the lower peak exceedance risk is selected first; if the peak exceedance risk is also the same, the candidate gain trajectory with the lower gain change rate is selected first. If all candidate gain trajectories are marked as high-risk trajectories, the audio processing unit does not use the original candidate results, but instead calls the hold-behind trajectory or the low change rate trajectory, so that the current analysis window can still output compensation results within the safety boundary.

[0061] Gain compensation and true peak protection are applied to the streaming accompaniment audio frame to obtain the compensated accompaniment audio stream. This includes: adjusting the gain of the sub-band signal according to the target gain trajectory and performing sub-band synthesis to obtain the compensated audio frame; and performing amplitude limiting on the compensated audio frame when the predicted peak value of the compensated audio frame is greater than the peak value limit threshold to obtain the compensated accompaniment audio stream.

[0062] In one embodiment, gain compensation and true peak protection are performed according to the sub-band execution path, enabling the target gain trajectory to be converted into actual audio output. After the sub-band signal corresponding to the current streaming accompaniment audio frame enters the execution window, the audio processing unit reads the target gain corresponding to each sub-band and control point in the target gain trajectory and performs band-by-band gain adjustment on each sub-band signal. When adjusting the sub-band signal gain, it is necessary to maintain the time alignment between the target gain trajectory and the current analysis window. If the energy of a sub-band is lower than the silence threshold within the current analysis window, the audio processing unit suppresses the target gain of that sub-band to avoid amplifying background noise, track remnants, or invalid frequency components. If a sub-band is close to the peak limit boundary, the audio processing unit prevents the target gain of that sub-band from increasing further, preventing the sub-band superposition from pushing the compensated audio frame beyond the peak limit threshold. After completing the band-by-band gain adjustment, the audio processing unit uses a reconstruction method matching the sub-band division to synthesize the sub-band signal, obtaining the compensated audio frame.

[0063] During sub-band synthesis, adjacent analysis windows can be overlapped or smoothly transitioned to maintain amplitude continuity and avoid abrupt changes at window boundaries. After the compensated audio frame is formed, the audio processing unit performs predicted peak detection on it. The predicted peak value represents the maximum peak level that the compensated audio frame may reach in the output link. The peak limit threshold can be set based on the digital full-amplitude upper limit, the digital-to-analog converter safety margin, and the power amplifier distortion tolerance, ensuring that the compensated audio frame retains necessary safety space before entering the subsequent playback link. When the predicted peak value exceeds the peak limit threshold, the audio processing unit enters the amplitude limiting processing path. The amplitude limiting processing determines the attenuation amount based on the magnitude by which the predicted peak value exceeds the peak limit threshold and applies controlled attenuation to the compensated audio frame, causing the peak value of the compensated audio frame to fall back to the allowable range.

[0064] If the predicted peak value only slightly exceeds the peak limit threshold, a smaller attenuation is used; if the predicted peak value significantly exceeds the peak limit threshold, a larger attenuation is used, and adjacent analysis windows are smoothly connected. When the predicted peak value does not exceed the peak limit threshold, the compensated audio frame is directly output as the compensated accompaniment audio stream. Before the compensated accompaniment audio stream is written to the playback buffer, a boundary continuity check can be performed; when the difference in boundary gain between the current analysis window and the previous analysis window exceeds the boundary correction range, the audio processing unit performs transition correction at the boundary. The boundary correction range can be set based on frame shift, playback buffer depth, and output device response characteristics.

[0065] The method further includes: when a target compensation template is matched, determining the loudness restoration result and peak over-limit detection result based on the compensated accompaniment audio stream; correcting the gain envelope information corresponding to the target compensation template based on the loudness restoration result and peak over-limit detection result; and writing the corrected target compensation template back to the historical compensation template set.

[0066] In one embodiment, the target compensation template is updated when the current analysis window already matches the target compensation template, allowing the historical compensation template set to absorb the results of this compensation execution. After the compensated accompaniment audio stream is output, the audio processing unit statistically analyzes the short-term loudness change and predicted peak value change within the readback interval corresponding to the current analysis window to obtain the loudness recovery result and the peak value exceeding the limit detection result.

[0067] Loudness recovery results indicate the degree of proximity between the compensated accompaniment audio stream and the target loudness. If the compensated accompaniment audio stream is still lower than the target loudness within the readback interval, it indicates insufficient compensation of the gain envelope information corresponding to the target compensation template; if the compensated accompaniment audio stream is significantly higher than the target loudness, it indicates excessive compensation of the gain envelope information corresponding to the target compensation template. Peak limit detection results indicate whether the compensated accompaniment audio stream approaches or exceeds the peak limit threshold within the readback interval. These can be categorized as no limit exceedance, close to limit exceedance, and already exceeded, or recorded according to the duration and magnitude of continuous limit exceedance. The audio processing unit maps the loudness recovery results and peak limit detection results to the gain envelope information in the target compensation template according to time alignment, and performs corrections in units of envelope control points.

[0068] For envelope control points with insufficient loudness recovery but no peak value exceeding the limit, the gain value can be appropriately increased; for envelope control points with excessive loudness recovery or peak value exceeding the limit, the gain value can be appropriately decreased; for envelope control points with loudness recovery meeting the set range and peak value remaining safe, the current gain value should be maintained. The correction range for a single envelope control point can be limited to the template correction range, which is set based on the dynamic margin of the vehicle audio link, the stability of historical compensation templates, and the allowable fluctuation range of accompaniment compensation. If abnormal silence, obvious popping, track separation errors, or continuous peak value exceeding the limit occur within the readback interval, the audio processing unit will pause the current template update, retain the original target compensation template, and avoid writing abnormal execution results into the historical compensation template set.

[0069] After the correction is completed, the audio processing unit writes the corrected target compensation template back to the historical compensation template set. The write-back method can be either an overwrite write-back or a version append write-back. When using version append write-back, the historical compensation template set maintains template priority according to the call time, matching degree, and most recent loudness recovery result, so that subsequent similar segments can read gain envelope information that is more consistent with the current audio stream state.

[0070] In one specific embodiment, a prototype in-vehicle entertainment system was selected as the execution platform. The audio input sampling rate was 48 kHz, the sampling bit width was 24 bits, the frame length of the streaming accompaniment audio frame was set to 20 milliseconds, and the frame shift was set to 10 milliseconds. For ease of illustration, 10 consecutive analysis windows were combined into one display point. The test track was a 12-second streaming accompaniment segment, with the first chorus segment from 3.0 seconds to 4.2 seconds and the second chorus segment from 7.2 seconds to 8.0 seconds. The second chorus segment was used as the verification interval for the historical compensation template reuse. The target loudness was set to -16.0 relative loudness units, and the peak limit threshold was set to -1.0 dB full scale. The historical compensation template set was established after the first chorus segment was completed, and the historical compensation template contained at least the historical segment description features and gain envelope information.

[0071] In the current embodiment, the system extracts compensation state parameters frame by frame from a 12-second streaming accompaniment segment. Display layer statistics show that the average short-time loudness of the track-by-track accompaniment is -22.25 relative loudness units, with an average absolute loudness error of 6.25; the average short-time loudness of the fixed-gain method is -16.75, with an average absolute error of 1.04 relative to the target loudness; and the average short-time loudness after processing with this invention is -16.61, with an average absolute error of 0.61 relative to the target loudness. Compared to the fixed-gain method, the average absolute error decreases by approximately 41.3%. Regarding peak safety, the fixed-gain method exhibits peak exceedance risk in all 121 display windows, with the maximum predicted peak reaching 2.22 dB full scale; the output of this invention, after true peak protection, has 0 peak exceedance windows, and the maximum predicted peak is -1.04 dB full scale. Therefore, this invention not only reduces loudness recovery error but also eliminates peak exceedance issues during accompaniment compensation.

[0072] In the compensation state parameter extraction stage, each streaming accompaniment audio frame is used as input to obtain the loudness difference, peak safety margin, segment matching confidence, transient intensity, and sub-band masking margin. At the beginning of the second chorus segment, the short-term loudness of the sub-track accompaniment at five consecutive display points (7.2 seconds, 7.4 seconds, 7.6 seconds, 7.8 seconds, and 8.0 seconds) are -21.36, -20.74, -20.37, -20.34, and -20.62, respectively, with loudness differences relative to the target loudness of 5.36, 4.74, 4.37, 4.34, and 4.62. The predicted peak values ​​at the same times are -3.62, -3.39, -3.28, -3.36, and -3.61, corresponding to peak safety margins of approximately 2.62, 2.39, 2.28, 2.36, and 2.61. The paragraph matching confidence scores were 0.45, 0.62, 0.76, 0.80, and 0.71, respectively, indicating that the 7.6-7.8 second interval had the highest similarity to the first chorus segment.

[0073] like Figure 2 As shown, this graph was obtained by merging the display layers of continuous analysis windows of a 12-second test segment. The horizontal axis represents time, and the vertical axis represents short-term loudness. The four curves in the graph represent the split-track accompaniment, fixed-gain output, the output of this invention, and the target loudness, respectively. The split-track accompaniment curve is mainly distributed between -23.8 and -20.3, indicating that the overall accompaniment is relatively low after splitting the tracks. The fixed-gain output curve is concentrated between -17.5 and -14.8, with a significant upward surge in the chorus. The output curve of this invention is mainly distributed between -16.9 and -16.1, maintaining a closer relationship with the target loudness line. Combining the chorus interval from 7.2 seconds to 8.0 seconds in the graph, it can be seen that the fixed-gain method exhibits excessive compensation when the energy rises, while the output of this invention remains near the target loudness, indicating that the loudness recovery of the convective segment by this invention is more stable.

[0074] In the look-ahead compensation triggering phase, a trigger determination value is formed based on the loudness difference, transient intensity, and segment matching confidence, and compensation conditions are determined in conjunction with the peak safety margin. In the current embodiment, the trigger threshold is set to 0.70, and the safety threshold is set to 1.20. The trigger determination values ​​at 7.2 seconds and 7.4 seconds are 0.55 and 0.62, respectively, both lower than the trigger threshold; the trigger determination value at 7.6 seconds reaches 0.70, and the trigger determination value at 7.8 seconds is greater than 0.70. Simultaneously, the peak safety margin is greater than the safety threshold in both cases, therefore, the look-ahead compensation conditions are met. Within this interval, the system calls the target compensation template from the historical compensation template set and reads the gain envelope information to form a priori gain envelope.

[0075] like Figure 3As shown in the figure, this graph is also obtained from the display layer data of a 12-second test segment. The horizontal axis represents time, and the vertical axis represents numerical values. The trigger judgment value curve in the graph exceeds the trigger threshold line in the 7.6-7.8 second range, and the segment matching confidence curve reaches 0.76-0.80 in this range, indicating that the current segment simultaneously possesses high compensation requirements and high template reuse reliability. The segment matching confidence also increases around 3.2 seconds, but since this belongs to the first chorus establishment stage, the system only records the template and does not execute historical template calls. This graph illustrates that the present invention does not directly call templates in all low-loudness ranges, but only imports the current segment into the template-driven path after the look-ahead compensation conditions are met.

[0076] In the subband gain budgeting stage, the current streaming accompaniment audio frame is divided into 8 frequency subbands, and the budget for each subband is determined based on loudness difference, subband masking margin, peak safety margin, and prior gain envelope. Taking the interval from 7.0 seconds to 8.8 seconds as an example, the system calculates the budget results every 0.2 seconds. The mid-frequency and mid-high-frequency subbands bear the main accompaniment texture and percussion information, and their budget values ​​are higher than those of the low-frequency subbands. At 7.8 seconds, the budget gains for low-frequency 1, low-frequency 2, mid-low-frequency 1, mid-low-frequency 2, mid-frequency 1, mid-frequency 2, high-frequency 1, and high-frequency 2 are 1.43, 1.87, 2.65, 3.13, 3.40, 3.20, 2.79, and 2.31, respectively. This result indicates that in the repeated chorus segments, the mid-frequency region allows for more aggressive compensation, while the low-frequency region maintains a lower budget level due to stronger peak constraints.

[0077] like Figure 4 As shown, this graph uses 10 time windows from 7.0 seconds to 8.8 seconds as horizontal samples and 8 frequency sub-bands as vertical samples, with color depth representing the budget gain. The graph shows that during the period from 7.6 seconds to 8.2 seconds, the budget values ​​for mid-frequency 1 and mid-frequency 2 sub-bands remain between 3.2 and 3.6, higher than the 1.3 to 2.0 for low-frequency 1 and low-frequency 2. This graph is derived from the loudness difference of the current segment, the energy distribution of each sub-band, the peak safety margin, and the a priori gain envelope, reflecting that this invention, without exceeding safety boundaries, prioritizes the allocation of limited compensation to frequency regions where the improvement in listening experience is more significant.

[0078] In the candidate gain trajectory screening stage, the gain change trend corresponding to the prior gain envelope is used as a reference trajectory to generate three types of candidate gain trajectories: low change rate, medium change rate, and high change rate within the budget range of each sub-band. At 7.6 seconds, the loudness recovery deviation corresponding to the low change rate trajectory is 0.48, the peak exceedance risk is low, and the gain change rate is 0.21; the loudness recovery deviation corresponding to the medium change rate trajectory is 0.16, the peak exceedance risk is low, and the gain change rate is 0.38; the loudness recovery deviation corresponding to the high change rate trajectory is 0.08, but the peak exceedance risk is significantly increased, and the gain change rate reaches 0.62. After comprehensive comparison, the system selects the medium change rate trajectory as the target gain trajectory. After performing sub-band gain adjustment and sub-band synthesis according to the target gain trajectory, the average short-time loudness of the output of this invention in the interval of 7.6 seconds to 8.0 seconds is -16.21, which is significantly better than the -20.64 of the track-by-track accompaniment, and is closer to the target loudness than the fixed gain method, while keeping the peak within the limit.

[0079] During the template update phase, once a target compensation template has been matched, the gain envelope information in the target compensation template is corrected based on the loudness restoration result and peak value exceeding detection result formed by the compensated accompaniment audio stream. Taking a local window of the second chorus segment as an example, the historical template envelope has a gain value of 2.55 at 0.8 seconds, while the actual executed envelope reaches 2.79 at the same position; the updated template envelope is corrected to 2.68. At 1.4 seconds, the historical template envelope is 1.72, the actual executed envelope is 1.80, and the updated template envelope is 1.78. The updated envelope retains the overall trend of the original template while incorporating the actual compensation result from this execution.

[0080] like Figure 5 As shown in the figure, this graph is obtained by extracting the historical template envelope, the actual execution envelope, and the updated template envelope within the same repeating segment. The horizontal axis represents the time within the segment, and the vertical axis represents the envelope gain. The figure shows that the updated template envelope consistently lies between the historical template envelope and the actual execution envelope throughout the entire interval, with the most significant correction occurring between 0.6 seconds and 1.0 seconds. This indicates that the invention does not simply overwrite the original template but performs controlled correction based on the loudness recovery results and peak value exceeding detection results. After this processing, when similar segments reappear subsequently, the template envelope invoked is closer to the actual compensation requirements.

[0081] In summary, this embodiment demonstrates that the present invention can complete a full closed loop in a real-world in-vehicle execution chain, including streaming accompaniment audio frame analysis, look-ahead compensation triggering, historical compensation template matching, sub-band level budget allocation, candidate gain trajectory screening, true peak protection, and template update write-back. Compared to the fixed gain method, the present invention reduces the mean absolute loudness error from 1.04 to 0.61, while reducing the number of peak exceedance windows from 121 to 0, enabling more stable and secure gain compensation in repetitive accompaniment segments.

[0082] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A method for compensating accompaniment gain based on streaming loudness matching, characterized in that, The method includes: Acquire streaming accompaniment audio frames, extract loudness difference, peak safety margin and segment matching confidence to obtain compensation state parameters; The prospective compensation triggering result is determined based on the compensation state parameters. When the prospective compensation triggering result meets the compensation conditions, the target compensation template is matched from the historical compensation template set based on the paragraph matching confidence to obtain the prior gain envelope. Given the prior gain envelope, the subband gain budget is determined based on the compensation state parameters and the prior gain envelope, and candidate gain trajectories are generated based on the subband gain budget. The target gain trajectory is determined based on the candidate gain trajectory, and gain compensation and true peak protection are performed on the streaming accompaniment audio frame based on the target gain trajectory to obtain the compensated accompaniment audio stream.

2. The method according to claim 1, characterized in that, The extraction of loudness difference, peak safety margin, and paragraph matching confidence yields compensation state parameters, including: The loudness difference is determined based on the short-time loudness value of the streaming accompaniment audio frame and the target loudness; The peak safety margin is determined based on the difference between the predicted peak value and the peak limit threshold of the streaming accompaniment audio frame. The segment matching confidence level is determined based on the similarity between the streaming accompaniment audio frame and the historical audio segment; The compensation state parameters also include transient intensity determined based on the short-time energy change rate of the streaming accompaniment audio frame, and sub-band masking margin determined based on the energy distribution of each sub-band of the streaming accompaniment audio frame.

3. The method according to claim 2, characterized in that, The step of determining the look-ahead compensation triggering result based on the compensation state parameters includes: The trigger determination value is determined based on the loudness difference, the transient intensity, and the paragraph matching confidence. If the trigger determination value is greater than the trigger threshold and the peak safety margin is greater than the safety threshold, it is determined that the forward compensation trigger result meets the compensation conditions.

4. The method according to claim 1, characterized in that, When the prospective compensation triggering result meets the compensation condition, the prior gain envelope is obtained by matching the target compensation template from the historical compensation template set based on the paragraph matching confidence. This includes: Extract segment description features from the streaming accompaniment audio frames; The target compensation template is determined based on the matching degree between the paragraph description features and the historical paragraph description features corresponding to each historical compensation template in the historical compensation template set; The prior gain envelope is obtained by reading the gain envelope information corresponding to the target compensation template.

5. The method according to claim 4, characterized in that, The historical compensation template set includes historical segment description features and gain envelope information. The historical segment description features are used to characterize the temporal energy distribution and spectral distribution of historical audio segments.

6. The method according to claim 2, characterized in that, The step of determining the sub-band gain budget based on the compensation state parameters and the prior gain envelope includes: The target compensation gain for each sub-band is determined based on the loudness difference. The perceptual allowable gain of each sub-band is determined based on the sub-band masking margin; The peak limit gain of each sub-band is determined based on the peak safety margin. The reference gain of each sub-band is determined based on the prior gain envelope; The minimum value among the compensation target gain, the perception allowable gain, the peak limit gain, and the reference gain is determined as the subband gain budget for the corresponding subband.

7. The method according to claim 6, characterized in that, The generation of candidate gain trajectories includes: Using the gain change trend corresponding to the prior gain envelope as a reference trajectory, multiple candidate gain trajectories with different gain change rates are generated within the range defined by the subband gain budget.

8. The method according to claim 1, characterized in that, Determining the target gain trajectory based on the candidate gain trajectories includes: Determine the loudness recovery deviation, peak exceedance risk, and gain change rate for each candidate gain trajectory; The trajectory cost of each candidate gain trajectory is determined based on the loudness recovery deviation, the peak value exceedance risk, and the gain change rate. The candidate gain trajectory with the minimum trajectory cost is determined as the target gain trajectory.

9. The method according to claim 1, characterized in that, The process of performing gain compensation and true peak protection on the streaming accompaniment audio frames to obtain the compensated accompaniment audio stream includes: The sub-band signal is gain-adjusted and sub-band synthesized according to the target gain trajectory to obtain a compensated audio frame. If the predicted peak value of the compensated audio frame is greater than the peak value limit threshold, the compensated audio frame is subjected to amplitude limiting to obtain the compensated accompaniment audio stream.

10. The method according to claim 1, characterized in that, The method further includes: If the target compensation template is matched, the loudness restoration result and peak value over-limit detection result are determined based on the compensated accompaniment audio stream; The gain envelope information corresponding to the target compensation template is corrected based on the loudness recovery result and the peak value exceeding the detection result. Write the revised target compensation template back into the historical compensation template set.

Images