High fidelity music adaptive enhancement system and method based on deep audio codec

Abstract

The application relates to the field of audio enhancement technology and discloses a high-fidelity music adaptive enhancement system and method based on deep audio coding, which comprises a frame-time-frequency module, a dynamic judgment module, a frequency division envelope extraction module, a frequency domain enhancement module, a time-frequency inverse conversion module and a damage compensation module, the frame-time-frequency module converts an original music audio signal into a frequency domain complex spectrum; the dynamic judgment module judges the paragraph attribute of the audio signal frame and marks the frame as a strong dynamic paragraph frame or a weak dynamic paragraph frame; the frequency division envelope extraction module determines the main frequency band boundary, extracts the main frequency band envelope and the detail frequency band envelope; the dynamic range normalization is performed on the main frequency band envelope, the harmonic expansion is performed on the detail frequency band envelope, and the first enhanced frequency domain signal is generated by combination; the first enhanced time domain signal is obtained by performing the frequency domain to time domain inverse conversion; the adaptive enhancement control signal is generated by extracting the coding damage feature; the enhanced music audio signal is generated by performing the damage compensation processing; and the application can improve the efficiency of music adaptive enhancement.

Description

Technical Field

[0001] This invention relates to the field of audio enhancement technology, and in particular to a high-fidelity music adaptive enhancement system and method based on deep audio codec. Background Technology

[0002] Currently, music enhancement systems based on deep audio codecs typically employ a fixed time-frequency transformation and a unified frequency domain gain processing architecture. These systems first convert the time-domain audio signal into a frequency-domain complex spectrum using a short-time Fourier transform. Then, they extract the signal envelope according to preset fixed frequency band division rules and apply gain adjustment and compression to the entire frequency band or several fixed sub-bands based on a unified dynamic range control strategy. Finally, they recover the time-domain signal through an inverse transform. The core premise of this approach is that the frequency band division rules and enhancement strategy parameters are fixed before system operation and do not dynamically change with the structural characteristics of the input signal itself. Therefore, the system design requires pre-configuring a large number of empirical threshold parameters for various types of music signals.

[0003] However, music signals exhibit significant differences in segment attributes along the timeline, with fundamental differences in spectral envelope shape, transient density, and energy distribution patterns between high-dynamic-range and low-dynamic-range segments. When using fixed frequency band division and a uniform enhancement strategy to process segments with different dynamic characteristics, transient impacts in high-dynamic-range segments are prone to overshoot distortion due to the application of a fixed compression ratio, while subtle details in low-dynamic-range segments may be misclassified into the main frequency band due to improper frequency band division, resulting in excessive smoothing of detail harmonics and loss of clarity. Simultaneously, the distribution patterns of quantization noise and transient distortion introduced during the encoding and decoding process differ across frequency bands, and the open-loop enhancement architecture cannot perceive the actual distribution of these impairments. This results in the enhanced signal having both uncompensated impairments and the potential introduction of new distortion components due to uniform gain adjustments. Summary of the Invention

[0004] This invention provides a high-fidelity music adaptive enhancement system and method based on deep audio codec to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides a high-fidelity music adaptive enhancement system based on deep audio codec, characterized in that the system includes a frame-segmentation time-frequency module, a dynamic segmentation module, a frequency division and network extraction module, a frequency domain enhancement module, a time-frequency inversion module, and an impairment compensation module, wherein:

[0006] The frame-segmentation time-frequency module is used to acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame.

[0007] The dynamic segmentation module is used to determine the segment attributes of the audio signal frame, and mark the audio signal frame as a strong dynamic segment frame or a weak dynamic segment frame according to the determination result, so as to obtain the segment marking result.

[0008] The frequency division and envelope extraction module is used to determine the main frequency division boundary based on the paragraph marking result, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary;

[0009] The frequency domain enhancement module is used to perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate a first enhanced frequency domain signal.

[0010] The time-frequency inverse transformation module is used to perform an inverse transformation from the frequency domain to the time domain on the first enhanced frequency domain signal to obtain the first enhanced time domain signal;

[0011] The impairment compensation module is used to extract encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction result, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

[0012] To address the aforementioned problems, this invention also provides a high-fidelity music adaptive enhancement method based on deep audio codec, the method comprising:

[0013] Pt.1. Acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame;

[0014] Pt.2. Perform paragraph attribute determination on the audio signal frame, and mark the audio signal frame as a strong dynamic paragraph frame or a weak dynamic paragraph frame according to the determination result to obtain the paragraph marking result.

[0015] Pt.3. Determine the main frequency division boundary based on the paragraph marking results, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary;

[0016] Pt.4. Perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate the first enhanced frequency domain signal;

[0017] Pt.5. Perform an inverse frequency-to-time transformation on the first enhanced frequency domain signal to obtain the first enhanced time domain signal;

[0018] Pt.6 Extract the encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction results, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

[0019] Compared with the prior art, the present invention has the following beneficial effects:

[0020] 1. This invention obtains the frequency domain complex spectrum of an audio signal through frame-by-frame time-frequency processing. Based on the distribution density of the spectral envelope extrema and the peak offset of the inter-frame cross-correlation, it determines the segment attributes of each signal frame. Based on the determination results, it adaptively determines the main frequency division boundary, and then performs differentiated enhancement processing on the main frequency band envelope and the detail frequency band envelope respectively. Simultaneously, it extracts encoding / decoding impairment features from the enhanced time-domain signal and generates an adaptive control signal to complete closed-loop impairment compensation. This technical solution fundamentally overcomes the processing mismatch problem caused by fixed frequency band division and uniform enhancement strategies described in the background technology. By forming a complete "perception-decision-execution-feedback" link in each stage of segment perception, frequency band segmentation, enhancement execution, and impairment compensation, the enhancement strategy dynamically adapts to the structural characteristics of the input signal and the actual distribution of encoding / decoding impairments, fundamentally eliminating the static mismatch defects caused by preset empirical thresholds.

[0021] 2. This invention separately measures the steepness characteristics of the rising and falling segments and compares each with an independent predetermined threshold. Only rising segments exceeding the rising threshold are subject to an attenuation coefficient to suppress overshoot, while falling segments exceeding the falling threshold are subject to a preservation coefficient to maintain the natural attenuation pattern. Rising and falling segments that do not exceed the corresponding thresholds are left unprocessed. This asymmetric regularization strategy, based on the frequency division enhancement achieved in the independent claim, further improves the refinement of dynamic regularization within the main frequency band, ensuring that the enhanced main frequency band maintains both the transient impact of the music signal and the natural extension characteristics of the musical tone's tail. Attached Figure Description

[0022] Figure 1 This is a system architecture diagram of a high-fidelity music adaptive enhancement system based on deep audio codec, provided in an embodiment of the present invention.

[0023] Figure 2 This is a flowchart illustrating a high-fidelity music adaptive enhancement method based on deep audio codec, provided in an embodiment of the present invention.

[0024] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments belong to some, but not all, embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “said” and “the” as used in the embodiments of this invention and the appended claims are also intended to include the plural forms, and “multiple” generally includes at least two unless the context clearly indicates otherwise.

[0027] Depending on the context, the word "if" or "if" as used here can be interpreted as "when," "when," "in response to determination," or "in response to detection." Similarly, depending on the context, the phrase "if determination" or "if detection (of the stated condition or event)" can be interpreted as "when determination," "in response to determination," "when detection (of the stated condition or event)," or "in response to detection (of the stated condition or event)."

[0028] Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0029] In practice, the server-side equipment deployed in a high-fidelity music adaptive enhancement system based on deep audio codec may consist of one or more devices. This high-fidelity music adaptive enhancement system based on deep audio codec can be implemented as a business instance, a virtual machine, or a hardware device. For example, this high-fidelity music adaptive enhancement system based on deep audio codec can be implemented as a business instance deployed on one or more devices in a cloud node. Simply put, this high-fidelity music adaptive enhancement system based on deep audio codec can be understood as software deployed on a cloud node, used to provide high-fidelity music adaptive enhancement to various user terminals. Alternatively, this high-fidelity music adaptive enhancement system based on deep audio codec can also be implemented as a virtual machine deployed on one or more devices in a cloud node. This virtual machine contains application software for managing various user terminals. Alternatively, this high-fidelity music adaptive enhancement system based on deep audio codec can also be implemented as a server composed of numerous identical or different types of hardware devices, with one or more hardware devices configured to provide high-fidelity music adaptive enhancement to various user terminals.

[0030] In terms of implementation, the high-fidelity music adaptive enhancement system based on deep audio codec and the user client are mutually compatible. That is, if the high-fidelity music adaptive enhancement system based on deep audio codec is implemented as an application installed on a cloud service platform, then the user client is implemented as a client that establishes a communication connection with the application; or if the high-fidelity music adaptive enhancement system based on deep audio codec is implemented as a website, then the user client is implemented as a webpage; or if the high-fidelity music adaptive enhancement system based on deep audio codec is implemented as a cloud service platform, then the user client is implemented as a mini-program in an instant messaging application.

[0031] like Figure 1 The figure shown is a system architecture diagram of a high-fidelity music adaptive enhancement system based on deep audio codec provided in an embodiment of the present invention.

[0032] The high-fidelity music adaptive enhancement system based on deep audio codec described in this invention can be set up in a cloud server. In terms of implementation, it can be used as one or more service devices, or as an application installed in the cloud (e.g., a mobile service operator's server, server cluster, etc.), or it can be developed into a website. Depending on the implemented functions, the high-fidelity music adaptive enhancement system based on deep audio codec may include a frame-segmentation time-frequency module, a dynamic segmentation module, a frequency division and network extraction module, a frequency domain enhancement module, a time-frequency inversion module, and an impairment compensation module. The module described in this invention can also be called a unit, which refers to a series of computer program segments that can be executed by the processor of an electronic device and can perform a fixed function, stored in the memory of the electronic device.

[0033] In this embodiment of the invention, in the high-fidelity music adaptive enhancement system based on deep audio codec, each of the above modules can be implemented independently and can call other modules. Here, "calling" can be understood as one module connecting to multiple modules of another type and providing corresponding services to those connected modules. In the high-fidelity music adaptive enhancement system based on deep audio codec provided by this embodiment of the invention, the applicable scope of the high-fidelity music adaptive enhancement system architecture based on deep audio codec can be adjusted by adding modules and directly calling them without modifying the program code, achieving cluster-based horizontal expansion to quickly and flexibly expand the high-fidelity music adaptive enhancement system based on deep audio codec. In practical applications, the above modules can be set in the same device or different devices, or they can be set in a virtual device, such as a service instance in a cloud server.

[0034] The following describes the components and workflow of a high-fidelity music adaptive enhancement system based on deep audio encoding and decoding, using specific embodiments as examples:

[0035] The frame-segmentation time-frequency module is used to acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame.

[0036] In this embodiment of the invention, in music audio signal processing, time-domain waveforms often exhibit non-stationary characteristics, with their spectral structure changing rapidly over time. If a single frequency domain transformation is performed on the entire signal, spectral difference information at different positions on the time axis will be lost. To address this, the frame-segmentation time-frequency module first acquires the sampling point sequence of the original music audio signal, and then divides the sampling point sequence into several audio signal frames along the time axis according to a preset frame length.

[0037] In this process, in order to suppress spectral leakage at frame boundaries and ensure the continuity of information between frames, there is partial overlap between adjacent audio signal frames.

[0038] Preferably, the frame length can be set to any value between 20 milliseconds and 60 milliseconds. For example, in one feasible implementation, the frame length is 40 milliseconds. In this case, for a digital audio signal with a sampling rate of 44.1 kHz, each frame contains approximately 1764 sampling points.

[0039] In another feasible implementation, the frame length can also be set to 30 milliseconds, which is suitable for music genres with higher transient response requirements.

[0040] In another feasible implementation, the frame length can also be set to 50 milliseconds, which is suitable for music passages with relatively gentle frequency spectrum changes.

[0041] The overlap length between adjacent frames can be 25% to 75% of the frame length. In one feasible implementation, the overlap length is 50% of the frame length. In another feasible implementation, the overlap length can also be 25% or 75% of the frame length.

[0042] The specific values ​​of the frame length and overlap length mentioned above can be preset by the operator based on a large number of experimental statistical results, or can be adaptively adjusted by those skilled in the art based on the sampling rate and duration characteristics of the target audio signal.

[0043] After the frame division is completed, a window function needs to be applied to each audio signal frame to obtain a windowed audio frame. The purpose of the window function is to make the amplitude at both ends of the frame transition smoothly to near zero, thereby reducing the spectral leakage introduced by subsequent time-frequency transformation.

[0044] Alternatively, the window function can be any of the Hanning window, Hamming window, or Blackman window.

[0045] One feasible implementation uses the Hanning window, whose expression is:

[0046] ;

[0047] in, The number of sampling points corresponding to the frame length. This represents the intra-frame sampling point number. In another feasible implementation, a Hamming window can be used, with the following expression:

[0048] ;

[0049] The choice of the specific type of window function depends on the trade-off between the main lobe width and the side lobe attenuation requirements. The Hanning window is suitable for scenarios with high requirements for side lobe suppression, while the Hamming window is suitable for scenarios with high requirements for frequency resolution accuracy.

[0050] Perform a time-frequency transformation on the windowed audio frame to map it from the time domain to the frequency domain, and obtain the frequency domain complex spectrum of the audio signal frame.

[0051] Preferably, the time-frequency transformation can be performed using a short-time Fourier transform, the calculation formula of which is:

[0052] ;

[0053] in, For frame number, For frequency index, The intra-frame sampling point number, For frame length, The frame shift step size, The first in the original signal The first frame One sampling point, For window functions, It is the imaginary unit.

[0054] Frequency Index The value range is 0 to The corresponding actual frequency value is ,in The sampling rate of the signal is denoted as . In one feasible implementation, the number of transformation points in the short-time Fourier transform can be the same as the frame length. In another feasible implementation, the number of transformation points can be an integer power of 2 greater than the frame length. For example, when the number of sampling points corresponding to the frame length is 1764, the number of transformation points can be 2048 to improve frequency resolution.

[0055] Alternatively, the time-frequency transformation can also employ an improved discrete cosine transform.

[0056] Through the above-mentioned framing, windowing, and time-frequency transformation operations, the time-domain signal is converted into a complex spectrum in the frequency domain, enabling subsequent processing to independently manipulate the amplitude and phase information of each frequency point in the frequency domain. This provides an operable frequency domain data foundation for subsequent frequency division processing of the main frequency band and detail frequency band.

[0057] The dynamic segmentation module is used to determine the segment attributes of the audio signal frame, and mark the audio signal frame as a strong dynamic segment frame or a weak dynamic segment frame according to the determination result, so as to obtain the segment marking result.

[0058] In this embodiment of the invention, in audio signal processing, the essential difference between strong dynamic segments and weak dynamic segments is not only reflected in the degree of fluctuation of energy amplitude, but also in the complexity of the spectral envelope shape and the similarity of the inter-frame spectral structure. A single energy threshold cannot distinguish between segments with similar energy amplitudes but completely different structural features. If a uniform enhancement strategy is used to process all segments, the transient details in strong dynamic segments will be damaged due to excessive smoothing, and the stable background in weak dynamic segments will cause auditory fatigue due to excessive enhancement.

[0059] To this end, the dynamic segmentation module first extracts the spectral envelope from the frequency domain complex spectrum of the audio signal frame, and detects the distribution density of local extrema points in the spectral envelope as the extrema distribution density of the audio signal frame.

[0060] Specifically, the spectral envelope can be obtained by smoothing and filtering the amplitude values ​​of the complex spectrum in the frequency domain.

[0061] In one feasible implementation, the local extremum point is detected as follows: in the spectral envelope sequence, if the amplitude value at a certain frequency point is greater than the amplitude values ​​at several adjacent frequency points to its left and right, then that frequency point is determined to be a local maximum point.

[0062] Optionally, the number of adjacent frequency points on the left and right can be set to 3 to 7.

[0063] In another feasible implementation, the number of left and right adjacent frequency points can also be adaptively determined according to the frame length of the current signal frame. For example, the number of left and right adjacent frequency points can be set to one percent to five percent of the total number of frequency points corresponding to the frame length.

[0064] The extreme value distribution density is the total number of local maxima divided by the total number of frequency points covered by the spectral envelope.

[0065] In one feasible implementation, the formula for calculating the extreme value distribution density is:

[0066] ;

[0067] in, This represents the total number of local maxima in the current frame's spectral envelope. This represents the total number of frequency points covered by the current frame's spectral envelope. This is the extreme value distribution density.

[0068] In another feasible implementation, the extreme value distribution density can also be expressed as the ratio of the number of local maxima to the bandwidth, in which case the unit of the ratio is per kilohertz.

[0069] The extreme value distribution density reflects the complexity of the current frame's spectral structure. The larger the density value, the more frequent the peak-valley alternation in the spectrum, the richer the harmonic structure contained in the signal, and the higher the probability that it belongs to a strong dynamic segment.

[0070] The dynamic segmentation module also performs cross-correlation comparison between the frequency domain amplitude spectrum of the audio signal frame and the frequency domain amplitude spectrum of the previous frame to obtain the cross-correlation peak offset of the audio signal frame.

[0071] Specifically, the formula for calculating cross-correlation alignment is:

[0072] ;

[0073] in, For the first Frame in frequency index The frequency domain amplitude value at that point, For the first Frame in frequency index The frequency domain amplitude value at that point, Offset index This represents the total number of frequency points.

[0074] The cross-correlation peak offset is such that The corresponding value when the maximum value is obtained value.

[0075] In one feasible implementation, the search range for cross-correlation can be set to... ,in The value can be 2 to 10 frequency index units.

[0076] In another feasible implementation, The value can also be determined adaptively based on the sampling rate, for example, set to the number of frequency points covered by the actual frequency range of 20Hz to 50Hz.

[0077] In another feasible implementation, the cross-correlation peak offset can also be calculated using a normalized cross-correlation function to eliminate the influence of the difference in absolute magnitude on the offset estimation.

[0078] The cross-correlation peak offset reflects the overall drift of the spectral structure between adjacent frames. The larger the offset, the more drastic the change in the spectral distribution between adjacent frames, and the higher the probability that it belongs to a highly dynamic segment.

[0079] The extreme value distribution density and the cross-correlation peak offset are combined as the judgment criteria.

[0080] In one feasible implementation, the criterion can be obtained by weighted summation:

[0081] ;

[0082] in, For extreme value distribution density, This represents the absolute value of the cross-correlation peak offset. and These are the weighting coefficients for the two respectively.

[0083] In another feasible implementation, the AND logic of both the extreme value distribution density and the cross-correlation peak offset exceeding their respective thresholds can be used as the criterion.

[0084] The judgment criteria are compared step by step with the thresholds at each level in the multi-level precision preset threshold system.

[0085] This threshold system includes at least one decision threshold value.

[0086] In one feasible implementation, the threshold system can include three levels of precision: high, medium, and low, each corresponding to a different degree of strictness in the judgment.

[0087] In another feasible implementation, the threshold system may contain only a single threshold value. The frame is marked as a strong dynamic segment frame if it is greater than the threshold value, and otherwise it is marked as a weak dynamic segment frame.

[0088] When the judgment criterion is greater than the corresponding threshold in the threshold system, the audio signal frame is marked as a strong dynamic segment frame.

[0089] When the judgment criterion is not greater than the corresponding threshold in the threshold system, the audio signal frame is marked as a weak dynamic segment frame.

[0090] In one feasible implementation, the threshold values ​​at each level in the threshold system can be preset by the operator based on the statistical characteristics of a large number of music samples.

[0091] In another feasible implementation, the threshold values ​​at each level in the threshold system can also be determined adaptively, for example, by dynamically adjusting the threshold values ​​based on the mean and variance of the judgment criteria of the currently processed frames.

[0092] The segment marking results are obtained by summarizing the marking results of all audio signal frames.

[0093] By combining the extreme value distribution density and cross-correlation peak offset, the segment attributes of each frame are comprehensively characterized from two aspects: spectral structure complexity and inter-frame variation intensity. This makes the distinction between strong dynamic segments and weak dynamic segments more accurate and reliable, providing an accurate segment attribute basis for the differentiated selection of subsequent frequency division enhancement strategies.

[0094] The frequency division and envelope extraction module is used to determine the main frequency division boundary based on the paragraph marking result, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary;

[0095] In this embodiment of the invention, in the music audio signal, the main frequency band carries the melody tonic and harmonic framework, while the detail frequency band carries the instrument overtones and spatial atmosphere information. The response characteristics of the two to enhancement processing are quite different. If a single fixed frequency is used as the frequency division boundary, it cannot adapt to the dynamic migration of the main energy concentration area in different segments, resulting in insufficient main enhancement or excessive detail processing.

[0096] The frequency division and network extraction module determines the main frequency division boundary based on the segment marking results. This boundary is used to divide the complex spectrum in the frequency domain into the main frequency band and the detail frequency band.

[0097] When the paragraph marking result is a highly dynamic paragraph frame, there are usually multiple local maxima in the spectral envelope. The local maxima with the largest amplitude corresponds to the frequency component with the most concentrated energy in the current frame. This frequency component often determines the position of the fundamental tone or dominant harmony of the paragraph.

[0098] The local maxima of the spectral envelope in the complex spectrum of the frequency domain are detected, and the local maxima with the largest amplitude are determined. The frequency corresponding to the local maxima with the largest amplitude is used as the main frequency division boundary.

[0099] Specifically, the method for detecting local maxima is to search for frequency points in the spectral envelope sequence where the amplitudes of adjacent frequency points are all less than the amplitude of the current frequency point.

[0100] In one feasible implementation, a three-point comparison method can be used in the spectral envelope sequence during detection. If the first... The amplitude value at each frequency point is simultaneously greater than that at the first frequency point. The frequency point and the first The amplitude value at the nth frequency point is then used to determine the nth frequency point. Each frequency point is a local maximum point.

[0101] In another feasible implementation, a five-point comparison method can also be used. If the first... If the amplitude value at a frequency point is greater than the amplitude values ​​at the two frequency points to its left and right, it is determined to be a local maximum point.

[0102] In another feasible implementation, in order to eliminate false extreme points caused by noise, an amplitude threshold can be set. Only when the amplitude value of a local maximum point is higher than the threshold will it be counted as a valid extreme point. The amplitude threshold can be set to 1.2 to 1.5 times the average amplitude of the current frame spectrum envelope.

[0103] When the paragraph marking result is a weak dynamic paragraph frame, the spectral envelope shape is relatively flat and there are no obvious towering spectral peaks. In this case, it is necessary to determine the location where the energy distribution changes significantly by the curvature of the envelope shape.

[0104] The spectral envelope is differentially processed to obtain a difference sequence. The position where the sign changes from positive to negative is determined in the difference sequence and this position is taken as the inflection point.

[0105] The calculation formula for differential processing is:

[0106] ;

[0107] in, Frequency Index The amplitude value of the spectral envelope, This represents the difference between adjacent frequency points. A value greater than zero indicates that the envelope is on an upward trend. A value less than zero indicates that the envelope is in a downward trend. The position where the sign changes from positive to negative is the inflection point where the envelope changes from rising to falling.

[0108] Select the inflection point with the largest amplitude value from the frequency points corresponding to each inflection point, and use the frequency corresponding to the inflection point with the largest amplitude value as the main frequency division boundary.

[0109] In one feasible implementation, if there are multiple positions in the differential sequence where the sign changes from positive to negative, the frequency point corresponding to each position and the amplitude value at that frequency point are recorded respectively, and the one with the largest amplitude value is selected as the basis for boundary determination.

[0110] In another feasible implementation, the inflection points can be sorted by amplitude value first, the top three inflection points with the largest amplitude values ​​can be selected, and the median frequency of the corresponding values ​​can be selected as the main frequency division boundary.

[0111] In another feasible implementation, if there is no position in the difference sequence where the sign changes from positive to negative, it indicates that the overall envelope is monotonically increasing or decreasing. In this case, the frequency point where the amplitude of the spectral envelope reaches its maximum value can be used as the main frequency division boundary.

[0112] The frequency bands below the main frequency division boundary are designated as the main frequency bands, and the frequency bands above the main frequency division boundary are designated as detail frequency bands.

[0113] The amplitude values ​​are extracted from the frequency range corresponding to the main frequency band in the complex spectrum of the frequency domain to form the envelope of the main frequency band.

[0114] The amplitude values ​​are extracted from the frequency range corresponding to the detail band in the complex spectrum of the frequency domain to form the envelope of the detail band.

[0115] In one feasible implementation, the amplitude value sequence of the main frequency band envelope is the set of amplitude values ​​of the complex spectrum in the frequency domain within the range of the frequency index corresponding to the main frequency division boundary, and the amplitude value sequence of the detail frequency band envelope is the set of amplitude values ​​within the range of the frequency index corresponding to the main frequency division boundary to the highest analysis frequency.

[0116] In another feasible implementation, a transition band can be set between the main frequency band and the detail frequency band. The amplitude value in the transition band participates in the weighted combination of the envelope of the main frequency band and the envelope of the detail frequency band to achieve a smooth transition between the two frequency bands.

[0117] By using the adaptive frequency division boundary determination method described above, the division between the main frequency band and the detail frequency band always follows the dynamic changes in the signal's own spectral structure. In the strong dynamic segment, the boundary is determined by the actual spectral peak position, and in the weak dynamic segment, the boundary is determined by the envelope inflection point. This provides a suitable frequency band range basis for the subsequent dynamic range regularization of the main frequency band and the harmonic expansion of the detail frequency band.

[0118] The frequency domain enhancement module is used to perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate a first enhanced frequency domain signal.

[0119] In this embodiment of the invention, in audio signal enhancement processing, the main frequency band carries the main melody and harmonic framework of the music, and its sensitivity to amplitude fluctuations is much higher than that of the detail frequency band. Excessive amplitude changes will directly lead to an unnatural auditory experience. On the other hand, the detail frequency band carries the overtones of instruments and spatial atmosphere information. The integrity of its spectral structure determines the delicacy and realism of the music. The response characteristics of the two to enhancement processing are fundamentally different. If a unified enhancement strategy is applied to both frequency bands at the same time, it will be impossible to meet both the stability requirements of the main frequency band and the richness requirements of the detail frequency band.

[0120] To this end, the frequency domain enhancement module performs dynamic range normalization on the main frequency band envelope.

[0121] Extract the amplitude value sequence from the main frequency band envelope, perform variational processing on the amplitude value sequence, and divide the amplitude value sequence into rising segment and falling segment according to the positive or negative sign of the variational processing result.

[0122] In one feasible implementation, variational processing is achieved by performing a first-order difference operation on the amplitude value sequence, with the positive difference value corresponding to the rising segment and the negative difference value corresponding to the falling segment.

[0123] ;

[0124] in, The first value in the main frequency band envelope amplitude value sequence The amplitude value at each frequency point This represents the difference between adjacent frequency points.

[0125] when When it is greater than zero, the first Frequency points up to the Each frequency range is classified as an ascending segment.

[0126] when When it is less than zero, the first Frequency points up to the Each frequency range is classified as a falling segment.

[0127] when When the value equals zero, the interval is classified as a flat segment, which can be merged into an adjacent rising or falling segment.

[0128] In one feasible implementation, to avoid a large number of false rising and falling segments caused by small fluctuations due to noise, a variational ignore threshold can be set. Only when the absolute value of the difference is greater than the threshold is it considered a valid rise or fall. The value of the threshold can be two to five percent of the average amplitude of the current frame's main frequency band envelope.

[0129] In another feasible implementation, variational processing can also be achieved by performing second-order difference operations on the amplitude value sequence, and the rising and falling inflection points of the envelope can be determined by detecting the sign change of the second-order difference.

[0130] The degree of amplitude change between the two ends of each rising segment is measured as the rise steepness characterization quantity. The rise steepness characterization quantity of each rising segment is compared with a predetermined rise threshold. When the rise steepness characterization quantity is greater than the predetermined rise threshold, an attenuation coefficient is applied to the amplitude value in that rising segment.

[0131] The degree of amplitude change between the two ends of each descent segment is measured as a descent steepness characterization quantity. The descent steepness characterization quantity of each descent segment is compared with a predetermined descent threshold. When the descent steepness characterization quantity is greater than the predetermined descent threshold, a holding coefficient is applied to the amplitude value in that descent segment.

[0132] The calculation methods for the ascending steepness and descending steepness characteristics are as follows:

[0133] ;

[0134] in, For the first The steepness of the ascent in each ascending segment is a characteristic quantity. This is the amplitude value at the end of the rising segment. This is the amplitude value at the beginning of the rising segment. This represents the number of frequency points contained in the rising segment. The descent steepness is calculated as follows:

[0135] ;

[0136] The meanings of the variables are similar to those in the calculation of the rising segment.

[0137] In one feasible implementation, the predetermined rise threshold and predetermined fall threshold can be preset by the operator based on statistical results from a large number of music samples.

[0138] In another feasible implementation, the predetermined rise threshold and the predetermined fall threshold can also be adaptively determined based on the global statistical characteristics of the current frame's main frequency band envelope, for example, by setting the threshold to 1.5 to 2.0 times the global average steepness value.

[0139] In another feasible implementation, the predetermined rise threshold of the rise steepness representation quantity and the predetermined fall threshold of the fall steepness representation quantity can be different values ​​to accommodate the difference in sensitivity of the rise and fall segments to amplitude changes.

[0140] The attenuation coefficient ranges from 0.3 to 0.9, indicating that 30% to 90% of the original amplitude is retained.

[0141] In one feasible implementation, the attenuation coefficient is set to 0.7, which means that 70% of the amplitude change is retained for the rising segment that exceeds the predetermined rising threshold.

[0142] In another feasible implementation, the attenuation coefficient can also be set as a dynamic value that is inversely proportional to the steepness characterization quantity; the steeper the steepness, the smaller the attenuation coefficient.

[0143] The coefficient is kept within the range of 0.9 to 1.1.

[0144] In one feasible implementation, the coefficient is kept to be 1.0, that is, the amplitude value remains unchanged for the descent segment that exceeds the predetermined descent threshold.

[0145] In another feasible implementation, the preservation coefficient can also be a value slightly greater than 1.0, such as 1.05, to apply slight gain compensation to the steep descent segment and avoid energy loss in the main frequency band due to excessive suppression.

[0146] The normalized main frequency band envelope is composed of the amplitude values ​​of the rising segment with applied attenuation coefficient and the amplitude values ​​of the falling segment with applied hold coefficient.

[0147] By applying dynamic range normalization as described above, attenuation is applied to steep rising segments exceeding the threshold to suppress overshoot, and hold is applied to steep falling segments exceeding the threshold to maintain natural attenuation, thereby controlling amplitude variations within the main frequency band within a comfortable listening range.

[0148] The frequency domain enhancement module also performs harmonic expansion on the detailed frequency band envelope.

[0149] Jump detection is performed on the amplitude value sequence in the detail frequency band envelope to detect jump points in the amplitude value sequence.

[0150] In one feasible implementation, transition detection is achieved by performing a differential operation on the sequence of amplitude values ​​of the detail frequency band envelope. When the absolute value of the difference between adjacent frequency points exceeds a preset transition threshold, the position is determined as a transition point.

[0151] In another feasible implementation, transition detection can also be achieved by performing a second-order difference operation on the amplitude value sequence of the detail band envelope. The location of the maximum value of the second-order difference value exceeding the preset threshold is determined as the transition point.

[0152] In another feasible implementation, the transition threshold can be set to three to five times the average amplitude value of the current frame detail band envelope.

[0153] The predetermined bandwidth is extended forward and backward, starting from each transition point.

[0154] In one feasible implementation, the predetermined bandwidth can be between 10 and 30 frequency points.

[0155] In another feasible implementation, the predetermined bandwidth can also be adaptively determined based on the total width of the detail band of the current frame, for example, set to five percent to fifteen percent of the total number of detail band frequency points.

[0156] Within each extended predetermined bandwidth, the amplitude value at the transition point is used as the peak value, and a continuously changing replacement amplitude value is generated according to a Gaussian attenuation law.

[0157] The formula for Gaussian decay law is:

[0158] ;

[0159] in, To extend the bandwidth Replacement amplitude value at each frequency point This represents the peak amplitude at the transition point. The frequency point location corresponding to the transition point. The standard deviation of the Gaussian function controls the smoothness of the decay. In one feasible implementation, the standard deviation of the Gaussian function... It can be set to one-third to one-half of the predetermined bandwidth. In another feasible implementation, a cosine attenuation law can be used instead of the Gaussian attenuation law, with the following formula:

[0160] ;

[0161] in, This is the total length of the extended bandwidth.

[0162] The original amplitude values ​​within the extended predetermined bandwidth are replaced with replacement amplitude values ​​to form the extended detail band envelope.

[0163] Through the above harmonic extension, the detail energy at the transition point is extended to the adjacent frequency band, and the extended detail frequency band envelope has a more continuous harmonic structure in the frequency domain.

[0164] The normalized main frequency band envelope is merged with the expanded detail frequency band envelope to generate the first enhanced frequency domain signal.

[0165] In one feasible implementation, the merging operation involves directly splicing the main frequency band envelope amplitude value sequence and the detail frequency band envelope amplitude value sequence according to their respective frequency point positions.

[0166] In another feasible implementation, the boundary between the main frequency band and the detail frequency band during the merging operation can be smoothed. For example, several frequency points on both sides of the boundary can be taken for weighted averaging to avoid amplitude discontinuity at the frequency band boundary.

[0167] By processing the dynamic range regularization of the main frequency band and the harmonic spread of the detail frequency band separately and then merging them, the amplitude stability of the main frequency band and the spectral continuity of the detail frequency band are both improved in a targeted manner, providing an enhanced frequency domain signal basis for subsequent time-frequency inverse transform.

[0168] The time-frequency inverse transformation module is used to perform an inverse transformation from the frequency domain to the time domain on the first enhanced frequency domain signal to obtain the first enhanced time domain signal;

[0169] In this embodiment of the invention, after the frequency domain enhancement processing is completed, the first enhanced frequency domain signal exists in the form of a frequency domain complex spectrum, wherein the amplitude value of each frequency point has been adjusted while the phase information remains consistent with the original frequency domain complex spectrum. This signal cannot be directly output as a playable or transmittable audio signal and must be restored from the frequency domain representation to the time domain waveform.

[0170] The time-frequency inverse transformation module performs an inverse transformation from the frequency domain to the time domain on the first enhanced frequency domain signal to obtain the first enhanced time domain signal.

[0171] This inverse transform is the inverse operation of the time-frequency transform used in the frame-segmentation time-frequency module.

[0172] In one feasible implementation, when the framing time-frequency module uses short-time Fourier transform, the corresponding time-frequency inverse transform module uses inverse short-time Fourier transform, and its calculation formula is as follows:

[0173] ;

[0174] in, For the first The first frame in the time-domain signal after frame enhancement The amplitude value of each sampling point For the first enhanced frequency domain signal, the first Frame in frequency index Complex values ​​at that location, The number of points in the inverse transform. The frame shift step size, The intra-frame sampling point number, For frequency index, It is the imaginary unit.

[0175] In another feasible implementation, when the frame-segmentation time-frequency module adopts the improved discrete cosine transform, the time-frequency inverse transformation module adopts the corresponding improved discrete cosine inverse transform.

[0176] Number of transform points in short-time inverse Fourier transform The number of transformation points is consistent with that used in the frame-segmentation time-frequency module.

[0177] In one feasible implementation, the number of transformation points is the same as the frame length.

[0178] In another feasible implementation, the number of transformation points can also be an integer power of 2 greater than the frame length.

[0179] Frame shift step size It is consistent with the frame shift step size in the frame-segmentation time-frequency module.

[0180] In one feasible implementation, the frame shift step size is fifty percent of the frame length.

[0181] In another feasible implementation, the frame shift step size can also be 25% or 75% of the frame length.

[0182] During the inverse transform process, the inverse transform is performed independently on the frequency domain complex spectrum of each frame to obtain the time domain segment corresponding to each frame.

[0183] Because the frame-segmentation time-frequency module sets overlap for adjacent frames, the time-domain segments corresponding to each frame have overlapping areas on the time axis. The time-frequency inversion module needs to perform overlap and addition processing on the time-domain segments of each frame.

[0184] The specific method of overlapping and adding is as follows: align the temporal segments of each frame on the time axis according to the frame shift step, and superimpose the temporal sampling points from adjacent frames in the overlapping area.

[0185] Within the overlapping region, the amplitude value of the superimposed sampling point is equal to the algebraic sum of the amplitude values ​​of the sampling points of each contributing frame at that location.

[0186] In one feasible implementation, the contribution of each frame to the overlapping area during superposition can be achieved using an equal-weight superposition method.

[0187] In another feasible implementation, the contribution of each frame in the overlapping region during superposition can also be achieved by using a weighted superposition method that matches the window function used in the frame-segmented time-frequency module. For example, each frame time-domain segment can be multiplied by its corresponding synthesis window before superposition.

[0188] The window function type is the same as the window function type used in the frame-based time-frequency module.

[0189] In one feasible implementation, when the frame-segmentation time-frequency module uses a Hanning window, the time-frequency inversion module uses the same Hanning window for weighted superposition.

[0190] In another feasible implementation, when the frame-segmentation time-frequency module uses a Hamming window, the time-frequency inversion module uses the same Hamming window for weighted superposition.

[0191] In another feasible implementation, the time-frequency inversion module can also use other window functions with the same shape but equal length as the window functions in the frame-segmented time-frequency module for weighted superposition.

[0192] To ensure that the amplitude of the time-domain signal after inverse transformation is consistent with the amplitude ratio of the original signal, the time-domain sampling points after overlapping and addition need to be normalized.

[0193] The normalization process is as follows: after overlapping and adding, the amplitude value of the sampling point at each time position is divided by the number of frames participating in the superposition at that position.

[0194] In one feasible implementation, normalization can also be achieved by dividing the amplitude values ​​of the sampled points at each time position after overlapping and summing by a fixed normalization coefficient, which is determined by the window function type and the frame shift step size.

[0195] For frame length is Frame shift step size is When using the Hanning window and with a 50% overlap rate, the normalization coefficient can be set to a value of [value missing]. .

[0196] In another feasible implementation, the normalization coefficient can also be set to a value of [value missing]. .

[0197] Through the above inverse transformation and overlapping addition processing, the first enhanced frequency domain signal after frequency domain enhancement is restored to a continuous first enhanced time domain signal, so that the enhanced audio data can be used for subsequent damage feature extraction and final output in the form of time domain waveform.

[0198] The impairment compensation module is used to extract encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction result, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

[0199] In this embodiment of the invention, during the encoding and decoding process, the nonlinear distortion introduced by the quantization operation in the frequency domain will manifest as various forms of damage related to the signal content in the time domain. The distribution of these damages is closely related to the spectral structure of the signal itself. The types and degrees of damage suffered by different frequency bands are different. If the damage features are not extracted and specifically compensated after the enhancement process, the distortion introduced by the encoding and decoding stage will be amplified or altered in form along with the original signal in subsequent processing, resulting in the final output enhanced signal containing both the encoding and decoding damage that has not been eliminated and the new distortion components introduced by the enhancement process itself.

[0200] The damage compensation module extracts the encoding and decoding damage features from the first enhanced time-domain signal.

[0201] The time-domain signals corresponding to the first enhanced time-domain signal and the original music audio signal are synchronized and aligned. Then, a sample-by-sample difference operation is performed on the two synchronized signals to obtain the difference time-domain signal.

[0202] The specific method of synchronization alignment is to use the time-domain waveform of the original music audio signal as a reference and adjust the time-shift of the first enhanced time-domain signal until the positions of the two signals on the time axis are completely corresponding.

[0203] In one feasible implementation, synchronization alignment can be achieved by calculating the cross-correlation function of the two signals, and the time shift corresponding to the peak value of the cross-correlation function is the time shift value that needs to be compensated.

[0204] In another feasible implementation, synchronization alignment can also be achieved by matching the starting points or zero-crossing points of the two signals.

[0205] The formula for calculating the interpolation operation point by point is:

[0206] ;

[0207] in, For the first The difference amplitude at each sampling point For the first enhanced time-domain signal in the 1st The amplitude value at each sampling point For the original music audio signal in the first The amplitude value at each sampling point.

[0208] In one feasible implementation, the amplitudes of the two signals can be normalized before the sample-by-sample difference operation to eliminate the influence of the overall gain difference on the difference result.

[0209] Perform a time-frequency transformation on the difference time-domain signal to obtain the difference time-frequency spectrum. This difference time-frequency spectrum characterizes the distribution of signal distortion introduced by the encoding and decoding process in the time-frequency domain.

[0210] The time-frequency transformation can use the same transformation method as the frame-based time-frequency module, such as short-time Fourier transform or improved discrete cosine transform.

[0211] In the difference time spectrum, a frequency region that extends continuously along the time axis is detected, and the detected region is marked as the quantization noise footprint introduced by encoding and decoding, which serves as the first-level impairment feature.

[0212] In the difference time spectrum, frequency regions that appear isolated on the time axis are detected, and these detected regions are marked as transient distortion footprints introduced by encoding and decoding, serving as second-level impairment features.

[0213] In one feasible implementation, the criterion for continuous extension along the time axis is that the difference energy at a certain frequency point is higher than a preset energy threshold in multiple consecutive time frames, and the number of consecutive frames can be set to 5 to 20 frames.

[0214] In another feasible implementation, the criterion for continuous extension along the time axis can also be adaptively set according to the actual frame length and frame shift step length, for example, set to cover a continuous time period of 50 milliseconds to 200 milliseconds.

[0215] In another feasible implementation, the criterion for determining isolated occurrences on the time axis is that the difference energy at a certain frequency point appears only in a single time frame or at most two consecutive time frames, and the difference energy at that frequency point in the adjacent time frames before and after it is lower than a preset energy threshold.

[0216] The amplitude corresponding to the frequency point covered by the quantization noise footprint is used as the noise density index, and the amplitude corresponding to the frequency point covered by the transient distortion footprint is used as the distortion impact index. Together, they serve as the encoding and decoding impairment features.

[0217] In one feasible implementation, the noise density index is the average or weighted average of the amplitudes at all frequency points covered by the quantized noise footprint.

[0218] In another feasible implementation, the distortion impact index is the maximum or peak value of the amplitude at all frequency points covered by the transient distortion footprint.

[0219] The damage compensation module generates an adaptive enhancement control signal based on the extraction results.

[0220] The noise density index is mapped to the detail compensation strength value, and the distortion impact index is mapped to the main protection strength value, forming a dual-channel compensation parameter corresponding to the first-level damage characteristics and the second-level damage characteristics.

[0221] In one feasible implementation, the mapping method can adopt a linear mapping formula:

[0222] ;

[0223] in, For detailed compensation strength values, The noise density index, The preset noise-to-detail compensation mapping coefficients range from 0.5 to 2.0.

[0224] The mapping formula for the main protection strength value is:

[0225] ;

[0226] in, The main protection strength value, The distortion impact index, The preset mapping coefficient from distortion to the main body protection is defined, and the value of the mapping coefficient ranges from 0.3 to 1.5.

[0227] In another feasible implementation, the mapping method can also adopt piecewise nonlinear mapping, using different mapping slopes for different input exponent intervals.

[0228] In another feasible implementation, the mapping method can also adopt a lookup table approach, which maps discrete index values ​​to corresponding intensity values ​​through a pre-configured mapping table.

[0229] The detailed compensation strength value and the main protection strength value are combined as parameters to generate an adaptive enhancement control signal.

[0230] In one feasible implementation, the parameters are combined by using the two intensity values ​​as two components of a multidimensional control vector to form a two-dimensional control signal.

[0231] In another feasible implementation, the parameter combination method can also be to encapsulate the two intensity values ​​as two independent fields of the control signal.

[0232] The damage compensation module uses an adaptive enhancement control signal to perform damage compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

[0233] Based on the detail compensation intensity value in the adaptive enhancement control signal, and considering the impact of encoding and decoding impairments on the high-frequency detail bands, an amplitude gain is applied to the high-frequency detail bands in the first enhanced time-domain signal that are higher than the main frequency division boundary in the audio signal frame, as the first level of compensation operation.

[0234] In one feasible implementation, the amplitude gain is specifically achieved by multiplying the amplitude value of each frequency point within the high-frequency detail band by a gain coefficient, which is positively correlated with the detail compensation intensity value.

[0235] In another feasible implementation, the amplitude gain can be achieved by adding an offset that is positively correlated with the detail compensation intensity value to the amplitude value of each frequency point in the high-frequency detail band.

[0236] Based on the main protection strength value in the adaptive enhancement control signal, and considering the impact of encoding and decoding impairments on the main frequency band, the dynamic compression ratio of the main frequency band in the first enhanced time domain signal that is lower than the main frequency division boundary in the audio signal frame is reduced as a second-level compensation operation.

[0237] In one feasible implementation, the specific way to reduce the dynamic compression ratio is to reduce the compression ratio parameter of the dynamic range compressor in the main frequency band from its original value to a target value that is negatively correlated with the main protection strength value.

[0238] In another feasible implementation, reducing the dynamic compression ratio can be achieved by increasing the inflection point threshold in the dynamic range compression processing of the main frequency band, so that more signal amplitudes are below the inflection point threshold and are not compressed.

[0239] The enhanced music audio signal is obtained by merging the high-frequency detail bands that have undergone amplitude gain and the main frequency bands that have undergone compression ratio adjustment.

[0240] In one feasible implementation, the merging method involves directly combining the main frequency band and the high-frequency detail band at their respective frequency points.

[0241] In another feasible implementation, the merging method can also be to perform cross-gradual merging of the main frequency band and the high-frequency detail band at the boundary to eliminate abrupt changes at the frequency band boundary.

[0242] By employing the two-level extraction of encoding and decoding impairment features, mapping and generating dual-channel compensation parameters, and differentiated compensation operations for the main frequency band and high-frequency detail band, the quantization noise and transient distortion introduced by encoding and decoding in the enhanced music audio signal are specifically suppressed, while the dynamic integrity of the main frequency band is maintained. The final enhanced signal output has a significant improvement in subjective listening quality.

[0243] Reference Figure 2 The diagram shown is a flowchart illustrating a high-fidelity music adaptive enhancement method based on depth audio codec according to an embodiment of the present invention. In this embodiment, the high-fidelity music adaptive enhancement method based on depth audio codec includes:

[0244] Pt.1. Acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame;

[0245] Pt.2. Perform paragraph attribute determination on the audio signal frame, and mark the audio signal frame as a strong dynamic paragraph frame or a weak dynamic paragraph frame according to the determination result to obtain the paragraph marking result.

[0246] Pt.3. Determine the main frequency division boundary based on the paragraph marking results, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary;

[0247] Pt.4. Perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate the first enhanced frequency domain signal;

[0248] Pt.5. Perform an inverse frequency-to-time transformation on the first enhanced frequency domain signal to obtain the first enhanced time domain signal;

[0249] Pt.6 Extract the encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction results, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

[0250] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention.

[0251] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0252] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention.

Claims

1. A high-fidelity music adaptive enhancement system based on deep audio codec, characterized in that, The system includes a frame-segmentation time-frequency module, a dynamic segmentation module, a frequency-segmentation network extraction module, a frequency domain enhancement module, a time-frequency inversion module, and an impairment compensation module, wherein: The frame-segmentation time-frequency module is used to acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame. The dynamic segmentation module is used to determine the segment attributes of the audio signal frame, and mark the audio signal frame as a strong dynamic segment frame or a weak dynamic segment frame according to the determination result, so as to obtain the segment marking result. The frequency division and envelope extraction module is used to determine the main frequency division boundary based on the paragraph marking result, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary; The frequency domain enhancement module is used to perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate a first enhanced frequency domain signal. The time-frequency inverse transformation module is used to perform an inverse transformation from the frequency domain to the time domain on the first enhanced frequency domain signal to obtain the first enhanced time domain signal; The impairment compensation module is used to extract encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction result, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

2. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 1, characterized in that, The frame-segmentation time-frequency module, when performing time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame, is specifically used for: Obtain the sample point sequence of the original music audio signal; The sampling point sequence is divided into audio signal frames according to the frame length, and there is overlap between adjacent audio signal frames; A window function is applied to the audio signal frame to obtain a windowed audio frame; Perform time-frequency transformation on the windowed audio frame to obtain the frequency domain complex spectrum of the audio signal frame.

3. The high-fidelity music adaptive enhancement system based on deep audio codec as described in claim 1, characterized in that, When the dynamic segmentation module performs segment attribute determination on the audio signal frame and marks the audio signal frame as a strong dynamic segment frame or a weak dynamic segment frame based on the determination result, and obtains the segment marking result, it is specifically used for: Extract the spectral envelope from the frequency domain complex spectrum of the audio signal frame, and detect the distribution density of local extrema points in the spectral envelope as the extrema distribution density of the audio signal frame; The cross-correlation peak offset of the audio signal frame is obtained by comparing the frequency domain amplitude spectrum of the audio signal frame with the frequency domain amplitude spectrum of the previous frame. The extreme value distribution density and the cross-correlation peak offset are combined as the determination criteria; The determination criteria are compared step by step with the thresholds at each level in the multi-level precision preset threshold system; When the determination criterion is greater than the corresponding threshold in the threshold system, the audio signal frame is marked as a strong dynamic segment frame; When the determination criterion is not greater than the corresponding threshold in the threshold system, the audio signal frame is marked as a weak dynamic segment frame; The segment marking results are obtained by summarizing the marking results of the audio signal frames.

4. The high-fidelity music adaptive enhancement system based on deep audio codec as described in claim 1, characterized in that, When the frequency division and network extraction module determines the main frequency division boundary based on the segment marking result, and extracts the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary, it is specifically used for: When the paragraph marking result is a strong dynamic paragraph frame, each local maximum point of the spectral envelope in the frequency domain complex spectrum is detected, and the local maximum point with the largest amplitude is determined. The frequency corresponding to the local maximum point with the largest amplitude is used as the main frequency division boundary. The frequency bands below the main frequency division boundary are designated as the main frequency bands, and the frequency bands above the main frequency division boundary are designated as detail frequency bands. The amplitude values ​​are extracted from the frequency range corresponding to the main frequency band in the complex spectrum of the frequency domain to form the envelope of the main frequency band; The amplitude values ​​are extracted from the frequency range corresponding to the detail band in the complex spectrum of the frequency domain to form the detail band envelope.

5. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 1, characterized in that, When the frequency division and network extraction module performs the following steps: determining the main frequency division boundary based on the segment marking result, and extracting the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary, it is further used for: When the paragraph marking result is a weak dynamic paragraph frame, the spectral envelope in the frequency domain complex spectrum is differentially processed to obtain a difference sequence. The position where the sign changes from positive to negative in the difference sequence is determined and the position is taken as the inflection point position. Select the inflection point with the largest amplitude value from the frequency points corresponding to each inflection point position, and use the frequency corresponding to the inflection point with the largest amplitude value as the main frequency division boundary. The frequency bands below the main frequency division boundary are designated as the main frequency bands, and the frequency bands above the main frequency division boundary are designated as detail frequency bands. The amplitude values ​​are extracted from the frequency range corresponding to the main frequency band in the complex spectrum of the frequency domain to form the envelope of the main frequency band; The amplitude values ​​are extracted from the frequency range corresponding to the detail band in the complex spectrum of the frequency domain to form the detail band envelope.

6. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 1, characterized in that, When the frequency domain enhancement module performs dynamic range normalization on the main frequency band envelope, it is specifically used for: The amplitude value sequence is extracted from the main frequency band envelope, and the amplitude value sequence is subjected to variational processing. The amplitude value sequence is divided into rising segment and falling segment according to the positive and negative signs of the variational processing result. The degree of amplitude change between the two ends of each rising segment is measured as the rising steepness characterization quantity. The rising steepness characterization quantity of each rising segment is compared with a predetermined rising threshold. When the rising steepness characterization quantity is greater than the predetermined rising threshold, an attenuation coefficient is applied to the amplitude value in the rising segment. The degree of amplitude change between the two ends of each descending segment is measured as a descending steepness characterization quantity. The descending steepness characterization quantity of each descending segment is compared with a predetermined descending threshold. When the descending steepness characterization quantity is greater than the predetermined descending threshold, a holding coefficient is applied to the amplitude value in the descending segment. The normalized main frequency band envelope is composed of the rising amplitude value after the attenuation coefficient has been applied and the falling amplitude value after the holding coefficient has been applied.

7. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 1, characterized in that, When performing harmonic spreading on the detailed frequency band envelope, the frequency domain enhancement module is specifically used for: Jump detection is performed on the amplitude value sequence in the detailed frequency band envelope to detect jump points in the amplitude value sequence, and a predetermined bandwidth is extended forward and backward with each jump point as the starting position. Within each of the aforementioned extended predetermined bandwidths, the amplitude value at the jump point is used as the peak value, and a continuously varying replacement amplitude value is generated according to a Gaussian attenuation law; The original amplitude values ​​within the extended predetermined bandwidth are replaced with the replacement amplitude values ​​to form the extended detail band envelope.

8. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 1, characterized in that, When the damage compensation module extracts the encoding / decoding damage features from the first enhanced time-domain signal, it is specifically used for: Synchronize and align the time-domain signals corresponding to the first enhanced time-domain signal and the original music audio signal, and perform a sample-by-sample-point difference operation on the two synchronized signals to obtain the difference time-domain signal; The time-frequency transformation of the difference time-domain signal is performed to obtain the difference time-frequency spectrum, which represents the distribution of signal distortion introduced by the encoding and decoding process in the time-frequency domain. In the difference time spectrum, a frequency region that extends continuously along the time axis is detected, and the detected region is marked as the quantization noise footprint introduced by the encoding and decoding, as the first-level impairment feature; In the difference time spectrum, frequency point regions that appear isolated on the time axis are detected, and the detected regions are marked as transient distortion footprints introduced by encoding and decoding, as second-level impairment features; The amplitude corresponding to the frequency point position covered by the quantization noise footprint is used as the noise density index, and the amplitude corresponding to the frequency point position covered by the transient distortion footprint is used as the distortion impact index, which together serve as the encoding / decoding impairment feature.

9. The high-fidelity music adaptive enhancement system based on depth audio codec as described in claim 8, characterized in that, When the damage compensation module generates an adaptive enhancement control signal based on the extraction result, and uses the adaptive enhancement control signal to perform damage compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal, it is specifically used for: The noise density index is mapped to a detail compensation strength value, and the distortion impact index is mapped to a main body protection strength value, forming dual-channel compensation parameters corresponding to the first level of damage characteristics and the second level of damage characteristics; The detailed compensation strength value and the main protection strength value are combined by parameters to generate an adaptive enhancement control signal; Based on the detail compensation intensity value in the adaptive enhancement control signal, and considering the impact of encoding / decoding impairments on high-frequency detail bands, an amplitude gain is applied to the high-frequency detail bands in the first enhanced time-domain signal that are higher than the main frequency division boundary in the audio signal frame, as a first-level compensation operation. Based on the main protection strength value in the adaptive enhancement control signal, and considering the impact of encoding / decoding impairments on the main frequency band, the dynamic compression ratio of the main frequency band in the first enhanced time domain signal that is lower than the main frequency division boundary in the audio signal frame is reduced as a second-level compensation operation. The enhanced music audio signal is obtained by merging the high-frequency detail bands that have undergone amplitude gain and the main frequency bands that have undergone compression ratio adjustment.

10. A high-fidelity music adaptive enhancement method based on deep audio codec, characterized in that, The method for using the high-fidelity music adaptive enhancement system based on depth audio encoding and decoding as described in claim 1: Pt.

1. Acquire the original music audio signal and perform time-domain segmentation and time-frequency transformation to obtain the frequency domain complex spectrum of the audio signal frame; Pt.

2. Perform paragraph attribute determination on the audio signal frame, and mark the audio signal frame as a strong dynamic paragraph frame or a weak dynamic paragraph frame according to the determination result to obtain the paragraph marking result. Pt.

3. Determine the main frequency division boundary based on the paragraph marking results, and extract the main frequency band envelope and detail frequency band envelope from the frequency domain complex spectrum based on the main frequency division boundary; Pt.

4. Perform dynamic range normalization on the main frequency band envelope, perform harmonic expansion on the detail frequency band envelope, and merge the normalized main frequency band envelope with the expanded detail frequency band envelope to generate the first enhanced frequency domain signal; Pt.

5. Perform an inverse frequency-to-time transformation on the first enhanced frequency domain signal to obtain the first enhanced time domain signal; Pt.6 Extract the encoding and decoding impairment features from the first enhanced time-domain signal, generate an adaptive enhancement control signal based on the extraction results, and use the adaptive enhancement control signal to perform impairment compensation processing on the first enhanced time-domain signal to generate an enhanced music audio signal.

Images