AI-based real-time track-elimination and replacement accompaniment system for one-man band

Abstract

The application relates to the technical field of audio signal processing, and discloses an AI-based real-time track elimination and replacement accompaniment system for a single-person band, which comprises the following modules: a live environment sound acoustic isolation module which filters out the external microphone channel signal interference and outputs a live performance characteristic stream; a streaming adaptive track elimination module which performs target voice part deduction according to the original song complex spectrum and the live performance characteristic stream, and outputs a complex spectrum residual accompaniment stream and related elimination parameters; an accompaniment stream three-component decoupling module which outputs three types of accompaniment components; a dynamic micro-clock control module which outputs a performer micro-clock curve; a multi-dimensional dynamic following alignment module which outputs a clock following accompaniment stream; a time-frequency constraint matrix generation module which generates a replacement accompaniment exclusion area and a compensation area; and a replacement accompaniment reconstruction output module which adjusts candidate replacement accompaniment and mixes and outputs an accompaniment audio stream. The application can reduce residual artifacts and mutation distortion in real-time track elimination.

Description

Technical Field

[0001] This invention relates to the field of audio signal processing technology, specifically to an AI-based real-time audio track removal and accompaniment replacement system for solo bands. Background Technology

[0002] Solo band performances refer to a performance style in which a single performer presents multi-part music through vocals, instruments, accompaniment equipment, or looping equipment. This is commonly seen in street performances, small stage performances, live performances, and personal practice scenarios. Real-time audio track elimination and accompaniment replacement technology refers to identifying, reducing, or eliminating specific parts of the original song during playback and introducing new accompaniment content, allowing the performer to take over some of the playing parts of the original song, thereby creating a sound that is closer to a multi-person ensemble.

[0003] Existing technologies typically employ a source separation model to process the original audio track by splitting it into individual tracks. The track corresponding to the target voice is then attenuated or removed from the original track, and a preset or external accompaniment is mixed with the remaining audio for output. However, in real-time use, the source separation model's phase estimation of the target voice across consecutive audio frames is easily affected by live performance sounds, voice overlap, and low-confidence separation results. Directly subtracting the target voice can lead to phase discontinuities across frames, resulting in residual artifacts, abrupt distortion, or unstable accompaniment sound. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides an AI-based real-time audio track elimination and accompaniment replacement system for solo bands, solving the problems of residual artifacts and abrupt distortion caused by cross-frame phase discontinuity during real-time target part subtraction.

[0005] To achieve the above objectives, the present invention provides the following technical solution: an AI-based real-time audio track removal and accompaniment replacement system for solo bands, comprising:

[0006] The ambient sound acoustic isolation module filters out crosstalk from the external microphone channel signal based on the external pickup channel signal, and outputs the live performance characteristic stream;

[0007] The streaming adaptive audio track removal module performs spectral analysis on the original audio to obtain the complex spectrum of the original track, and performs target voice subtraction based on the complex spectrum of the original track and the live performance feature stream, outputting the complex spectrum residual accompaniment stream, separation confidence, target track amplitude estimate and energy difference before and after removal;

[0008] The three-component decoupling module of the accompaniment stream decouples the components of the complex spectrum residual accompaniment stream and outputs transient components, low-frequency periodic components and continuous harmonic components;

[0009] The dynamic micro-clock control module outputs the performer's micro-clock curve based on the live performance characteristic flow;

[0010] A multi-dimensional dynamic follow alignment module is used to adjust the time scale of the transient components, the low-frequency periodic components and the continuous harmonic components according to the performer's micro-clock curve, and output a clock-following accompaniment stream.

[0011] The time-frequency constraint matrix generation module generates a replacement accompaniment forbidden region and a replacement accompaniment compensation region based on the separation confidence, the target track amplitude estimate, the complex spectrum residual accompaniment stream, the energy difference before and after elimination, and the live performance feature stream.

[0012] The replacement accompaniment reconstruction output module receives candidate replacement accompaniments, adjusts the energy distribution of the candidate replacement accompaniments on a two-dimensional time-frequency grid based on the replacement accompaniment forbidden region and the replacement accompaniment compensation region, and mixes the adjusted candidate replacement accompaniments with the clock-following accompaniment stream to output the accompaniment audio stream.

[0013] Preferably, the ambient acoustic isolation module includes:

[0014] The priori time-frequency analysis unit performs a short-time Fourier transform on the external microphone channel signal and outputs the microphone time-frequency characteristic stream;

[0015] The transfer function estimation unit estimates the spatial acoustic transfer function from the external microphone channel to the external microphone channel based on the time-frequency characteristic stream of the microphone and the complex spectrum of the signal from the external microphone channel.

[0016] The crosstalk filtering subtraction unit generates a crosstalk estimation spectrum based on the spatial acoustic transfer function and the time-frequency characteristic stream of the pickup, and subtracts the crosstalk estimation spectrum from the complex spectrum of the external microphone channel signal to output the live performance characteristic stream.

[0017] Preferably, the crosstalk filtering subtraction unit includes:

[0018] Buffer the time-domain audio frames of the external microphone channel signal, and calculate the time delay difference between the time-domain audio frames of the external microphone channel signal and the corresponding time-domain audio frames of the external pickup channel signal;

[0019] The spatial acoustic transfer function is time-axis aligned and corrected based on the time delay difference to obtain the aligned spatial acoustic transfer function.

[0020] A crosstalk estimation spectrum is generated based on the aligned spatial acoustic transfer function and the time-frequency characteristic stream of the microphone, and the crosstalk estimation spectrum is subtracted from the complex spectrum of the external microphone channel signal to obtain the microphone complex spectrum after crosstalk filtering.

[0021] The fundamental frequency trajectory, harmonic distribution, transient position, and energy envelope are extracted from the complex spectrum of the microphone after crosstalk filtering to generate the live performance feature stream.

[0022] Preferably, the streaming adaptive audio track cancellation module includes:

[0023] The amplitude and phase estimation unit is used to input the complex spectrum of the original piece and the live performance feature stream into the trained track separation model, and output the target track amplitude estimate, the target track phase estimate and the separation confidence.

[0024] The cancellation kernel dynamic generation unit generates a cross-frame phase-continuous target audio track cancellation kernel based on the target audio track phase estimate, frequency group delay, and separation confidence.

[0025] The complex spectrum subtraction execution unit uses the target track cancellation check to perform target part subtraction on the complex spectrum of the original song, outputs the complex spectrum residual accompaniment stream, and calculates the energy difference of the complex spectrum of the original song before and after the target part subtraction to obtain the energy difference before and after the elimination.

[0026] Preferably, the offset kernel dynamic generation unit and the complex spectrum subtraction execution unit specifically include:

[0027] The phase change trend is calculated based on the target audio track phase estimate between adjacent audio frames, and the frequency group delay is determined based on the phase change trend.

[0028] Determine whether the separation confidence level corresponding to the current audio frame is lower than a preset confidence level threshold;

[0029] When the separation confidence level is not lower than the confidence threshold, a target audio track cancellation kernel for the current audio frame is generated based on the target audio track amplitude estimate, the target audio track phase estimate, and the frequency group delay.

[0030] When the separation confidence is lower than the confidence threshold, the phase angle extrapolation correction is performed on the target audio track phase estimate of the current audio frame according to the frequency group delay evolution trend corresponding to the historical audio frames with a separation confidence not lower than the confidence threshold, and the target audio track cancellation kernel of the current audio frame is generated according to the corrected target audio track phase estimate and the target audio track amplitude estimate.

[0031] The complex spectrum subtraction execution unit performs complex spectrum subtraction on the complex spectrum of the corresponding audio frame in the complex spectrum of the original song according to the target track cancellation kernel of the current audio frame, and outputs the complex spectrum residual accompaniment stream.

[0032] Preferably, the three-component decoupling module for the accompaniment stream includes:

[0033] The transient feature splitting unit calculates the energy differential value of the complex spectrum residual accompaniment stream within a short time window, and extracts transient class components within the corresponding time frame when the energy differential value is greater than a preset transient differential threshold.

[0034] The low-frequency period extraction unit extracts the low-frequency signal of the complex spectrum residual accompaniment stream through a preset low-pass filter, and locks the fundamental frequency phase in the low-frequency signal through an adaptive phase-locked loop, and outputs low-frequency periodic components.

[0035] The sustained harmony stripping unit removes the transient components and the low-frequency periodic components from the complex spectrum residual accompaniment stream, and uses the residual complex spectrum components obtained after the removal as sustained harmony components.

[0036] Preferably, the multi-dimensional dynamic following alignment module includes:

[0037] The temporal segment rearrangement unit rearranges the transient components and adjusts the segment spacing according to the performer's micro-clock curve.

[0038] The time-domain adaptive resampling unit dynamically adjusts the resampling rate of the low-frequency periodic components based on the performer's micro-clock curve.

[0039] The time-frequency domain phase-locked scaling unit performs pitch-phase-locked time-frequency domain time-scale scaling on the sustained harmony components according to the performer's micro-clock curve, and synthesizes the outputs of the time-domain segment rearrangement unit, the time-domain adaptive resampling unit, and the time-frequency domain phase-locked scaling unit into the clock-following accompaniment stream.

[0040] Preferably, the time-domain adaptive resampling unit specifically includes:

[0041] Retrieve zero-crossing points in the time-domain sampled values ​​of the low-frequency periodic components;

[0042] The zero-crossing point is determined as the splicing alignment point;

[0043] The target resampling rate is determined based on the rate of change of speed indicated by the performer's micro-clock curve.

[0044] The low-frequency periodic components are resampled in the time domain according to the target resampling rate, and splicing correction is performed at the splicing alignment point according to the first derivative continuity condition.

[0045] Preferably, the time-frequency constraint matrix generation module includes:

[0046] The residual artifact mapping unit determines the time-frequency grid coordinates occupied by the residual artifact based on the separation confidence, the target track amplitude estimate and the time-frequency correlation between the complex spectrum residual accompaniment stream, and generates the first boundary matrix of the replacement accompaniment forbidden region;

[0047] The live performance mapping unit calculates the time-frequency energy distribution of the live performance based on the live performance feature flow, determines the time-frequency grid coordinates occupied by the main vocal part of the live performance, and generates the second boundary matrix of the replacement accompaniment forbidden area;

[0048] The functional void detection unit determines the time-frequency grid coordinates corresponding to the missing vocal part based on the energy difference before and after elimination, and generates the third boundary matrix of the replacement accompaniment compensation area.

[0049] Preferably, the replacement accompaniment reconstruction output module includes:

[0050] Replace the accompaniment mapping unit to convert the candidate replacement accompaniment to the two-dimensional time-frequency grid and obtain the original time spectrum of the candidate replacement accompaniment;

[0051] The forbidden region attenuation unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the first boundary matrix and the second boundary matrix, and applies an attenuation coefficient to the candidate replacement accompaniment grid points to obtain the attenuated candidate replacement accompaniment time spectrum;

[0052] The compensation region gain unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the third boundary matrix, and applies a compensation gain coefficient to the candidate replacement accompaniment grid points according to the energy difference before and after elimination to obtain the compensated candidate replacement accompaniment time spectrum.

[0053] The complex spectrum mixing output unit adds and mixes the compensated candidate replacement accompaniment spectrum with the complex spectrum corresponding to the clock-following accompaniment stream, performs an inverse short-time Fourier transform on the added and mixed complex spectrum, and outputs the accompaniment audio stream.

[0054] This invention provides an AI-based real-time audio track removal and accompaniment replacement system for solo bands. It offers the following advantages:

[0055] 1. This invention generates a target audio track cancellation kernel by combining the complex spectrum of the original song and the characteristic flow of the live performance. When the separation confidence is insufficient, it performs phase extrapolation correction based on the historical frequency group delay evolution trend, so that the target voice subtraction process maintains phase connection between consecutive audio frames, reducing residual artifacts and abrupt distortion in real-time audio track elimination.

[0056] 2. This invention decomposes the complex spectrum residual accompaniment stream into transient components, low-frequency periodic components, and continuous harmonic components, and performs segment rearrangement, resampling rate adjustment, and time-frequency domain time scale scaling respectively, so that the residual accompaniment can follow the changes of the performer's micro-clock curve, thereby improving the time consistency between solo performance and accompaniment.

[0057] 3. This invention performs forbidden attenuation and compensated gain processing on candidate replacement accompaniment based on residual artifacts, the time-frequency grid regions corresponding to the main vocal part of the live performance and the missing vocal part, so that the replacement accompaniment avoids the area occupied by the main vocal part of the live performance and fills the accompaniment gap formed after the target vocal part is deducted. Attached Figure Description

[0058] Figure 1 This is an architecture diagram of the AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to the present invention.

[0059] Figure 2 This is an architectural diagram of the streaming adaptive audio track cancellation module of the present invention;

[0060] Figure 3 This is an architectural diagram of the replacement accompaniment reconstruction output module of the present invention. Detailed Implementation

[0061] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0062] Please see the appendix Figure 1 -Appendix Figure 3 This invention provides an AI-based real-time audio track removal and accompaniment replacement system for solo bands, comprising:

[0063] The ambient sound acoustic isolation module filters out crosstalk from the external microphone channel signal based on the external pickup channel signal, and outputs the live performance characteristic stream;

[0064] Furthermore, the on-site environmental acoustic isolation module includes:

[0065] The priori time-frequency analysis unit performs a short-time Fourier transform on the external microphone channel signal and outputs the microphone time-frequency characteristic stream;

[0066] The transfer function estimation unit estimates the spatial acoustic transfer function from the external microphone channel to the external microphone channel based on the microphone time-frequency characteristic flow and the complex spectrum of the external microphone channel signal.

[0067] The crosstalk filtering subtraction unit generates a crosstalk estimation spectrum based on the spatial acoustic transfer function and the pickup time-frequency characteristic stream, and subtracts the crosstalk estimation spectrum from the complex spectrum of the external microphone channel signal to output the live performance characteristic stream.

[0068] Specifically, in one implementation, the system receives an external pickup channel signal and an external microphone channel signal. The external pickup channel signal is formed by a pickup connected to the performer's instrument, and the external microphone channel signal is formed by a live microphone. The live ambient sound acoustic isolation module buffers the two channels at a uniform sampling rate and forms corresponding audio frames according to a preset frame length and frame shift, so that the two channels participate in subsequent processing under the same time index.

[0069] The prior time-frequency analysis unit performs windowing processing on each frame of time-domain sampled data of the external microphone channel signal and performs short-time Fourier transform to obtain complex spectrum data corresponding to each time frame and each frequency point. The complex spectrum data retains the amplitude and phase information of the microphone channel signal and forms a microphone time-frequency feature stream in chronological order. For example, with a sampling rate of 48 kHz, a frame length of 1024 points, and a frame shift of 256 points, the continuous audio frames of the microphone channel can be converted into continuous complex spectrum to form a microphone time-frequency feature stream.

[0070] The transfer function estimation unit reads the corresponding complex spectrum of the pickup time-frequency characteristic stream and the external microphone channel signal, and estimates the spatial acoustic transfer function from the external pickup channel to the external microphone channel based on the amplitude ratio, phase difference and short-time correlation between the two channels at the same time frame and the same frequency point. This spatial acoustic transfer function is used to characterize the amplitude and phase changes of the instrument component in the pickup channel when it enters the microphone channel after propagating through the field space. In order to reduce the influence of instantaneous noise, the transfer function estimation unit can smooth the estimation results of several adjacent audio frames.

[0071] The crosstalk filtering subtraction unit maps the time-frequency characteristic flow of the pickup based on the spatial acoustic transfer function to generate the crosstalk estimation spectrum in the external microphone channel. Subsequently, the crosstalk filtering subtraction unit subtracts the crosstalk estimation spectrum from the complex spectrum of the external microphone channel signal to obtain the microphone complex spectrum after crosstalk filtering. During the subtraction process, the subtraction weight can be set according to the correlation between the pickup channel and the microphone channel at each frequency point so that the crosstalk components related to the pickup channel are reduced.

[0072] After crosstalk filtering is completed, the crosstalk filtering subtraction unit extracts the fundamental frequency trajectory, harmonic distribution, transient position, and energy envelope from the complex spectrum of the microphone after crosstalk filtering, and combines these time-updated features into a live performance feature stream. The live performance feature stream is then passed to the subsequent streaming adaptive track cancellation module as live performance reference data when subtracting the target voice.

[0073] Furthermore, the crosstalk filtering subtraction unit includes:

[0074] Buffer the time-domain audio frames of the external microphone channel signal and calculate the time delay difference between the time-domain audio frames of the external microphone channel signal and the corresponding time-domain audio frames of the external pickup channel signal;

[0075] The spatial acoustic transfer function is corrected for time axis alignment based on the time delay difference to obtain the aligned spatial acoustic transfer function;

[0076] The crosstalk estimation spectrum is generated based on the aligned spatial acoustic transfer function and the time-frequency characteristic flow of the microphone, and the crosstalk estimation spectrum is subtracted from the complex spectrum of the external microphone channel signal to obtain the microphone complex spectrum after crosstalk filtering.

[0077] The fundamental frequency trajectory, harmonic distribution, transient position, and energy envelope are extracted from the complex spectrum of the microphone after crosstalk filtering to generate a live performance feature stream.

[0078] Specifically, the crosstalk filtering subtraction unit receives continuous time-domain audio frames from the external microphone channel signal and retains the time-domain audio frames corresponding to the external pickup channel signal. The audio frames from both channels are buffered using the same sampling rate, frame length, and frame shift for comparison within the same time range. The crosstalk filtering subtraction unit sets a finite-length delay search range near the current frame and compares the similarity between the time-domain audio frames of the external microphone channel signal and the corresponding time-domain audio frames of the external pickup channel signal, thereby determining the delay difference between them. This delay difference reflects the propagation delay between the instrument sound being directly collected by the pickup, passing through the air, and entering the microphone channel. For example, at a sampling rate of 48 kHz, if the maximum correlation position of the corresponding frames of the two channels differs by 96 sampling points, the delay difference of the current frame can be determined to be two milliseconds.

[0079] After determining the time delay difference, the crosstalk filtering and subtraction unit uses this time delay difference to perform time axis alignment correction on the spatial acoustic transfer function. Specifically, the spatial acoustic transfer function obtained by the transfer function estimation unit represents the amplitude and phase change relationship between the pickup channel signal and the microphone channel signal. However, this relationship will be affected by air propagation delay in the actual field. The crosstalk filtering and subtraction unit adjusts the correspondence of the spatial acoustic transfer function in the time direction according to the time delay difference of the current frame, so that the instrument component in the pickup time-frequency characteristic stream corresponds to the crosstalk component in the complex spectrum of the microphone channel within the same time frame. The spatial acoustic transfer function obtained after correction is used for subsequent crosstalk estimation spectrum generation, reducing the spectrum subtraction deviation caused by time misalignment.

[0080] Subsequently, the crosstalk filtering subtraction unit applies the aligned spatial acoustic transfer function to the microphone time-frequency feature stream. Each time frame and each frequency point in the microphone time-frequency feature stream contains corresponding amplitude and phase information. The aligned spatial acoustic transfer function is used to describe the propagation amplitude and phase changes at that frequency point. The crosstalk filtering subtraction unit generates a crosstalk estimation spectrum based on this. This crosstalk estimation spectrum corresponds to the estimated crosstalk component formed by the instrument's air propagation in the external microphone channel signal. After completing the crosstalk estimation, the crosstalk filtering subtraction unit subtracts the crosstalk estimation spectrum from the complex spectrum of the external microphone channel signal to obtain the microphone complex spectrum after crosstalk filtering. During the subtraction process, the subtraction weight can be set according to the correlation between the two channels at the corresponding frequency points, so that the highly correlated instrument crosstalk components are subtracted, while other sound source components in the field microphone that are less correlated with the microphone channel are retained.

[0081] After obtaining the complex spectrum of the microphone after crosstalk filtering, the crosstalk filtering subtraction unit further extracts time-frequency features to describe the state of the live performance. The fundamental frequency trajectory can be determined by the fundamental frequency position corresponding to the stable harmonic sequence in each time frame. The harmonic distribution can be determined by the energy distribution of the fundamental frequency and its integer multiples near the frequency. The transient position can be determined by the energy change position between adjacent time frames. The energy envelope can be determined by the change of the overall spectral energy of each time frame over time. The above features are combined in the order of time frames to form the live performance feature stream.

[0082] The streaming adaptive audio track removal module performs spectral analysis on the original audio to obtain the complex spectrum of the original track, and performs target voice subtraction based on the complex spectrum of the original track and the live performance characteristic stream. It outputs the complex spectrum residual accompaniment stream, separation confidence, target track amplitude estimate and energy difference before and after removal.

[0083] Furthermore, the streaming adaptive audio track cancellation module includes:

[0084] The amplitude and phase estimation unit is used to input the complex spectrum of the original song and the feature stream of the live performance into the trained track separation model, and output the target track amplitude estimate, the target track phase estimate and the separation confidence.

[0085] The cancellation kernel dynamic generation unit generates a cross-frame phase-continuous target audio track cancellation kernel based on the target audio track phase estimate, frequency group delay, and separation confidence.

[0086] The complex spectrum subtraction execution unit uses the target track cancellation check to perform target part subtraction on the complex spectrum of the original song, outputs the complex spectrum residual accompaniment stream, and calculates the energy difference of the complex spectrum of the original song before and after the target part subtraction, thus obtaining the energy difference before and after elimination.

[0087] Specifically, the streaming adaptive audio track elimination module receives the original audio and the aforementioned live performance feature stream. The original audio can be a local audio file, a real-time playback audio stream, or audio data from a backing track library. This module first divides the original audio into frames according to the same frame length and frame shift as the live performance feature stream, and performs windowing and spectrum analysis on each frame to obtain the complex spectrum of the original audio arranged by time frame and frequency point. The complex spectrum of the original audio retains the amplitude and phase information of the original audio at each frequency point. The subsequent target part subtraction uses this complex spectrum as the direct processing object, rather than only performing overall attenuation on the time domain waveform.

[0088] Within the current processing frame, the amplitude and phase estimation unit aligns the complex spectrum of the original piece with the live performance feature stream using time indexing. The fundamental frequency trajectory, harmonic distribution, transient position, and energy envelope in the live performance feature stream are organized into conditional features with the same frame order as the complex spectrum of the original piece. This enables the trained track separation model to simultaneously read the time-frequency content of the original piece and the live performance state. The input data of the model includes the amplitude and phase components of the complex spectrum of the original piece and the feature vector of the corresponding frame of the live performance feature stream. The model estimates the amplitude and phase distribution of the target track at each time frame and each frequency point, and generates a separation confidence score based on the degree of distinction between the target voice and other voices. The separation confidence score can be obtained by combining the target voice probability output by the model, the estimation stability of adjacent frames, and the time-frequency masking intensity of the target voice. It is used to indicate the reliability of the target track estimation result in the current frame for subtraction.

[0089] After receiving the target track amplitude estimate, target track phase estimate, and separation confidence, the cancellation kernel dynamic generation unit does not directly use the target track estimated by the model as a fixed waveform for subtraction. Instead, it first determines the frequency group delay based on the target track phase change between adjacent audio frames, and then uses the separation confidence to generate the target track cancellation kernel by directly using or modifying the phase of the current frame. The cancellation kernel contains the target track amplitude subtraction amount and phase direction at each frequency point and is continuously updated according to time frames to keep the cancellation direction between adjacent frames smooth. For example, when the original song contains a rhythm guitar part and the live performance feature stream shows that the performer is playing a guitar in a similar frequency band, the model will give a higher target track amplitude estimate in the mid-frequency harmonic region occupied by that part. The cancellation kernel dynamic generation unit will then form corresponding subtraction kernels in these time-frequency regions accordingly.

[0090] The complex spectrum subtraction execution unit applies the target track cancellation kernel to the complex spectrum of the original music. For each frequency point in the current audio frame, the complex spectrum subtraction execution unit subtracts the corresponding target voice component from the complex spectrum of the original music according to the amplitude and phase indicated by the target track cancellation kernel, obtaining the subtracted complex spectrum data. The subtracted complex spectra of consecutive time frames are arranged in chronological order to form a complex spectrum residual accompaniment stream. This residual accompaniment stream retains the accompaniment components in the original music that were not subtracted from the target voice and serves as the input data for subsequent three-component decoupling processing.

[0091] While performing target voice subtraction, the complex spectrum subtraction execution unit also calculates the energy change before and after the target voice subtraction. Specifically, the unit counts the energy of the original complex spectrum at each frequency point in the current frame, and the energy of the subtracted complex spectrum at the corresponding frequency point. The difference between the two forms the energy difference before and after subtraction. This energy difference is saved with time frames and frequency points and is used to represent the energy gap formed on the time-frequency grid after the target voice is subtracted. The energy difference before and after subtraction is then passed to the time-frequency constraint matrix generation module to determine the replacement accompaniment compensation area. The separation confidence, target track amplitude estimate and complex spectrum residual accompaniment stream are also saved simultaneously and used for subsequent residual artifact mapping, accompaniment stream decoupling and replacement accompaniment reconstruction processing, respectively.

[0092] Furthermore, the offset kernel dynamic generation unit and the complex spectrum subtraction execution unit specifically include:

[0093] The phase change trend is calculated based on the target audio track phase estimate between adjacent audio frames, and the frequency group delay is determined based on the phase change trend.

[0094] Determine whether the separation confidence level corresponding to the current audio frame is lower than the preset confidence level threshold;

[0095] When the separation confidence level is not lower than the confidence threshold, the target audio track cancellation kernel of the current audio frame is generated based on the target audio track amplitude estimate, the target audio track phase estimate, and the frequency group delay.

[0096] When the separation confidence is lower than the confidence threshold, the phase angle extrapolation correction is performed on the target audio track phase estimate of the current audio frame based on the frequency group delay evolution trend corresponding to the historical audio frames with a separation confidence not lower than the confidence threshold. The target audio track cancellation kernel of the current audio frame is generated based on the corrected target audio track phase estimate and the target audio track amplitude estimate.

[0097] The complex spectrum subtraction execution unit performs complex spectrum subtraction on the complex spectrum of the corresponding audio frame in the complex spectrum of the original song based on the target track cancellation kernel of the current audio frame, and outputs the complex spectrum residual accompaniment stream.

[0098] Specifically, in one implementation, the cancellation kernel dynamic generation unit receives the target audio track phase estimate values ​​from consecutive audio frames and compares the phase changes of adjacent frames at the same frequency point in time frame order. Since the phase values ​​exhibit periodic jumps, the cancellation kernel dynamic generation unit performs phase expansion on the target audio track phase estimate values ​​before calculating the phase change trend, ensuring that the phase difference between adjacent frames remains within a continuously varying range. Subsequently, based on the change in the phase difference between adjacent frames with frequency points, a frequency group delay is determined to characterize the phase advancement trend of the target audio track at different frequency points. This frequency group delay is not used as the output audio alone, but rather as constraint data for the phase continuity of the cancellation kernel in the current frame.

[0099] The cancellation kernel dynamic generation unit reads the separation confidence level within the current audio frame and compares it with a preset confidence threshold. The separation confidence level can be the confidence data at the corresponding frequency point in the current time frame, or it can be the confidence statistics of the main frequency band of the target sound part in the current frame. The confidence threshold is used to distinguish whether the target audio track estimation result is suitable for direct participation in complex spectrum subtraction. For example, the confidence threshold can be set to 0.7. When the separation confidence level of a certain frequency point in the current frame is 0.82, the frequency point uses the model output phase to participate in the cancellation kernel generation; when the separation confidence level is 0.55, the frequency point enters the phase extrapolation correction process.

[0100] When the separation confidence of the current audio frame is not lower than the confidence threshold, the cancellation kernel dynamic generation unit generates the target audio track cancellation kernel for the current audio frame using the target audio track amplitude estimate and the target audio track phase estimate. The cancellation kernel represents the component to be subtracted from the target sound part at the corresponding time frame and frequency point in the complex spectral domain. Its generation process can be expressed as follows:

[0101] ;

[0102] in, Indicates the first Each audio frame at frequency point The target audio track cancellation kernel, This represents the amplitude estimate of the target audio track. This represents the target audio track phase value used in the current audio frame. When the separation confidence level is not lower than the confidence threshold, this phase value is the estimated target audio track phase value. Represents the imaginary unit. Represents a complex direction determined by the phase;

[0103] When the separation confidence of the current audio frame is lower than the confidence threshold, the cancellation kernel dynamic generation unit does not directly use the target audio track phase estimate of the current frame. Instead, it calls historical audio frames whose separation confidence is not lower than the confidence threshold. Based on the evolution trend of frequency group delay in these historical audio frames, this unit determines the phase change at the corresponding frequency point of the current frame and extrapolates and corrects the target audio track phase estimate of the current frame. The phase extrapolation correction can be expressed as:

[0104] ;

[0105] in, This represents the phase value of the target audio track after correction for the current low-confidence audio frame. Indicates the current audio frame number. This indicates the most recent audio frame with a satisfactory confidence level preceding the current audio frame. Indicates the first Frame at frequency point Phase estimation of the target audio track, This represents the average phase change obtained based on the historical frequency group delay evolution trend. Through this processing, low-confidence frames can still generate cancellation kernels by following the historical phase continuity trend, avoiding abrupt changes in the cancellation direction between adjacent frames.

[0106] After obtaining the target track cancellation kernel for the current audio frame, the complex spectrum subtraction execution unit reads the complex spectrum of the corresponding audio frame from the original song's complex spectrum and performs complex spectrum subtraction according to the target track cancellation kernel. The subtraction process does not merely reduce amplitude; rather, it subtracts the target voice components within the complex spectrum plane according to the amplitude and phase direction of the cancellation kernel. This process can be represented as:

[0107] ;

[0108] in, The complex spectral residual accompaniment flow represents the first... audio frames and frequency points The complex spectrum on Represents the complex spectrum of the original melody. This indicates the target audio track cancellation kernel for the current audio frame. This represents the subtraction strength determined by the separation confidence level. When the separation confidence level is high, A larger value can be taken; when the separation confidence level is close to the confidence threshold, A smaller value can be chosen to reduce the risk of over-deduction;

[0109] For example, at the 800 Hz frequency point of the 200th frame, when the target track amplitude estimate is 0.6, the separation confidence is 0.8, and the confidence threshold is 0.7, the cancellation kernel dynamic generation unit uses the target track phase estimate of that frame to generate the cancellation kernel. At the same frequency point of the 201st frame, when the separation confidence drops to 0.56, the unit corrects the phase based on the frequency group delay evolution trend of the previous confidence-compliant frames and then generates the corresponding cancellation kernel. The complex spectrum subtraction results of consecutive frames are arranged in chronological order to form a complex spectrum residual accompaniment stream, which continues to participate in the three-component decoupling processing of subsequent accompaniment streams.

[0110] The three-component decoupling module for accompaniment stream decouples the components of the complex spectrum residual accompaniment stream and outputs transient components, low-frequency periodic components, and continuous harmonic components.

[0111] Furthermore, the three-component decoupling module for the accompaniment stream includes:

[0112] The transient feature splitting unit calculates the energy differential value of the complex spectrum residual accompaniment stream within a short time window, and extracts transient class components within the corresponding time frame when the energy differential value is greater than the preset transient differential threshold.

[0113] The low-frequency period extraction unit extracts the low-frequency signal of the complex spectrum residual accompaniment stream through a preset low-pass filter, and locks the fundamental frequency phase in the low-frequency signal through an adaptive phase-locked loop, outputting low-frequency periodic components.

[0114] The sustained harmony stripping unit removes transient and low-frequency periodic components from the complex spectrum residual accompaniment stream, and uses the residual complex spectrum components obtained after the removal as sustained harmony components.

[0115] Specifically, in one implementation, the three-component decoupling module of the accompaniment stream receives the complex spectrum residual accompaniment stream output from the previous stage and maintains the time frame order and frequency point index of the complex spectrum residual accompaniment stream. Since the complex spectrum residual accompaniment stream simultaneously contains drum-like transients, low-frequency periodic parts, and sustained backing parts, if the entire stream is directly adjusted in subsequent time scales, it is easy to stretch short transients or shift the phase of low-frequency periods. Therefore, this module first performs layered processing on the complex spectrum residual accompaniment stream within the same two-dimensional time-frequency structure, so that residual accompaniment components with different acoustic characteristics enter different subsequent alignment paths.

[0116] The transient feature splitting unit reads the complex spectrum residual accompaniment stream in short time windows and counts the energy changes between adjacent windows. This energy change can be obtained from the difference between the spectral energy of the current window and the spectral energy of the previous window. If the energy differential value in a certain time frame is greater than the preset transient differential threshold, it indicates that there is a rapid energy rise acoustic event in that time frame. The transient feature splitting unit further extracts the frequency points with significant energy rise in that time frame and marks the corresponding complex spectrum components as transient components. For example, in the complex spectrum residual accompaniment stream, if a certain frame has a significant energy surge in the range of 200 Hz to 5 kHz, and the duration of the surge is shorter than that of several adjacent continuous sound parts frames, then the corresponding time-frequency components of that frame can be extracted as transient components.

[0117] After the transient component is split, the low-frequency period extraction unit extracts the low-frequency band from the complex spectrum residual accompaniment stream. The range of the low-frequency band can be preset according to the system sampling rate and accompaniment type. For example, 20 Hz to 250 Hz can be selected as the low-frequency analysis range. The low-frequency period extraction unit retains the complex spectrum components in this frequency band through a preset low-pass filter and converts the low-frequency band signal into a time domain or phase sequence that is easy to track. Subsequently, the adaptive phase-locked loop locks the fundamental frequency phase according to the period peak, zero-crossing position or phase advancement relationship in the low-frequency band signal, so that the fundamental frequency phase remains trackable in a continuous time frame. The low-frequency complex spectrum components constrained by the locked fundamental frequency phase are output as low-frequency period components for subsequent dynamic adjustment of the resampling rate.

[0118] After obtaining transient and low-frequency periodic components, the sustained harmony stripping unit performs residual component calculations on the complex spectrum residual accompaniment stream. Specifically, under the same time frame and frequency point index, this unit subtracts the complex spectrum components corresponding to transient and low-frequency periodic components from the complex spectrum residual accompaniment stream and smooths the remaining complex spectrum. This remaining complex spectrum usually has a long duration, slow energy change, and relatively stable harmonic distribution, and is therefore regarded as a sustained harmony component. The resulting transient, low-frequency periodic, and sustained harmony components maintain the same time frame index as the original complex spectrum residual accompaniment stream. The subsequent multi-dimensional dynamic follow-alignment module can apply different time scale adjustment methods to the three types of components respectively.

[0119] The dynamic micro-clock control module outputs the performer's micro-clock curve based on the characteristics of the live performance.

[0120] Specifically, the dynamic micro-clock control module receives the live performance feature stream and reads the changes in fundamental frequency trajectory, transient position, and energy envelope over time frames. This module uses continuous transient positions as the performer's accent points, stable segments of the fundamental frequency trajectory as sustained performance segments, and rising segments of the energy envelope as the starting position reference, thereby forming a time sequence of the performer's current playing beat. Subsequently, the dynamic micro-clock control module smooths this time sequence, filtering out local abrupt changes caused by pickup jitter, occasional noise, or weak start, resulting in a continuously changing performer micro-clock curve. This performer micro-clock curve is used to represent the local speed changes, start offset, and short-term beat drift of the live performance relative to the original accompaniment, and is then passed to the multi-dimensional dynamic follow-alignment module as the control basis for time-scale adjustments of transient components, low-frequency periodic components, and sustained harmonic components.

[0121] The multi-dimensional dynamic follow alignment module is used to adjust the time scale of transient components, low-frequency periodic components and continuous harmonic components according to the performer's micro-clock curve, and output a clock-following accompaniment stream.

[0122] Furthermore, the multi-dimensional dynamic follow alignment module includes:

[0123] The temporal segment rearrangement unit rearranges transient components and adjusts segment spacing based on the performer's micro-clock curve.

[0124] The time-domain adaptive resampling unit dynamically adjusts the resampling rate of low-frequency periodic components based on the performer's micro-clock curve;

[0125] The time-frequency domain phase-locked scaling unit performs time-frequency domain time-scale scaling of the continuous harmony components with pitch phase locking according to the performer's micro-clock curve, and synthesizes the outputs of the time-domain segment rearrangement unit, the time-domain adaptive resampling unit, and the time-frequency domain phase-locked scaling unit into a clock-following accompaniment stream.

[0126] Specifically, in one implementation, the multi-dimensional dynamic follow alignment module receives the performer's micro-clock curve, transient components, low-frequency periodic components, and sustained harmonic components. The performer's micro-clock curve is indexed by time frames and records the local speed changes, start offsets, and short-term beat shifts of the live performance relative to the original accompaniment. This module does not perform uniform stretching on the entire complex spectrum residual accompaniment stream, but adjusts the time scale separately according to the acoustic characteristics of the three types of components, so that transient, low-frequency periodic, and sustained harmonic components use different data processing methods when following the same micro-clock curve.

[0127] The temporal segment rearrangement unit first converts transient components into a temporal segment sequence, and determines the start, peak, and end positions of each segment based on the transient position. When the performer's micro-clock curve indicates the time offset of the current performance relative to the original accompaniment, the temporal segment rearrangement unit adjusts the arrangement of each transient segment on the time axis. When the performer's micro-clock curve indicates the speed to increase or decrease, the unit adjusts the spacing between adjacent transient segments. This processing mainly changes the occurrence time and interval of transient segments, without significantly stretching individual transient waveforms, thereby maintaining the onset boundary of percussive transients. For example, if the performer shortens the interval between two consecutive beats from 500 milliseconds to 480 milliseconds, the unit will synchronously compress the segment spacing of the corresponding transient segment by 20 milliseconds.

[0128] After receiving low-frequency periodic components, the time-domain adaptive resampling unit first reads the velocity change rate corresponding to the current time frame in the performer's micro-clock curve, and then determines the target resampling rate for the low-frequency periodic components. These components typically contain periodic waveforms of bass or low-frequency drum sounds. If only segment movement is used for tracking, phase discontinuities at the junctions of adjacent periods can easily occur. Therefore, this unit dynamically adjusts the resampling rate within the continuous sampling sequence of the low-frequency periodic waveform, causing the period length to change with the performer's micro-clock curve. For segments with slightly increased speed, the sampling interval of the low-frequency periodic components is compressed accordingly; for segments with decreased speed, the sampling interval is extended accordingly. This processing result is used as the low-frequency aligned output and enters the subsequent synthesis stage.

[0129] The time-frequency domain phase-locked scaling unit receives sustained harmonic components and tracks their fundamental tone and harmonic phase relationships in the time-frequency domain. Since the energy changes of sustained harmonic components are relatively gradual, simple time-domain slicing can easily lead to harmonic phase drift. Therefore, this unit determines the scaling ratio for each time frame based on the performer's micro-clock curve and maintains the relative relationship between the fundamental tone phase and the main harmonic phase during the scaling process. For sustained undertones, the unit achieves time-scale scaling by adjusting the advance interval between time-frequency frames and performs phase continuity processing when synthesizing adjacent frames, enabling sustained harmonic components to follow the changes in live performance tempo.

[0130] After aligning the three types of components separately, the multi-dimensional dynamic follow alignment module converts the outputs of the time-domain segment rearrangement unit, the time-domain adaptive resampling unit, and the time-frequency domain phase-locked scaling unit to a unified time axis. If the three outputs are in different representation domains, they are first unified into the corresponding complex spectrum or time-domain audio frames, and then superimposed and synthesized according to the same frame order. The synthesized data forms a clock-following accompaniment stream and continues to be passed to the replacement accompaniment reconstruction output module for subsequent mixing with the time-frequency energy adjustment results of the candidate replacement accompaniment.

[0131] Furthermore, the time-domain adaptive resampling unit specifically includes:

[0132] Retrieve zero-crossing points in the time-domain sampled values ​​of low-frequency periodic components;

[0133] The zero-crossing point is determined as the splicing alignment point;

[0134] The target resampling rate is determined based on the rate of change of speed indicated by the performer's micro-clock curve;

[0135] The low-frequency periodic components are resampled in the time domain according to the target resampling rate, and the splicing is corrected at the splicing alignment point according to the first derivative continuity condition.

[0136] Specifically, after receiving low-frequency periodic components, the time-domain adaptive resampling unit first converts the low-frequency periodic components into a continuous time-domain sampled value sequence and retains the correspondence between the sequence and the original time frame. Since low-frequency periodic components usually have obvious periodic waveforms, the time-domain adaptive resampling unit detects the sign change of adjacent sampled values ​​point by point in the sampled value sequence and determines the zero-crossing point by combining the position where the sampled value amplitude is close to zero. In the case of noise disturbance in the low-frequency waveform, the unit can first perform short-window smoothing on the low-frequency periodic components and then search for zero-crossing points to reduce false crossover positions caused by small jitter.

[0137] After determining the zero-crossing point, the time-domain adaptive resampling unit uses the zero-crossing point as the splicing alignment point. The splicing alignment point is used to divide the continuous waveform segments of the low-frequency periodic components, so that subsequent resampling and segment splicing are performed at positions where the waveform amplitude is close to zero. Compared with direct segmentation at arbitrary sampling points, the amplitude jump near the zero-crossing point is smaller, so it is more suitable as a segment boundary. For example, in the low-frequency periodic components with a sampling rate of 48 kHz, if the continuous waveform changes from a positive value to a negative value near the 1200th sampling point, this position can be determined as a splicing alignment point, and the adjacent periodic segments are processed with this position as the boundary.

[0138] Subsequently, the time-domain adaptive resampling unit reads the velocity change rate corresponding to the current time frame from the performer's micro-clock curve and determines the target resampling rate accordingly. When the performer's micro-clock curve indicates that the live performance speed is higher than the original accompaniment speed, the target resampling rate is increased relative to the original sampling rate, shortening the duration of low-frequency periodic components after resampling; when the performer's micro-clock curve indicates that the live performance speed is lower than the original accompaniment speed, the target resampling rate is decreased relative to the original sampling rate, lengthening the duration of low-frequency periodic components after resampling. This target resampling rate does not change the source of the low-frequency periodic components; it is only used to control the scaling of the low-frequency periodic waveform on the time axis.

[0139] During time-domain interpolation resampling, the time-domain adaptive resampling unit interpolates the waveform values ​​between adjacent sampling points according to the target resampling rate, generating a new low-frequency periodic sampling sequence. The interpolation method can be linear or cubic interpolation. Based on the splicing alignment point, the resampled low-frequency periodic segments are rearranged to the time positions corresponding to the performer's micro-clock curve. For the splicing boundary between adjacent segments, the unit compares the slope of the sample value change between the end of the previous segment and the beginning of the next segment at the splicing alignment point, and performs smoothing corrections on several sampling points near the boundary to ensure the splicing point satisfies the first derivative continuity condition. The resulting low-frequency periodic components maintain the continuity of the low-frequency periodic waveform while following the speed change of the performer's micro-clock curve, and are used as the low-frequency aligned output in the subsequent clock-following accompaniment stream synthesis process.

[0140] The time-frequency constraint matrix generation module generates a replacement accompaniment forbidden region and a replacement accompaniment compensation region based on the separation confidence, the target track amplitude estimate, the complex spectrum residual accompaniment stream, the energy difference before and after elimination, and the live performance feature stream.

[0141] Furthermore, the time-frequency constraint matrix generation module includes:

[0142] The residual artifact mapping unit determines the time-frequency grid coordinates occupied by the residual artifacts based on the separation confidence, the target track amplitude estimate, and the time-frequency correlation between the complex spectrum residual accompaniment stream, and generates the first boundary matrix of the replacement accompaniment forbidden region.

[0143] The live performance mapping unit calculates the time-frequency energy distribution of the live performance based on the live performance feature flow, determines the time-frequency grid coordinates occupied by the main vocal part of the live performance, and generates a second boundary matrix for the replacement accompaniment forbidden area;

[0144] The functional void detection unit determines the time-frequency grid coordinates corresponding to the missing vocal part based on the energy difference before and after elimination, and generates the third boundary matrix of the replacement accompaniment compensation area.

[0145] Specifically, in one implementation, the time-frequency constraint matrix generation module receives the separation confidence, target track amplitude estimate, complex spectrum residual accompaniment stream, and energy difference before and after elimination from the previous streaming adaptive track elimination module. At the same time, it receives the live performance feature stream from the live environment acoustic isolation module. All types of data are mapped to a two-dimensional time-frequency grid according to the same time frame index and frequency point index, so that the subsequently generated boundary matrix can correspond to the time spectrum of the candidate replacement accompaniment. The replacement accompaniment forbidden area generated by this module is used to restrict the candidate replacement accompaniment from entering the residual artifact area and the live performance main voice area. The replacement accompaniment compensation area is used to indicate the energy loss position caused by the subtraction of the target voice.

[0146] The residual artifact mapping unit reads the target track amplitude estimate and the complex spectrum residual accompaniment stream within the current time frame, and calculates the time-frequency correlation between them in the neighborhood of each time-frequency grid to be analyzed. If the target track amplitude estimate still maintains a high correlation with the complex spectrum residual accompaniment stream at a certain time-frequency position, and the separation confidence at that position is lower than the confidence threshold, then that position is more likely to correspond to the residual artifact left after the target track is subtracted. The first boundary matrix can be generated by the following formula:

[0147] ;

[0148] in, Indicates the first boundary matrix at the th audio frames and frequency points The value on, This represents the amplitude estimate of the target audio track. This represents the amplitude spectrum of the complex spectral residual accompaniment stream. Indicates the first audio frames and frequency points The time-frequency neighborhood centered on the center This represents the correlation calculated within the time-frequency neighborhood. Indicates the relevance threshold. Indicates the confidence level of separation. Indicates the confidence threshold. This indicates a conditional indicator function. The value is one when the condition is true and zero when the condition is false. For example, when the time-frequency correlation at a certain frequency point is 0.78, the correlation threshold is 0.65, the separation confidence is 0.52, and the confidence threshold is 0.7, the first boundary matrix corresponding to that time-frequency grid takes the value of one.

[0149] The live performance mapping unit generates the live performance time-frequency energy distribution based on the live performance feature stream. The fundamental frequency trajectory in the live performance feature stream is used to locate the main frequency of the live performance, the harmonic distribution is used to determine the harmonic occupancy range corresponding to the main frequency, the transient position is used to supplement the short-time initiation area, and the energy envelope is used to determine the overall energy change of each time frame. The live performance mapping unit maps the above features to a two-dimensional time-frequency grid and forms a second boundary matrix at the positions where the live performance time-frequency energy exceeds the live performance energy threshold. The second boundary matrix can be generated by the following formula:

[0150] ;

[0151] in, The second boundary matrix is ​​represented in the first... audio frames and frequency points The value on, This represents the time-frequency energy distribution of a live performance calculated based on the characteristic flow of the live performance. Indicates the energy threshold for live performance. This represents the conditional indicator function. The second boundary matrix, together with the first boundary matrix, constitutes a forbidden region for replacement accompaniment, ensuring that candidate replacement accompaniment avoids the time-frequency positions occupied by the main vocal part in the live performance during subsequent reconstruction.

[0152] The functional void detection unit reads the energy difference before and after elimination and determines the location of the energy loss formed after the target sound part is subtracted according to the time frame and frequency point index. If the energy difference before and after elimination on a certain time-frequency grid is greater than the preset energy difference threshold, then the location is determined as the compensation location corresponding to the missing sound part, and the third boundary matrix can be generated by the following formula:

[0153] ;

[0154] in, The third boundary matrix is ​​represented in the first... audio frames and frequency points The value of ; This represents the energy difference of the target acoustic part on the time-frequency grid before and after subtraction; Indicates the energy difference threshold; This represents the conditional indication function. The functional hole detection unit determines the replacement accompaniment compensation region using the third boundary matrix. The first, second, and third boundary matrices are then passed to the replacement accompaniment reconstruction output module for no-entry attenuation and gain compensation processing on the candidate replacement accompaniments.

[0155] The replacement accompaniment reconstruction output module receives candidate replacement accompaniments, adjusts the energy distribution of the candidate replacement accompaniments on a two-dimensional time-frequency grid based on the replacement accompaniment forbidden region and the replacement accompaniment compensation region, and mixes the adjusted candidate replacement accompaniments with the clock-following accompaniment stream to output the accompaniment audio stream.

[0156] Furthermore, the replacement accompaniment reconstruction output module includes:

[0157] Replace the accompaniment mapping unit to transform the candidate replacement accompaniment to a two-dimensional time-frequency grid, and obtain the original time spectrum of the candidate replacement accompaniment;

[0158] The forbidden region attenuation unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the first boundary matrix and the second boundary matrix, and applies an attenuation coefficient to the candidate replacement accompaniment grid points to obtain the attenuated candidate replacement accompaniment time spectrum.

[0159] The compensation region gain unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the third boundary matrix, and applies a compensation gain coefficient to the candidate replacement accompaniment grid points according to the energy difference before and after elimination to obtain the compensated candidate replacement accompaniment time spectrum.

[0160] The complex spectrum mixing output unit adds the compensated candidate replacement accompaniment spectrum with the complex spectrum corresponding to the clock-following accompaniment stream, and performs an inverse short-time Fourier transform on the added mixed complex spectrum to output the accompaniment audio stream.

[0161] Specifically, in one implementation, the accompaniment accent output module receives candidate replacement accompaniments, a first boundary matrix, a second boundary matrix, a third boundary matrix, an accompaniment stream with energy difference before and after elimination, and a clock-following accompaniment stream. The candidate replacement accompaniment can be an accompaniment audio stream corresponding to the tempo and key of the original song. The replacement accompaniment mapping unit performs windowing and spectral analysis on the candidate replacement accompaniment according to the same frame length, frame shift, and frequency resolution as the complex spectrum residual accompaniment stream, and maps the obtained complex spectrum to a two-dimensional time-frequency grid. The time frame index and frequency point index of this two-dimensional time-frequency grid are consistent with the first boundary matrix, the second boundary matrix, and the third boundary matrix, thereby obtaining the original time-frequency spectrum of the candidate replacement accompaniment. This original time-frequency spectrum serves as the basis data for subsequent attenuation and compensation processing.

[0162] The disabled region attenuation unit reads the original time spectrum of the candidate replacement accompaniment and retrieves the candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the first and second boundary matrices, where the first boundary matrix corresponds to the region where residual artifacts are located and the second boundary matrix corresponds to the region where the main vocal part of the live performance is located. For the aforementioned overlapping grid points, the disabled region attenuation unit does not delete the entire audio segment of the candidate replacement accompaniment, but instead applies an attenuation coefficient at the corresponding time frame and frequency point. This process can be expressed as:

[0163] ;

[0164] in, Indicates the candidate replacement accompaniment in the 1st... audio frames and frequency points The original time spectrum on This represents the spectrum of the candidate replacement accompaniment after decay. This represents the value of the first boundary matrix on this time-frequency grid. This indicates the value of the second boundary matrix on this time-frequency grid. This represents the attenuation coefficient corresponding to the first boundary matrix; This represents the attenuation coefficient corresponding to the second boundary matrix, and and The values ​​of are all in the range of zero to one. Through the product attenuation method, the spectral amplitude can continue to be non-negative when both forbidden boundaries are hit at the same time.

[0165] The compensation region gain unit reads the third boundary matrix and the energy difference before and after elimination, and extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the third boundary matrix where the value is one. The third boundary matrix corresponds to the compensation region formed after the target voice is subtracted. The compensation region gain unit applies compensation gain to the attenuated candidate replacement accompaniment time-frequency spectrum based on the energy difference before and after elimination at the corresponding grid points. This process can be expressed as:

[0166] ;

[0167] in, The spectrum of the candidate replacement accompaniment after compensation. This represents the spectrum of the candidate replacement accompaniment after decay. The third boundary matrix is ​​represented in the first... audio frames and frequency points The value on, This represents the energy difference before and after elimination on the time-frequency grid. Indicates the compensation gain coefficient. This represents a positive constant to prevent the denominator from being zero. For example, at a certain time-frequency grid... As one, When the energy is relatively large, and the candidate replacement accompaniment already has a certain energy at that position, the compensation region gain unit increases the energy of the candidate replacement accompaniment at that grid point according to the above gain relationship. When the value is zero, no gain compensation is performed on the grid point;

[0168] The complex spectrum mixing output unit receives the compensated candidate replacement accompaniment time spectrum and the complex spectrum corresponding to the clock-following accompaniment stream. That is, it performs complex spectrum addition at the same time frame and frequency point, so that the compensated part of the candidate replacement accompaniment and the clock-following result of the previous residual accompaniment are located in the same time-frequency structure. After completing the addition mixing, the complex spectrum mixing output unit performs inverse short-time Fourier transform on the added complex spectrum and performs overlapping addition according to the frame shift to obtain a continuous accompaniment audio stream. This accompaniment audio stream is the system output result and maintains a corresponding relationship with the performance part of the live performer in terms of time scale and time-frequency occupation.

[0169] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An AI-based real-time audio track removal and accompaniment replacement system for solo bands, characterized in that, include: The ambient sound acoustic isolation module filters out crosstalk from the external microphone channel signal based on the external pickup channel signal, and outputs the live performance characteristic stream; The streaming adaptive audio track removal module performs spectral analysis on the original audio to obtain the complex spectrum of the original track, and performs target voice subtraction based on the complex spectrum of the original track and the live performance feature stream, outputting the complex spectrum residual accompaniment stream, separation confidence, target track amplitude estimate and energy difference before and after removal; The three-component decoupling module of the accompaniment stream decouples the components of the complex spectrum residual accompaniment stream and outputs transient components, low-frequency periodic components and continuous harmonic components; The dynamic micro-clock control module outputs the performer's micro-clock curve based on the live performance characteristic flow; A multi-dimensional dynamic follow alignment module is used to adjust the time scale of the transient components, the low-frequency periodic components and the continuous harmonic components according to the performer's micro-clock curve, and output a clock-following accompaniment stream. The time-frequency constraint matrix generation module generates a replacement accompaniment forbidden region and a replacement accompaniment compensation region based on the separation confidence, the target track amplitude estimate, the complex spectrum residual accompaniment stream, the energy difference before and after elimination, and the live performance feature stream. The replacement accompaniment reconstruction output module receives candidate replacement accompaniments, adjusts the energy distribution of the candidate replacement accompaniments on a two-dimensional time-frequency grid based on the replacement accompaniment forbidden region and the replacement accompaniment compensation region, and mixes the adjusted candidate replacement accompaniments with the clock-following accompaniment stream to output the accompaniment audio stream.

2. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 1, characterized in that, The on-site environmental acoustic isolation module includes: The priori time-frequency analysis unit performs a short-time Fourier transform on the external microphone channel signal and outputs the microphone time-frequency characteristic stream; The transfer function estimation unit estimates the spatial acoustic transfer function from the external microphone channel to the external microphone channel based on the time-frequency characteristic stream of the microphone and the complex spectrum of the signal from the external microphone channel. The crosstalk filtering subtraction unit generates a crosstalk estimation spectrum based on the spatial acoustic transfer function and the time-frequency characteristic stream of the pickup, and subtracts the crosstalk estimation spectrum from the complex spectrum of the external microphone channel signal to output the live performance characteristic stream.

3. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 2, characterized in that, The crosstalk filtering subtraction unit includes: Buffer the time-domain audio frames of the external microphone channel signal, and calculate the time delay difference between the time-domain audio frames of the external microphone channel signal and the corresponding time-domain audio frames of the external pickup channel signal; The spatial acoustic transfer function is time-axis aligned and corrected based on the time delay difference to obtain the aligned spatial acoustic transfer function. A crosstalk estimation spectrum is generated based on the aligned spatial acoustic transfer function and the time-frequency characteristic stream of the microphone, and the crosstalk estimation spectrum is subtracted from the complex spectrum of the external microphone channel signal to obtain the microphone complex spectrum after crosstalk filtering. The fundamental frequency trajectory, harmonic distribution, transient position, and energy envelope are extracted from the complex spectrum of the microphone after crosstalk filtering to generate the live performance feature stream.

4. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 1, characterized in that, The streaming adaptive audio track cancellation module includes: The amplitude and phase estimation unit is used to input the complex spectrum of the original piece and the live performance feature stream into the trained track separation model, and output the target track amplitude estimate, the target track phase estimate and the separation confidence. The cancellation kernel dynamic generation unit generates a cross-frame phase-continuous target audio track cancellation kernel based on the target audio track phase estimate, frequency group delay, and separation confidence. The complex spectrum subtraction execution unit uses the target track cancellation check to perform target part subtraction on the complex spectrum of the original song, outputs the complex spectrum residual accompaniment stream, and calculates the energy difference of the complex spectrum of the original song before and after the target part subtraction to obtain the energy difference before and after the elimination.

5. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 4, characterized in that, The offset kernel dynamic generation unit and the complex spectrum subtraction execution unit specifically include: The phase change trend is calculated based on the target audio track phase estimate between adjacent audio frames, and the frequency group delay is determined based on the phase change trend. Determine whether the separation confidence level corresponding to the current audio frame is lower than a preset confidence level threshold; When the separation confidence level is not lower than the confidence threshold, a target audio track cancellation kernel for the current audio frame is generated based on the target audio track amplitude estimate, the target audio track phase estimate, and the frequency group delay. When the separation confidence is lower than the confidence threshold, the phase angle extrapolation correction is performed on the target audio track phase estimate of the current audio frame according to the frequency group delay evolution trend corresponding to the historical audio frames with a separation confidence not lower than the confidence threshold, and the target audio track cancellation kernel of the current audio frame is generated according to the corrected target audio track phase estimate and the target audio track amplitude estimate. The complex spectrum subtraction execution unit performs complex spectrum subtraction on the complex spectrum of the corresponding audio frame in the complex spectrum of the original song according to the target track cancellation kernel of the current audio frame, and outputs the complex spectrum residual accompaniment stream.

6. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 1, characterized in that, The three-component decoupling module for the accompaniment stream includes: The transient feature splitting unit calculates the energy differential value of the complex spectrum residual accompaniment stream within a short time window, and extracts transient class components within the corresponding time frame when the energy differential value is greater than a preset transient differential threshold. The low-frequency period extraction unit extracts the low-frequency signal of the complex spectrum residual accompaniment stream through a preset low-pass filter, and locks the fundamental frequency phase in the low-frequency signal through an adaptive phase-locked loop, and outputs low-frequency periodic components. The sustained harmony stripping unit removes the transient components and the low-frequency periodic components from the complex spectrum residual accompaniment stream, and uses the residual complex spectrum components obtained after the removal as sustained harmony components.

7. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 6, characterized in that, The multi-dimensional dynamic following alignment module includes: The temporal segment rearrangement unit rearranges the transient components and adjusts the segment spacing according to the performer's micro-clock curve. The time-domain adaptive resampling unit dynamically adjusts the resampling rate of the low-frequency periodic components based on the performer's micro-clock curve. The time-frequency domain phase-locked scaling unit performs pitch-phase-locked time-frequency domain time-scale scaling on the sustained harmony components according to the performer's micro-clock curve, and synthesizes the outputs of the time-domain segment rearrangement unit, the time-domain adaptive resampling unit, and the time-frequency domain phase-locked scaling unit into the clock-following accompaniment stream.

8. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 7, characterized in that, The time-domain adaptive resampling unit specifically includes: Retrieve zero-crossing points in the time-domain sampled values ​​of the low-frequency periodic components; The zero-crossing point is determined as the splicing alignment point; The target resampling rate is determined based on the rate of change of speed indicated by the performer's micro-clock curve. The low-frequency periodic components are resampled in the time domain according to the target resampling rate, and splicing correction is performed at the splicing alignment point according to the first derivative continuity condition.

9. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 1, characterized in that, The time-frequency constraint matrix generation module includes: The residual artifact mapping unit determines the time-frequency grid coordinates occupied by the residual artifact based on the separation confidence, the target track amplitude estimate and the time-frequency correlation between the complex spectrum residual accompaniment stream, and generates the first boundary matrix of the replacement accompaniment forbidden region; The live performance mapping unit calculates the time-frequency energy distribution of the live performance based on the live performance feature flow, determines the time-frequency grid coordinates occupied by the main vocal part of the live performance, and generates the second boundary matrix of the replacement accompaniment forbidden area; The functional void detection unit determines the time-frequency grid coordinates corresponding to the missing vocal part based on the energy difference before and after elimination, and generates the third boundary matrix of the replacement accompaniment compensation area.

10. The AI-based real-time audio track elimination and accompaniment replacement system for solo bands according to claim 9, characterized in that, The replacement accompaniment reconstruction output module includes: Replace the accompaniment mapping unit to convert the candidate replacement accompaniment to the two-dimensional time-frequency grid and obtain the original time spectrum of the candidate replacement accompaniment; The forbidden region attenuation unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the first boundary matrix and the second boundary matrix, and applies an attenuation coefficient to the candidate replacement accompaniment grid points to obtain the attenuated candidate replacement accompaniment time spectrum; The compensation region gain unit extracts candidate replacement accompaniment grid points that coincide with the time-frequency grid coordinates of the third boundary matrix, and applies a compensation gain coefficient to the candidate replacement accompaniment grid points according to the energy difference before and after elimination to obtain the compensated candidate replacement accompaniment time spectrum. The complex spectrum mixing output unit adds and mixes the compensated candidate replacement accompaniment spectrum with the complex spectrum corresponding to the clock-following accompaniment stream, performs an inverse short-time Fourier transform on the added and mixed complex spectrum, and outputs the accompaniment audio stream.

Images