An audio separation method, device, apparatus and computer readable storage medium

Abstract

The application discloses an audio separation method, device and equipment and a computer readable storage medium, and is applied to the technical field of audio, and comprises the following steps: acquiring a scene separation model trained by using a teacher pseudo label as a supervision signal, wherein a teacher vocal track in the teacher pseudo label is generated by using double model cascade separation and by performing alignment and time-frequency adaptive fusion on residual vocals; extracting an initial vocal track from original mixed audio by using the model; cutting the initial vocal track into segments according to a subtitle timestamp and performing singing voice detection; performing blind source separation on the singing voice-containing segments to obtain candidate audio tracks; selecting a target dialogue track according to singing voice features and speech features of the candidate audio tracks, and generating a final vocal track based on the target dialogue track. The application trains a single model by using double model cascade to generate high-quality pseudo labels, and performs segment-level singing voice detection and target dialogue track selection in combination with a subtitle timestamp, so that the purity of reference audio is improved, and the quality of downstream speech synthesis is ensured.

Description

Technical Field

[0001] This invention relates to the field of audio technology, and in particular to an audio separation method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] In scenarios such as film and television dubbing, the original audio typically contains various sound components, including character dialogue, background music, and environmental sound effects. Existing methods still fall short in terms of sound separation purity, making it difficult to balance clean voice extraction with the efficiency required for practical applications. Therefore, how to extract high-quality sound signals from such mixed audio is a key technical challenge in this field. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide an audio separation method, apparatus, device and computer-readable storage medium, which solves the problem of insufficient sound separation purity in the prior art.

[0004] To solve the above-mentioned technical problems, the present invention provides an audio separation method, comprising: A scenario-based separation model trained with teacher pseudo-labels as supervision signals is obtained. The teacher voice track in the teacher pseudo-labels is generated by dual-model cascade separation and alignment and time-frequency adaptive fusion of the residual voice. The scene-based separation model is used to separate human voices from background sounds in the original mixed audio to be processed, thereby obtaining an initial human voice track. The initial voice track is divided into at least two voice segments based on the subtitle timestamp, and singing detection is performed on each voice segment; Blind source separation is performed on vocal segments detected as containing singing to obtain at least two candidate audio tracks; The target dialogue track is selected based on the singing and speech features of the candidate audio tracks, and the final human voice track is generated based on the target dialogue track.

[0005] Optionally, the training process of the scenario-based separation model includes: The original mixed audio used for training is separated into a first vocal track and a first background audio track using the first separation model; The first background audio track is separated a second time using the second separation model to obtain the second background audio track and the residual human voice track, and the second background audio track is used as the teacher's background audio track. The residual vocal track is subjected to amplitude alignment, phase alignment, and time delay alignment to obtain the aligned vocal track. Based on time-frequency adaptive fusion weights, the aligned human voice track and the first human voice track are fused, and the teacher's human voice track is determined based on the fused human voice track; Based on the teacher's background audio track, it is augmented using multi-source background audio data to obtain augmented background audio. The teacher's voice track and the augmented background audio are then mixed to obtain training mixed audio. The training mixed audio is used as the input to the third separation model, and the teacher's voice track and the teacher's background audio track are used as the supervision targets of the third separation model. The third separation model is trained based on the constructed loss function, and the trained third separation model is used as the scene-based separation model.

[0006] Optionally, before fusing the aligned vocal track and the first vocal track based on time-frequency adaptive fusion weights, and determining the teacher's vocal track based on the fused vocal track, the method further includes: Calculate the voice confidence of the residual voice track, the energy ratio of the residual voice track to the second background sound track, the spectral correlation between the residual voice track and the first voice track, and the change in reconstruction error before and after fusion; The time-frequency adaptive fusion weights are determined based on the voice confidence level, the energy ratio, the spectral correlation, and the change in reconstruction error.

[0007] Optionally, determining the teacher's voice track based on the fused voice track includes: The fused human voice track is enhanced using a speech enhancement model to obtain an enhanced human voice track; The fusion coefficient is determined based on the enhancement quality score, and the fused vocal track and the enhanced vocal track are adaptively fused based on the fusion coefficient to obtain the teacher vocal track; Among them, the enhanced quality score is positively correlated with background suppression gain, timbre similarity, and speech recognition consistency, and negatively correlated with enhancement distortion risk; the fusion coefficient increases with the increase of the enhanced quality score.

[0008] Optionally, the process of constructing the loss function includes: Construct vocal reconstruction loss, background noise reconstruction loss, hybrid reconstruction consistency loss, and vocal quality constraint loss; The human voice quality constraint loss includes a timbre consistency constraint term and a background residual energy constraint term. The timbre consistency constraint term is used to constrain the consistency of the predicted human voice track and the teacher's human voice track in terms of timbre characteristics. The background residual energy constraint term is used to constrain the background sound energy in the predicted human voice track.

[0009] Optionally, singing detection can be performed on each vocal segment, including: Obtain the singing probability, singing timestamp, singing duration, and singing proportion of the segment duration corresponding to the human voice segment; When the probability of the singing voice is greater than a preset threshold, or the timestamp of the singing voice has a valid range, or the duration of the singing voice or the proportion of the singing voice to the segment duration meets a preset condition, it is determined that the human voice segment contains singing voice.

[0010] Optionally, a target dialogue track is selected based on the singing and speech features of the candidate audio tracks, and a final vocal track is generated based on the target dialogue track, including: The two candidate audio tracks obtained by blind source separation are subjected to singing detection, speech activity detection, subtitle text matching, and timbre matching with adjacent human voice segments, respectively. If only one candidate audio track meets the conditions for singing detection, then the other candidate audio track is determined as the target dialogue track. If both candidate audio tracks meet or do not meet the conditions for singing detection, then the target dialogue track is determined from the candidate audio tracks based on at least one of the results of the speech activity detection, the results of the subtitle text matching, and the timbre matching with adjacent human voice segments. The target white track is backfilled to the corresponding subtitle timestamp position, and the adjacent target white tracks are connected by boundary processing. The final voice track is generated based on the backfilled target white tracks.

[0011] The present invention also provides an audio separation device, comprising: The model acquisition module is used to acquire the scene-based separation model trained with teacher pseudo-labels as supervision signals. The teacher voice track in the teacher pseudo-label is generated by using dual-model cascade separation and by aligning the residual voice and time-frequency adaptive fusion. The initial separation module is used to separate human voices from background sounds in the original mixed audio to be processed using the scene-based separation model, thereby obtaining an initial human voice track. The singing detection module is used to divide the initial voice track into at least two voice segments based on the subtitle timestamp, and to perform singing detection on each voice segment; The blind source separation module is used to perform blind source separation on human voice segments detected as containing singing, to obtain at least two candidate audio tracks; The dialogue track selection module is used to select a target dialogue track based on the singing and speech features of the candidate audio tracks, and generate the final voice track based on the target dialogue track.

[0012] The present invention also provides an audio separation device, comprising: Memory, used to store computer programs; A processor for implementing the audio separation method described above when executing the computer program.

[0013] The present invention also provides a computer-readable storage medium storing computer-executable instructions, which, when loaded and executed by a processor, implement the audio separation method described above.

[0014] The present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the audio separation method described above.

[0015] As can be seen from the above technical solution, the present invention obtains a scene-based separation model trained with teacher pseudo-labels as the supervision signal. The teacher's voice track in the teacher pseudo-label is generated by using dual-model cascade separation and aligning and time-frequency adaptive fusion of the residual voice. The scene-based separation model is used to separate the voice from the background sound in the original mixed audio to be processed, and an initial voice track is obtained. The initial voice track is divided into at least two voice segments according to the subtitle timestamp, and singing detection is performed on each voice segment. Blind source separation is performed on the voice segments that are detected to contain singing to obtain at least two candidate audio tracks. The target dialogue track is selected according to the singing and speech features of the candidate audio tracks, and the final voice track is generated based on the target dialogue track. The beneficial effects of this invention are as follows: by aligning and time-frequency adaptively fusing the residual human voice recovered in the dual-model cascade separation, a pseudo-label for a teacher containing supplementary information of human voice in the background is generated, enabling the scene-based separation model to obtain high-quality human voice purity and low background residue in a single inference; at the same time, by combining the subtitle timestamp to perform segment-level singing detection on the human voice track separated by the scene-based separation model, and performing blind source separation and candidate track selection on the singing segment, the target dialogue track is determined based on singing features and speech features, forming a full-process processing from human voice separation to singing separation, effectively avoiding singing pollution of the reference audio, and providing a more reliable reference signal for downstream speech secondary development.

[0016] In addition, the present invention also provides an audio separation device, apparatus, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of an audio separation method provided in an embodiment of the present invention; Figure 2This is a schematic diagram of the structure of an audio separation device provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of an audio separation device provided in an embodiment of the present invention. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] First, several terms used in this invention will be explained: Original mixed audio: refers to the original audio track extracted from film, television, short drama or other video content, which may simultaneously contain character dialogue, background music, background singing, ambient sounds, onomatopoeia, special sound effects and reverb.

[0021] Voice track: This mainly refers to the audio of character dialogue used for subsequent dubbing, voice cloning, or speech synthesis. It should preserve the character's timbre, tone, emotion, and rhythm as much as possible, while removing background music, noise, sound effects, and singing interference.

[0022] Background audio track: This refers to the audio remaining after subtracting the character dialogue from the original mixed audio. It can include background music, ambient sounds, sound effects, and ambient sounds. This background audio track can be remixed with the target language dubbing later to restore the scene atmosphere of the original video.

[0023] Singing interference refers to non-target dialogue singing sounds in film and television clips, including singing in background songs, singing by characters within the scene, off-screen music and vocals, humming, etc. Since both singing and dialogue are produced by human voices, conventional human voice / background sound separation models often struggle to distinguish between the two, easily preserving singing in the human voice track, thus affecting the subsequent speech synthesis model's learning of normal speaking timbre and intonation.

[0024] In current film and television dubbing processes, the original audio corresponding to the subtitle segment is typically extracted from the original video as reference audio. This reference audio is then input into speech synthesis, voice cloning, or cross-language dubbing models. However, there are significant differences between original film and television audio and studio recording audio. Without specialized processing, this can significantly impact the quality of subsequent dubbing. Original film and television audio is usually multi-source mixed audio. Character dialogue is often superimposed with background music, environmental noise, prop sound effects, scene ambient sounds, reverberation, and other character voices. For subsequent speech synthesis models, these sounds unrelated to the character's timbre contaminate the reference audio, causing problems such as background noise residue, timbre drift, unclear pronunciation, or degraded sound quality when synthesizing target language audio. Currently, general voice separation models cannot fully meet the needs of film and television dubbing scenarios, and still have shortcomings in sound separation purity, making it difficult to balance clean voice extraction with efficient practical application requirements. Therefore, how to extract high-quality sound signals from mixed audio is a technical problem facing this field.

[0025] To address the aforementioned problems, this invention provides an audio separation method, for details of which please refer to [reference needed]. Figure 1 , Figure 1 A flowchart illustrating an audio separation method provided in an embodiment of the present invention. The method may include: S101: Obtain the scene-based separation model trained with teacher pseudo-labels as the supervision signal. The teacher's voice track in the teacher pseudo-label is generated by separating the dual-model cascade and aligning and time-frequency adaptively fusing the residual voice.

[0026] It should be noted that this method can be applied in various scenarios, including multilingual translation, localization of film and television scripts, AI (artificial intelligence) dubbing, voice cloning, character voice transfer, and post-production of film and television audio. For example, relatively clean character voice audio can be extracted from Chinese short dramas and combined with translated texts in English, Portuguese, Spanish, Indonesian, Thai, Arabic, etc., to generate dubbing in the target language; or in film and television post-production, the original mixed audio can be split into dialogue tracks and background audio tracks, facilitating dialogue replacement, noise reduction, remixing, and secondary content creation.

[0027] The scene-based separation model in this embodiment is trained using a dual-model teacher separation pipeline in the offline stage to generate high-quality pseudo-labels. This allows it to output high-quality human voice and background audio tracks with only a single forward computation in the online inference stage, thereby reducing GPU memory consumption and improving the efficiency of large-scale short drama processing. This embodiment does not specifically limit the alignment method, nor does it specifically limit the fusion method.

[0028] Furthermore, to improve the separation quality of the scene-based separation model, the specific training process of the aforementioned scene-based separation model may include: Step 11: Use the first separation model to separate the original mixed audio used for training into a first vocal track and a first background audio track.

[0029] The raw mixed audio 'x' is extracted from movies, short dramas, or other video content for training. To further improve separation quality, preprocessing can be performed on 'x' before inputting it into the first separation model, including but not limited to resampling, loudness normalization, silence trimming, long audio sliding window segmentation, overlapping region setting, and timestamp recording. This raw mixed audio typically contains target character dialogue, background music, background vocals, ambient sounds, sound effects, and reverb.

[0030] This embodiment does not specifically limit the first separation model. For example, the first separation model Fv can be a BS-Roformer-based human voice target separation model, with input x and outputs a first human voice track v1 and a first background sound track b1: (v1, b1) = Fv(x). The first human voice track v1 mainly contains the target character's dialogue, but may contain some background music, ambient sounds, sound effects, or singing. The first background sound track b1 mainly contains background sounds, but in complex film and television scenes, it may still contain weak human voices, dialogue tails, reverberant human voices, or human voices masked by music.

[0031] Step 12: Use the second separation model to perform a second separation of the first background audio track to obtain the second background audio track and the residual human voice track, and use the second background audio track as the teacher's background audio track.

[0032] To recover the residual human voice information in the first background audio track b1, a second separation model Fb is used to perform a secondary separation on the first background audio track b1. This embodiment does not specifically limit the second separation model. For example, the second separation model can be the same as the first separation model, or it can be different. In this embodiment, the second separation model can be a background sound target separation model also based on BS-Roformer, which outputs a second background audio track b2 and a residual human voice track v2: (b2, v2) = Fb(b1).

[0033] It should be noted that in this embodiment, the second background audio track b2 can be directly used as the teacher's background audio track bt, or other methods can be used, such as modifying it to bt=x-vt according to the consistency of the mixed reconstruction, and then weighting and fusing it with b2 to ensure that the sum of vt and bt is as close as possible to the original mixed audio x.

[0034] Step 13: Perform amplitude alignment, phase alignment, and time delay alignment on the remaining vocal track to obtain the aligned vocal track.

[0035] This embodiment aligns the amplitude, phase, and time delay of the residual vocal track v2 to obtain the aligned vocal track Align(v2). This embodiment does not specifically limit the alignment methods. For example, the amplitude spectrum, phase spectrum, and cross-correlation results of the first vocal track v1 and the residual vocal track v2 can be calculated in the short-time Fourier transform domain respectively: (1) In terms of amplitude, the amplitude scaling factor is determined according to the root mean square energy ratio or loudness difference between the first vocal track v1 and the residual vocal track v2 in the speech activity region, so that the overall loudness of the residual vocal track v2 is consistent with that of the first vocal track v1; (2) In terms of time delay, the relative delay is estimated by using the cross-correlation peak position of the first vocal track v1 and the residual vocal track v2 in the speech activity region, and the residual vocal track v2 is shifted at the sample level or frame level; (3) In terms of phase, the phase of the original mixed audio x can be used as the reference phase, or a phase representation with a smaller phase difference from the first vocal track v1 can be selected in the overlapping time-frequency unit to reduce the cancellation and artifacts caused by direct superposition.

[0036] Step 14: Based on the time-frequency adaptive fusion weight, fuse the aligned human voice track and the first human voice track, and determine the teacher's human voice track based on the fused human voice track.

[0037] It should be noted that the time-frequency adaptive fusion weights in this embodiment are not fixed constants, but are determined comprehensively based on reliability indicators. This embodiment does not impose specific limitations on the reliability indicators.

[0038] Furthermore, before fusing the aligned vocal track and the first vocal track based on time-frequency adaptive fusion weights, and determining the teacher's vocal track based on the fused vocal track, the process may further include: Step 21: Calculate the voice confidence of the residual human voice track, the energy ratio of the residual human voice track and the second background sound track, the spectral correlation between the residual human voice track and the first human voice track, and the change in reconstruction error before and after fusion.

[0039] Specifically, the voice confidence Cv(t,f) of this time-frequency unit, the energy ratio Rv(t,f) of the residual voice track v2 relative to the second background track b2, the coherence Co(t,f) between the residual voice track v2 and the first voice track v1, and the improvement in the reconstruction error of the mixed audio after adding the residual voice track v2 (i.e., the change in reconstruction error) ΔE(t,f). t represents time, and f represents frequency.

[0040] Among them, the confidence of human voice can be output by the speech activity detection model or the human voice / background classifier; the energy ratio can be calculated by the ratio of the local spectral energy of the residual human voice track v2 to the local spectral energy of the second background sound track b2; the coherence can be calculated by the short-time spectral correlation coefficient or cosine similarity between the first human voice track v1 and the residual human voice track v2; the change in reconstruction error can be obtained by comparing the differences between the original mixed audio x and v1+b2, and between the original mixed audio x and v1+v2+b2.

[0041] Step 22: Determine the time-frequency adaptive fusion weights based on human voice confidence, energy ratio, spectral correlation, and reconstruction error change.

[0042] The above indicators can be normalized to 0 to 1 and then weighted and summed. A truncation function is then used to limit the sum to a preset range. The formula for the time-frequency adaptive fusion weight w(t, f) is: w(t,f)=clip(γ1Cv(t,f)+γ2Rv(t,f)+γ3Co(t,f)+γ4ΔE(t,f),0,wmax).

[0043] Wherein, γ1 to γ4 are weight coefficients obtained by preset or validation set tuning, and wmax is the maximum fusion upper limit. When a certain time-frequency region of the residual vocal track v2 is more likely to be a real residual vocal and can improve reconstruction consistency, the time-frequency adaptive fusion weight w(t,f) is larger; when the residual vocal track v2 is more likely to be background leakage, music residue or separation artifact, the time-frequency adaptive fusion weight w(t,f) is smaller.

[0044] Based on this, the fused vocal track vt can be expressed as: vt = v1 + w·Align(v2).

[0045] It should be noted that this implementation considers directly using the merged human voice track VT as the teacher's human voice track v. Alternatively, it can be enhanced, and the enhanced vocal track can be used as the teacher's vocal track. Furthermore, the above-mentioned determination of the teacher's voice track based on the fused voice track may specifically include: enhancing the fused voice track using a speech enhancement model to obtain an enhanced voice track; determining a fusion coefficient based on the enhancement quality score; and adaptively fusing the fused voice track and the enhanced voice track based on the fusion coefficient to obtain the teacher's voice track; wherein, the enhancement quality score is positively correlated with background suppression gains, timbre similarity, and speech recognition consistency, and negatively correlated with enhancement distortion risk; the fusion coefficient increases with the increase of the enhancement quality score.

[0046] It should be noted that although the dual-model teacher separation pipeline can obtain high-quality pseudo-labels for human voices, the fused human voice track vt may still contain weak background music, environmental noise, sound effects, reverberation, or separation artifacts. Therefore, this embodiment further utilizes a speech enhancement model E, such as MossFormer2_SE, to enhance the fused human voice track vt, resulting in the enhanced human voice track ve: ve = E(vt). Furthermore, to avoid over-processing by the speech enhancement model and resulting in damage to the character's timbre, this embodiment does not directly use the enhanced human voice track ve as the final training human voice label (i.e., the teacher's human voice track). Instead, it determines the fusion coefficient α based on the enhancement quality score Q and adaptively fuses the results before and after enhancement: v =α·ve+(1-α)·vt.

[0047] Among them, v This represents the final vocal label used for training, i.e., the teacher's vocal track. The value of α can be determined based on both enhancement gains and timbre fidelity.

[0048] Specifically, this embodiment can calculate the following indicators: (1) the timbre similarity of the human voice before and after enhancement, Sspk, for example, by using the speaker embedding model to extract the timbre vectors of the fused human voice track vt and the enhanced human voice track ve and calculating the cosine similarity; (2) the ASR recognition consistency before and after enhancement, Sasr, for example, by comparing the recognition text or phoneme sequence similarity between the fused human voice track vt and the enhanced human voice track ve; (3) the background suppression gain after enhancement, Sden, for example, by comparing the degree of energy reduction in non-speech regions, the degree of energy reduction in noise estimation, or the change in the output of the human voice / background classifier before and after enhancement; (4) the enhancement distortion risk, Sdis, for example, by estimating based on the degree of spectrum oversmoothing, high-frequency loss, popping, discontinuity, or abnormal changes in the energy of speech activity regions.

[0049] In one specific implementation, the enhanced mass fraction Q can be constructed first: Q=η1Sden+η2Sspk+η3Sasr-η4Sdis.

[0050] Where η1 to η4 are weighting coefficients. Q is then mapped to α: α=clip((Q-Qmin) / (Qmax-Qmin), αmin, αmax).

[0051] Here, Qmin and Qmax are the preset lower and upper limits of the quality score, while αmin and αmax are the minimum and maximum values ​​of the fusion coefficients. For example, when the enhanced background suppression is significant, the timbre similarity and ASR consistency are high, and the risk of distortion is low, α is close to αmax, and the enhanced result is used more. When the enhanced result reduces noise but the timbre changes significantly or the recognition consistency decreases, α is close to αmin, and the fused vocal track (vt) is preserved more. In this way, a balance can be achieved between removing residual noise and protecting the character's timbre.

[0052] Step 15: Based on the teacher's background audio track, augment it using multi-source background audio data to obtain augmented background audio, and mix the teacher's voice track and the augmented background audio to obtain training mixed audio.

[0053] For the background sound portion, this embodiment utilizes background sound data from multiple sources for expansion, including but not limited to open-source ambient sound data such as ESC-50, film and television sound effects data, real film and television background sound, instrumental accompaniment, and music clips containing vocals. The teacher's voice track v With expanded background sound b The mixed audio can be constructed and trained by mixing different signal-to-noise ratios, loudness ratios, reverberation conditions, and time offsets. : x =g1·v +g2·b +g3·n.

[0054] Where g1, g2, and g3 represent the loudness control coefficients for human voice, background sound, and optional noise or sound effects, respectively, and n represents optional noise, reverberation, or transient sound effects. By randomly changing the proportion of different sound sources, this embodiment can construct various challenging film and television dubbing scenarios, such as indoor dialogue, outdoor dialogue, dialogue with music overlay, dialogue with singing overlay, and dialogue with strong sound effects overlay, thereby improving the robustness of subsequent single models on real business data.

[0055] Step 16: Use the training mixed audio as input to the third separation model, and use the teacher's voice track and teacher's background audio track as supervision targets for the third separation model. Train the third separation model based on the constructed loss function, and use the trained third separation model as the scene-based separation model.

[0056] This embodiment does not specifically limit the third separation model. For example, it can be the same as or different from the architecture of the first / second separation model. This embodiment does not specifically limit the loss function. For example, it can include voice reconstruction loss; or it can also include background sound reconstruction loss. After training, the separation capabilities obtained by the dual-model teacher pipeline are transferred to the single separation model. In the actual deployment stage, it is no longer necessary to run two sets of separation models simultaneously. Instead, a single model can output high-quality voice and background sound tracks with a single inference, thereby reducing GPU memory consumption and improving the efficiency of large-scale short drama processing.

[0057] For example, this embodiment uses the BS-Roformer separation model as the third separation model Fs, and fine-tunes it for film and television dubbing scenarios. During training, the model input is a mixed training audio x. Output the predicted human voice track v and the predicted background audio track b: (v, b) = Fs(x ).

[0058] Where, x Can be made by teacher's voice track v With expanded background sound b The mixture yields v This represents the human voice supervision signal obtained after dual-model teacher separation and speech enhancement fusion, b This indicates the corresponding background sound monitoring signal.

[0059] Furthermore, the construction process of the above loss function may specifically include: constructing voice reconstruction loss, background sound reconstruction loss, hybrid reconstruction consistency loss, and voice quality constraint loss; wherein, the voice quality constraint loss includes a timbre consistency constraint term and a background residual energy constraint term. The timbre consistency constraint term is used to constrain the consistency of the predicted voice track with the teacher's voice track in timbre features, and the background residual energy constraint term is used to constrain the background sound energy in the predicted voice track.

[0060] In one implementation, the overall loss function for model training can be expressed as: L=λ1·Lv+λ2·Lb+λ3·Lrec+λ4·Lq.

[0061] Where Lv represents the human voice reconstruction loss, Lb represents the background sound reconstruction loss, Lrec represents the hybrid reconstruction consistency loss, Lq represents the human voice quality constraint loss, and λ1 to λ4 are the weight coefficients corresponding to each loss term.

[0062] Specifically, (1) the voice reconstruction loss Lv is used to constrain the predicted voice trajectory v to approach the voice supervision signal v. It can be expressed as: Lv=d(v, v ).

[0063] (2) The background sound reconstruction loss Lb is used to constrain the predicted background sound track b to approach the background sound supervision signal b. It can be expressed as: Lb=d(b, b ).

[0064] Here, d() represents the audio distance function, which can be the time-domain L1 distance, the spectral distance, or a weighted combination of the two. Using the human voice reconstruction loss Lv and the background sound reconstruction loss Lb, the model can learn the separation results of the target human voice and the background sound, respectively.

[0065] (3) The hybrid reconstruction consistency loss is used to constrain that the sum of the predicted human voice track and the predicted background audio track can reconstruct the input mixed audio, and can be expressed as: Lrec=d(x (v+b).

[0066] This loss is used to reduce energy loss and additional artifacts that occur during the separation process, so that the output vocal track and background audio track are consistent with the original mixed audio as a whole.

[0067] (4) The voice quality constraint loss Lq is used to reduce background residue in the predicted voice and to preserve the character's timbre and speech content as much as possible. In one implementation, Lq can be expressed as: Lq = Lspk + Lbg. Wherein, Lspk is used to constrain the predicted voice to maintain consistency with the voice supervision signal in timbre characteristics, and Lbg is used to constrain the background sound energy in the predicted voice to be as low as possible. Further, the following simplified form can be adopted: Lspk=1-cos(Φ(v),Φ(v)) )).

[0068] Lbg=Ebg(v).

[0069] Where Φ() represents the speaker feature extraction model, cos() represents the cosine similarity, and Ebg() represents the estimated value of the background residual energy in the predicted human voice, which can be obtained from the background sound detection model, the energy of the non-speech region, or the energy of the dominant background frequency band.

[0070] Using the loss function described above, the model can simultaneously learn objectives such as voice extraction, background noise preservation, mixed audio reconstruction, and voice quality optimization.

[0071] S102: Use a scene-based separation model to separate human voices from background sounds in the original mixed audio to obtain the initial human voice track.

[0072] During the inference process, this embodiment utilizes the offline-trained scene-based separation model to separate the human voice from the background sound in the original mixed audio to be processed, so as to obtain a preliminary human voice track and a preliminary background sound track: (vp, bp) = Fs(x'). Wherein, vp and bp represent the initial human voice track and the preliminary background sound track, respectively, Fs represents the scene-based separation model, and x' represents the original mixed audio to be processed.

[0073] This embodiment does not specifically limit the scene-based separation model. For example, the model could be a voice target separation model based on BS-Roformer.

[0074] Furthermore, to improve separation quality, the original mixed audio to be processed can be extracted from the video and preprocessed, including resampling, loudness normalization, and sliding window segmentation. The preprocessed original mixed audio is then input into a trained scene-based separation model to obtain preliminary human voice tracks and preliminary background audio tracks. For long audio files, overlapping sliding window inference can be used, and overlapping addition and boundary smoothing can be performed at the output stage to avoid discontinuities or artifacts at segment boundaries.

[0075] S103: Divide the initial voice track into at least two voice segments based on the subtitle timestamp, and perform singing detection on each voice segment.

[0076] This embodiment divides the initial voice track into several (e.g., two or three) voice segments Pi based on the subtitle timestamp: Pi = vp[ti_start, ti_end]. Here, ti_start and ti_end represent the start and end times of the i-th subtitle or dubbing text, respectively. To avoid cutting off phonemes or ending sounds, the subtitle timestamp can be used to determine the approximate boundaries of the dialogue. Then, a preset length is extended before and after the subtitle boundaries, and the boundaries are fine-tuned based on the VAD detection results to preserve the start, end, and contextual information of the voice. It should be noted that if there is no subtitle timestamp, other fixed or non-fixed division methods can also be used.

[0077] This embodiment also performs singing detection on each voice segment Pi. This embodiment does not specifically limit the singing detection method; for example, a singing detection model can be used, and this embodiment does not specifically limit the singing detection model either. Thus, compared to performing uniform singing detection on the entire audio segment, segment-level detection based on subtitle timestamps can more accurately determine whether each dubbing reference segment is affected by singing, avoiding unnecessary processing of segments without singing, and reducing the impact of false detections on the entire voice track.

[0078] Furthermore, the aforementioned singing detection for each voice segment can specifically include: obtaining the singing probability, singing timestamp, singing duration, and singing proportion of the segment duration corresponding to the voice segment; when the singing probability is greater than a preset threshold, or the singing timestamp has a valid range, or the singing duration or singing proportion of the segment duration meets preset conditions, the voice segment is determined to contain singing. In this embodiment, the singing probability Ps, singing timestamp, singing duration Ds, and singing proportion of the segment duration Rs for each voice segment can be obtained through a singing detection model. When any of the above conditions are met, the segment is determined to contain singing interference and needs to enter the singing separation process; otherwise, the segment is directly retained as a usable voice segment.

[0079] S104: Perform blind source separation on the detected vocal segments containing singing to obtain at least two candidate audio tracks.

[0080] This embodiment does not limit the specific method of blind source separation, nor does it limit the number of candidate audio tracks. For example, for a vocal segment P determined to contain singing interference, a blind source separation model G, such as MossFormer2_SS, is invoked to perform two-source or multi-source separation on the segment, obtaining candidate audio tracks y1 and y2: (y1, y2) = G(P). Since the output order of the blind source separation model is usually not fixed, y1 and y2 do not necessarily correspond to dialogue and singing respectively. Therefore, subsequent identification of each candidate track is required to determine the target dialogue track.

[0081] S105: Select the target dialogue track based on the singing and speech features of the candidate audio tracks, and generate the final human voice track based on the target dialogue track.

[0082] This embodiment obtains two candidate audio tracks, y1 and y2, and inputs them into a singing detection model. Based on singing and speech features, a target dialogue track is selected, and the final vocal track is generated based on this target dialogue track. This embodiment does not specifically limit the singing and speech features. For example, singing features may include at least one of singing detection probability, singing duration, or singing confidence; speech features may include at least one of speech activity detection confidence, matching degree with subtitle text, or timbre similarity with adjacent vocal segments. This embodiment also does not specifically limit the target singing detection model. For example, the FireRedVAD model based on deep neural networks can be used, or other detection models capable of outputting speech / singing classification labels and timestamps can be used.

[0083] Furthermore, the above-mentioned selection of the target dialogue track based on the singing and speech features of the candidate audio tracks, and the generation of the final vocal track based on the target dialogue track, can specifically include: Step 51: Perform singing detection, speech activity detection, subtitle text matching, and timbre matching with adjacent human voice segments on the two candidate audio tracks obtained from blind source separation.

[0084] This embodiment does not specifically limit the singing detection method. For details, please refer to step S103 above. Singing detection can determine whether a candidate track contains singing components (such as singing, humming, etc.); speech activity detection can identify which time periods in the candidate track contain human voices (dialogue), distinguishing between silence / noise and valid speech, used to measure the continuity and confidence of the dialogue; subtitle text matching refers to comparing the automatically recognized text of the candidate track (via ASR) with the original subtitles or translated text, calculating the matching degree; the higher the matching degree, the more likely it is to be the correct dialogue track; timbre matching with adjacent human voice segments refers to extracting the timbre features of the candidate track (such as speaker embedding vectors) and comparing the similarity with the timbre of adjacent (temporally preceding and following) non-singing human voice segments; identical timbre indicates the same character and should be retained preferentially.

[0085] Step 52: If only one candidate audio track meets the conditions for singing detection, then the other candidate audio track is determined as the target dialogue track.

[0086] If only y1 detects singing, then y2 is returned as the human voice audio; if only y2 detects singing, then y1 is returned as the human voice audio.

[0087] Step 53: If both candidate audio tracks meet or do not meet the conditions for singing detection, then determine the target dialogue track from the candidate audio tracks based on at least one of the following: the result of speech activity detection, the result of subtitle text matching, and the timbre matching with adjacent human voice segments.

[0088] When both candidate audio tracks meet or fail to meet the singing detection criteria (i.e., both are detected to contain singing, or neither is detected to contain singing), the target dialogue track cannot be determined directly by excluding singing tracks. In this case, other features must be considered: the activity and confidence of the dialogue in the candidate tracks can be evaluated based on the speech activity detection results, selecting the track with more continuous speech activity; the text matching results can be used to compare the text of the candidate tracks after automatic speech recognition with the original subtitles, selecting the track with a higher matching degree; and the timbre matching results with adjacent non-singing vocal segments can be used to select the track with timbre features closer to the same character. Through a comprehensive comparison of at least one of these indicators, the optimal track is selected as the target dialogue track. If the two candidate tracks cannot be distinguished in terms of singing detection, and further comprehensive evaluation using indicators such as speech activity, subtitle matching, and timbre comparison still indicates that both candidate tracks are of poor quality, then the process reverts to the original segment Pi, and mild speech enhancement or spectral gating can be applied to avoid excessive separation causing dialogue damage.

[0089] If no singing is detected in either of the two candidate audio tracks y1 and y2, no selection can be made, and the system can directly return to Pi before separation.

[0090] Step 54: Backfill the target white track to the corresponding subtitle timestamp position, and perform boundary connection processing on adjacent target white tracks. Generate the final voice track based on the backfilled target white tracks.

[0091] After all segments are processed, the vocal segments that have undergone vocal detection and selective vocal separation are backfilled into the original timeline, and loudness matching, boundary fade-in and fade-out, overlap addition and spectral smoothing are performed to obtain the final vocal track Vfinal.

[0092] It should be noted that the final background audio track Bfinal can be taken from the initial background audio track bp, or it can be reconstructed and corrected based on the final vocal track. That is, the corrected background audio is obtained by using the difference between the original mixed audio x' and the final vocal track Vfinal, and then weighted and fused with the initial background audio track bp to ensure the naturalness of the background audio and the consistency of the overall reconstruction.

[0093] The final output, the Vfinal voice track, can be used as a reference audio for multilingual speech synthesis or voice cloning, while the final background audio track, Bfinal, can be used to remix with the target language dubbing, thus forming a complete audio processing workflow for film and television dubbing.

[0094] An audio separation method provided by this invention is applied in the following steps: S101: Obtain a scene-based separation model trained with teacher pseudo-labels as supervision signals. The teacher's voice track in the teacher pseudo-labels is generated by dual-model cascade separation and alignment and time-frequency adaptive fusion of residual voices; S102: Use the scene-based separation model to separate the voices from the background sounds in the original mixed audio to be processed, obtaining an initial voice track; S103: Divide the initial voice track into at least two voice segments according to the subtitle timestamps, and perform singing detection on each voice segment; S104: Perform blind source separation on the voice segments detected as containing singing, obtaining at least two candidate audio tracks; S105: Select a target dialogue track based on the singing and speech features of the candidate audio tracks, and generate the final voice track based on the target dialogue track. This invention aligns and adaptively fuses residual human voices recovered during dual-model cascade separation to generate pseudo-labels for teachers that include supplementary information about human voices in the background. This enables the scene-based separation model to achieve high-quality human voice purity and low background residue in a single inference. Simultaneously, it combines subtitle timestamps to perform segment-level singing detection on the human voice tracks separated by the scene-based separation model, and performs blind source separation and candidate track selection on the singing segments. Based on singing features and speech features, it determines the target dialogue track, forming a complete process from human voice separation to singing separation. This effectively avoids singing contamination of the reference audio and provides a more reliable reference signal for downstream speech secondary development.

[0095] To better understand this solution, please refer to the following example: (1) During the offline training phase, large-scale audio data from movies, short dramas, and related audio sources were first collected. The original audio was then preprocessed, including sampling rate unification, loudness normalization, segmentation, and subtitle timestamp alignment. Subsequently, a dual-model teacher separation pipeline was used to perform preliminary processing on the original mixed audio: the first separation model, targeting human voices, separated the original mixed audio into a first human voice track and a first background audio track; the second separation model, targeting background sounds, further separated the first background audio track to recover any remaining human voice information, resulting in a second background audio track and a remaining human voice track. The first human voice track and the remaining human voice track were then aligned, quality evaluated, and adaptively fused to obtain the teacher's human voice track, and the second background audio track or the background sound corrected based on the mixed reconstruction was used as the teacher's background audio track. To improve the purity of the teacher's human voice track, a speech enhancement model was further used to enhance the teacher's human voice track, suppressing any remaining background music, environmental noise, sound effects, and reverberation. Meanwhile, to avoid excessive damage to the character's timbre from voice enhancement, enhancement fusion coefficients are determined based on the similarity of timbre before and after enhancement, speech recognition consistency, enhancement distortion risk, and the degree of residual background noise suppression (background noise suppression benefit). The enhancement result is then adaptively fused with the original voice. For the background noise component, open-source environmental sound data such as ESC-50, film and television sound effects data, music accompaniment data, music data containing vocals, and real film and television background noise data are used to expand the training samples, constructing a comprehensive dataset covering various film and television acoustic conditions. Based on this data, the third separation model is fine-tuned using the enhanced voice labels and expanded background noise labels, resulting in a voice / background noise separation model specifically suited for film and television dubbing scenarios. Unlike directly deploying a dual-model pipeline, this model only requires one forward computation during actual inference to output the voice track and background noise track, thereby reducing GPU memory consumption and improving inference efficiency.

[0096] (2) In the online inference stage, the original audio of the video to be processed is first input into the trained scene separation model to obtain the preliminary human voice track and the preliminary background audio track. Then, according to the timestamps in the translated text or the original subtitles, the preliminary human voice track is divided into several human voice segments, and each segment is input into the singing detection system. If a segment is detected to contain singing, the segment is further separated by blind source separation models such as MossFormer2_SS to obtain multiple candidate audio tracks. Then, singing detection, speech activity detection, subtitle text matching, and timbre consistency evaluation are performed on the candidate tracks respectively, and the audio track that is more likely to belong to the character's dialogue is selected as the final human voice audio. For segments that do not contain singing, the human voice segments output by the scene separation model are directly retained. Finally, the system fills the processed segments back into the original timeline to generate the final human voice track and outputs the final background audio track to support subsequent multilingual dubbing synthesis and background sound remixing.

[0097] It is evident that this method has the following beneficial effects: (1) Combining dual-model teacher separation with single-model deployment, balancing separation quality and inference efficiency: In the offline stage, a dual-model cascaded pipeline of human voice target separation model and background sound target separation model is adopted to further recover residual human voice from the first separation of background sound, generating higher quality human voice and background sound pseudo-labels; in the deployment stage, the capabilities of the dual-model pipeline are transferred to a single BS-Roformer separation model through film and television scene-based fine-tuning. This design not only takes advantage of the high separation quality of dual-model, but also avoids the problems of high memory consumption and long time consumption in the online inference stage, making it suitable for large-scale batch processing scenarios such as short dramas.

[0098] (2) Introducing an interpretable adaptive fusion mechanism to improve the quality of voice pseudo-labels: This invention does not simply add the primary voice and the secondary recovered voice directly, but first aligns the amplitude, phase, and time delay of the residual voice, and then determines the time-frequency adaptive fusion weights based on voice confidence, energy proportion, spectral coherence, and reconstruction error improvement. For the speech enhancement results, this invention also determines the enhancement fusion coefficients based on background suppression gains, timbre similarity, ASR consistency, and enhancement distortion risk. The above mechanism makes the pseudo-label generation process more stable and interpretable, and can achieve a balance between completing weak voices and avoiding the introduction of artifacts.

[0099] (3) Constructing training data for film and television dubbing scenarios to improve generalization ability in complex acoustic environments: This invention addresses common issues in short film and television dramas such as background music, ambient sounds, onomatopoeia, strong sound effects, and singing covering dialogue. It expands the training data using real film and television background sounds, ESC-50 and other ambient sound and onomatopoeia data, music, and segments containing singing, and constructs multi-condition mixed samples by combining voice tags after speech enhancement. This approach makes the training model closer to the real application scenario of film and television dubbing, which is different from the model trained only for general voice / accompaniment separation tasks.

[0100] (4) Singing detection and candidate track discrimination based on subtitle timestamps to reduce interference from singing in dubbing synthesis: This invention utilizes the existing subtitle timestamps in the dubbing process to segment the human voice track and then perform singing detection and separation, which can accurately locate dubbing reference segments contaminated by singing. For candidate tracks obtained by blind source separation, this invention further combines singing probability, ordinary speech confidence, subtitle text matching degree, and timbre consistency of adjacent segments to automatically select the character dialogue track that is truly suitable for dubbing reference. This mechanism solves the problem that singing itself is also a human voice and is easily preserved by ordinary human voice separation models, which can effectively reduce singing style, melodic intonation, and timbre drift in target language dubbing.

[0101] The audio separation device provided in the embodiments of the present invention will be described below. The audio separation device described below can be referred to in correspondence with the audio separation method described above.

[0102] Please refer to the details. Figure 2 , Figure 2 A schematic diagram of an audio separation device provided in an embodiment of the present invention may include: The model acquisition module 100 is used to acquire the scene-based separation model trained with teacher pseudo-labels as supervision signals. The teacher voice track in the teacher pseudo-labels is generated by using dual-model cascade separation and by aligning the residual voice and time-frequency adaptive fusion. The initial separation module 200 is used to separate human voices from background sounds in the original mixed audio to be processed using the scene-based separation model to obtain an initial human voice track. The singing detection module 300 is used to divide the initial human voice track into at least two human voice segments according to the subtitle timestamp, and to perform singing detection on each human voice segment; Blind source separation module 400 is used to perform blind source separation on human voice segments detected as containing singing, to obtain at least two candidate audio tracks; The dialogue track selection module 500 is used to select a target dialogue track based on the singing and speech features of the candidate audio tracks, and generate a final voice track based on the target dialogue track.

[0103] Based on the above embodiments, the model acquisition module 100 may include: The first separation unit is used to separate the original mixed audio used for training into a first vocal track and a first background audio track using the first separation model; The second separation unit is used to perform secondary separation of the first background audio track using the second separation model to obtain the second background audio track and the residual human voice track, and to use the second background audio track as the teacher's background audio track. An alignment unit is used to perform amplitude alignment, phase alignment and time delay alignment on the residual vocal track to obtain an aligned vocal track. The fusion unit is used to fuse the aligned human voice track and the first human voice track based on the time-frequency adaptive fusion weight, and to determine the teacher's human voice track based on the fused human voice track. An expansion unit is used to expand the teacher's background audio track using multi-source background audio data to obtain expanded background audio, and to mix the teacher's voice track and the expanded background audio to obtain training mixed audio. The training unit is used to take the training mixed audio as input to the third separation model, take the teacher's voice track and the teacher's background audio track as supervision targets of the third separation model, train the third separation model based on the constructed loss function, and use the trained third separation model as the scene-based separation model.

[0104] Furthermore, based on the above embodiments, the audio separation device may further include: The calculation module is used to calculate the voice confidence of the residual human voice track, the energy ratio of the residual human voice track and the second background sound track, the spectral correlation between the residual human voice track and the first human voice track, and the change in reconstruction error before and after fusion. The determination module is used to determine the time-frequency adaptive fusion weights based on the human voice confidence, the energy ratio, the spectral correlation, and the change in reconstruction error.

[0105] Furthermore, based on the above embodiments, the fusion unit may specifically include: An enhancement subunit is used to enhance the fused human voice track using a speech enhancement model to obtain an enhanced human voice track; The fusion subunit is used to determine the fusion coefficient based on the enhancement quality score, and adaptively fuse the fused vocal track and the enhanced vocal track according to the fusion coefficient to obtain the teacher vocal track; Among them, the enhanced quality score is positively correlated with background suppression gain, timbre similarity, and speech recognition consistency, and negatively correlated with enhancement distortion risk; the fusion coefficient increases with the increase of the enhanced quality score.

[0106] Furthermore, based on the above embodiments, the training unit may specifically include: The loss construction subunit is used to construct the human voice reconstruction loss, background sound reconstruction loss, hybrid reconstruction consistency loss, and human voice quality constraint loss. The human voice quality constraint loss includes a timbre consistency constraint term and a background residual energy constraint term. The timbre consistency constraint term is used to constrain the consistency of the predicted human voice track and the teacher's human voice track in terms of timbre characteristics. The background residual energy constraint term is used to constrain the background sound energy in the predicted human voice track.

[0107] Furthermore, based on the above embodiments, the singing detection module 300 may specifically include: The acquisition unit is used to acquire the singing probability, singing timestamp, singing duration, and singing proportion of the segment duration corresponding to the human voice segment. The determination unit is used to determine that the human voice segment contains singing when the probability of the singing is greater than a preset threshold, or the timestamp of the singing has a valid interval, or the duration of the singing or the proportion of the singing to the segment duration meets a preset condition.

[0108] Furthermore, based on the above embodiments, the white track selection module 500 may specifically include: The processing unit is used to perform singing detection, speech activity detection, subtitle text matching, and timbre matching with adjacent human voice segments on the two candidate audio tracks obtained by blind source separation. The first determining unit is used to determine the other candidate audio track as the target dialogue track if only one candidate audio track meets the singing detection condition. The second determining unit is used to determine the target dialogue track from the candidate audio tracks if both candidate audio tracks meet or do not meet the conditions for singing detection, based on at least one of the results of the speech activity detection, the results of the subtitle text matching, and the timbre matching with adjacent human voice segments. The backfill unit is used to backfill the target white track to the corresponding subtitle timestamp position, perform boundary connection processing on adjacent target white tracks, and generate the final voice track based on the backfilled target white tracks.

[0109] It should be noted that the order of the modules and units in the above-mentioned audio separation device can be changed without affecting the logic.

[0110] An audio separation device provided in this embodiment of the invention includes a model acquisition module 100, which acquires a scenario-based separation model trained with teacher pseudo-labels as a supervised signal. The teacher's voice track in the teacher pseudo-label is generated by dual-model cascade separation and alignment and time-frequency adaptive fusion of residual voices. An initial separation module 200 uses the scenario-based separation model to separate the voices from the background sounds in the original mixed audio to be processed, obtaining an initial voice track. A singing detection module 300 divides the initial voice track into at least two voice segments based on the subtitle timestamps and performs singing detection on each voice segment. A blind source separation module 400 performs blind source separation on the voice segments detected as containing singing, obtaining at least two candidate audio tracks. A white track selection module 500 selects a target white track based on the singing and speech features of the candidate audio tracks and generates the final voice track based on the target white track. This device aligns and adaptively fuses residual human voices recovered during dual-model cascade separation, generating pseudo-labels for teachers that include supplementary information about human voices in the background. This enables the scene-based separation model to achieve high-quality human voice purity and low background residue in a single inference. Simultaneously, it performs segment-level singing detection on the human voice tracks separated by the scene-based separation model by combining subtitle timestamps, and performs blind source separation and candidate track selection on singing segments. Based on singing features and speech features, it determines the target dialogue track, forming a complete process from human voice separation to singing separation. This effectively avoids singing contamination of the reference audio and provides a more reliable reference signal for downstream speech secondary development.

[0111] The audio separation device provided in the embodiments of the present invention will be described below. The audio separation device described below and the audio separation method described above can be referred to each other.

[0112] Please refer to Figure 3 , Figure 3 A schematic diagram of an audio separation device provided in an embodiment of the present invention may include: Memory 10 is used to store computer programs; Processor 20 is used to execute computer programs to implement the audio separation method described above.

[0113] The memory 10, processor 20, and communication interface 31 all communicate with each other through the communication bus 32.

[0114] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions: A scenario-based separation model trained with teacher pseudo-labels as supervision signals is obtained. The teacher voice track in the teacher pseudo-labels is generated by separating the dual-model cascade and aligning the residual voice and time-frequency adaptive fusion. A scene-based separation model is used to separate human voices from background sounds in the original mixed audio to be processed, thus obtaining the initial human voice track. The initial audio track is divided into at least two audio segments based on the subtitle timestamps, and singing is detected in each audio segment. Blind source separation is performed on vocal segments detected as containing singing to obtain at least two candidate audio tracks; The target dialogue track is selected based on the singing and speech features of the candidate audio tracks, and the final human voice track is generated based on the target dialogue track.

[0115] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0116] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0117] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0118] Communication interface 31 can be an interface for the communication module, used to connect with other devices or systems.

[0119] Of course, it should be noted that, Figure 3 The structure shown does not constitute a limitation on the audio separation device in the embodiments of the present invention. In practical applications, the audio separation device may include more than Figure 3 More or fewer components as shown, or combinations of certain components.

[0120] It is understood that if the audio separation method in the above embodiments is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the current technology, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), electrically erasable programmable ROM, register, hard disk, removable disk, CD-ROM, magnetic disk, or optical disk, and other media capable of storing program code.

[0121] Based on this, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the audio separation method described above.

[0122] The following describes a computer program product provided by an embodiment of this application. The computer program product described below can be referred to in conjunction with other embodiments described herein.

[0123] A computer program product includes a computer program / instructions that, when executed by a processor, implement the steps of the aforementioned disclosed audio separation method.

[0124] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0125] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0126] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0127] The above provides a detailed description of an audio separation method, apparatus, device, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. An audio separation method, characterized in that, include: A scenario-based separation model trained with teacher pseudo-labels as supervision signals is obtained. The teacher voice track in the teacher pseudo-labels is generated by dual-model cascade separation and alignment and time-frequency adaptive fusion of the residual voice. The scene-based separation model is used to separate human voices from background sounds in the original mixed audio to be processed, thereby obtaining an initial human voice track. The initial voice track is divided into at least two voice segments based on the subtitle timestamp, and singing is detected in each voice segment. Blind source separation is performed on the detected human voice segments containing singing to obtain at least two candidate audio tracks; the at least two candidate audio tracks correspond to dialogue or singing respectively. The target dialogue track is selected based on the singing and speech features of the candidate audio tracks, and the final human voice track is generated based on the target dialogue track. The training process of the scenario-based separation model includes: The original mixed audio used for training is separated into a first vocal track and a first background audio track using the first separation model; The first background audio track is separated a second time using the second separation model to obtain the second background audio track and the residual human voice track, and the second background audio track is used as the teacher's background audio track. The residual vocal track is subjected to amplitude alignment, phase alignment, and time delay alignment to obtain the aligned vocal track. Based on time-frequency adaptive fusion weights, the aligned human voice track and the first human voice track are fused, and the teacher's human voice track is determined based on the fused human voice track. Based on the teacher's background audio track, it is augmented using multi-source background audio data to obtain augmented background audio. The teacher's voice track and the augmented background audio are then mixed to obtain training mixed audio. The training mixed audio is used as the input to the third separation model, and the teacher's voice track and the teacher's background audio track are used as the supervision targets of the third separation model. The third separation model is trained based on the constructed loss function, and the trained third separation model is used as the scene-based separation model.

2. The audio separation method according to claim 1, characterized in that, Before fusing the aligned vocal track and the first vocal track based on time-frequency adaptive fusion weights, and determining the teacher's vocal track based on the fused vocal track, the method further includes: Calculate the voice confidence of the residual voice track, the energy ratio of the residual voice track to the second background sound track, the spectral correlation between the residual voice track and the first voice track, and the change in reconstruction error before and after fusion; The time-frequency adaptive fusion weights are determined based on the voice confidence level, the energy ratio, the spectral correlation, and the change in reconstruction error.

3. The audio separation method according to claim 1, characterized in that, Determining the teacher's voice track based on the fused voice track includes: The fused human voice track is enhanced using a speech enhancement model to obtain an enhanced human voice track; The fusion coefficient is determined based on the enhancement quality score, and the fused vocal track and the enhanced vocal track are adaptively fused based on the fusion coefficient to obtain the teacher vocal track; Among them, the enhanced quality score is positively correlated with background suppression gain, timbre similarity, and speech recognition consistency, and negatively correlated with enhancement distortion risk; the fusion coefficient increases with the increase of the enhanced quality score.

4. The audio separation method according to claim 1, characterized in that, The process of constructing the loss function includes: Construct vocal reconstruction loss, background noise reconstruction loss, hybrid reconstruction consistency loss, and vocal quality constraint loss; The human voice quality constraint loss includes a timbre consistency constraint term and a background residual energy constraint term. The timbre consistency constraint term is used to constrain the consistency of the predicted human voice track and the teacher's human voice track in terms of timbre characteristics. The background residual energy constraint term is used to constrain the background sound energy in the predicted human voice track.

5. The audio separation method according to claim 1, characterized in that, Perform vocal testing on each vocal segment, including: Obtain the singing probability, singing timestamp, singing duration, and singing proportion of the segment duration corresponding to the human voice segment; When the probability of the singing voice is greater than a preset threshold, or the timestamp of the singing voice has a valid range, or the duration of the singing voice or the proportion of the singing voice to the segment duration meets a preset condition, it is determined that the human voice segment contains singing voice.

6. The audio separation method according to any one of claims 1 to 5, characterized in that, Selecting a target dialogue track based on the singing and speech features of the candidate audio tracks, and generating a final vocal track based on the target dialogue track, includes: The two candidate audio tracks obtained by blind source separation are subjected to singing detection, speech activity detection, subtitle text matching, and timbre matching with adjacent human voice segments, respectively. If only one candidate audio track meets the conditions for singing detection, then the other candidate audio track is determined as the target dialogue track. If both candidate audio tracks meet or do not meet the conditions for singing detection, then the target dialogue track is determined from the candidate audio tracks based on at least one of the results of the speech activity detection, the results of the subtitle text matching, and the timbre matching with adjacent human voice segments. The target white track is backfilled to the corresponding subtitle timestamp position, and the adjacent target white tracks are connected by boundary processing. The final voice track is generated based on the backfilled target white tracks.

7. An audio separation device, characterized in that, include: The model acquisition module is used to acquire the scene-based separation model trained with teacher pseudo-labels as supervision signals. The teacher voice track in the teacher pseudo-label is generated by using dual-model cascade separation and by aligning the residual voice and time-frequency adaptive fusion. The initial separation module is used to separate human voices from background sounds in the original mixed audio to be processed using the scene-based separation model, thereby obtaining an initial human voice track. The singing detection module is used to divide the initial human voice track into at least two human voice segments according to the subtitle timestamp, and to perform singing detection on each human voice segment; The blind source separation module is used to perform blind source separation on human voice segments detected as containing singing, to obtain at least two candidate audio tracks; the at least two candidate audio tracks correspond to dialogue or singing respectively; The dialogue track selection module is used to select a target dialogue track based on the singing and speech features of the candidate audio tracks, and generate the final voice track based on the target dialogue track. The model acquisition module includes: The first separation unit is used to separate the original mixed audio used for training into a first vocal track and a first background audio track using the first separation model; The second separation unit is used to perform secondary separation of the first background audio track using the second separation model to obtain the second background audio track and the residual human voice track, and to use the second background audio track as the teacher's background audio track. An alignment unit is used to perform amplitude alignment, phase alignment and time delay alignment on the residual vocal track to obtain an aligned vocal track. The fusion unit is used to fuse the aligned human voice track and the first human voice track based on the time-frequency adaptive fusion weight, and to determine the teacher's human voice track based on the fused human voice track. An expansion unit is used to expand the teacher's background audio track using multi-source background audio data to obtain expanded background audio, and to mix the teacher's voice track and the expanded background audio to obtain training mixed audio. The training unit is used to take the training mixed audio as input to the third separation model, take the teacher's voice track and the teacher's background audio track as supervision targets of the third separation model, train the third separation model based on the constructed loss function, and use the trained third separation model as the scene-based separation model.

8. An audio separation device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the audio separation method as described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the audio separation method as described in any one of claims 1 to 6.

Images