A method, apparatus and electronic device for suppressing hallucinated speech

Abstract

This application provides a method, apparatus, and electronic device for suppressing hallucinatory speech, aiming to solve the problem of hallucinatory speech in generated audio in scenarios where only sound effects or music are needed. In this embodiment, after obtaining a pre-trained V2X generation model, the model is fine-tuned based on the generated supervised audio (excluding human voice) and the original video frame sequence after processing the human voice component. This allows the V2X generation model to forget the speech generation pattern while retaining and further learning the ability to generate sound effects and music. Therefore, when generating audio based on this V2X generation model, hallucinatory speech can be effectively suppressed, improving the quality and accuracy of the generated audio.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence multimedia generation technology, and specifically relates to a method, apparatus and electronic device for suppressing hallucinatory speech. Background Technology

[0002] In recent years, multimodal generation technology has become an important research direction in the field of artificial intelligence. End-to-End Video-to-Audio Generation (V2A) is an important branch of this technology, aiming to directly generate audio that is semantically synchronized with the video content, using a sequence of video frames as input. It has broad application prospects in scenarios such as film and television post-production, animation dubbing, virtual reality, and game sound effect design.

[0003] Currently, V2A generation models have made significant progress, capable of generating sound effects and music synchronized with visuals. However, a prominent problem commonly exists in practical applications of these V2A generation models—phantom speech. Phantom speech refers to audio segments output by V2A generation models that possess acoustic features of human speech (including fundamental frequency, formants, syllable rhythm, etc.) but do not carry any intelligible language information. In scenarios where only sound effects or music are needed, the generation of phantom speech severely degrades the overall quality of the generated audio, harming the user experience.

[0004] The main reason for the generation of hallucinatory speech is that most V2A generation models adopt a joint visual-text-audio modeling paradigm, which only establishes semantic alignment at the sound event level on multimodal data and does not have the ability to model language-level information.

[0005] Therefore, there is an urgent need for an effective method to suppress hallucinatory speech, in order to improve the quality of generated audio and user experience in scenarios where only sound effects or music are required. Summary of the Invention

[0006] This application provides a method, apparatus, and electronic device for suppressing hallucinatory speech, aiming to solve the problem of hallucinatory speech in the generated audio in scenarios where only sound effects or music are needed.

[0007] In a first aspect, embodiments of this application provide a method for suppressing hallucinatory speech, the method comprising: Obtain any sample pair from a pre-saved sample set, wherein the sample pair contains a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence; Using the video frame sequence and the supervised audio, the pre-trained V2X generation model is fine-tuned by forgetting to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

[0008] In one possible implementation, the human voice component of the original audio corresponding to the video frame sequence is processed by at least one of the following methods to obtain the supervised audio: The original audio corresponding to the video frame sequence is separated into human voice channel audio and non-human voice channel audio using a source separation model, and the non-human voice channel audio is used as the supervision audio. The semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence are identified using the speech recognition model ASR. The semantic speech segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio. The speech activity detection (VAD) model is used to detect speech activity segments in the original audio corresponding to the video frame sequence. The speech activity segments include sub-speech segments that do not have semantic meaning. The speech activity segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio.

[0009] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If so, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the first deviation between the output audio and the supervisory audio; The speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

[0010] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If not, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio of the V2X generation model. Determine the second deviation between the output audio and the supervisory audio; The sound effect parameters of the V2X generation model are fine-tuned based on the second bias and the preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

[0011] In one possible implementation, whether the current stage is the first fine-tuning phase is determined as follows: If this is the first time the V2X generation model has been fine-tuned, then it is determined that the current stage is the first fine-tuning stage; Otherwise, obtain the target output audio of the V2X generation model when it was last fine-tuned; determine the hallucination rate in the output audio using a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; determine whether the hallucination rate is lower than a preset threshold, if yes, determine that it is currently in the second fine-tuning stage, if no, determine that it is currently in the first fine-tuning stage.

[0012] In one possible implementation, determining the hallucination rate in the output audio using a pre-tuned large language model includes: The VAD model, which has been pre-trained, identifies speech segments in the output audio. Identify the target segment of speech pronunciation within the aforementioned speech segment; The target fragment is transcribed using a pre-trained transcription model. Based on a pre-tuned large language model, it is determined whether the transcribed text is a random combination of characters or syllables. If so, the target segment is determined to be hallucinatory speech. Based on the proportion of the hallucinatory speech in the duration of the output audio, the hallucination rate of the output audio is determined.

[0013] In one possible implementation, identifying the target segment of speech pronunciation within the speech segment includes: The speech segments are classified by an audio classifier, and the segments containing spoken language are retained to obtain the target segments; wherein, the segments containing non-spoken language include at least one of humming, laughter and crying.

[0014] In one possible implementation, the method further includes: Receive the sequence of video frames to be set to music; The sequence of video frames to be set to music is input into the target V2X generation model to obtain the target audio output by the target V2X generation model.

[0015] Secondly, embodiments of this application also provide a device for suppressing hallucinatory speech, the device comprising: The acquisition module is used to acquire any sample pair from a pre-saved sample set, wherein the sample pair includes a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence. The processing module is used to perform forgetting fine-tuning on the pre-trained V2X generation model using the video frame sequence and the supervised audio to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

[0016] In one possible implementation, the processing module is specifically configured to perform at least one of the following processing steps on the human voice component of the original audio corresponding to the video frame sequence to obtain the supervised audio: using a source separation model to separate the original audio corresponding to the video frame sequence into human voice channel audio and non-human voice channel audio, and using the non-human voice channel audio as the supervised audio; using ASR to identify semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence, replacing the semantic speech segments with a preset non-human voice residual signal, and using the replaced audio as the supervised audio; using a VAD model to detect speech activity segments in the original audio corresponding to the video frame sequence, wherein the speech activity segments include sub-speech segments without semantic meaning, replacing the speech activity segments with a preset non-human voice residual signal, and using the replaced audio as the supervised audio.

[0017] In one possible implementation, the processing module is specifically used to determine whether it is currently in the first fine-tuning stage. If so, it inputs the video frame sequence into the pre-trained V2X generation model to obtain the output audio output by the V2X generation model; determines the first deviation between the output audio and the supervised audio; and fine-tunes the speech parameters of the V2X generation model based on the first deviation and a preset first learning rate.

[0018] In one possible implementation, the processing module is specifically used to determine whether it is currently in the first fine-tuning stage. If not, it inputs the video frame sequence into the pre-trained V2X generation model to obtain the output audio output by the V2X generation model; determines the second deviation between the output audio and the supervised audio; and fine-tunes the sound effect parameters of the V2X generation model based on the second deviation and a preset second learning rate. The second learning rate is less than the preset first learning rate.

[0019] In one possible implementation, the processing module is specifically configured to determine whether it is currently in the first fine-tuning stage by: if this is the first time the V2X generation model has been fine-tuned, then it is determined that it is currently in the first fine-tuning stage; otherwise, it obtains the target output audio of the V2X generation model when it was last fine-tuned; it determines the hallucination rate in the output audio using a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; it determines whether the hallucination rate is lower than a preset threshold, and if so, it is determined that it is currently in the second fine-tuning stage, otherwise, it is determined that it is currently in the first fine-tuning stage.

[0020] In one possible implementation, the processing module is specifically configured to: identify speech segments in the output audio using a pre-trained VAD model; identify target segments of speech articulation within the speech segments; transcribe the target segments into text using a pre-trained transcription model; determine whether the transcribed text is a random combination of characters or syllables based on a pre-fine-tuned large language model; if so, determine that the target segment is hallucination speech; and determine the hallucination rate of the output audio based on the proportion of the hallucination speech in the duration of the output audio.

[0021] In one possible implementation, the processing module is specifically used to classify the speech segments using an audio classifier, retaining the segments of spoken language to obtain the target segment; wherein, the segments of non-spoken language include at least one of humming, laughter, and crying.

[0022] In one possible implementation, the processing module is further configured to receive a sequence of video frames to be accompanied by music; input the sequence of video frames to be accompanied by music into the target V2X generation model, and obtain the target audio output by the target V2X generation model.

[0023] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is configured to execute a computer program stored in the memory to implement the steps of the method for suppressing phantom speech as described in any of the preceding claims.

[0024] In this embodiment, after obtaining the pre-trained V2X generation model, the pre-trained V2X generation model is fine-tuned based on the generated supervised audio (without human voice) and the original video frame sequence after processing the human voice component. This allows the V2X generation model to forget the speech generation pattern while retaining and further learning the ability to generate sound effects and music. As a result, when generating audio based on this V2X generation model, it is possible to effectively suppress phantom speech and improve the quality and accuracy of the generated audio. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating a method for suppressing hallucinatory speech provided in an embodiment of this application; Figure 2 A schematic diagram illustrating the generation of supervised audio as provided in an embodiment of this application; Figure 3 A schematic diagram illustrating the process of determining the illusion rate provided in this application embodiment; Figure 4 A schematic diagram of the spectrum visualization of supervised audio and real values ​​provided in an embodiment of this application; Figure 5 A schematic diagram of a device for suppressing hallucinatory speech provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To make the objectives and implementation methods of this application clearer, the exemplary implementation methods of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0028] It should be noted that the brief descriptions of terms in this application are only for the convenience of understanding the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise stated, these terms should be understood in their ordinary and common meaning.

[0029] The terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar or related objects or entities, and do not necessarily imply a specific order or sequence, unless otherwise specified. It should be understood that such terms are interchangeable where appropriate.

[0030] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0031] The term "module" refers to any known or subsequently developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0032] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0033] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

[0034] The terms used in this application are explained as follows: V2X: refers to a generative model system that takes a sequence of video frames as input and directly generates audio waveforms or spectrograms that are semantically synchronized with the video content.

[0035] Hallucinatory speech: Audio segments output by a V2A generation model that possess acoustic features of human speech (fundamental frequency, formants, syllable rhythm) but do not carry any intelligible language information. Its content typically manifests as: random syllable strings; meaningless repetitions; and a mixture of various language fragments that do not conform to grammatical rules.

[0036] Voice Activity Detection (VAD): A binary classification model that determines the presence of human voice activity in audio at the frame level. The output mask m_vad(t)∈{0,1}, where 1 indicates that speech or speech-like energy has been detected. This application employs pre-trained models such as the Silo VAD, with adjustable thresholds to minimize the missed detection of non-speech human voices.

[0037] Distribution-Matching Metrics: These are quantitative metrics used to measure the consistency between generated and real audio distributions in the embedding space. This application involves: Fréchet Distance (FD): Based on features extracted by a deep audio classification network, the Fréchet Inception Distance (FID) distance between feature distributions is calculated. The deep audio classification network includes: VGGish, Large-scale Pretrained Audio Neural Networks (PANNs), and PaSST; Kullback-Leibler (KL) Divergence: Using the posterior of the output class of the same classifier, the difference between the predicted distributions of generated and real audio is calculated.

[0038] Semantic Alignment Metrics (ImageBind Score, IB-Score): The ImageBind multimodal large model is used to extract video embeddings and audio embeddings respectively, calculate cosine similarity, and evaluate the semantic consistency between "image and sound".

[0039] De-Synchronization (DeSync): Based on the Synchformer model, it predicts the time offset between audio and video. The smaller the value, the better the synchronization.

[0040] High-quality video-to-audio (Multimodal Audio, MMAudio) model trained using multimodal methods: This application presents a publicly available state-of-the-art end-to-end V2A generative model trained using multimodal methods, with a 44.1kHz waveform output. This application uses its L-44.1kHz-v2 version as the baseline model and the starting point for fine-tuning.

[0041] VGGSound: A large-scale audio-visual dataset used to train V2A models, containing a large amount of speech data.

[0042] Understandably, multimodal generation has become an important research topic in generative artificial intelligence in recent years. Video-to-audio generation, as a multimodal generation technique, has enormous potential for widespread application in multimedia creation, such as film, animation, and video game production. Current V2A generation models have made significant progress, capable of producing suitable audiovisual synchronized sound effects and music. However, these models often produce phantom speech, i.e., audio that mimics the acoustic properties of natural language but lacks a correspondence with any real language system or semantic meaning. In scenarios where only sound effects or music are needed, the generation of phantom speech has become a significant problem because it degrades the overall audio quality.

[0043] The two main reasons for this problem are: First, most state-of-the-art V2A models employ a uniform visual-text-audio joint modeling paradigm, establishing semantic alignment only at the sound event level across all multimodal data, without the ability to model language-level information. Second, most multimodal datasets used to train V2A models (such as VGGSound) contain a significant proportion of speech data. During training, speech samples are treated the same as other sound effects and music, thus converting the rich context and linguistic semantic information inherent in speech into harmful redundancy. Speech in the training data leads to the generation of phantom speech that deviates from the target during inference, significantly impairing the quality of the generated audio.

[0044] To simultaneously obtain intelligible speech and realistic background audio, some recent studies have attempted to introduce additional modal information to enhance generation capabilities. For example, DualDub proposed using a cross-modal aligner and a multimodal large language model to jointly model four modalities: visual, audio, speech, and text, achieving the goal of generating intelligible speech for videos. However, while enhancing speech generation capabilities, this method significantly weakens the modeling ability for sound effects and music, resulting in insufficient performance in terms of environmental atmosphere, emotional expression, and richness of detail in the generated audio. Given that in practical applications, most video content (such as scenery, actions, and object interactions) relies more on high-quality environmental sound effects and music to enhance immersion than intelligible speech, this application argues that the core objective of V2A tasks should prioritize generating reliable, high-quality sound effects and music, rather than pursuing speech intelligibility. Therefore, there is an urgent need for a V2A technology solution that can effectively suppress phantom speech generation and maintain the quality of generated sound effects and music without introducing intelligible speech.

[0045] Based on the above background, this application embodiment systematically introduces the speech forgetting mechanism into the end-to-end video-to-audio generation framework for the first time. Through a collaborative strategy of data-level speech suppression and model-level fine-tuning of forgetting, it achieves effective elimination of hallucinatory speech and high preservation of the quality of sound effects / music generation.

[0046] This application aims to address the common problem of hallucinatory speech in V2A generation tasks, where the model outputs fake speech segments that have acoustic features of human voices but lack intelligible semantic content. This problem stems from an excessively high proportion of speech samples in the training corpus, causing the model to incorrectly synthesize hallucinatory speech even in scenarios where only sound effects or music generation is required, significantly degrading audio quality and user experience. Existing methods either rely on additional text modalities to improve speech intelligibility or employ simple post-processing filtering, which increases inference complexity, fails to eradicate the hallucinatory speech generation mechanism, and easily disrupts the distribution consistency of sound effects / music.

[0047] To address this, this application proposes a data suppression-forgetting fine-tuning collaborative framework. For the first time, it actively removes speech components from the training supervision target and leverages the catastrophic forgetting characteristics of neural networks to fine-tune the speech generation pathway of the directional suppression model in a lightweight manner. This framework requires no external text annotation or speech labeling, and can reduce the incidence of hallucinatory speech while preserving the high-fidelity generation capabilities of sound effects and music, achieving zero-hallucination speech output from the V2A model in pure sound effect scenarios. Ultimately, this promotes the implementation of high-quality, high-reliability audio-video synchronous generation technology, meeting the needs of film and television post-production, animation, games, and virtual reality fields for automated audio creation without speech interference and with a strong sense of immersion.

[0048] Based on this, embodiments of this application provide a method, apparatus, and electronic device for suppressing hallucinatory speech.

[0049] The method for suppressing hallucinatory speech includes: an electronic device acquiring any sample pair from a pre-saved sample set, wherein the sample pair contains a video frame sequence and corresponding supervised audio; wherein the supervised audio is supervised audio generated without human voice after processing the human voice component of the original audio corresponding to the video frame sequence; using the video frame sequence and supervised audio, a pre-trained end-to-end video-to-audio V2X generation model is fine-tuned by forgetting to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

[0050] Example 1: Figure 1 A schematic diagram of a method for suppressing hallucinatory speech provided in this application embodiment, the process including the following steps: S101: Obtain any sample pair from the pre-saved sample set, wherein the sample pair contains a video frame sequence and the corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence.

[0051] The method for suppressing hallucinatory speech provided in this application is applied to an electronic device, which can be a smart device such as a PC or a server.

[0052] In the data preprocessing stage, this application proposes a data-level speech suppression scheme, which aims to eliminate speech components from the source of training data to the supervised signal of the V2X generative model, thereby constructing a sample set of video-audio pairs containing only clean sound effects and music.

[0053] In one possible implementation, for any training sample pair in a pre-saved sample set, the sample pair includes a video frame sequence and the corresponding supervisory audio. The supervisory audio is not the original audio mixture signal, but rather a clean audio signal without human voices obtained by selectively processing the human voice component in the original audio corresponding to the video frame sequence. It is understood that the human voice described here is broadly defined, encompassing not only human speech but also non-verbal vocalizations such as laughter, humming, and crying. Non-human voices cover environmental background sounds, instrumental music, object collisions, and other non-verbal events; thus, the non-human voice residual can be referred to as rnon(t). Through this data-level intervention, the V2X generative model is unable to learn any mapping patterns related to speech or human voices from the supervisory target during the training phase, fundamentally cutting off the knowledge source for phantom speech generation.

[0054] It is understandable that, compared to post-processing correction at the output or inference stage of the V2X generative model, the advantage of data-level speech suppression described in this application is its pre-emptive and fundamental nature. That is, when there is no speech-related supervision signal in the sample pairs used for training, the V2X generative model will be unable to recall or reproduce speech patterns that have never been seen before in the inference process of actual applications.

[0055] This application employs a data-level speech suppression scheme to construct a sample pair of visual input and clean non-speech audio output from the source of the training data. In this approach, the supervised signal (i.e., the supervised audio) faced by the V2X generation model no longer contains any form of speech component. This means that the V2X generation model cannot establish a path from video semantic features to speech patterns from the training sample pair, thereby effectively cutting off the process of generating hallucinatory speech.

[0056] S102: Using the video frame sequence and the supervised audio, the pre-trained V2X generation model is fine-tuned by forgetting to obtain the target V2X generation model, which is used to suppress phantom speech in video generation.

[0057] In this embodiment, the electronic device uses the acquired sample pairs, namely clean video and audio, to perform forgetting fine-tuning on the pre-trained V2X generation model, thereby obtaining a target V2X generation model that can effectively suppress the phenomenon of hallucinatory speech in video generation.

[0058] The forgetting fine-tuning technique differs fundamentally from conventional model fine-tuning. Conventional fine-tuning aims to adapt the model to new data distributions or to learn new task capabilities, while the forgetting fine-tuning proposed in this application aims to leverage the catastrophic forgetting mechanism in neural networks to guide the V2X generative model to selectively forget previously learned speech generation patterns from training data containing a large amount of speech during the adaptation to speech-free data distributions, while retaining and maintaining its ability to generate sound effects and music. This technique does not require explicitly locating and deleting speech-related neurons or parameter connections in the V2X generative model. Instead, by changing the data distribution during the fine-tuning phase, the V2X generative model naturally and gradually weakens its response strength to speech features during parameter update iterations.

[0059] Regarding the selection of the pre-trained model, this application can adopt the currently publicly available state-of-the-art end-to-end video-to-audio generation model architecture as the baseline model for fine-tuning. In a preferred implementation, the MMAudio-L-44.1kHz-v2 version is selected as the base model. This model employs a multimodal joint training strategy, is capable of outputting high-quality audio waveforms at a sampling rate of 44.1kHz, and has undergone sufficient pre-training on large-scale audio and video datasets such as VGGSound during the original training phase, possessing the basic ability to generate audio-visual synchronized sound effects and music. However, precisely because the VGGSound dataset contains a large amount of speech data, this baseline model exhibits a significant tendency to generate hallucinatory speech during inference, that is, even when there are no speech cues in the visual scene, the model still tends to fabricate speech content that should not exist in the generated audio. This phenomenon is essentially a statistical learning result caused by the bias in the distribution of training data, rather than a defect in the model architecture itself. Therefore, this bias is corrected by changing the data distribution during the fine-tuning phase.

[0060] In the data preparation stage, this application employs the aforementioned data-level speech suppression method to systematically process the original training dataset. In one possible implementation, the original VGGSound training set is used as the processing object. Speech components are detected, separated, and masked in the original audio of each sample in the sample set, constructing a large-scale fine-tuned sample set containing only clean sound effects and music. In the processed dataset, the non-human voice residual rnon(t) serves as the supervision target for model fine-tuning, while the video frame sequence remains unchanged, thus forming a clean visual-audio mapping pair. The key to this data preparation process is that the fine-tuned dataset is completely consistent with the original training dataset in terms of video content distribution; the only change is the presence or absence of human voice components in the audio supervision target. This allows subsequent observed changes in model behavior to be clearly attributed to the removal of the speech supervision signal, rather than other confounding factors.

[0061] In the specific execution of forgetting fine-tuning, this application carefully configures the training parameters to achieve selective knowledge forgetting. Specifically, the learning rate can be set to 1×102. -5 This relatively low value ensures the stability of the fine-tuning process, allowing the V2X generative model to move smoothly along the gradient direction in the parameter space, avoiding excessive forgetting of non-target knowledge (i.e., sound effects and music generation capabilities) due to excessively large step sizes. Other training parameters in the V2X generative model, such as the optimizer, batch size, and number of training epochs, are consistent with the training configuration of the original MMAudio model to minimize the interference of training strategy differences on the fine-tuning effect. The sample set size is set to be comparable to the original VGGSound training set size, ensuring that the V2X generative model obtains sufficient data iterations during fine-tuning without unnecessarily extending the training cycle due to excessive data volume. The parameter configuration in the inference phase is also consistent with the baseline model to objectively compare performance changes before and after fine-tuning under identical evaluation conditions, ensuring the reliability and reproducibility of experimental conclusions.

[0062] From the perspective of the forgetting mechanism's implementation principle, when the V2X generation model optimizes its parameters on the aforementioned processed non-speech data distribution, the supervision signal for its loss function only comes from the non-voice residual rnon(t). This means that any feature responses related to speech generation cannot obtain positive feedback for gradient updates under this data distribution; instead, the network parameters corresponding to these feature responses will be gradually weakened or covered during iteration. Meanwhile, since the supervision signal still retains complete background sound, environmental sound effects, and music information, the relevant parameters in the model responsible for generating these non-speech audio elements will continue to receive positive feedback and be consolidated. This selective forgetting effect—partial retention and partial forgetting—is precisely the ideal fine-tuning result pursued in this application. During inference, the V2X generation model preferentially activates generation paths related to sound effects and music, while significantly suppressing the activation probability of speech paths.

[0063] The aforementioned forgetting-tuned target V2X generation model can be deeply coupled with intelligent transmission networks and generative intelligent transmission to further improve the deployment efficiency and practical value of end-to-end video-to-audio generation tasks in weak network environments. The core of intelligent transmission networks lies in the deep integration of communication, networking, and artificial intelligence. Unlike traditional communication technologies that merely transfer raw information symbolically, generative intelligent transmission uses the core concept of trading computation for bandwidth. Instead of directly transmitting massive amounts of raw data, it transmits lexical units that condense the semantics, features, and conditions within the data, and then reconstructs the information at the receiving end. Based on this architecture, the audio generation results output by the forgetting-tuned target V2X generation model do not need to be transmitted raw in the form of complete waveform data. Instead, at the sending end, the key semantic features and acoustic conditions in the audio generation results can be condensed into compact lexical representations, transmitted via the network link, and then the audio content can be reconstructed at the receiving end using generative intelligent transmission technology. This coupling scheme is particularly suitable for scenarios where speech generation is not required. Once the target V2X generation model has effectively suppressed the tendency to generate hallucinatory speech through forgetting fine-tuning, the pure sound effects and music content it generates can be further distributed efficiently with the help of the lexical transmission mechanism of the intelligent transmission network, so that the receiving end can still obtain an auditory experience that is basically consistent with the original generation result under extremely low bandwidth conditions.

[0064] In this embodiment, after obtaining the pre-trained V2X generation model, the pre-trained V2X generation model is fine-tuned based on the generated supervised audio (without human voice) and the original video frame sequence after processing the human voice component. This allows the V2X generation model to forget the speech generation pattern while retaining and further learning the ability to generate sound effects and music. As a result, when generating audio based on this V2X generation model, it is possible to effectively suppress phantom speech and improve the quality and accuracy of the generated audio.

[0065] Example 2: To obtain the supervisory audio accurately and effectively, based on the above embodiments, in this embodiment, the human voice component of the original audio corresponding to the video frame sequence is processed by at least one of the following methods to obtain the supervisory audio: The original audio corresponding to the video frame sequence is separated into human voice channel audio and non-human voice channel audio using a source separation model, and the non-human voice channel audio is used as the supervision audio. The ASR model is used to identify semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence. The semantic speech segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio. The VAD model is used to detect speech activity segments in the original audio corresponding to the video frame sequence. The speech activity segments include sub-speech segments that do not have semantic meaning. The speech activity segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio.

[0066] This application introduces three complementary data-level speech suppression strategies, which process the original audio in the sample set and then fine-tune the V2A model based on the processed supervised audio, thereby reducing the supervision signal of the V2X generation model for speech patterns at the source. These three strategies each have different technical focuses and scopes of application, making them suitable for different application scenarios. They can also be cascaded and combined according to actual needs to achieve a flexible balance between speech suppression strength, semantic accuracy, and computational cost.

[0067] The first strategy is the Separation-guided Global Masking (Sep-GM) strategy. This strategy utilizes a mature source separation model to parse the original audio, decomposing the input audio into two independent audio streams in the channel dimension: a vocal channel and a non-vocal channel. After channel separation, the entire content of the vocal channel can be discarded, retaining only the audio signal of the non-vocal channel as the supervision target. The advantage of this approach is that it achieves global-level removal of vocal components, completely eliminating the possibility of learning speech mappings in the model from the physical level of audio composition, providing the strongest upper bound for speech removal in training environments without speech backgrounds. Because the vocal channel is completely stripped away, any form of language content, speech-like vocalizations, and even non-speech vocalizations are excluded at once, making it impossible for the V2X generative model to obtain any spectral patterns related to vocal cord vibration from the supervision audio. Meanwhile, thanks to the source separation model's preservation of the audio temporal structure during the separation process, the background sounds and music components in the non-human voice channel maintain their original temporal continuity and spectral integrity, and will not produce temporal breakpoints or phase errors due to the separation operation. This is crucial for the subsequent V2X generative model to learn the fine temporal relationship of audiovisual synchronization.

[0068] The second strategy is a semantic masking strategy guided by Automatic Speech Recognition (ASR). This strategy uses a timestamped ASR model as a semantic localization tool. First, ASR recognition is applied to the original audio. In one possible implementation, a pre-trained ASR model is used to perform ASR recognition on the human voice channel in the original audio, obtaining word-level or phrase-level time boundary information containing lexical content and corresponding start and end timestamps. Based on these time boundaries, a binary mask sequence m can be constructed. asr(t) ∈ {0, 1}, this mask takes a value of 1 only on the speech span containing intelligible semantic content, and a value of 0 at other time-domain positions. Subsequently, this mask is used to perform selective replacement operations on the original audio: when m asr When m(t) is 0, the original audio signal a(t) remains unchanged; when m asr When (t) is 1, the audio content of the corresponding time span is replaced with the pre-saved non-human voice residual signal r. non (t). Its mathematical expression can be: asr (t) = (1 - m asr (t))·a(t) + m asr (t)·r non (t), where, asr (t) represents the supervised audio obtained at time t, m asr (t) represents the original audio at time t, a(t) represents the original audio at time t, and r non (t) represents the preset non-human voice residual signal. This method boasts extremely high semantic precision, removing not only audio segments carrying clearly understandable language content but also avoiding semantically insensitive human voice components (such as laughter, sighs, coughs, etc.) and performing no processing on background ambient sounds. This makes Automatic Speech Recognition-guided Semantic Masking (ASR-SM) highly efficient when processing samples where the speech content accounts for only a small portion of the total audio duration. It preserves the natural texture of the original audio to the greatest extent possible, avoiding excessive masking that leads to low information density in the supervisory signal. Furthermore, ASR-SM's timestamp localization capability allows for precise start and end point positioning of the replaced semantic segments. The non-human voice residual signal in the replaced audio and the original background audio can achieve a smooth transition at the boundary, avoiding auditory abruptness caused by discontinuous boundaries.

[0069] The third strategy is the VAD-guided activity masking strategy. This strategy uses a lightweight frame-level speech activity detection model to distinguish the original audio a(t) frame by frame, marking all time-domain intervals where speech activity exists, and forming a binary activity mask m. vad(t)∈{0,1}, where frames identified as having speech activity have a value of 1, and the remaining frames have a value of 0. Unlike ASR-SM, the speech activity segments identified by the VAD model include not only intelligible semantic content but also all frames with speech acoustic features, even if these frames do not contain any identifiable lexical information. This means that Voice Activity Detection-guided Activity Masking (VAD-AM) covers a wider range, including not only spoken content but also sub-speech segments that do not have semantic carrying capacity, such as voiceless consonants, vowel transitions, breath sounds, and labiodental fricatives, as well as certain speech-like sounds similar to speech spectral features. After obtaining the VAD mask, this strategy replaces all speech activity segments marked as 1 with pre-saved non-human voice residual signals, keeping the unmarked areas unchanged. Its mathematical form is: = (1 - )· + · ,in, This indicates that after VAD masking, at time... The obtained supervisory audio signal; This is a binary activity mask, whose values ​​are determined based on the VAD model for time. The determination depends on the result of the audio frame discrimination. When the frame is determined to contain speech or speech-like activity, ,otherwise ; The original audio signal representing that moment; This refers to the pre-extracted non-human voice residual signal, which is the background sound and music information retained after removing the human voice component from the original audio. When When no speech activity is detected at that moment, the supervisory audio retains the original audio completely. ;when When a signal indicates the presence of speech activity, the original signal is replaced with a clean, non-human voice residual signal. The core advantages of VAD-AM lie in its broad coverage and low computational overhead. VAD models typically employ lightweight network structures, resulting in inference speeds far exceeding those of source separation models and ASR models, making them suitable for efficient data cleaning in large-scale dataset processing scenarios. Furthermore, because VAD performs discrimination at the frame level, its mask temporal resolution typically reaches the millisecond level, enabling it to capture extremely short speech segments. This characteristic is particularly advantageous when processing audio samples interspersed with sporadic speech or non-speech sounds. However, the decision boundaries of VAD models can sometimes become blurred, especially when the signal-to-noise ratio between speech and background noise is low. This can lead to the misclassification of some non-speech events as speech activities, resulting in unintentional information loss.

[0070] The three strategies mentioned above differ fundamentally at the operational level. The separation-driven strategy operates on the channel dimension, physically removing human voice components across the entire frequency band through source separation; while ASR-SM and VAD-AM operate on the time domain dimension, replacing signals by applying masks over specific time spans. The separation operation can remove all types of human voice components at once, unrestricted by semantic content or activity state, providing the strongest upper bound guarantee for speech-free backgrounds; however, ASR-SM and VAD-AM, in their masking operations, may unintentionally suppress non-speech events coexisting with the masked speech in the time domain. For example, when a line of dialogue and background music occur simultaneously, ASR-SM, when replacing the time span of that semantic segment, will equally replace the background music with r. non (t), resulting in the loss of music information within that time period; similarly, when VAD-AM replaces a frame containing speech activity, it also removes the ambient sound effects superimposed on the same frame. This incidental loss is an inherent characteristic of temporal masking strategies and a significant advantage of separation-driven strategies compared to the previous two. In practical implementation, the above three strategies can either be applied independently to the original audio or cascaded according to the actual needs of the application scenario. For example, VAD-AM can be used for coarse-grained, rapid full coverage of speech activity, and then ASR-SM can be used for semantic-level fine-tuning of the blurred boundary areas, thereby achieving an optimal balance between masking accuracy and computational overhead. The final output is a new audio track aligned with the video frame but without any human voices or human-like voices, which can be used directly as a clean label in the subsequent forgetting fine-tuning stage.

[0071] The three data-level speech suppression strategies described above bring significant improvements in multiple dimensions. Firstly, in reducing phantom speech generation, experimental results show that compared to the baseline model MMAudio's phantom speech frequency of 0.143, the Sep-GM strategy reduces the phantom speech frequency to 0.011, a reduction of approximately 92%. ASR-SM and VAD-AM reduce it by approximately 64% and 60%, respectively, significantly improving the quality of the generated audio. This improvement is particularly valuable in scenarios where only sound effects and music are needed, offering direct engineering benefits for practical applications such as video soundtracks and background sound effect generation. Secondly, in terms of flexibility and adaptability, the three data processing strategies offer different options for speech suppression strength and precision. Sep-GM provides the strongest upper bound for speech removal, suitable for demanding scenarios with zero tolerance for speech; ASR-SM achieves precise semantic-level suppression by accurately identifying the span of intelligible speech, suitable for applications that need to retain non-semantic human voice components; VAD-AM features broad coverage, low computational overhead, and strong adaptability, suitable for resource-constrained or large-scale data scenarios requiring rapid processing. This diversity of strategies allows for flexible selection of the optimal method based on specific application needs. Third, in terms of improving the semantic and temporal alignment of audio and video, by suppressing speech redundancy in the training data, the fine-tuned V2X generation model has a higher matching degree with the target audio distribution when generating audio. Spectrogram visualization shows that the audio processed by ASR-SM, VAD-AM, and Sep-GM is closer to the ground truth. Further validation on the VGG-Subset and VGG-Sep datasets in the absence of phantom speech interference also confirms the effectiveness of data-level speech suppression in maintaining audiovisual consistency.

[0072] Figure 2 This is a schematic diagram illustrating the generation of supervised audio, as provided in an embodiment of this application.

[0073] like Figure 2 As shown, supervised audio can be obtained by processing the original audio based on a speech activity detection (VAD) model, an speech recognition (ASR) model, or a source separation model.

[0074] The forgetting fine-tuning technique proposed in this application brings several significant benefits. First, in maintaining sound quality and music generation capabilities, because the supervision objective during the fine-tuning stage fully preserves the non-human voice residual rnon(t), the sound quality and music generation quality of the V2X generation model are not substantially impaired during the forgetting process. Experimental evaluation on the VGG-Sep dataset shows that the fine-tuned model using the Sep-GM strategy outperforms the original MMAudio baseline model in distribution matching metrics such as Fréchet distance and KL divergence. Simultaneously, semantic alignment and temporal synchronization metrics such as IB-Score and DeSync remain close, fully demonstrating that forgetting fine-tuning effectively suppresses phantom speech without negatively impacting sound quality and music generation capabilities. Second, in improving audio-video alignment performance, by selectively forgetting speech patterns, the fine-tuned model can focus more attention on semantic alignment at the sound event level, thereby improving audio-video synchronization. Experimental data from the DeSync metric show that the ASR-SM and VAD-AM strategies outperform the baseline model in temporal alignment on both the VGG-Subset and VGG-Sep test sets, indicating that forgetting fine-tuning enhances audiovisual consistency overall. Thirdly, regarding model adaptability and stability, a uniform 1×102 metric was used. -5 The training parameter configuration, with the learning rate consistent with the original model, ensures the stability and controllability of the entire fine-tuning process. Ablation experiments further demonstrate that continuous training on the original VGGSound dataset offers limited performance improvement, while forgetting fine-tuning on speech suppression data significantly optimizes the model's adaptability in speech-free scenarios. Fourthly, regarding its broad applicability and engineering value, the forgetting fine-tuning method proposed in this application is adaptable to various mainstream V2A model architectures and achieves efficient fine-tuning convergence with a dataset size comparable to the original training set. Spectral visualization results further confirm that the audio spectrum generated by the fine-tuned model is closer to the ground truth, intuitively verifying the effectiveness and reliability of this method in practical deployment.

[0075] Example 3: To accurately and effectively fine-tune the V2X generative model, based on the above embodiments, in this embodiment, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If so, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the first deviation between the output audio and the supervisory audio; The speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

[0076] During model fine-tuning, this application can employ a staged parameter update strategy to achieve selective forgetting of speech patterns.

[0077] In one possible implementation, the electronic device can first determine whether the current training iteration is in a preset first fine-tuning stage, which corresponds to the initial stage of V2X generative model fine-tuning, a period when the parameters have not yet fully adapted to the distribution of speechless data. In this first fine-tuning stage, the electronic device can input a video frame sequence processed with data-level speech suppression into the pre-trained V2X generative model. In one possible implementation, the V2X generative model receives the video frame sequence as input in an end-to-end manner, propagates it layer by layer through an encoder and decoder architecture, and finally outputs an audio waveform whose duration is aligned with the input video frame sequence. This audio waveform serves as the model's output audio. After obtaining the output audio from the V2X generative model, the electronic device can compare this output audio with the supervised audio, i.e., the clean audio without human voice obtained after speech suppression processing, frame-by-frame or time-step by time, calculating the first deviation between the two. This deviation can be quantified using various loss functions, including but not limited to mean squared error loss, multi-scale spectral loss, or perceptual weighted loss, with the aim of accurately measuring the degree of difference between the generated audio and the supervised target in the time and frequency domains.

[0078] The electronic device can calculate gradient information based on this first deviation and perform backpropagation updates on the speech parameters in the V2X generative model according to a preset first learning rate. The speech parameters referred to here are the set of network weights within the V2X generative model that are sensitive to acoustic features such as speech spectral envelope, fundamental frequency trajectory, and formant structure. These parameters are fully optimized during the initial pre-training phase due to the large amount of speech content in the training data, thus becoming the root cause of hallucinatory speech generated by the V2X generative model during inference. By driving these parameters to move in a direction that reduces the deviation between the generated audio and the clean supervised audio through the first learning rate, the V2X generative model gradually weakens its encoding ability for speech features during the iteration process, ultimately achieving the technical goal of forgetting fine-tuning. This phased judgment and parameter differential update strategy ensures the stability of the fine-tuning process in the early stages of training.

[0079] Example 4: To accurately and effectively fine-tune the V2X generative model, based on the above embodiments, in this embodiment, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If not, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio of the V2X generation model. Determine the second deviation between the output audio and the supervisory audio; The sound effect parameters of the V2X generation model are fine-tuned based on the second bias and the preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

[0080] In the specific implementation of V2X generative model fine-tuning, this application adopts a phased parameter update strategy to achieve selective forgetting and capability retention of the pre-trained V2X generative model.

[0081] In one possible implementation, the electronic device can first determine whether the current training iteration is in a preset first fine-tuning stage. This stage corresponds to the critical window period in the early stages of V2X generative model training, when the parameters have not yet fully adapted to the distribution of speech-free data. If it is determined that the current stage is not the first fine-tuning stage, it means that the V2X generative model has completed the initial coarse-grained forgetting of speech patterns, and then enters the second fine-tuning stage. This stage is for fine-tuning the sound effect generation capability of the V2X generative model. In this second fine-tuning stage, the electronic device can input a video frame sequence after data-level speech suppression processing into the pre-trained V2X generative model. The V2X generative model receives the video frame sequence as input in an end-to-end manner, and through layer-by-layer forward propagation of the encoder and decoder architecture, finally outputs an audio waveform whose duration is aligned with the input video frame sequence, i.e., the output audio of the V2X generative model. After obtaining the output audio, the electronic device can compare the output audio with the supervised audio, i.e., the clean audio target without human voice obtained after speech suppression processing, frame by frame or time step by time, and calculate the second deviation between the two. This deviation can also be quantified using various loss functions such as mean squared error loss, multi-scale spectral loss, or perceptual weighted loss. Its physical meaning is the degree of distortion between the generated audio and the real sound effect target.

[0082] The electronic device can calculate gradient information based on the aforementioned second deviation and perform backpropagation updates on the audio parameters in the V2X generation model according to a preset second learning rate. Here, audio parameters refer to the set of parameters such as network weights responsible for generating non-speech audio content like ambient background sounds, musical instruments, and object collision sounds in the V2X generation model. These parameters directly determine the quality and expressiveness of the V2X generation model in terms of sound effects and music generation. The second learning rate is set to be smaller than the preset first learning rate. That is, in the first fine-tuning stage, a larger first learning rate is used to drive rapid coarse-grained forgetting of speech-related parameters, allowing the V2X generation model to quickly break free from excessive dependence on speech patterns in the initial iterations. After entering the second fine-tuning stage, the core task of the V2X generation model shifts from suppressing speech to consolidating audio effects. At this point, a smaller learning rate is needed to perform refined gradient updates on the audio parameters to avoid unnecessary perturbations or bounces to the forgotten speech patterns when adjusting audio parameters due to an excessively large learning rate, while also preventing over-correction of the V2X generation model's existing audio generation capabilities. Through this coarse-to-fine phased parameter update strategy, the V2X generation model effectively suppresses the tendency to generate hallucinatory speech while preserving or even further improving the quality of its sound effects and music generation. Finally, after the second fine-tuning phase, the V2X generation model parameters are sufficiently weakened in the speech-related dimension and finely optimized in the sound effect-related dimension. The entire forgetting fine-tuning process is thus completed, and the resulting target V2X generation model can prioritize the activation of generation paths related to sound effects and music during the inference phase, significantly suppressing the activation probability of speech paths and achieving the expected technical effect of suppressing speech without compromising sound effects.

[0083] Example 5: In order to accurately and effectively fine-tune the V2X generation model, based on the above embodiments, in this application embodiment, whether the current stage is the first fine-tuning stage is determined in the following way: If this is the first time the V2X generation model has been fine-tuned, then it is determined that the current stage is the first fine-tuning stage; Otherwise, obtain the target output audio of the V2X generation model when it was last fine-tuned; determine the hallucination rate in the output audio using a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; determine whether the hallucination rate is lower than a preset threshold, if yes, determine that it is currently in the second fine-tuning stage, if no, determine that it is currently in the first fine-tuning stage.

[0084] In the specific process of model fine-tuning, this application adopts a two-stage forgetting strategy. By dynamically judging the fine-tuning stage, it achieves differentiated updates of speech parameters and sound effect parameters in the V2X generation model, thereby effectively suppressing the generation of hallucinatory speech while maximizing the maintenance of the model's original sound effect generation quality and audiovisual synchronization capability. Forgetting fine-tuning is performed directly on the pre-trained V2A model without modifying the network structure. Instead, the parameters are divided into two groups: speech parameters and sound effect parameters. A large learning rate is used to accelerate forgetting of speech parameters, while a small learning rate is used to lock in the capabilities of sound effect parameters. The training data only uses pure sound effect-music tracks after data-level speech suppression, combined with the original video frame sequence. First, speech priors are quickly erased, and then high-frequency details are finely restored. The entire process can use hallucinatory speech detection to calculate the hallucination rate in real time. Once the index is lower than the set threshold, it stops immediately to prevent excessive forgetting. After fine-tuning, the V2X generation model loses the impulse to generate speech but fully retains the original ability to synchronize sound effects, music, and video, and can be deployed directly.

[0085] To determine whether it is currently in the first fine-tuning phase, the electronic device can determine if this is the first time fine-tuning has been performed on the V2X generative model. If it is determined to be the first fine-tuning, it means that the parameters of the V2X generative model have not yet undergone any gradient updates for speech suppression, and the speech parameters are still in their original pre-training state, with a high response strength to speech patterns. In this case, it is directly determined that it is currently in the first fine-tuning phase, that is, driving the speech parameters to perform coarse-grained rapid forgetting with a large initial learning rate. If it is determined not to be the first fine-tuning, the electronic device can obtain the target output audio of the V2X generative model in the previous fine-tuning iteration. The electronic device can input this output audio into a pre-fine-tuned large language model, which will automatically identify and evaluate the hallucination speech of the output audio to determine the hallucination rate corresponding to the output audio. The hallucination rate is used to characterize the proportion of meaningless hallucinatory speech segments in the total duration of the output audio. The higher the value, the more non-existent speech content the V2X generation model has created during the generation process, i.e., the more severe the hallucination phenomenon. The lower the value, the purer the generated audio is, without meaningless speech-like components.

[0086] The electronic device compares the calculated hallucination rate with a preset threshold. If the hallucination rate is lower than the preset threshold, it indicates that the speech generation tendency of the current V2X generation model has been effectively suppressed, and the forgetting of speech parameters has achieved the expected effect. At this point, it is determined that the current stage should switch to the second fine-tuning stage, that is, to perform fine-grained detail recovery and capability consolidation of the sound effect parameters with a smaller second learning rate. If the hallucination rate is not lower than the preset threshold, it indicates that the current V2X generation model still has a strong speech generation impulse, and the forgetting of speech parameters is not yet sufficient. At this point, it is determined that the current stage is still in the first fine-tuning stage, and the speech parameters continue to be driven to perform further rapid forgetting with a larger first learning rate.

[0087] By using this method of determining the fine-tuning stage based on real-time feedback of the hallucination rate, this application realizes the determination of the fine-tuning process. The V2X generation model always automatically decides whether to continue coarse-grained forgetting or switch to fine-grained recovery under the guidance of the indicator of whether forgetting is sufficient. It can smoothly transition from the first stage to the second stage at the appropriate time without manual intervention.

[0088] During training, the hallucination speech detection pipeline runs continuously, calculating the hallucination rate after each iteration in real time. Once this metric consistently falls below a set threshold, the electronic device triggers an early stop mechanism, immediately terminating the fine-tuning process to prevent the V2X generation model from excessively forgetting non-target knowledge due to overtraining. This approach ensures that the entire forgetting fine-tuning process maintains speech suppression effectiveness while minimizing the risk of impaired sound effect generation capabilities due to excessive forgetting, ensuring that the fine-tuned model achieves an optimal balance between speech suppression and sound effect preservation. The final fine-tuned target V2X generation model loses its tendency to generate hallucination speech while fully retaining the original ability to synchronize sound effects, music, and visuals, making it directly deployable to real-world applications.

[0089] Example 6: To accurately and effectively determine the hallucination rate of the output audio, based on the above embodiments, in this embodiment, determining the hallucination rate of the output audio using a pre-fine-tuned large language model includes: The VAD model, which has been pre-trained, identifies speech segments in the output audio. Identify the target segment of speech pronunciation within the aforementioned speech segment; The target fragment is transcribed using a pre-trained transcription model. Based on a pre-tuned large language model, it is determined whether the transcribed text is a random combination of characters or syllables. If so, the target segment is determined to be hallucinatory speech. Based on the proportion of the hallucinatory speech in the duration of the output audio, the hallucination rate of the output audio is determined.

[0090] During model fine-tuning and evaluation, this application constructs a hallucination speech detection method to calculate the hallucination rate of the output audio of the V2X generation model. This method balances detection accuracy and computational efficiency, and can quickly and accurately identify whether there is any hallucination speech content that should not appear in the generated audio without human intervention.

[0091] In one possible implementation, the electronic device can first employ a pre-trained VAD model to identify all audio segments (speech-like segments) from the output audio that exhibit speech or speech-like acoustic properties. This VAD model, trained on a large amount of labeled data, is capable of accurately locating any temporal region resembling human voice amidst complex background noise and music interference. This approach can be understood as a signal search process that does not concern itself with whether the audio segments contain intelligible language content; it only starts from the underlying acoustic features of the output audio waveform, such as the spectral envelope, fundamental frequency trajectory, and harmonic structure, to extract all regions resembling human voice from the background noise.

[0092] After acquiring these speech fragments, the electronic device can further filter them to identify target fragments containing spoken language. This involves removing non-verbal sounds that, while possessing human vocal characteristics, do not constitute spoken language, such as laughter, crying, humming, and sighing. Since non-verbal sounds are removed during the data-level speech suppression stage by being categorized under generalized human speech, they still constitute unwanted hallucinations in the output audio during the inference stage. Therefore, they must be treated differently from genuine spoken language to accurately assess whether the model has generated semantic-level hallucinations.

[0093] After acquiring target segments containing spoken language, the electronic device can use a pre-trained transcription model (with multilingual recognition capabilities) to transcribe these segments one by one, converting the speech content in the audio signal into corresponding text sequences. This transcription model, specially trained and adapted, can not only recognize standard speech but also transcribe speech segments with unclear pronunciation, incomplete syllables, or cross-linguistic mixing. Even if the segment lacks complete semantic intelligibility, the transcription model can still output its corresponding phoneme or syllable sequences, providing necessary text input for subsequent semantic judgment. After obtaining the transcribed text, the electronic device can input it into a pre-fine-tuned large language model, which performs deep semantic understanding and judgment on the text content. The large language model analyzes whether the transcribed text is a random combination at the character or syllable level, whether it lacks a complete grammatical structure, whether it cannot constitute a clear semantic expression in any language, and whether it only contains fragmented cross-linguistic vocabulary. If the large language model determines that the transcribed text belongs to the above-mentioned random combination type, that is, confirms that the text does not constitute any intelligible language content, the electronic device can mark the target segment as hallucinatory speech. This means that the V2X generation model has created an audio content that is acoustically similar to speech but semantically empty and meaningless in a position where speech should not be generated.

[0094] After identifying and labeling all candidate segments in the output audio, the electronic device can calculate the cumulative duration of all segments labeled as hallucinatory speech and determine the proportion of this cumulative duration to the total duration of the output audio. This proportion is the hallucination rate of the output audio. The hallucination rate is a dimensionless value between 0 and 1. The closer the value is to 0, the purer the generated audio and the weaker the model's tendency to generate hallucinatory speech; the closer the value is to 1, the more meaningless the content of the generated audio is, indicating that the model has a serious problem of speech fabrication.

[0095] After thorough validation on diverse test sets, the detection accuracy reaches 0.885, demonstrating high-precision hallucinatory speech recognition capabilities. This effectively verifies the practical effects of data-level speech suppression and forgetting fine-tuning in reducing hallucinatory speech generation. The hallucination rate output by this detection can serve not only as an early stopping criterion during fine-tuning (terminating fine-tuning once the index consistently falls below a preset threshold to prevent excessive forgetting), but also as a standardized evaluation metric for V2A generation tasks, allowing for horizontal comparison of the performance of different model architectures, training strategies, or data processing schemes in hallucinatory speech suppression. Furthermore, the diverse test sets cover test samples from different language types, signal-to-noise ratios, and audio / video content categories, ensuring the detection method maintains robustness and generalization ability across scenarios and domains, providing reliable technical support for the evaluation of V2A and related cross-modal generation tasks. Through the output of this pipeline, researchers can quantitatively and intuitively measure the effectiveness and reliability of the proposed technical solution in suppressing hallucinatory speech, providing clear feedback signals for subsequent model optimization and iteration.

[0096] Example 7: To accurately and effectively determine the illusion rate of the output audio, based on the above embodiments, in this embodiment, the identification of the target segment of speech production in the speech segment includes: The speech segments are classified by an audio classifier, and the segments containing spoken language are retained to obtain the target segments; wherein, the segments containing non-spoken language include at least one of humming, laughter and crying.

[0097] In the non-verbal sound filtering process of the hallucination speech detection pipeline, the electronic device can first call a pre-trained audio classifier to perform acoustic category discrimination on speech segments. This audio classifier can include a non-verbal sound filter (NVs filter) and a laughter detector. This audio classifier can employ a large-scale pre-trained audio neural network, fully pre-trained on a large-scale audio and video dataset, possessing strong classification capabilities for a wide range of audio event types (covering human speech, non-verbal voices, environmental sound effects, musical instruments, etc.). Through this classifier, the electronic device can classify each speech segment into its corresponding acoustic category, selecting and retaining segments belonging to verbal sound production, and using these retained segments as target segments for subsequent ASR transcription steps. All segments judged as non-verbal sound production are marked and removed. Such non-verbal sound production segments include, but are not limited to, humming, laughter, crying, and other sounds produced by humans that, while possessing human vocal spectral characteristics, do not constitute language content. After completing the above classification and screening, the electronic device can also merge adjacent surviving segments. If two adjacent segments are close in time and may belong to the continuous components of the same discourse, they are spliced ​​into a complete segment to restore the semantic continuity of the discourse and avoid destroying the context required for subsequent semantic analysis due to over-segmentation.

[0098] The aforementioned nonverbal vocalization filter plays a crucial role in the entire hallucination speech detection pipeline. Previous VAD models identified all speech segments that sounded like human voices from audio based solely on acoustic features. While this method was highly sensitive, its specificity was limited, easily confusing nonverbal vocalizations with spoken vocalizations. The NVs Filter, however, discriminates between these speech segments as spoken and nonverbal vocalizations. Accurate identification of nonverbal vocalizations such as humming, laughter, and crying is particularly important because these sounds often overlap with speech in the acoustic spectrum, making accurate differentiation difficult with a VAD model alone. A dedicated laughter detector further enhances the detection capability of periodic laughter segments, effectively reducing false negatives due to insufficient classification confidence. In this way, the NVs Filter ensures that the segments fed into the subsequent ASR module are as pure as possible, primarily containing target segments with linguistic content. This provides high-quality input for accurate ASR transcription and semantic validity verification of the large language model, significantly improving the detection accuracy and robustness of the entire pipeline.

[0099] This application also constructed a diverse test dataset containing 14,000 audio clips to comprehensively evaluate the aforementioned hallucination speech detection pipeline. The test set covers three main categories of audio content: the first category is human speech or singing, containing 6,000 clips across 8 subcategories; the second category is non-speech human sound effects, containing 3,000 clips across 5 subcategories; and the third category is other sound effects and music, containing 5,000 clips across 217 subcategories. Samples for each category are evenly divided into real-world recorded audio and synthesized audio generated using the MMAudio model, with all samples sampled at a uniform rate of 44.1 kHz. The real-world speech data is partly derived from the FLEURS dataset, ensuring coverage of speech samples from multiple languages ​​to verify the pipeline's generalization ability in multilingual scenarios. Positive samples of hallucination speech are defined as 3,000 speech or singing clips synthesized using MMAudio. Experimental results show that the proposed hallucination speech detection pipeline achieves a balanced accuracy of 0.885, a Matthews correlation coefficient of 0.804, and an F1 score of 0.842 on the Raw TestSet, all of which significantly outperform the baseline multimodal large language model. This pipeline, serving as a feedback tool in the V2A model fine-tuning process, can accurately measure the suppression effect of hallucination speech and provide standardized quantitative metrics for subsequent evaluation of V2A generation tasks.

[0100] Figure 3 This is a schematic diagram illustrating a process for determining the illusion rate, as provided in an embodiment of this application.

[0101] like Figure 3 As shown, the VAD model can be used to identify language segments in the output audio first, then the target segments of language pronunciation can be retained through the audio classification area, and then the transcribed text can be obtained through the ASR model. Finally, the large language model can be used to detect whether the target segment belongs to a random combination of character-level or syllable-level (i.e., whether it is a hallucination segment).

[0102] Figure 4 This is a schematic diagram of a spectrogram visualization of supervised audio and real values ​​provided in an embodiment of this application.

[0103] Figure 4 The embodiments of this application demonstrate that the supervised audio generated by the semantic masking strategy guided by automatic speech recognition (i.e., using a speech recognition model), the activity masking strategy guided by speech activity detection (i.e., using a speech activity detection model), and the separation-driven global masking (i.e., using a source separation model) is closer to the real value (i.e., ideal audio) and does not contain phantom speech, while MMAudio (i.e., Figure 4 The prior art shown 1) and VGG-CT (i.e. Figure 4 The existing technology shown in 2) generates audio that contains phantom speech.

[0104] To accurately and effectively generate the target audio, based on the above embodiments, the method in this application embodiment further includes: Receive the sequence of video frames to be set to music; The sequence of video frames to be set to music is input into the target V2X generation model to obtain the target audio output by the target V2X generation model.

[0105] After completing the forgetting fine-tuning of the pre-trained V2X generative model, the target V2X generative model has the ability to generate high-quality sound effects and music without speech interference, and can be deployed to real-world video-to-audio generation application scenarios.

[0106] In the inference deployment phase of practical applications, the electronic device first receives the video frame sequence to be used for background music. This video frame sequence can be a set of image frames extracted in chronological order after decoding a complete video file uploaded by the user, or it can be a continuous frame sequence extracted from a real-time video stream at a fixed sampling rate. As the sole input signal for the target V2X generative model's inference, the video frame sequence carries all the visual semantic information required to drive audio generation, including but not limited to scene category, object motion trajectory, character actions, ambient lighting changes, and temporal boundaries of visual events related to sound events. After acquiring the video frame sequence, the electronic device can preprocess it according to the input format requirements of the target V2X generative model. This preprocessing includes adjusting frame size, standardizing pixel values, and sampling at fixed time intervals to ensure that the temporal resolution of the frame sequence is consistent with the frame rate configuration used during model training, thereby avoiding degradation in generation quality due to input format mismatch. After preprocessing, the electronic device can input the preprocessed video frame sequence into the target V2X generative model. The target model is the optimal model weight obtained after data-level speech suppression and two-stage forgetting fine-tuning. The parameters of this target V2X generation model have been sufficiently weakened in the speech dimension and finely optimized in the sound effect dimension. It can encode visual features into acoustic features that are temporally aligned with the input frame sequence during forward propagation, and finally output the target audio waveform that precisely matches the duration of the video frame sequence through the decoder. The output target audio is the background music generated by the system for the video to be set to music. Its content only includes environmental sound effects, background music and other non-speech sound events that are consistent with the semantics of the visual scene, and does not contain any hallucinatory speech components. It can be directly used for downstream tasks such as video post-production, real-time playback or streaming media distribution. Through this input and output process, end-to-end automatic generation from video content to pure audio background music is realized, which has broad application prospects in film and television production, short video editing, game sound effect generation, virtual reality content creation and other fields.

[0107] Example 8: Figure 5 A schematic diagram of a device for suppressing hallucinatory speech provided in this application embodiment is shown. The device includes: The acquisition module 501 is used to acquire any sample pair from a pre-saved sample set, wherein the sample pair includes a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence. Processing module 502 is used to perform forgetting fine-tuning on a pre-trained V2X generation model using the video frame sequence and the supervised audio to obtain a target V2X generation model, which is used to suppress phantom speech in video generation.

[0108] In one possible implementation, the processing module 502 is specifically configured to perform at least one of the following processing steps on the human voice component of the original audio corresponding to the video frame sequence to obtain the supervised audio: using a source separation model to separate the original audio corresponding to the video frame sequence into human voice channel audio and non-human voice channel audio, and using the non-human voice channel audio as the supervised audio; using ASR to identify semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence, replacing the semantic speech segments with a preset non-human voice residual signal, and using the replaced audio as the supervised audio; using a VAD model to detect speech activity segments in the original audio corresponding to the video frame sequence, wherein the speech activity segments include sub-speech segments without semantic meaning, replacing the speech activity segments with a preset non-human voice residual signal, and using the replaced audio as the supervised audio.

[0109] In one possible implementation, the processing module 502 is specifically used to determine whether the current stage is the first fine-tuning stage. If so, the video frame sequence is input into the pre-trained V2X generation model to obtain the output audio of the V2X generation model; the first deviation between the output audio and the supervised audio is determined; and the speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

[0110] In one possible implementation, the processing module 502 is specifically used to determine whether the current stage is the first fine-tuning stage. If not, the video frame sequence is input into the pre-trained V2X generation model to obtain the output audio output by the V2X generation model; a second deviation between the output audio and the supervised audio is determined; and the sound effect parameters of the V2X generation model are fine-tuned based on the second deviation and a preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

[0111] In one possible implementation, the processing module 502 is specifically configured to determine whether it is currently in the first fine-tuning stage by: if this is the first time the V2X generation model has been fine-tuned, then it is determined that it is currently in the first fine-tuning stage; otherwise, it obtains the target output audio of the V2X generation model when it was last fine-tuned; it determines the hallucination rate in the output audio through a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; it determines whether the hallucination rate is lower than a preset threshold, and if so, it is determined that it is currently in the second fine-tuning stage, and if not, it is determined that it is currently in the first fine-tuning stage.

[0112] In one possible implementation, the processing module 502 is specifically used to identify speech segments in the output audio using a pre-trained VAD model; identify target segments of speech pronunciation within the speech segments; transcribe the target segments into text using a pre-trained transcription model; determine whether the transcribed text belongs to a random combination of characters or syllables based on a pre-fine-tuned large language model; if so, determine that the target segment is hallucination speech; and determine the hallucination rate of the output audio based on the proportion of the hallucination speech in the duration of the output audio.

[0113] In one possible implementation, the processing module 502 is specifically used to classify the speech segments using an audio classifier, retain the segments of spoken language, and obtain the target segment; wherein, the segments of non-spoken language include at least one of humming, laughter, and crying.

[0114] In one possible implementation, the processing module 502 is further configured to receive a sequence of video frames to be accompanied by music; input the sequence of video frames to be accompanied by music into the target V2X generation model, and obtain the target audio output by the target V2X generation model.

[0115] The method provided in this application does not perform post-processing at the output of the V2X generative model. Instead, it removes or replaces the human voice component in the original audio during the training data construction stage, ensuring that the V2X generative model is not exposed to any speech-related supervision signals during training. This pre-emptive intervention ensures that the model cannot establish a visual semantic-to-speech pattern mapping at the mapping level, resulting in greater thoroughness and robustness.

[0116] Furthermore, this application provides three data-level speech suppression strategies that can be selected independently or combined in cascades: Sep-GM achieves global human voice removal through source separation, suitable for demanding scenarios with zero tolerance for speech; ASR-SM removes only understandable speech content through semantic-level precise localization, suitable for applications that need to retain non-semantic human voice components; and VAD-AM achieves wide-coverage speech activity masking with low computational overhead, suitable for resource-constrained or large-scale data processing scenarios. This diversity of strategies gives this application broad adaptability to different business needs.

[0117] In the data-level speech suppression process, the non-human voice residual signal is fully preserved and used as a supervision target, ensuring that the model can still obtain sufficient gradient feedback from background sounds, environmental sound effects, and music during training. Experimental results show that the audio generated by the model after processing with the Sep-GM strategy outperforms the unprocessed baseline model in distribution matching metrics (FD, KL), proving that this implementation method effectively suppresses hallucinatory speech while maintaining or even improving the quality of non-target audio content generation.

[0118] This application utilizes the catastrophic forgetting mechanism in neural networks to naturally weaken the response strength of the V2X generative model to speech features during parameter updates by changing the data distribution during the fine-tuning phase, rather than forcibly suppressing the speech generation path through explicit methods such as parameter pruning or adding adversarial constraints. This implicit and gradual forgetting method is less destructive to the overall model architecture and avoids introducing new model biases or performance instability due to explicit intervention.

[0119] Example 9: Based on the above embodiments, this application also provides an electronic device, such as... Figure 6 As shown, it includes: processor 601, communication interface 602, memory 603 and communication bus 604, wherein processor 601, communication interface 602 and memory 603 communicate with each other through communication bus 604.

[0120] The memory 603 stores a computer program, which, when executed by the processor 601, causes the processor 601 to perform the following steps: Obtain any sample pair from a pre-saved sample set, wherein the sample pair contains a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence; Using the video frame sequence and the supervised audio, the pre-trained V2X generation model is fine-tuned by forgetting to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

[0121] In one possible implementation, the human voice component of the original audio corresponding to the video frame sequence is processed by at least one of the following methods to obtain the supervised audio: The original audio corresponding to the video frame sequence is separated into human voice channel audio and non-human voice channel audio using a source separation model, and the non-human voice channel audio is used as the supervision audio. The semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence are identified using the speech recognition model ASR. The semantic speech segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio. The speech activity detection (VAD) model is used to detect speech activity segments in the original audio corresponding to the video frame sequence. The speech activity segments include sub-speech segments that do not have semantic meaning. The speech activity segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio.

[0122] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If so, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the first deviation between the output audio and the supervisory audio; The speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

[0123] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If not, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio of the V2X generation model. Determine the second deviation between the output audio and the supervisory audio; The sound effect parameters of the V2X generation model are fine-tuned based on the second bias and the preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

[0124] In one possible implementation, whether the current stage is the first fine-tuning phase is determined as follows: If this is the first time the V2X generation model has been fine-tuned, then it is determined that the current stage is the first fine-tuning stage; Otherwise, obtain the target output audio of the V2X generation model when it was last fine-tuned; determine the hallucination rate in the output audio using a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; determine whether the hallucination rate is lower than a preset threshold, if yes, determine that it is currently in the second fine-tuning stage, if no, determine that it is currently in the first fine-tuning stage.

[0125] In one possible implementation, determining the hallucination rate in the output audio using a pre-tuned large language model includes: The VAD model, which has been pre-trained, identifies speech segments in the output audio. Identify the target segment of speech pronunciation within the aforementioned speech segment; The target fragment is transcribed using a pre-trained transcription model. Based on a pre-tuned large language model, it is determined whether the transcribed text is a random combination of characters or syllables. If so, the target segment is determined to be hallucinatory speech. Based on the proportion of the hallucinatory speech in the duration of the output audio, the hallucination rate of the output audio is determined.

[0126] In one possible implementation, identifying the target segment of speech pronunciation within the speech segment includes: The speech segments are classified by an audio classifier, and the segments containing spoken language are retained to obtain the target segments; wherein, the segments containing non-spoken language include at least one of humming, laughter and crying.

[0127] In one possible implementation, the method further includes: Receive the sequence of video frames to be set to music; The sequence of video frames to be set to music is input into the target V2X generation model to obtain the target audio output by the target V2X generation model.

[0128] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0129] The communication interface 602 is used for communication between the above-mentioned electronic device and other devices.

[0130] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0131] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0132] Example 10: Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps: Obtain any sample pair from a pre-saved sample set, wherein the sample pair contains a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence; Using the video frame sequence and the supervised audio, the pre-trained V2X generation model is fine-tuned by forgetting to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

[0133] In one possible implementation, the human voice component of the original audio corresponding to the video frame sequence is processed by at least one of the following methods to obtain the supervised audio: The original audio corresponding to the video frame sequence is separated into human voice channel audio and non-human voice channel audio using a source separation model, and the non-human voice channel audio is used as the supervision audio. The semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence are identified using the speech recognition model ASR. The semantic speech segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio. The speech activity detection (VAD) model is used to detect speech activity segments in the original audio corresponding to the video frame sequence. The speech activity segments include sub-speech segments that do not have semantic meaning. The speech activity segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio.

[0134] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If so, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the first deviation between the output audio and the supervisory audio; The speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

[0135] In one possible implementation, the forgetting fine-tuning of the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If not, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio of the V2X generation model. Determine the second deviation between the output audio and the supervisory audio; The sound effect parameters of the V2X generation model are fine-tuned based on the second bias and the preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

[0136] In one possible implementation, whether the current stage is the first fine-tuning phase is determined as follows: If this is the first time the V2X generation model has been fine-tuned, then it is determined that the current stage is the first fine-tuning stage; Otherwise, obtain the target output audio of the V2X generation model when it was last fine-tuned; determine the hallucination rate in the output audio using a pre-fine-tuned large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucination speech segments in the output audio; determine whether the hallucination rate is lower than a preset threshold, if yes, determine that it is currently in the second fine-tuning stage, if no, determine that it is currently in the first fine-tuning stage.

[0137] In one possible implementation, determining the hallucination rate in the output audio using a pre-tuned large language model includes: The VAD model, which has been pre-trained, identifies speech segments in the output audio. Identify the target segment of speech pronunciation within the aforementioned speech segment; The target fragment is transcribed using a pre-trained transcription model. Based on a pre-tuned large language model, it is determined whether the transcribed text is a random combination of characters or syllables. If so, the target segment is determined to be hallucinatory speech. Based on the proportion of the hallucinatory speech in the duration of the output audio, the hallucination rate of the output audio is determined.

[0138] In one possible implementation, identifying the target segment of speech pronunciation within the speech segment includes: The speech segments are classified by an audio classifier, and the segments containing spoken language are retained to obtain the target segments; wherein, the segments containing non-spoken language include at least one of humming, laughter and crying.

[0139] In one possible implementation, the method further includes: Receive the sequence of video frames to be set to music; The sequence of video frames to be set to music is input into the target V2X generation model to obtain the target audio output by the target V2X generation model.

[0140] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0141] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0142] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0143] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0144] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for suppressing hallucinatory speech, characterized in that, The method includes: Obtain any sample pair from a pre-saved sample set, wherein the sample pair contains a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence; Using the video frame sequence and the supervised audio, a pre-trained end-to-end video-to-audio V2X generation model is fine-tuned by forgetting to obtain a target V2X generation model, which is used to suppress phantom speech in video generation.

2. The method according to claim 1, characterized in that, The supervised audio is obtained by performing at least one of the following processes on the human voice component of the original audio corresponding to the video frame sequence: The original audio corresponding to the video frame sequence is separated into human voice channel audio and non-human voice channel audio using a source separation model, and the non-human voice channel audio is used as the supervision audio. The semantic speech segments with semantic meaning in the original audio corresponding to the video frame sequence are identified using the speech recognition model ASR. The semantic speech segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio. The speech activity detection (VAD) model is used to detect speech activity segments in the original audio corresponding to the video frame sequence. The speech activity segments include sub-speech segments that do not have semantic meaning. The speech activity segments are replaced with preset non-human voice residual signals, and the replaced audio is used as the supervision audio.

3. The method according to claim 1, characterized in that, The step of performing forgetting fine-tuning on the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If so, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the first deviation between the output audio and the supervisory audio; The speech parameters of the V2X generation model are fine-tuned based on the first deviation and the preset first learning rate.

4. The method according to claim 1, characterized in that, The step of performing forgetting fine-tuning on the pre-trained V2X generative model using the video frame sequence and the supervised audio includes: Determine whether the current stage is the first fine-tuning stage. If not, input the video frame sequence into the pre-trained V2X generation model and obtain the output audio output by the V2X generation model. Determine the second deviation between the output audio and the supervisory audio; The sound effect parameters of the V2X generation model are fine-tuned based on the second bias and the preset second learning rate; wherein the second learning rate is less than the preset first learning rate.

5. The method according to claim 3 or 4, characterized in that, Whether we are currently in the first fine-tuning phase can be determined in the following way: If this is the first time the V2X generation model has been fine-tuned, then it is determined that the current stage is the first fine-tuning stage; Otherwise, obtain the target output audio of the V2X generation model when it was last fine-tuned; The hallucination rate in the output audio is determined by pre-fine-tuning a large language model; wherein the hallucination rate is used to characterize the proportion of meaningless hallucinatory speech segments in the output audio; it is determined whether the hallucination rate is lower than a preset threshold. If it is, it is determined that the current stage is the second fine-tuning stage; if not, it is determined that the current stage is the first fine-tuning stage.

6. The method according to claim 5, characterized in that, The determination of the hallucination rate in the output audio using a pre-fine-tuned large language model includes: The VAD model, which has been pre-trained, identifies speech segments in the output audio. Identify the target segment of speech pronunciation within the aforementioned speech segment; The target fragment is transcribed using a pre-trained transcription model. Based on a pre-tuned large language model, it is determined whether the transcribed text is a random combination of characters or syllables. If so, the target segment is determined to be hallucinatory speech. Based on the proportion of the hallucinatory speech in the duration of the output audio, the hallucination rate of the output audio is determined.

7. The method according to claim 6, characterized in that, The target segment for identifying the speech pronunciation in the speech segment includes: The speech segments are classified by an audio classifier, and the segments containing spoken language are retained to obtain the target segments; wherein, the segments containing non-spoken language include at least one of humming, laughter and crying.

8. The method according to claim 1, characterized in that, The method further includes: Receive the sequence of video frames to be set to music; The sequence of video frames to be set to music is input into the target V2X generation model to obtain the target audio output by the target V2X generation model.

9. A device for suppressing hallucinatory speech, characterized in that, The device includes: The acquisition module is used to acquire any sample pair from a pre-saved sample set, wherein the sample pair includes a video frame sequence and a corresponding supervisory audio; wherein the supervisory audio is supervisory audio that does not contain human voice generated after processing the human voice component of the original audio corresponding to the video frame sequence. The processing module is used to perform forgetting fine-tuning on the pre-trained V2X generation model using the video frame sequence and the supervised audio to obtain a target V2X generation model, which is used to suppress hallucinatory speech in video generation.

10. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, the processor being configured to execute a computer program stored in the memory to implement the steps of the method for suppressing phantom speech as described in any one of claims 1-8.

Images