Environment perception speech synthesis method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of artificial intelligence, can be applied to the technical field of medical health and finance, and discloses an environment perception speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a reference voice, a target environment voice, a target text and an environment perception value; inputting the reference voice, the target environment voice, the target text and the environment perception value into a synthesis model, converting the text into a vector sequence through a text coding layer by the synthesis model, and applying a mask strategy to the reference voice, processing the environment perception value and embedding a time step to regulate and control the intensity of the target environment voice, voice generation with environment perception is carried out, and environment perception voice is formed; and outputting the environment perception voice. By implementing the method provided by the embodiment of the invention, the naturalness and reality sense of environmental perception speech synthesis can be remarkably improved, and particularly, clearer and more natural speech communication experience is provided in remote medical consultation and financial customer service systems.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, healthcare, and finance, and more specifically to environmentally perceptive speech synthesis methods, apparatus, computer equipment, and storage media. Background Technology

[0002] Environment-Aware Text-to-Speech (EA-TTS) technology aims to generate natural and realistic speech output that adapts to various environmental conditions to meet the needs of different application scenarios. However, the current development of EA-TTS technology faces three core challenges. First, from the Lombard effect to seamless integration in telemedicine and financial customer service, traditional methods such as VoiceLDM and WavCraft typically employ a strategy of "generating speech first and then adding background noise." This approach ignores the dynamic interaction between speech and background noise. For example, in noisy environments, human speakers unconsciously raise their voice pressure and change their tone to ensure they are heard; this phenomenon is known as the Lombard effect. Because traditional methods fail to fully consider this collaborative change, the synthesized speech is distorted and unnatural. In telemedicine consultations, this distortion can directly affect doctors' understanding of patients' conditions and the accuracy of their diagnoses. Similarly, in financial customer service systems, customers may have difficulty understanding customer service personnel's instructions or information due to speech distortion, thus affecting service efficiency and customer satisfaction.

[0003] Secondly, existing systems are weak in handling background noise, failing to effectively simulate the "voice-environment fusion" of the real world. This means that the consistency and realism of voice output are insufficient across different environmental scenarios, thus affecting the authenticity and immersion of the user experience. In the healthcare field, which requires high personalization and accuracy, this may lead to patients not clearly understanding doctors' advice, affecting treatment outcomes. In the fintech industry, poor voice quality may reduce customer trust and service experience, thereby impacting business performance.

[0004] Finally, training efficient environment-aware models requires a large amount of paired data, including speech, corresponding ambient sounds, and text annotations. However, obtaining this paired data in practical applications is very difficult, especially in the healthcare and fintech industries, where strict privacy and data security requirements further increase the difficulty of data collection and use.

[0005] Therefore, it is necessary to design a new method to significantly improve the naturalness and realism of environmentally conscious speech synthesis, especially to provide a clearer and more natural voice communication experience in remote medical consultation and financial customer service systems. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an environmentally aware speech synthesis method, apparatus, computer equipment, and storage medium.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: an environment-aware speech synthesis method, comprising:

[0008] Acquire reference speech, target ambient sound, target text, and environmental perception values;

[0009] The reference speech, the target ambient sound, the target text, and the environment-aware value are input into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the environment-aware value and time step embedding to adjust the intensity of the target ambient sound in order to generate environment-aware speech and form environment-aware speech.

[0010] Output the environmentally conscious speech.

[0011] The further technical solution is as follows: the synthetic model is a model formed by combining a diffusion model architecture with a conditional flow matching method.

[0012] The further technical solution is as follows: the synthesis model includes an input layer, a text encoding layer, an encoding layer and a multilayer perceptron, and an output layer. The text encoding layer includes a text encoding layer based on the ConvNeXt V2 structure. The encoding layer and the multilayer perceptron are used to process environmental perception values ​​and work together with time step embedding on the network to regulate the intensity of the target environmental sound.

[0013] The further technical solution is as follows: the training process of the synthetic model includes:

[0014] The training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes training triplets containing time information and environmental perception values, is obtained by using a diffusion model architecture combined with a conditional flow matching method.

[0015] The further technical solution is as follows: the training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes a model containing temporal information and environmental perception values, is obtained by using a diffusion model architecture combined with a conditional flow matching method. This includes:

[0016] Get the mixed audio;

[0017] The mixed audio was transcribed using Whisper-large-v2, and speech and ambient sounds were extracted using VAD or source separation techniques to synthesize training triples containing time steps and SER values.

[0018] Construct a model that combines a diffusion model architecture with a conditional flow matching method;

[0019] The model is trained using the training triples to obtain the synthetic model.

[0020] The further technical solution is as follows: the reference speech, the target ambient sound, the target text, and the environment-aware value are input into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the environment-aware value and time step embedding to adjust the intensity of the target ambient sound, thereby generating environment-aware speech and forming environment-aware speech.

[0021] The reference speech, the target ambient sound, the target text, and the environmental perception value are input into the synthesis model;

[0022] The character sequence of the target text is converted into a text conditional vector sequence, and a masking strategy is applied to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments in order to obtain the first processing result.

[0023] The environmental perception values ​​are processed and combined with temporal step embedding to act on the network to regulate the intensity of the target ambient sound;

[0024] The modulated target ambient sound and the first processing result are synthesized to obtain environment-aware speech.

[0025] A further technical solution is as follows: processing the environmental perception value and combining it with time step embedding to act on the network to regulate the intensity of the target environmental sound includes:

[0026] The environmental perception values ​​are processed using sine encoding combined with MLP to form a vector;

[0027] The vector is superimposed with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

[0028] The present invention also provides an environment-aware speech synthesis device, comprising:

[0029] The acquisition unit is used to acquire reference speech, target ambient sound, target text, and environmental perception values;

[0030] The synthesis unit is used to input the reference speech, the target ambient sound, the target text, and the environment-aware value into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the environment-aware value and time step embedding to adjust the intensity of the target ambient sound in order to generate environment-aware speech and form environment-aware speech.

[0031] The output unit is used to output the environmentally aware speech.

[0032] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the above-described method.

[0033] The present invention also provides a storage medium, characterized in that the storage medium stores a computer program, which, when executed by a processor, implements the above-described method.

[0034] The beneficial effects of this invention compared to existing technologies are as follows: By acquiring reference speech, target ambient sound, target text, and environmental awareness values, and inputting this information into a synthesis model constructed based on a diffusion model architecture combined with a conditional flow matching method, the system can generate environment-aware speech, thus producing highly adaptive, environmentally-aware speech. This method significantly improves the naturalness and realism of speech synthesis because it not only considers the influence of background noise but also specifically adjusts the speech output to match the current environmental conditions. In application scenarios such as telemedicine consultation and financial customer service systems, even in the face of complex background noise or different acoustic environments, it can ensure a clearer and more natural voice communication experience, greatly enhancing user satisfaction and interaction efficiency. This technological advancement is of great significance for improving service quality and enhancing user experience.

[0035] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0036] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0037] Figure 1 This is a schematic diagram illustrating an application scenario of the environment-aware speech synthesis method provided in an embodiment of the present invention.

[0038] Figure 2 This is a flowchart illustrating the environmentally perceptive speech synthesis method provided in an embodiment of the present invention.

[0039] Figure 3 A schematic diagram of a sub-process of the environment-aware speech synthesis method provided in an embodiment of the present invention;

[0040] Figure 4 A schematic diagram of a sub-process of the environment-aware speech synthesis method provided in an embodiment of the present invention;

[0041] Figure 5A schematic block diagram of an environment-aware speech synthesis device provided in an embodiment of the present invention;

[0042] Figure 6 A schematic block diagram of the synthesis unit of the environmentally aware speech synthesis device provided in an embodiment of the present invention;

[0043] Figure 7 A schematic block diagram of the processing subunit of the environmentally aware speech synthesis device provided in an embodiment of the present invention;

[0044] Figure 8 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0045] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0046] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0047] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0048] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0049] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the environmentally perceptive speech synthesis method provided in an embodiment of the present invention. Figure 2This is a schematic flowchart illustrating the environment-aware speech synthesis method provided in this embodiment of the invention. The method is applied in a server. The server interacts with the terminal, automatically extracting speech and ambient sounds from mixed audio by combining a diffusion model architecture and conditional flow matching technology, utilizing Whisper-large-v2 and VAD or source separation techniques. It then trains the synthesis model by constructing training triples using temporal information and emotion recognition values ​​(SER). This method can construct large-scale datasets without manual annotation, thereby generating environment-aware speech with high naturalness and realism. Especially in application scenarios such as remote medical consultation and financial customer service systems, it can provide a clearer and more natural voice communication experience, greatly improving the user's interactive experience and service quality. By using a text encoding layer based on the ConvNeXt V2 structure and applying a masking strategy to the Mel spectrogram, the accuracy and fluency of speech synthesis are further enhanced, making the final output speech closer to the communication effect in real-world scenarios.

[0050] Figure 2 This is a flowchart illustrating the environmentally perceptive speech synthesis method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S130.

[0051] S110, acquire reference speech, target ambient sound, target text, and ambient perception values.

[0052] In this embodiment, step S110 is fundamental to realizing the environment-aware speech synthesis method. It involves acquiring four key input elements: reference speech, target ambient sound, target text, and environment-aware values. These elements work together to provide the necessary conditions for subsequent model training and speech generation.

[0053] Reference speech refers to samples of speaker characteristics or voice style used to guide the speech synthesis process. This can be obtained by recording audio clips of a specific speaker or by selecting suitable samples from an existing speech database. Importantly, the selected reference speech should have high intelligibility and contain as much contextual information as possible similar to the intended output. This helps ensure that the final generated speech not only accurately conveys the content of the target text but also preserves the original speaker's vocal characteristics.

[0054] Target ambient sound refers to the background noise or environmental sound effects that are desired to be reflected in the final voice output. For example, in a telemedicine consultation scenario, it might be necessary to simulate the faint sounds of conversation or keyboard typing in an office; while in a financial customer service system, it might be the background noise during a telephone call. By collecting sounds from the actual environment or using audio editing software to synthesize specific environmental sound effects, target ambient sounds suitable for different application scenarios can be created.

[0055] The target text is the text content that needs to be converted into speech. It can be any type of text information, such as dialogues, announcements, instructions, etc. To improve the quality of speech synthesis, the target text is usually preprocessed, including but not limited to word segmentation, stop word removal, and spelling error correction. In addition, considering the characteristics of different languages ​​and regions, optimization for multilingual support is also an essential part.

[0056] SER (Speech-to-Environment Ratio) is a parameter used to adjust the intensity of background noise. It determines the relative loudness of ambient noise relative to the speech itself in the final synthesized speech. By adjusting the SER value, the balance between speech clarity and naturalness can be flexibly controlled in different application scenarios. For example, in a noisy environment, appropriately lowering the SER value can make the speech stand out more, making it easier for the listener to understand; conversely, in a quieter environment, adding some background noise can make the speech sound more realistic and natural.

[0057] Successful execution of step S110 is crucial to the entire environment-aware speech synthesis process. It not only provides a high-quality data foundation for subsequent model training but also ensures the possibility of customized speech generation based on specific needs. By accurately acquiring and configuring the above four elements, the naturalness and realism of the synthesized speech can be significantly improved, thereby better serving various application scenarios such as remote medical consultation and financial customer service systems.

[0058] S120. The reference speech, the target ambient sound, the target text, and the ambient awareness value are input into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the ambient awareness value and time step embedding to adjust the intensity of the target ambient sound, thereby generating ambient-aware speech and forming ambient-aware speech.

[0059] The synthesis model converts text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes environmental awareness values ​​and time step embeddings to adjust the intensity of the target environmental sound in order to generate speech with environmental awareness.

[0060] In this embodiment, environment-aware speech refers to the generation of speech through specific algorithms and architecture design, which not only contains clear target text content, but also integrates natural and scene-appropriate background sound effects, thereby enhancing the realism and immersion of the auditory experience.

[0061] The synthesis model described is formed by combining a diffusion model architecture with a conditional flow matching method. This model is particularly suitable for processing complex audio signals, starting from Gaussian noise and progressively refining it until high-quality speech output is generated. Its core lies in using a conditional control mechanism to achieve precise adjustment of the generation process.

[0062] The synthesis model includes an input layer, a text encoding layer, an encoder and a multilayer perceptron, and an output layer. The text encoding layer includes a text encoding layer based on the ConvNeXt V2 structure. The encoder and the multilayer perceptron are used to process environmental perception values ​​and work together with temporal step embeddings on the network to regulate the intensity of the target environmental sound.

[0063] Specifically, the input layer is responsible for receiving four main types of input: reference speech, target ambient sound, target text, and environmental perception values. These inputs provide the necessary information foundation for subsequent layers.

[0064] Text encoding layer: Built on the ConvNeXt V2 architecture, it is used to convert target text into a machine-understandable form. During this process, characters and padding tokens are fused to form a text conditional vector sequence, ensuring the effective transmission of text information.

[0065] Encoding and Multilayer Perceptron: This part primarily processes environmental perception values ​​and combines them with temporal step embeddings to work together at different layers of the network to modulate the intensity of the target ambient sound. This mechanism ensures that the salience of the background sound can be flexibly adjusted even in different application scenarios.

[0066] Output layer: Finally, a speech spectrogram with environmental awareness features is generated, which is then recovered into actual waveform data by a vocoder.

[0067] The training process of the synthetic model includes:

[0068] The training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes training triplets containing time information and environmental awareness, is trained using a diffusion model architecture combined with a conditional flow matching method.

[0069] In one embodiment, the training of the training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes training triplets containing temporal information and environmental awareness values, adopts a diffusion model architecture combined with a conditional flow matching method, and includes:

[0070] Get the mixed audio;

[0071] The mixed audio was transcribed using Whisper-large-v2, and speech and ambient sounds were extracted using VAD or source separation techniques to synthesize training triples containing time steps and SER values.

[0072] Construct a model that combines a diffusion model architecture with a conditional flow matching method;

[0073] The model is trained using the training triples to obtain the synthetic model.

[0074] Specifically, the synthetic model is obtained through an innovative training strategy, which involves extracting reference speech and ambient sound from mixed audio and synthesizing training triples containing temporal information and environmental awareness values ​​to train the model. The specific steps are as follows:

[0075] Speech (xs) and ambient sound (xe) were extracted from unlabeled audio using VAD (Voice Activity Detection) or source separation techniques, and the transcribed text (y) was automatically generated using Whisper-large-v2.

[0076] Training triple construction: Combine the information extracted above into training triples of the form (x, xs, xe, y), where x represents the combined representation of time step embedding plus SER value.

[0077] Specifically, in diffusion models, the generation process is divided into multiple steps, each corresponding to a different time point. To enable the model to understand the current time point, the temporal information is typically encoded into a vector, known as a time step embedding. This helps the model understand the progress of the transition from noise to clear speech.

[0078] Environmental Awareness Value (SER): This is a user-defined parameter used to control the intensity of background noise in the generated speech. By adjusting the SER value, fine-tuning of the salience of background noise can be achieved, thereby meeting the needs of different application scenarios.

[0079] x is actually a combination of these two elements, namely x = TimeEmbed(t) + SEREmbed(ser). This means that at any given time point t, combined with a specific environment-aware value ser, the model can adjust the intensity of background noise in the generated speech according to these conditions.

[0080] Train the triple (x, xspeech, xenv, y);

[0081] x: A conditional vector consisting of time step embeddings and environment-aware values.

[0082] xspeech: The extracted reference speech portion.

[0083] xenv: Target ambient sound extracted from audio.

[0084] y: The corresponding transcribed text information.

[0085] The purpose of this training triplet is to enable the model to learn how to extract and regenerate speech with appropriate background noise from mixed audio based on a given time step and context-aware settings.

[0086] The model is trained using a diffusion model architecture combined with a conditional flow matching method, enabling it to learn how to generate a high-quality speech + background sound combination from noise based on given conditions.

[0087] Specifically, diffusion models are generative models that generate data from pure noise by gradually adding noise and then learning how to reverse this process. This approach performs well in fields such as image generation and audio processing. When combined with conditional flow matching methods, diffusion models can generate customized content based on specific conditions.

[0088] The core idea of ​​the diffusion model is to gradually transform the data into noise and then learn how to reverse this process. Specific steps include:

[0089] Forward process: gradually add noise to the data until the data becomes completely random noise.

[0090] The reverse process involves learning a model that can gradually recover the original data from noise.

[0091] Conditional flow matching methods allow the introduction of additional control variables or conditions during the generation process. This means that the model can not only generate data, but also adjust the generated data according to these conditions. For example, in speech synthesis tasks, conditions can include time step embeddings and environment-aware values ​​(SER) to control the intensity of background noise.

[0092] In the task of generating speech plus background noise, the specific application of this method is as follows:

[0093] A large number of speech samples were collected, and each sample was accompanied by a corresponding text transcription and the desired background sound information (such as background sound intensity).

[0094] For each pair of speech and background sounds, features are extracted and used as part of the training data.

[0095] For each training sample, an input x is constructed, which includes a time-step embedding and an environment-aware value (SER). This represents the conditions for the desired generated speech and background noise.

[0096] The goal of the model is to learn to start from a point in the forward process, gradually remove noise based on given conditions (x), and finally generate clear speech and the desired background noise.

[0097] Use an appropriate loss function to measure the difference between the model-generated results and the real samples, and guide the model to optimize its parameters.

[0098] During the training phase, the model receives a series of noisy speech data and their corresponding clean speech and background noise as input, and attempts to learn how to recover the target output from the noise based on given conditions.

[0099] Conditional flow matching technology is used to ensure that the generated speech is of high quality and meets the preset background sound conditions.

[0100] This method not only produces natural and fluent speech, but also allows for the flexible addition of different types of background noise as needed, greatly improving the adaptability and practicality of the speech synthesis system. Whether in film and television dubbing, game sound design, or virtual assistant scenarios, it provides a richer and more realistic auditory experience.

[0101] An unsupervised data construction method was used to automatically separate speech, ambient sound, and text information from unlabeled data. This not only solves the problem of traditional speech synthesis systems relying on a large amount of manual annotation, but also significantly reduces training costs and time. By using VAD or source separation techniques, clean speech (xspeak) and ambient sound (xenv) can be extracted from the original audio, and combined with the Whisper-large-v2 model to generate the corresponding transcribed text. This method can quickly and efficiently construct large-scale training datasets, providing a solid foundation for subsequent model training.

[0102] In one embodiment, please refer to Figure 3 The above-mentioned step S120 may include steps S121 to S124.

[0103] S121. Input the reference speech, the target ambient sound, the target text, and the environmental perception value into the synthesis model.

[0104] In this embodiment, a reference speech (as a template for speech style or quality), a target ambient sound (background sound to be added to the generated speech), a target text (text content to be converted into speech), and an environment-aware value (a value or a set of values ​​used to adjust the intensity or other characteristics of the background sound) are input into a synthesis model.

[0105] S122. Convert the character sequence of the target text into a text conditional vector sequence, and apply a masking strategy to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments to obtain the first processing result.

[0106] In this embodiment, the first processing result refers to the data representation generated after converting the target text into a character sequence and applying a masking strategy to the Mel spectrogram of the reference speech.

[0107] The character sequence of the target text is converted into a series of vectors through a certain encoding method (such as one-hot encoding, word2vec, etc.). These vectors are called text conditional vector sequences, which represent the information of the text.

[0108] The acoustic features of the reference speech are typically represented as a Mel spectrogram. Applying a masking strategy means selecting certain portions of the Mel spectrogram as known information, while the rest are considered as parts to be predicted. The purpose of this is to simulate situations in real-world scenarios where only parts of the speech are clearly discernible, thereby training the model to learn how to fill in the missing information. The first result is the data obtained after this processing.

[0109] S123. Process the environmental perception value and work together with the time step embedding on the network to regulate the intensity of the target environmental sound.

[0110] In one embodiment, please refer to Figure 4 The above step S123 may include steps S1231 to S1232.

[0111] S1231. The environmental perception values ​​are processed using sine encoding combined with MLP to form a vector.

[0112] In this embodiment, a vector refers to an information unit that represents the adjustment of the target ambient sound intensity after the ambient sound value is processed by sin encoding and multilayer perceptron (MLP).

[0113] Encoding environmental perception values ​​using a sine function is a location encoding technique commonly used in transformer models to capture information at different locations.

[0114] S1232. The vector is superimposed with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

[0115] In this embodiment, the composite vector refers to the composite information carrier generated by combining the aforementioned vector with time step embedding, which is used to participate in network condition control to regulate the intensity of target ambient sound.

[0116] The sine-encoded environmental perception values ​​are input into a multilayer perceptron for further processing, generating a vector. This vector contains information about how to adjust the target ambient sound intensity.

[0117] The vector generated in the above steps is combined with the time step embedding (representing the current processing time point or stage) to form a so-called "synthetic vector". This synthetic vector is then used to participate in the conditional control of the network, specifically to adjust the intensity of the target ambient sound to ensure that the generated speech can adapt to different background environments.

[0118] S124. The adjusted target ambient sound and the first processing result are synthesized to obtain ambient-aware speech.

[0119] The final step is to combine the regulated target ambient sound with the first processing result obtained in step S122, and generate the final environment-aware speech through a synthesis model. This step realizes the conversion from text to speech, and adjusts the quality of the output speech and background sound effects according to given conditions (such as environment-aware values) to achieve a more natural effect that meets the needs of specific scenarios.

[0120] This detailed process design aims to improve the quality of speech synthesis, ensuring that the generated speech not only accurately reflects the content of the original text but also appropriately integrates the required background sound effects, thereby enhancing the user experience.

[0121] This embodiment employs a joint modeling strategy to synthesize both speech and background noise simultaneously, rather than the traditional "speech synthesis + background mixing" approach. This ensures a natural blend between speech and background noise, significantly enhancing the realism and immersion of the synthesized speech. Furthermore, a multimodal conditional input mechanism is introduced, utilizing the ConvNeXt V2 structure to encode the text sequence and fusing characters with padding tokens to form a text conditional vector sequence. For the speech portion, a mask mechanism is applied to mask the Mel spectrogram, simulating the "known part" and the "part to be predicted," thereby effectively training the model to reconstruct the occluded portion.

[0122] To enhance the system's control capabilities, a SER control factor is introduced during the inference phase. Users can precisely adjust the intensity of background noise by setting different SER values, enabling speech synthesis in environments ranging from completely quiet to noisy. This mechanism uses sine encoding and MLP to form vectors, which are then superimposed with time-step embeddings to participate in network conditional control, providing users with a flexible and powerful tool to customize the environmentally aware characteristics of the synthesized speech.

[0123] The synthetic model is designed based on a conditional flow matching architecture, achieving conditional control, high-fidelity generation, and efficient inference under a unified model. Compared to traditional diffusion models, CFM (Conditional Flow Matching) eliminates the need for complex sampling steps or multi-step diffusion processes, thereby significantly improving inference speed while ensuring the quality and detail of the generated content.

[0124] As an environmental perception TTS system, the above improvements bring several significant advantages:

[0125] By using joint modeling, the synchronous generation of speech and background sound is achieved directly at the model level, avoiding potential inconsistencies during the later mixing process and making the final output more realistic.

[0126] The introduced SER mechanism allows users to adjust the intensity of background noise according to specific needs, increasing the flexibility and customizability of application scenarios.

[0127] The unsupervised data construction solution overcomes the dependence of traditional speech synthesis systems on high-quality labeled data, lowers the deployment threshold, and is suitable for different languages ​​and application scenarios.

[0128] The CFM-based architecture optimizes the inference process, significantly improving processing speed while maintaining high fidelity compared to traditional diffusion models.

[0129] These improvements give the system enormous potential for creating immersive speech synthesis content, making it particularly suitable for applications requiring highly customized and realistic voices, such as game voice-over, virtual assistants, intelligent customer service, and educational broadcasting.

[0130] S130, Output the environmentally aware voice.

[0131] The environmentally aware voice is output to the terminal for playback.

[0132] For example, a bank needs to automatically make outbound calls to a large number of customers to remind them of credit card repayments, investment maturity dates, and anti-fraud alerts. To make customers feel like a "real customer service representative is calling from a branch," the following steps are required:

[0133] Preserve the customer service representative's original voice and friendly tone;

[0134] The background includes a slight service hall atmosphere (distant calls and keyboard sounds) but does not excessively mask the speech.

[0135] The background intensity can be dynamically adjusted according to the customer's calling environment (quiet / noisy);

[0136] First, collect the four key elements:

[0137] Reference audio: 10 high-quality 30-second recordings selected from top customer service representatives, sampled at 48kHz, mono, with no background noise.

[0138] Target ambient sound: Record a 5-minute lobby environment during peak business hours and use SourceSeparation to extract pure ambient sound (keyboard, footsteps, distant conversations).

[0139] Target text: Outbound call script, such as "Dear Mr. Zhang, your credit card bill for this period is 5280 yuan, and the final payment date is August 25th. If you need to pay in installments, please press 1."

[0140] SER: Default 1.2 (slight background noise), if the client side has high noise, adjust to 0.7 in real time; if the client is in a quiet office, adjust to 1.6.

[0141] When the system receives an outbound call task, it automatically retrieves the above four elements.

[0142] The target text is encoded into a text conditional vector using ConvNeXt-V2.

[0143] The reference speech mel masking strategy is to retain 60% of the known segments and train the model to complete the rest.

[0144] SER=1.2 is used to form an adjustment vector through sin-MLP, which is then superimposed with the time step embedding to control the ambient sound intensity in the business hall.

[0145] The model generates 48kHz audio in one go, featuring both customer service voice and a business hall background.

[0146] Outbound calls are made to the customer's mobile phone via a SIP gateway; if the customer's environment is noisy, the noise energy is identified in real time, the SER is reduced to 0.7 within 50ms, and the second half of the sentence is resynthesized for seamless connection.

[0147] When a customer answers, it feels like a familiar, live customer service representative is making a call near a branch counter; the background noise is just right, ensuring key numbers are easily audible. If the customer is on the subway, the system immediately lowers the background volume to guarantee that the "repayment date, August 25th" is clearly identifiable.

[0148] Here's another example: A top-tier hospital's post-operative follow-up center needs to send personalized follow-up questions (pain scores, medication reminders, and appointment scheduling) to thousands of discharged patients daily. Requirements:

[0149] Use the voice of the attending physician that patients trust most during their hospitalization to improve compliance;

[0150] Adding subtle background noises of a hospital corridor evokes a sense of reassurance that one is still being cared for in the hospital.

[0151] Elderly patients are hard of hearing, so it is necessary to emphasize speech; younger patients can retain more environmental realism.

[0152] First, collect the four key elements:

[0153] Reference audio: 15 one-minute educational audio clips recorded by the attending physician in a quiet clinic.

[0154] Target ambient sound: Record footsteps in the corridor and the sound of nursing carts being pushed during morning rounds, use VAD to cut out sudden screams, and retain stable background noise.

[0155] Dynamically generate target text, such as "Ms. Wang, today is the 3rd day after your appendectomy. What is your current abdominal pain score on a scale of 0-10? Please reply with the number."

[0156] SER: Automatically set based on patient age and hearing profile. SER is 0.6 for patients ≥65 years old (mild background), and SER is 1.3 for patients <65 years old.

[0157] The HIS system automatically pushes the patient ID, doctor ID, and follow-up question script to the speech synthesis platform.

[0158] The text is encoded using ConvNeXt-V2.

[0159] Reference speech mel masking strategy: Preserve the doctor's signature pauses and intonation features.

[0160] SER0.6 or 1.3 is adjusted by sin-MLP and then embedded with time steps.

[0161] The model outputs 16kHz follow-up audio with a slight background noise from the hospital corridor.

[0162] Through the hospital's official WeChat account / telephone outbound call, patients hear the voice of "their attending physician," and the soft background sound of the nursing cart makes them feel that they are still being cared for in the hospital.

[0163] The final experience is as follows:

[0164] Elderly patients: speech is prominent, background noise is almost inaudible, and the "pain rating" can be clearly heard.

[0165] Younger patients: The background is slightly more visible, enhancing the realism, while key figures remain clear.

[0166] The aforementioned environment-aware speech synthesis method acquires reference speech, target ambient sound, target text, and environment-aware values, and inputs this information into a synthesis model constructed based on a diffusion model architecture combined with a conditional flow matching method. The system can then generate environment-aware speech, producing highly adaptive, environment-aware speech. This method significantly improves the naturalness and realism of speech synthesis because it not only considers the influence of background noise but also specifically adjusts the speech output to match the current environmental conditions. In application scenarios such as telemedicine consultations and financial customer service systems, even in the face of complex background noise or different acoustic environments, it ensures a clearer and more natural voice communication experience, greatly enhancing user satisfaction and interaction efficiency. This technological advancement is of great significance for improving service quality and enhancing user experience.

[0167] Figure 5 This is a schematic block diagram of an environmentally aware speech synthesis device 300 provided in an embodiment of the present invention. Figure 5 As shown, corresponding to the above-described environment-aware speech synthesis method, the present invention also provides an environment-aware speech synthesis apparatus 300. This environment-aware speech synthesis apparatus 300 includes a unit for performing the above-described environment-aware speech synthesis method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 5 The environmentally aware speech synthesis device 300 includes an acquisition unit 301, a synthesis unit 302, and an output unit 303.

[0168] The acquisition unit 301 is used to acquire reference speech, target ambient sound, target text, and environment-aware values; the synthesis unit 302 is used to input the reference speech, target ambient sound, target text, and environment-aware values ​​into the synthesis model, whereby the synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech, processes the environment-aware values ​​and time step embedding to adjust the intensity of the target ambient sound, so as to generate environment-aware speech and form environment-aware speech; the output unit 303 is used to output the environment-aware speech.

[0169] The device further includes a model training unit, the model training unit comprising:

[0170] The audio acquisition subunit is used to acquire mixed audio; the triple generation subunit is used to perform speech transcription on the mixed audio using Whisper-large-v2, extract speech and ambient sound using VAD or source separation technology, and synthesize training triples containing time steps and SER values; the construction subunit is used to construct a model formed by combining a diffusion model architecture with a conditional flow matching method; and the training subunit is used to train the model using the training triples to obtain the synthesized model.

[0171] In one embodiment, such as Figure 6 As shown, the synthesis unit 302 includes an input subunit 3021, an analog subunit 3022, a processing subunit 3023, and a speech synthesis subunit 3024.

[0172] Input subunit 3021 is used to input the reference speech, the target ambient sound, the target text, and the environment-aware value into the synthesis model; simulation subunit 3022 is used to convert the character sequence of the target text into a text conditional vector sequence, and apply a masking strategy to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments to obtain a first processing result; processing subunit 3023 is used to process the environment-aware value and work together with the time-step embedding on the network to regulate the intensity of the target ambient sound; speech synthesis subunit 3024 is used to synthesize the regulated target ambient sound and the first processing result to obtain environment-aware speech.

[0173] In one embodiment, such as Figure 7 As shown, the processing subunit 3023 includes: an encoding module 30231 and an overlay module 30232.

[0174] The encoding module 30231 is used to process the environmental perception value using sine encoding combined with MLP to form a vector; the superposition module 30232 is used to superimpose the vector with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

[0175] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned environmental perception speech synthesis device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0176] The aforementioned environmentally aware speech synthesis device 300 can be implemented as a computer program, which can, for example... Figure 8 It runs on the computer device shown.

[0177] Please see Figure 8 , Figure 8 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0178] See Figure 8 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0179] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform an environment-aware speech synthesis method.

[0180] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0181] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute an environment-aware speech synthesis method.

[0182] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0183] The processor 502 is used to run a computer program 5032 stored in the memory to perform the following steps:

[0184] Acquire reference speech, target ambient sound, target text, and environment-aware values; input the reference speech, target ambient sound, target text, and environment-aware values ​​into a synthesis model, where the synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech, processes the environment-aware values ​​and time step embedding to adjust the intensity of the target ambient sound, thereby generating environment-aware speech; output the environment-aware speech.

[0185] The synthetic model is formed by combining a diffusion model architecture with a conditional flow matching method.

[0186] The synthesis model includes an input layer, a text encoding layer, an encoder and a multilayer perceptron, and an output layer. The text encoding layer includes a text encoding layer based on the ConvNeXt V2 structure. The encoder and the multilayer perceptron are used to process environmental perception values ​​and work together with temporal step embeddings on the network to regulate the intensity of the target environmental sound.

[0187] The training process of the synthetic model includes:

[0188] The training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes training triplets containing time information and environmental awareness, is trained using a diffusion model architecture combined with a conditional flow matching method.

[0189] In one embodiment, when the processor 502 implements the step of extracting reference speech and ambient sound from mixed audio and synthesizing training triples containing temporal information and environmental awareness values, and training a model formed by combining a diffusion model architecture with a conditional flow matching method, the specific implementation steps are as follows:

[0190] Acquire mixed audio; use Whisper-large-v2 to perform speech transcription on the mixed audio, extract speech and ambient sound by combining VAD or source separation technology, and synthesize training triples containing time steps and SER values; construct a model formed by combining a diffusion model architecture with a conditional flow matching method; train the model using the training triples to obtain the synthesized model.

[0191] In one embodiment, when the processor 502 implements the steps of inputting the reference speech, the target ambient sound, the target text, and the ambient awareness value into the synthesis model, converting the text into a vector sequence through a text encoding layer and applying a masking strategy to the reference speech, and processing the ambient awareness value and time step embedding to adjust the intensity of the target ambient sound, in order to generate ambient-aware speech, the specific implementation is as follows:

[0192] The reference speech, the target ambient sound, the target text, and the environment-aware value are input into the synthesis model; the character sequence of the target text is converted into a text conditional vector sequence, and a masking strategy is applied to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments to obtain a first processing result; the environment-aware value is processed and combined with temporal step embedding to act on the network to regulate the intensity of the target ambient sound; the regulated target ambient sound and the first processing result are synthesized to obtain environment-aware speech.

[0193] In one embodiment, when the processor 502 implements the step of processing the environmental perception value and working in conjunction with the temporal step embedding on the network to regulate the intensity of the target ambient sound, it specifically implements the following steps:

[0194] The environmental perception values ​​are processed using sine coding combined with MLP to form a vector; the vector is then superimposed with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

[0195] It should be understood that in the embodiments of this application, the processor 502 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor.

[0196] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0197] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the following steps:

[0198] Acquire reference speech, target ambient sound, target text, and environment-aware values; input the reference speech, target ambient sound, target text, and environment-aware values ​​into a synthesis model, where the synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech, processes the environment-aware values ​​and time step embedding to adjust the intensity of the target ambient sound, thereby generating environment-aware speech; output the environment-aware speech.

[0199] The synthetic model is formed by combining a diffusion model architecture with a conditional flow matching method.

[0200] The synthesis model includes an input layer, a text encoding layer, an encoder and a multilayer perceptron, and an output layer. The text encoding layer includes a text encoding layer based on the ConvNeXt V2 structure. The encoder and the multilayer perceptron are used to process environmental perception values ​​and work together with temporal step embeddings on the network to regulate the intensity of the target environmental sound.

[0201] The synthesis model is formed by extracting reference speech and ambient sound from mixed audio and synthesizing training triples containing time information and environmental perception values. The training adopts a diffusion model architecture combined with a conditional flow matching method.

[0202] In one embodiment, when the processor executes the computer program to implement the synthesis model, which involves extracting reference speech and ambient sound from mixed audio and synthesizing a training triplet containing temporal information and environmental awareness values, and training a model formed using a diffusion model architecture combined with a conditional flow matching method, the specific steps are as follows:

[0203] Acquire mixed audio; use Whisper-large-v2 to perform speech transcription on the mixed audio, extract speech and ambient sound by combining VAD or source separation technology, and synthesize training triples containing time steps and SER values; construct a model formed by combining a diffusion model architecture with a conditional flow matching method; train the model using the training triples to obtain the synthesized model.

[0204] In one embodiment, when the processor executes the computer program to implement the step of inputting the reference speech, the target ambient sound, the target text, and the ambient awareness value into the synthesis model, whereby the synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech, and processes the ambient awareness value and time step embedding to adjust the intensity of the target ambient sound, thereby generating ambient-aware speech, the specific implementation of the following steps is as follows:

[0205] The reference speech, the target ambient sound, the target text, and the environment-aware value are input into the synthesis model; the character sequence of the target text is converted into a text conditional vector sequence, and a masking strategy is applied to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments to obtain a first processing result; the environment-aware value is processed and combined with temporal step embedding to act on the network to regulate the intensity of the target ambient sound; the regulated target ambient sound and the first processing result are synthesized to obtain environment-aware speech.

[0206] In one embodiment, when the processor executes the computer program to process the environmental awareness values ​​and work in conjunction with time-step embedding on the network to regulate the intensity of the target ambient sound, it specifically implements the following steps:

[0207] The environmental perception values ​​are processed using sine coding combined with MLP to form a vector; the vector is then superimposed with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

[0208] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0209] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0210] In the several embodiments provided by this invention, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For example, the division of each unit is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0211] The steps in the method of this invention can be adjusted, merged, or reduced in order according to actual needs. The units in the device of this invention can be merged, divided, or reduced according to actual needs. Furthermore, the functional units in the various embodiments of this invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0212] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a terminal, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0213] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An environment-aware speech synthesis method, characterized in that, include: Acquire reference speech, target ambient sound, target text, and environmental perception values; The reference speech, the target ambient sound, the target text, and the environment-aware value are input into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the environment-aware value and time step embedding to adjust the intensity of the target ambient sound in order to generate environment-aware speech and form environment-aware speech. Output the environmentally conscious speech.

2. The environmentally perceptive speech synthesis method according to claim 1, characterized in that, The synthetic model is formed by combining a diffusion model architecture with a conditional flow matching method.

3. The environmentally perceptive speech synthesis method according to claim 2, characterized in that, The synthesis model includes an input layer, a text encoding layer, an encoder and a multilayer perceptron, and an output layer. The text encoding layer includes a text encoding layer based on the ConvNeXt V2 structure. The encoder and the multilayer perceptron are used to process environmental perception values ​​and work together with temporal step embeddings on the network to regulate the intensity of the target environmental sound.

4. The environmentally perceptive speech synthesis method according to claim 3, characterized in that, The training process of the synthetic model includes: The training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes training triplets containing time information and environmental perception values, is obtained by using a diffusion model architecture combined with a conditional flow matching method.

5. The environmentally perceptive speech synthesis method according to claim 4, characterized in that, The training triplet, which extracts reference speech and ambient sound from mixed audio and synthesizes a model containing temporal information and environmental awareness values, is obtained by using a diffusion model architecture combined with a conditional flow matching method. This includes: Get the mixed audio; The mixed audio was transcribed using Whisper-large-v2, and speech and ambient sounds were extracted using VAD or source separation techniques to synthesize training triples containing time steps and SER values. Construct a model that combines a diffusion model architecture with a conditional flow matching method; The model is trained using the training triples to obtain the synthetic model.

6. The environmentally perceptive speech synthesis method according to claim 4, characterized in that, The process involves inputting the reference speech, the target ambient sound, the target text, and the ambient awareness value into a synthesis model. The synthesis model then converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the ambient awareness value and time step embedding to adjust the intensity of the target ambient sound, thereby generating ambient-aware speech and forming ambient-aware speech. This includes: The reference speech, the target ambient sound, the target text, and the environmental perception value are input into the synthesis model; The character sequence of the target text is converted into a text conditional vector sequence, and a masking strategy is applied to the Mel spectrogram corresponding to the reference speech to simulate known and predicted speech segments in order to obtain the first processing result. The environmental perception values ​​are processed and combined with temporal step embedding to act on the network to regulate the intensity of the target ambient sound; The modulated target ambient sound and the first processing result are synthesized to obtain environment-aware speech.

7. The environmentally perceptive speech synthesis method according to claim 6, characterized in that, The process of processing the environmental perception values ​​and integrating them with temporal step embeddings into the network to modulate the intensity of the target ambient sound includes: The environmental perception values ​​are processed using sine encoding combined with MLP to form a vector; The vector is superimposed with the time step to form a comprehensive vector, which participates in network condition control to regulate the intensity of the target ambient sound.

8. An environmentally perceptive speech synthesis device, characterized in that, include: The acquisition unit is used to acquire reference speech, target ambient sound, target text, and environmental perception values; The synthesis unit is used to input the reference speech, the target ambient sound, the target text, and the environment-aware value into the synthesis model. The synthesis model converts the text into a vector sequence through a text encoding layer and applies a masking strategy to the reference speech. It processes the environment-aware value and time step embedding to adjust the intensity of the target ambient sound in order to generate environment-aware speech and form environment-aware speech. The output unit is used to output the environmentally aware speech.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images