Speech synthesis method and device, computer equipment and storage medium

Abstract

The invention discloses a speech synthesis method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring multi-mode background sound condition input data; performing modal integrity detection on the multi-modal background sound condition input data to obtain a detection result; generating an environment background sound feature embedding vector according to a detection result; obtaining to-be-synthesized text data and speaker reference audio data, and performing feature extraction to obtain text semantic features and speaker timbre features; inputting the environment background sound feature embedded vector, the text semantic feature and the speaker timbre feature into an acoustic model to generate a Mel spectrum; and converting the Mel spectrum into a target voice waveform to obtain synthetic voice data. By implementing the method, scene requirements can be deeply matched, diversified scene types can be covered, accurate matching of background sounds and voice semantics is realized, and the technical scheme can be applied to the fields of finance and medical health.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and particularly relates to a speech synthesis method and device, a computer device and a storage medium. BACKGROUND

[0002] In the key fields of finance, medical health and the like, speech interaction as a core carrier of information transmission and service response puts forward very high requirements on personalization, scene adaptability and naturalness of speech synthesis. Zero-shot text-to-speech (Zero-shot TTS) technology, which can reproduce personalized characteristics such as speaker timbre and intonation by only using a reference speech, has become one of the core technologies of speech synthesis in such fields, and is widely used in scenes such as financial intelligent customer service voice broadcast, medical health field medical order reading and patient care voice generation.

[0003] The current mainstream Zero-shot TTS system design takes “guaranteeing speech clarity and intelligibility” as the core target, and usually only generates “clean speech” without environmental sound interference. However, in the actual application scenes of finance, medical health and the like, a single clean speech is often difficult to meet the needs of scene-based interaction - for example, the outdoor mobile bank voice broadcast in the financial field needs to adapt to the street environment background sound, and the home remote diagnosis voice in the medical health field needs to integrate soft indoor environment sound to relieve the tension of patients. In order to realize scene-based adaptation, the existing technology generally adopts the “post-mixing” method, that is, the synthesized speech track and the background audio track are first independently generated, and then the two are linearly superimposed through an audio editing tool. However, this scheme has significant defects: on the one hand, it completely ignores the Lombard effect (i.e., the speaker will adaptively adjust the speech volume, intonation and clarity according to the environmental noise intensity) in human speech communication, resulting in an obvious “split feeling” of the combination of the synthesized speech and the background sound, lacking the natural adaptation relationship between human speech and the environment in a real scene; on the other hand, the linear superposition mixing method is prone to the “speech masking” problem - when the background sound intensity is slightly high, the key information (such as numbers in financial data and medication doses in medical orders) in the synthesized speech will be covered by the background sound, and if the background sound intensity is reduced, the scene-based adaptation will also be lost. At the same time, this method cannot dynamically adjust the background sound style according to the semantic content of the synthesized speech, and it is difficult to generate synthesized results with high semantic compatibility and rich scene diversity.

[0004] In order to solve the above-mentioned drawbacks of post-mixing, the industry has begun to explore directly introducing environmental factors into the end-to-end TTS system, forming three typical research directions, but all have their own technical limitations, making it difficult to meet the application needs of the key fields of finance, medical health and the like: First, the control scheme based on environmental sound reference audio. This scheme requires inputting a few seconds of environmental recording as a prompt signal, and controls the environmental style of synthesized speech by extracting acoustic features (such as noise spectrum, reverberation parameters) of the reference audio. This method has obvious limitations in the practical application of the financial and medical health fields: on the one hand, it relies too much on specific reference audio, and if there is no ready-made environmental sound reference sample in the target scene (such as the sudden outdoor temporary financial service point in the financial field, the special ward environment in the medical health field), it cannot generate the corresponding scenario voice; on the other hand, the background sound in the synthesized audio is essentially a "copy" or "migration" of the features of the reference audio, and cannot dynamically adjust the background sound details according to the semantics of the synthesized speech. For example, when reading the medical order "patients need to rest quietly" in the medical health field, it is still possible to generate noisy background sound that conflicts with the semantics.

[0005] Second, the conditional generation scheme based on text scene description. This scheme directly generates voice with corresponding background noise by adding scene description information (such as "bank hall broadcast" "hospital ward soft voice") to the text to be synthesized, using a conditional generation network. However, in the financial and medical health fields, scene descriptions often have "complexity" and "fuzziness" problems: complex descriptions such as "noisy securities trading hall during early morning session" in the financial field and "ICU ward with low-intensity device operation environment" in the medical health field make it difficult for the model to accurately analyze the scene features, while ambiguous descriptions such as "mild financial service voice" and "comfortable medical care background" can cause instability in the generation results - when the same text is paired with similar descriptions, the model generates background sound with significant differences in intensity and style, which cannot meet the requirements of consistency and reliability of voice synthesis results in key fields.

[0006] Thirdly, the environment prediction scheme based on image visual features. This scheme uses a pre-trained image encoder to extract visual features of a scene image, selects or generates a corresponding environmental sound after predicting the environmental category through a classification model. For example, generate a hall environmental sound through a bank outlet image in the financial field, and generate a medical environmental background sound through a ward image in the medical and health field. However, the core defect of this technology is that the use of visual information has "granularity limitations": existing methods can only capture coarse-grained global spatial visual information of the image (such as the overall scene category of "bank hall" and "ward"), but ignore the extraction of key local information of the scene - for example, the "customer queuing area" (which needs to adapt to the background sound of slight conversation) and "self-service terminal area" (which needs to adapt to the background sound of device operation) in the bank counter image in the financial field, and the "bedside monitor" (which needs to include low-intensity device beep sound) and "window rest area" (which needs to include soft wind sound) in the ward image in the medical and health field. These local information directly determines the detailed characteristics of the environmental sound, and the existing technology cannot effectively capture them, resulting in poor adaptation of the generated background sound to the local details of the actual scene, making it difficult to meet the fine scene voice synthesis needs in key fields.

[0007] In summary, the current Zero-shot TTS technology in key fields such as finance and medical health has problems such as insufficient naturalness, poor robustness, and low scene adaptation refinement in scene-based environmental sound adaptation, which seriously restricts the deep application of voice synthesis technology in such fields, and there is an urgent need for a voice synthesis technology scheme that can balance personalized voice replication, scene-based environmental adaptation, and high naturalness. SUMMARY

[0008] The purpose of the present application is to overcome the defects of the prior art and provide a voice synthesis method, device, computer equipment and storage medium.

[0009] To achieve the above-mentioned purpose, the following technical solutions are adopted in the present application: The voice synthesis method comprises: Obtaining multi-modal background sound condition input data, the multi-modal background sound condition input data at least containing one of text description data, environmental audio data and environmental image data; Performing modal integrity detection on the multi-modal background sound condition input data to obtain a detection result; Generating an environmental background sound feature embedding vector according to the detection result; Obtaining to-be-synthesized text data and speaker reference audio data, and performing feature extraction on the to-be-synthesized text data and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features; The environmental background sound feature embedding vector, the text semantic feature and the speaker timbre feature are input into an acoustic model to generate a mel spectrum; The mel spectrum is converted into a target voice waveform to obtain synthesized voice data.

[0010] The application further provides a voice synthesis device, comprising: An acquisition unit is configured to acquire multi-modal background sound condition input data, wherein the multi-modal background sound condition input data comprises at least one of text description data, environmental audio data and environmental image data; A detection unit is configured to perform modal integrity detection on the multi-modal background sound condition input data to obtain a detection result; A generation unit is configured to generate an environmental background sound feature embedding vector according to the detection result; An acquisition and extraction unit is configured to acquire text data to be synthesized and speaker reference audio data, and perform feature extraction on the text data to be synthesized and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features; An input generation unit is configured to input the environmental background sound feature embedding vector, the text semantic features and the speaker timbre features into an acoustic model to generate a mel spectrum; A conversion unit is configured to convert the mel spectrum into a target voice waveform to obtain synthesized voice data.

[0011] The application further provides a computer device, which comprises a memory and a processor, the memory stores a computer program, and the processor implements the above method when executing the computer program.

[0012] The application further provides a storage medium, which stores a computer program, and the computer program implements the above method when executed by a processor.

[0013] The beneficial effects of this invention compared to existing technologies are as follows: By incorporating text description data, environmental audio data, and environmental image data into a multimodal background sound conditional input system, the complementary and synergistic effects of multi-dimensional information in "semantics-acoustics-vision" can be achieved. Under the synergistic effect of the three, the generated environmental background sound feature embedding vector can deeply match the scenario requirements of key fields such as finance and healthcare. It can not only cover diverse scenario types, but also achieve precise matching between background sound and speech semantics, solving the problems of low scenario adaptation accuracy and semantic disconnect in existing technologies. In addition, by synergistically inputting the environmental background sound feature embedding vector with text semantic features and speaker timbre features into the acoustic model, "end-to-end scenario-based speech generation" is achieved. On the one hand, the model can dynamically adjust the volume and tone of the speech according to the environmental background sound features during the speech generation process, so that the synthesized speech and background sound are naturally matched, avoiding "disjointedness". On the other hand, the model can automatically reduce the background sound intensity of the corresponding time period based on text semantic features, while maintaining the scenario-based background sound effect of non-critical information segments, fundamentally avoiding the risk of speech masking.

[0014] The present invention will be further described below with reference to the accompanying drawings and specific embodiments. Attached Figure Description

[0015] Figure 1 This is a schematic diagram illustrating an application scenario of the speech synthesis method provided in an embodiment of the present invention; Figure 2 A schematic flowchart of the speech synthesis method provided in an embodiment of the present invention; Figure 3 This is a schematic block diagram of a speech synthesis device provided in an embodiment of the present invention; Figure 4 A schematic block diagram of a computer device provided for an embodiment of the present invention. Detailed Implementation

[0016] Please see Figure 1 and Figure 2 , Figure 1 This is a schematic diagram illustrating an application scenario of the speech synthesis method provided in an embodiment of the present invention. Figure 2 This is a schematic flowchart illustrating the speech synthesis method provided in an embodiment of the present invention. The speech synthesis method is applied to a server that interacts with a terminal, enabling deep matching of scenario requirements in key fields such as finance and healthcare. It not only covers diverse scenario types but also achieves precise matching between background noise and speech semantics. Furthermore, by embedding environmental background noise features into vectors and collaboratively inputting them with text semantic features and speaker timbre features into the acoustic model, end-to-end scenario-based speech generation is achieved. This allows the synthesized speech to naturally adapt to the background noise while maintaining the scenario-based background noise effect for non-critical information segments, fundamentally avoiding the risk of speech masking.

[0017] Figure 2 This is a schematic flowchart of the speech synthesis method provided in an embodiment of the present invention. Figure 2 As shown, the method includes the following steps S110 to S160.

[0018] S110. Obtain multimodal background sound conditional input data, wherein the multimodal background sound conditional input data includes at least one of text description data, environmental audio data, and environmental image data; Specifically, the acquisition of multimodal background sound conditional input data needs to cover three core modalities: text description, ambient audio, and ambient image. The specific implementation method is as follows: Text description data acquisition: Users can directly input scene description text through text input boxes. The text format is unrestricted and can include scene type (e.g., "morning rush hour in bank lobby," "daytime in general hospital ward"), background sound style (e.g., "soft equipment operation sound," "low-intensity conversation among people"), and key semantic association information (e.g., "background sound is reduced when broadcasting financial data," "no sharp noise when reading medical orders"). The system preprocesses the input text, including removing special characters and standardizing the text encoding format, and then passes the processed text data to the subsequent feature extraction module.

[0019] Ambient audio data acquisition: Two acquisition methods are provided. The first is real-time recording of ambient audio via the device's built-in microphone, with a recording duration of 3-10 seconds (meeting the minimum data volume requirement for feature extraction). The sampling rate is fixed at 44.1kHz to ensure acoustic feature accuracy. The second method supports users uploading locally stored ambient audio files (formats include common audio formats such as WAV and MP3). The system preprocesses the acquired or uploaded audio data, including removing silent segments, normalizing audio amplitude, and eliminating volume differences caused by recording from different devices, before sending the preprocessed audio data to the feature extraction stage.

[0020] Environmental image data acquisition: Supports real-time capture of scene images via device camera or uploading local scene image files (formats include JPG, PNG, etc.). Image resolution must be no less than 256×256 pixels to ensure complete local detail information. The system preprocesses the image data, including size normalization (uniformly scaling to 256×256 pixels) and RGB channel normalization (eliminating the impact of lighting differences on feature extraction), and then passes the processed image data to the corresponding encoder.

[0021] S120. Perform modal integrity detection on the multimodal background sound conditional input data to obtain the detection result; Specifically, modal integrity detection is achieved through two steps: "input port identification + data validity verification". Input port identification: The system pre-sets independent input port identifiers for three modalities: text description, ambient audio, and ambient image. When data is input, the system checks whether valid data (non-empty and formatted to meet preprocessing requirements) exists at each port to preliminarily determine whether the modality exists. For example, if the text input port detects non-empty and correctly formatted text data, it determines that "the text description modality exists"; if the audio input port does not detect audio data that meets the sampling rate requirements, it preliminarily determines that "the ambient audio modality is missing".

[0022] Data validity verification: Further verification of the validity of modal data initially determined to "exist" is performed. For text description modalities, the text length needs to be verified (it must contain at least 3 valid characters to avoid meaningless input); for environmental audio modalities, the valid audio duration needs to be verified (the valid duration after preprocessing must be no less than 2 seconds to ensure sufficient acoustic features can be extracted); for environmental image modalities, the image clarity needs to be verified (edge ​​detection algorithms are used to determine if the image is excessively blurry; if the blur exceeds a preset threshold, the modal data is deemed invalid, equivalent to a missing modality). Finally, based on the results of these two steps, a clear detection result is output, such as "Text + Audio + Image modalities are complete" or "Only the text modality exists, audio + image modalities are missing," etc.

[0023] In one embodiment, performing modal integrity detection on the multimodal background noise conditional input data to obtain a detection result includes: Automatically classify multimodal background sound conditional input data to separate text description data blocks, environmental audio data blocks, and environmental image data blocks; Specifically, the system sets up independent input channels for three modalities: text description, ambient audio, and ambient image (such as the "text input box," "audio upload button," and "image upload button" on the front-end interface). Each channel corresponds to a unique modality identifier (text modality identifier "T," audio modality identifier "A," and image modality identifier "P"). When a user inputs data through any channel, the system automatically adds the corresponding initial modality identifier to the data, initially distinguishing the data type.

[0024] Text description data block classification: After confirming that the data is text type through character encoding detection (such as UTF-8 and GBK encoding recognition) and initial semantic validity judgment (using a lightweight text classification model to determine whether the data contains understandable scene description words, such as "bank lobby" and "hospital ward", excluding pure garbled characters or meaningless character combinations), a "text description data block" is generated and associated with the initial "T" identifier.

[0025] Environmental audio data block classification: After confirming that the data is an audio type through audio header file detection (identifying file header features of WAV, MP3 and other formats) and acoustic feature extraction (calculating the sampling rate and number of channels of the data, which must meet the input requirements of the subsequent CLAP audio encoder of the system - sampling rate 44.1kHz, mono), an "environmental audio data block" is generated and associated with the initial "A" identifier.

[0026] Environmental image data block classification: After confirming that the data is an image type by image format detection (recognizing file header signatures of formats such as JPG and PNG) and visual feature verification (using edge detection algorithms to determine whether the data contains an image pixel matrix and the resolution is not less than 256×256 pixels to meet the requirements of CLIP image encoder for local detail extraction), an "environmental image data block" is generated and associated with the initial "P" identifier.

[0027] After classification, the system stores the three types of data blocks in separate cache areas. Each data block carries a "modal identifier - classification verification result" label (such as "T - valid text", "A - valid audio", "P - valid image") to provide a data foundation for subsequent format verification.

[0028] The text description data block, environmental audio data block, and environmental image data block are format-validated according to the preset modal integrity rules to generate a validation status. Specifically, the preset modal integrity rules are defined as follows: Text description data block rules: 1. Data length: ≥3 valid characters (avoid meaningless inputs such as "empty text" or "single word" to ensure that the core semantics of the scene can be extracted); 2. Semantic relevance: contain at least one scene-related keyword (such as "counter" and "transaction" in the financial field, and "ward" and "medical order" in the medical field, which are verified by matching the preset domain keyword library).

[0029] Environmental audio data block rules: 1. Valid duration: The audio duration after preprocessing (removing silence segments and normalizing amplitude) is ≥2 seconds (to meet the minimum data volume requirement for the CLAP audio encoder to extract acoustic features and avoid feature loss due to insufficient duration); 2. Acoustic quality: Signal-to-noise ratio (SNR) ≥20dB (by calculating the power ratio of audio signal to noise, audio that is too noisy and cannot extract effective acoustic style is excluded).

[0030] Environmental image data block rules: 1. Resolution: ≥256×256 pixels (to ensure that the CLIP image encoder can capture local details, such as local information such as "bedside monitor" and "IV stand" in medical scenes); 2. Sharpness: blur ≤ preset threshold (the image edge gradient is calculated by the Laplacian operator, and if the gradient value is lower than the threshold, it is judged as blurry, so as to avoid the deviation of visual feature extraction due to image blur).

[0031] The system calls a separate verification module to perform verification on each of the three types of data blocks in the cache area: Text description data block: First, count the number of valid characters to determine if the length requirement is met; then, use a keyword matching algorithm to check if it contains scene-related keywords. If both conditions are met, the verification passes; otherwise, the verification fails.

[0032] Environmental audio data block: First, the effective duration after preprocessing is calculated using audio analysis tools to determine whether it meets the standard; then, the acoustic quality is evaluated using signal-to-noise ratio calculation tools (such as noise estimation methods based on short-time Fourier transform). If both conditions are met, the verification passes; otherwise, the verification fails.

[0033] Environmental image data block: First, read the image resolution parameters to determine if they meet the requirements; then, calculate the sharpness using the Laplacian blur detection algorithm. If both requirements are met, the verification passes; otherwise, the verification fails.

[0034] After each type of data block is validated, a corresponding validation status label is generated, which is divided into two categories: "passed" and "failed". For example, if a text description data block meets all the rules, it is marked "text validation passed"; if an environmental audio data block is less than 2 seconds in duration, it is marked "audio validation failed"; and if an environmental image data block exceeds the blur limit, it is marked "image validation failed".

[0035] The detection results are generated based on the verification status.

[0036] Specifically, the detection results need to clearly define "whether each modality exists effectively," providing a decision-making basis for the subsequent "generation of environmental background sound feature embedding vectors in scenarios with complete / missing modalities." The specific implementation process is as follows: Establish a corresponding rule for "verification status - modality existence": if the verification status of a certain type of data block is "passed", then the modality is determined to be "validally existing"; if the verification status is "failed" (or there is no corresponding data block input), then the modality is determined to be "missing".

[0037] Based on the existence combinations of the three modalities, standardized detection results are generated. The results must cover all scenarios that the system may encounter, specifically including: Fully modal scenario: Text, audio, and image modalities are all "validally present", and the detection result is marked as "T+A+P complete".

[0038] Bimodal existence scenario: any two types of modalities are "validally present", and a third type is "missing", such as "T+A exists, P is missing", "T+P exists, A is missing", "A+P exists, T is missing".

[0039] Single-modal existence scenario: Only one type of modality "exists" and the other two types are "missing", such as "only T exists, A+P is missing", "only A exists, T+P is missing", "only P exists, T+A is missing".

[0040] Full-modal missing scenario: All three modalities are "missing", and the detection result is marked as "no valid modal input" (at this time, the system needs to trigger a front-end prompt, requiring the user to supplement at least one type of valid data).

[0041] The standardized detection results are synchronized to the "Environmental Background Sound Feature Embedding Vector Generation Module". The module automatically selects a feature generation strategy based on the results. If "T+A+P are complete", the "Modal-Specific Feature Extraction + Shared Feature Fusion" process is executed. If it is a dual-modal / single-modal scenario, the "Existing Modal Feature Extraction + Missing Modal Feature Inference" process is executed to ensure that the subsequent feature generation process is seamlessly connected with the modal detection results.

[0042] In other words, the dual filtering of "automatic classification + format verification" effectively eliminates interference from invalid data (such as garbled text, noisy audio, and blurry images) on subsequent CLAP and CLIP encoder feature extraction. For example, if noisy audio with a signal-to-noise ratio (SNR) below 10dB is input into the CLAP audio encoder without format verification, the extracted acoustic features will be distorted, leading to a deviation between the environmental background sound feature embedding vector and the actual scene. However, this implementation, through a verification rule with an SNR ≥ 20dB, ensures that the audio data input to the encoder has clear acoustic style information, improving the accuracy of the subsequently generated environmental background sound feature embedding vector by more than 30%, providing data support for the natural integration of speech and scene. Furthermore, standardized detection results can directly guide the "environmental background sound feature embedding vector generation module" in selecting an adaptation strategy, avoiding process chaos caused by modal state ambiguity. For example, in outdoor mobile service scenarios in the financial sector, users can only provide "text description (bank outdoor service point) + environmental image (outdoor scene photo)". The detection result is marked as "T+P exists, A is missing". The module can automatically trigger the process of "text semantic feature extraction + image visual feature extraction + audio feature inference" without manual intervention. If modal integrity detection is lacking, the module cannot determine whether the audio modality is missing, and may mistakenly execute the "three-modal feature fusion" process, resulting in feature generation failure. In addition, through pre-processing modal integrity detection, invalid data can be filtered out before the data enters the feature extraction stage, avoiding redundant calculations of meaningless data by CLAP and CLIP encoders. For example, if a user mistakenly uploads a 1-second audio clip to the system, the "valid duration ≥ 2 seconds" rule in the format verification will determine that the verification fails and directly mark it as "audio missing." There is no need to input it into the CLAP audio encoder for feature extraction, which can reduce the computational cost of the encoder. At the same time, standardized detection results can eliminate the need for the subsequent feature generation module to repeatedly judge the validity of the data, shortening the process time. Especially in scenarios with high real-time requirements, such as high-frequency financial trading broadcasts and emergency medical order generation, this efficiency improvement can ensure that the speech synthesis response time is controlled within 1 second, meeting the real-time interaction needs of key areas.

[0043] S130. Generate an environmental background sound feature embedding vector based on the detection results; Specifically, based on the modal integrity detection results, environmental background sound feature embedding vectors are generated for two scenarios: "modal complete" and "modal missing". The core relies on the encoders of the Contrastive Language-Audio Pre-training System (CLAP) and the Contrastive Language-Image Pre-training System (CLIP) and the cross-modal feature mapping mechanism. A scene with all modalities (text, audio, and images are all present); Modality-specific feature extraction: The preprocessed text description data is input into CLAP's text encoder, which extracts scene-related semantic feature vectors (512-dimensional) from the text through a semantic mapping network trained by contrastive learning; the preprocessed environmental audio data is input into CLAP's audio encoder to extract acoustic style feature vectors (512-dimensional), including noise spectrum distribution, reverberation parameters, etc.; the preprocessed environmental image data is input into CLIP's image encoder to extract global and local visual feature vectors (512-dimensional), covering not only global scene categories (such as "bank lobby"), but also local details (such as "self-service terminal area in the lobby" and "queuing crowd").

[0044] Shared Feature Learning and Fusion: The modality-specific feature vectors of the above three types are fed into a shared encoder (using a 3-layer fully connected network with ReLU activation function). The intermodal correlation features are learned through a multimodal attention mechanism, outputting a 512-dimensional shared feature vector. Subsequently, the modality-specific feature vectors (512-dimensional each for text, audio, and image) are concatenated with the shared feature vector (512-dimensional) to obtain a 2048-dimensional complete environmental background sound feature embedding vector. This embedding contains both modality-specific information and cross-modal correlation information.

[0045] Modality missing scenarios (e.g., only text and images exist, but audio is missing); Existing modal feature extraction: Following the method for scenarios with complete modalities described above, extract the semantic feature vector (512-dimensional) of the text description and the visual feature vector (512-dimensional) of the environment image, and obtain the corresponding shared feature vector (512-dimensional) through a shared encoder.

[0046] Missing modality feature inference: A cross-modal mapping module (based on a multilayer perceptron) is used as input. Taking the feature vectors of existing modalities (textual semantic features + image visual features + shared features) as input, the missing environmental audio feature vectors are inferred through the trained mapping relationship. The inference formula is as follows: Where k is the number of existing modes (here k=2). The feature vectors of existing modalities (textual semantic features, image visual features). The inferred audio feature vector (512 dimensions).

[0047] Feature fusion: The inferred audio feature vector, the feature vector of the existing modality, and the shared feature vector are concatenated to obtain a 2048-dimensional environmental background sound feature embedding vector, ensuring that the feature dimension is consistent with the scene with complete modalities, and providing a unified format for subsequent acoustic model input.

[0048] In one embodiment, generating an ambient background sound feature embedding vector based on the detection result includes: If the detection result indicates that no modality is missing, feature representations of text description data, environmental audio data, and environmental image data are extracted separately by a modality-specific encoder to obtain modality-specific features. At the same time, the text description data, environmental audio data, and environmental image data are fed into a shared encoder for unified representation learning to obtain modality-shared features. The modality-specific features and modality-shared features are then concatenated to generate an environmental background sound feature embedding vector. Specifically, when the detection results show that all three modalities are validly present, the environmental background sound feature embedding vector needs to be generated through three steps: "modality-specific feature extraction - modality-shared feature learning - feature concatenation and fusion". The core relies on the pre-trained encoders of CLAP and CLIP and the shared learning network. The specific implementation is as follows: Modality-specific feature extraction (based on CLAP and CLIP encoders). Text description data feature extraction: A contrastive language-audio pre-trained system (CLAP) is used to semantically encode pre-processed text description data (such as "bank lobby morning peak transaction broadcast" in a financial scenario and "ICU ward doctor's orders reading" in a medical scenario). This encoder converts the text into a 512-dimensional text semantic feature vector through a semantic mapping network trained by contrastive learning. The vector contains core scenario attributes (such as "bank lobby" and "ICU ward") and semantic association information (such as "transaction broadcasts should highlight numbers" and "doctor's orders should not have sharp background noise"), and is in the same semantic space as the output features of the CLAP audio encoder.

[0049] Environmental audio data feature extraction: Using CLAP's audio encoder, acoustic features are encoded into preprocessed environmental audio data (such as conversations in a bank lobby or low-level equipment humming in a hospital ward). The encoder first converts the audio into a spectrogram using a short-time Fourier transform, then extracts spectral features using a convolutional neural network (CNN), ultimately outputting a 512-dimensional acoustic style feature vector. This vector encompasses key acoustic information such as noise spectrum distribution, reverberation parameters, and audio intensity, accurately representing the style attributes of the environmental audio.

[0050] Environmental image data feature extraction: The image encoder, based on the Contrastive Language-Image Pre-training System (CLIP), encodes visual features into preprocessed environmental image data (such as images of bank lobbies with self-service terminals and hospital wards with bedside monitors). The encoder captures global scene features (e.g., "bank lobby" and "hospital ward") through convolutional layers of a ResNet architecture, while simultaneously extracting local detail features (e.g., "self-service terminal operation area" and "monitoring equipment") through an attention mechanism. The final output is a 512-dimensional visual semantic feature vector, ensuring alignment of visual information with the text and audio semantic spaces.

[0051] Modal shared feature learning (based on shared encoder): The extracted text semantic feature vector, acoustic style feature vector, and visual semantic feature vector are input together into a shared encoder (using a 3-layer fully connected network with ReLU activation). The shared encoder learns the relationships between the three modalities (such as the semantic relationship between the text "bank lobby" and the audio "crowd conversation" and the image "self-service terminal") through a multimodal attention mechanism. By comparing the cross-entropy loss function (with a loss function design finely tuned similar to CLAP / CLIP), the network parameters are optimized, and finally, a 512-dimensional modal shared feature vector is output. This vector no longer distinguishes between modal types but integrates the common scene information of the three modalities to form a unified scene feature representation.

[0052] Feature concatenation and fusion (generating complete feature embeddings): The text semantic feature vector, acoustic style feature vector, visual semantic feature vector, and modal shared feature vector are concatenated in the order of "text-audio-image-shared" to form a 512×4=2048-dimensional ambient background sound feature embedding vector. The concatenated ambient background sound feature embedding vector contains both modal-specific detailed information (such as the semantic emphasis of the text, the acoustic details of the audio, and the local features of the image) and cross-modal correlation information (such as "the bank lobby scene needs to match the sound of the crowd + the visual features of the self-service terminal"), which can fully support the subsequent scene-based speech generation of the acoustic model.

[0053] If the detection result indicates the presence of a missing modality, features of the existing modality data are extracted using a modality-specific encoder. These features are then input into a cross-modality mapping module to infer the semantic features of the missing modality. Subsequently, the features of the existing modality data and the semantic features of the missing modality are fed into a shared encoder for unified representation learning to obtain the unified representation learning result. Finally, the features of the existing modality data, the semantic features of the missing modality, and the unified representation learning result are concatenated to generate an environmental background sound feature embedding vector.

[0054] Specifically, when the detection result indicates that one or two modalities are missing (e.g., only text + image exists, only audio exists), feature embedding needs to be generated through four steps: "existing modal feature extraction - missing modal feature inference - shared feature learning - feature concatenation". The core relies on the cross-modal mapping module and the shared encoder. The specific implementation takes "only text + image exists, audio is missing" as an example (the process is the same for other missing scenarios): Existing modal feature extraction (same as modality-specific extraction in scenarios without missing modal features): The CLAP text encoder is used to extract 512-dimensional semantic features from the text description data, and the CLIP image encoder is used to extract 512-dimensional visual features from the environmental image data. The extraction process is completely consistent with that in scenarios without missing modal features, ensuring the accuracy of existing modal features and consistency with the semantic space.

[0055] Missing Modality Feature Inference (Based on Cross-Modal Mapping Module): A cross-modal mapping module (based on a 2-layer multilayer perceptron MLP) is constructed. Existing modal features, including semantic and visual features, are input into the cross-modal mapping module. The module infers a 512-dimensional acoustic style feature vector matching the missing audio modality using pre-trained cross-modal mapping relationships (training data consists of a large amount of text-audio-image trimodal alignment data to ensure mapping accuracy). During inference, the cross-modal mapping module dynamically adjusts the acoustic parameters of the acoustic style feature vector based on semantic features (e.g., "bank lobby") and visual features (e.g., "densely populated area") to ensure semantic consistency between the inferred features and existing modal features (e.g., "bank lobby + crowd visual features" corresponds to the acoustic features of "moderate-intensity crowd conversation").

[0056] Shared feature learning (based on a shared encoder): Existing modal features (semantic features, visual features) and inferred missing modal features (acoustic style feature vector) are input into a shared encoder (using the same network structure as in the missing-feature scenario). A multimodal attention mechanism is used to learn the relationships between the three types of features, outputting a 512-dimensional shared modal feature vector. This process is consistent with the shared learning logic in the missing-feature scenario, ensuring that the shared features can integrate scene information from both the existing and inferred modalities, maintaining the consistency of feature representation.

[0057] Feature concatenation and fusion (generating complete feature embedding): Existing modal features (semantic features and visual features, each 512-dimensional), inferred missing modal features (acoustic style feature vector, 512-dimensional), and shared features (modal shared feature vector, 512-dimensional) are concatenated to form a 512×4=2048-dimensional ambient background sound feature embedding vector. The dimensions of this embedding are completely consistent with the ambient background sound feature embedding vector for scenes without missing features, ensuring that subsequent acoustic models do not need to adjust the input format and achieve seamless adaptation to different modal scenes.

[0058] In other words, by extracting modality-specific features through the CLAP / CLIP encoder and learning cross-modal associations through the shared encoder, the generated feature embedding can achieve deep synergy of three dimensions of information: semantics, acoustics, and vision. This solves the shortcomings of existing technologies, such as coarse adaptation to single-modal input scenarios and semantic disconnect. For example, in the scenario of "transaction broadcasting in the bank lobby during morning rush hour" in the financial field, text features clearly define the semantic requirement of "transaction broadcasting" (emphasizing numbers), audio features provide the acoustic style of "crowd conversation + keyboard typing," and image features capture the local details of the "self-service terminal area." After the three are fused with the shared features, the generated feature embedding can accurately guide the acoustic model: when broadcasting "transaction amount of 50,000 yuan," the background noise is weakened to highlight the numbers; when broadcasting non-critical content, the background noise of the "crowd + device" scenario is maintained, which improves the scene adaptation accuracy of the synthesized speech, and the semantic and background noise fit is significantly better than existing technologies. Furthermore, relying on the cross-modal mapping module and shared learning mechanism, the system can still generate logically coherent and scenario-adaptive feature embeddings even in the event of modal missingness, solving the problem of performance collapse in existing technologies when modalities are missing. For example, in the "home-based remote consultation" scenario in the medical field, if only "medical order text (with soft background noise) + home environment image (sofa + curtains)" (without ambient audio) can be obtained, the system can infer the acoustic features of "low-intensity indoor ambient noise (such as soft wind)" through text and image features. The generated feature embedding can guide the acoustic model to generate consultation voice with "doctor's voice + soft home background noise", avoiding the problems of "no background noise" or "disordered background noise" caused by missing audio. This effect improves the success rate of feature generation in scenarios with limited data collection, such as financial outdoor services and medical emergency rescue, and significantly enhances robustness. Furthermore, regardless of whether the scene has no or some modal missing features, the generated ambient background sound feature embedding vector is 2048-dimensional, and all features (including the inferred missing modal features) are in a unified semantic space aligned with CLAP / CLIP, eliminating the need for input format adjustments to subsequent AudioLDM-based acoustic models. For example, when receiving ambient background sound feature embedding vectors with or without missing features, the acoustic model can directly map the 2048-dimensional features to the input dimension of the adaptation diffusion model through a linear layer, avoiding increased model adaptation costs due to differences in feature dimensions or semantic space. This improves the training and inference efficiency of the acoustic model while ensuring stylistic consistency of synthesized speech across different modalities.

[0059] In one embodiment, if the detection result indicates no modality loss, feature representations of the text description data, environmental audio data, and environmental image data are extracted separately using a modality-specific encoder to obtain modality-specific features; simultaneously, the text description data, environmental audio data, and environmental image data are fed into a shared encoder for unified representation learning to obtain modality-shared features; then, the modality-specific features and the modality-shared features are concatenated to generate an environmental background sound feature embedding vector, including: The modality-specific encoder includes a text encoder, an audio encoder, and an image encoder, which respectively output text semantic features, acoustic style features, and visual semantic features; Specifically, modality-specific encoders include three types of dedicated encoders: text encoders, audio encoders, and image encoders, which correspond to feature extraction from text description data, environmental audio data, and environmental image data, respectively. Each encoder is optimized based on a pre-trained model to ensure feature extraction accuracy and semantic space consistency. Text Encoder and Semantic Feature Extraction: A CLAP pre-trained text encoder for audio-text contrastive learning was employed, and fine-tuned for scenario-specific texts in target domains such as finance and healthcare (e.g., "bank lobby morning rush hour transaction broadcast" and "ICU ward doctor's orders reading"). The fine-tuning process used an internal domain dataset (containing 5000+ scenario text-audio pairings) as training samples, optimizing encoder parameters by comparing cross-entropy loss functions to enhance semantic capture of domain-specific scenario vocabulary (e.g., "annualized rate of return" in finance and "drug dosage" in healthcare).

[0060] Extraction process: The preprocessed text description data (special characters removed, unified UTF-8 encoding) is converted into a character-level vector (based on the BPE word segmentation algorithm) and input into the fine-tuned CLAP text encoder. The encoder captures the semantic relationships of the text context through a multi-layer Transformer structure, and finally outputs a 512-dimensional text semantic feature vector. The vector contains core scene attributes (such as "bank lobby" and "ICU ward") and semantic emphasis markers (such as the positional weights of "numbers" and "key instructions" in the text).

[0061] Audio Encoder and Acoustic Style Feature Extraction: A CLAP pre-trained audio encoder, compatible with the CLAP text encoder, was employed. Fine-tuning was performed using a domain-specific environmental audio dataset (containing over 3000 ambient sounds from financial / medical scenarios, such as keystrokes in a bank and low-level noises from hospital equipment). Fine-tuning aimed for "audio-text semantic alignment," enhancing the encoder's ability to extract typical acoustic features from the domain scenario (such as the sound of a banknote counter in a financial scenario and the beeping sound of a monitor in a medical scenario) by adjusting the encoder's convolutional layer parameters.

[0062] Extraction Process: The preprocessed environmental audio data (sampling rate 44.1kHz, mono, silence removed) is converted into a spectrogram (frame length 20ms, frame shift 10ms) using a Short Time Fourier Transform (STFT), and then input into the fine-tuned CLAP audio encoder. The encoder extracts the time-frequency domain features of the spectrum through a 6-layer CNN structure, and then maps them to a 512-dimensional acoustic style feature vector through a fully connected layer. The vector covers key acoustic parameters such as noise spectrum distribution, reverberation time, and audio intensity, which can accurately characterize the style attributes of the environmental audio.

[0063] Image Encoder and Visual Semantic Feature Extraction: A CLIP pre-trained image encoder (based on the ResNet-50 architecture) was used and fine-tuned for target domain scene images (such as bank branch images and hospital ward images). The fine-tuning dataset contains 2000+ "scene image-text description" pairs. The encoder's attention mechanism was optimized through contrastive learning to make it pay more attention to local key information in the scene (such as "self-service terminal" in bank images and "bedside monitor" in hospital ward images).

[0064] Extraction process: The preprocessed environmental image data (256×256 pixels resolution, RGB channel normalization) is input into the fine-tuned CLIP image encoder. The encoder captures global scene features of the image through the convolutional layer of ResNet-50, and then focuses on local detail features through the self-attention module, finally outputting a 512-dimensional visual semantic feature vector. The vector realizes the alignment of visual information with the semantic space of text and audio (e.g., the visual features of "ward image" are in the same semantic dimension as the semantics of the text "ward" and the acoustic features of "equipment sound").

[0065] The shared encoder adopts a fully connected neural network structure to perform unified dimensional mapping on text description data, environmental audio data, and environmental image data in order to output modal shared features. Specifically, the shared encoder adopts a fully connected neural network (FCN) structure corresponding to the model fine-tuning architecture. Its core function is to perform unified dimensional mapping and association learning on specific features of the three modalities, generating modal shared features that integrate common information from multiple modalities. Network structure design: The shared encoder contains 3 fully connected layers, with 1024, 512 and 512 neurons in each layer, respectively, and ReLU is used as the activation function in all layers; the input layer receives a concatenated vector of text semantic feature vector, acoustic style feature vector and visual semantic feature vector (dimension 512×3=1536), and the output layer outputs a 512-dimensional modality shared feature vector.

[0066] Unified Dimension Mapping and Association Learning Process: The concatenated vectors of received text semantic feature vectors, acoustic style feature vectors, and visual semantic feature vectors are concatenated in the order of "text-audio-image" to form a 1536-dimensional multimodal initial vector. This vector is then input into the first fully connected layer of the shared encoder. The dimension is compressed to 1024 dimensions through linear transformation. At the same time, a non-linear feature mapping is introduced through the ReLU activation function to initially fuse the basic features of the three modalities.

[0067] The 1024-dimensional features output from the first layer are input into the second fully connected layer, which further compresses them to 512 dimensions. At this point, the network learns the relationships between modalities (such as the correspondence between the text semantics of "bank lobby" and the acoustic features of "people talking" and the visual features of "counter area"), weakening modal-specific differences and strengthening common information of the scene.

[0068] The 512-dimensional features output from the second layer are input into the third fully connected layer (without dimensionality change). Residual connections preserve the effective features from the preceding layers, ultimately outputting a 512-dimensional modality-shared feature vector. This modality-shared feature vector no longer distinguishes between modal types but focuses on the "overall attributes of the scene." For example, the shared features for a "bank lobby morning rush hour" scene will incorporate common information such as "transaction broadcast semantics," "crowd acoustic style," and "counter visual details."

[0069] The text semantic features, acoustic style features, visual semantic features, and modal sharing features are concatenated into vectors and then fused through a linear layer to generate an environmental background sound feature embedding vector.

[0070] Specifically, through two steps of "feature concatenation - linear layer fusion", modality-specific features and modality-shared features are integrated into a unified environmental background sound feature embedding vector. The specific process is as follows: Vector concatenation: The text semantic feature vector (512-dimensional), acoustic style feature vector (512-dimensional), visual semantic feature vector (512-dimensional), and modal shared feature vector (512-dimensional) are concatenated in a fixed order (text semantic feature vector → acoustic style feature vector → visual semantic feature vector → modal shared feature vector) to form a 512×4=2048-dimensional multimodal feature concatenation vector. The concatenation process strictly maintains the dimensional order to ensure that subsequent linear layers can stably recognize the semantic meaning of each feature.

[0071] Linear Layer Feature Fusion: A linear transformation layer (input dimension 2048, output dimension 2048) is set up, and the concatenated vector of multimodal features is input into this layer for feature fusion. The linear layer learns the weight parameters of each feature dimension (e.g., assigning higher weights to "textual semantic emphasis" and "acoustic style details"), filters redundant information in the concatenated vector, and enhances effective information, finally outputting a 2048-dimensional ambient sound feature embedding vector. This ambient sound feature embedding vector contains both "modality-specific details" (such as key semantics in text, acoustic parameters in audio, and local features in images) and "cross-modal common information" (such as the overall style of the scene), which can be directly used as conditional input for subsequent acoustic models.

[0072] In other words, based on the fine-tuned CLAP / CLIP encoder, textual semantic features can accurately capture the core needs of the domain scenario (such as the key instruction "take medication twice a day" in medical text), acoustic style features can restore the unique acoustic details of the scenario (such as the sound of a money counter in a financial scenario), and visual semantic features can capture key local information of the scenario (such as "monitoring equipment" in a ward). The extraction accuracy of these three types of features is improved compared to the un-fine-tuned general encoder, avoiding scenario adaptation bias caused by the blurring of modality-specific features, and providing high-quality basic data for subsequent feature fusion and speech generation. In addition, the shared encoder learns the correlation between modalities through a fully connected network, and the generated modal shared features can eliminate the semantic barriers between different modalities, realizing the collaborative expression of "text-audio-image". For example, in the "morning rush hour in the bank lobby" scenario, the shared features will integrate the textual semantics of "transaction broadcast", the acoustic style of "crowd conversation", and the visual details of "counter area" into a unified "overall scene feature", avoiding the problem of "speech and background sound separation" caused by the isolation of modal information in subsequent acoustic models. Furthermore, the linear layer uses weight learning to perform redundant filtering and effective information enhancement on the spliced ​​features, thereby increasing the information density of the embedded vector of environmental background sound features. For example, in the "ICU ward medical orders" scenario, the linear layer assigns higher weights to "key information in the medical order text," "low-frequency acoustic parameters of the equipment," and "visual details of the monitor," while filtering redundant modifiers in the text and irrelevant noise in the audio. This ensures that the subsequent acoustic model can prioritize generating speech based on key information—for example, when announcing "dosage of 5mg," the background noise is automatically weakened to highlight the key data, while the equipment background noise is maintained when announcing non-key content. This makes the synthesized speech both meet the needs of the scenario and avoid the problem of speech masking, resulting in a significant improvement in naturalness compared to traditional post-mixing techniques.

[0073] S140. Obtain the text data to be synthesized and the speaker reference audio data, and extract features from the text data to be synthesized and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features. Specifically, the text data to be synthesized is obtained through user input or file import (supporting formats such as TXT and DOC), and the content can cover transaction broadcast text in the financial field, medical prescription text in the medical field, etc.; the speaker reference audio data is obtained through microphone recording (duration 10-30 seconds, to ensure that stable timbre features can be extracted) or local file upload (supporting WAV format), and must include natural speech segments of the speaker (such as sentence reading at normal speaking speed).

[0074] Text semantic feature extraction: A bidirectional long short-term memory network (Bi-LSTM) is used as a text encoder. The text to be synthesized is converted into character-level or word-level vectors and then input into the network. By capturing the semantic relationship of the text context, a 512-dimensional text semantic feature vector is output. This vector not only contains the literal meaning of the text, but also covers the semantic focus (such as numbers in financial texts and drug dosages in medical texts).

[0075] Speaker timbre feature extraction: Using a speaker encoder (based on ResNet architecture), frame-level feature extraction is performed on the preprocessed speaker reference audio data (silence removed and normalized). The spectral features of the audio are captured by Mel-frequency cepstral coefficients (MFCC), and then a 256-dimensional speaker timbre feature vector is obtained through network mapping. This vector can accurately represent the speaker's timbre, intonation and other personalized features, meeting the requirements of zero-sample replication.

[0076] In one embodiment, the step of acquiring the text data to be synthesized and the speaker reference audio data, and extracting features from the text data to be synthesized and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features, includes: A pre-trained language model is used to extract context encoding from the text data to be synthesized, generating text semantic feature vectors. Specifically, a pre-trained Transformer language model adapted to text semantic understanding (such as the BERT-base model) was selected and fine-tuned based on key domain scenario texts (such as financial transaction broadcast texts and medical prescription texts) in finance and healthcare. The fine-tuning dataset contains 5000+ domain-specific text data (each text is annotated with semantic emphasis, such as numbers in financial texts and drug dosages in medical texts). Through joint training using "masked language modeling (MLM)" and "semantic classification tasks," the model's ability to capture domain-specific vocabulary (such as "annualized rate of return" and "intravenous injection") and key information was optimized, ensuring that the semantic features output by the model are highly adapted to the domain scenario.

[0077] Perform standardized preprocessing on the synthesized text data (supports importing TXT, DOC, and other formats or direct user input): Text cleaning: Remove special symbols (such as "#" and "@"), redundant spaces and meaningless characters (such as pure numeric garbled characters), and retain the core semantic content (such as medical orders "oral medication 3 times a day, 5mg each time"). Text segmentation and encoding: Long texts (such as financial product introductions exceeding 512 characters) are segmented according to the input length limit of the pre-trained model (such as the 512 token limit of BERT-base). The text is converted into a token sequence using the model's matching tokenizer (such as BERTTokenizer), and corresponding attention masks are generated (marking the positions of valid tokens and excluding the interference of padding). The preprocessed token sequence and attention mask are input into the fine-tuned pre-trained language model to perform context encoding: The model captures the contextual relationships of text through a multi-layer Transformer encoder (such as the 12-layer Transformer in BERT-base). For example, in the medical prescription "take oral medication 3 times a day, 5mg each time", the model can identify the logical association between "3 times" and "5mg", as well as the semantic binding between "oral" and "medication". It extracts the output vector corresponding to the model token (768 dimensions, which is the default output dimension of the BERT-base model), and this vector integrates the global semantic information of the entire text. A linear transformation layer maps the 768-dimensional vector into a 512-dimensional text semantic feature vector (which has the same modality-specific feature dimension as the ambient background sound feature embedding vector, facilitating subsequent feature concatenation). This vector is the final text semantic feature used for speech generation, which can accurately represent the semantic logic, core information and domain attributes of the text.

[0078] The speaker reference audio data is processed by a speaker audio encoder, which is based on a VAE architecture, to extract the latent feature representation of the speaker's timbre in order to obtain the speaker's timbre features.

[0079] Specifically, the speaker timbre feature extraction aims to "reproduce the speaker's personalized timbre with zero samples," and relies on the VAE (Variational Autoencoder) architecture to achieve efficient extraction of latent timbre features. The specific implementation steps are as follows: Speaker Audio Encoder VAE Architecture Design: The Speaker Audio Encoder adopts a VAE architecture adapted for timbre feature extraction, which is divided into three parts: "Encoder", "Latent Space", and "Decoder". The core focus is on the encoder's timbre feature extraction capabilities. Encoder structure: It adopts a 3-layer convolutional neural network (CNN) + 2-layer fully connected layer design. The input is preprocessed speaker reference audio data (sampling rate 22.05kHz, mono). The CNN layer extracts the frame-level acoustic features of the audio (such as Mel-frequency cepstral coefficients MFCC and spectral envelope). Then, the fully connected layer maps the frame-level features to the latent space and outputs a 256-dimensional timbre latent feature vector (this dimension takes into account both the accuracy of timbre representation and the efficiency of subsequent feature fusion). Latent space constraint: The latent feature vectors are constrained to follow a normal distribution (N(0,I)) by using the KL divergence loss function of VAE, which ensures that the timbre features of different speakers are evenly distributed in the latent space, avoids the overlap of similar timbre features, and improves the distinguishability of timbre replication; Decoder-assisted training: The decoder adopts a structure of "2 fully connected layers + 3 deconvolution layers" that is symmetrical to the encoder. It reconstructs the latent feature vectors into audio waveforms. The encoder parameters are jointly optimized by "reconstruction loss (such as MSE loss)" and KL divergence loss to ensure that the latent features extracted by the encoder can accurately restore the speaker's timbre.

[0080] Speaker reference audio data preprocessing: Perform preprocessing on speaker reference audio data (supports microphone recording or local WAV format file upload; recording duration should be 10-30 seconds to ensure stable timbre characteristics): Audio cleaning: Remove silent segments from audio using silence detection algorithms (such as the energy threshold-based VAD algorithm) while retaining the speaker's valid speech segments; Standardization processing: Amplitude normalization is performed on effective speech segments (the audio amplitude is uniformly mapped to the [-1, 1] interval) to eliminate volume differences caused by different recording devices; Frame segmentation: The standardized audio is segmented into a frame sequence with a frame length of 20ms and a frame shift of 10ms, providing the input basis for frame-level feature extraction by the encoder.

[0081] Speaker timbre feature extraction process: Input the preprocessed audio frame sequence into the trained VAE encoder: The encoder extracts low-frequency acoustic features of the audio (such as fundamental frequency and basic attributes related to the speaker's timbre) through the first layer of CNN (3×3 kernel size, stride 1). The second CNN layer (convolution kernel size 5×5, stride 2) further extracts mid-frequency features (such as harmonic structure, which determines the richness of timbre). The third CNN layer (7×7 kernel size, stride 2) captures high-frequency detail features (such as formants, distinguishing the timbre differences of different speakers). The features output by the CNN are mapped to the latent space to generate a 256-dimensional speaker timbre feature vector. This vector can uniquely represent the speaker's timbre, intonation and other personalized features, meeting the personalized replication needs of Zero-shot TTS technology.

[0082] In other words, the fine-tuned pre-trained language model can deeply understand the contextual logic of domain-specific texts. For example, in the financial text "Account balance 15,000 yuan, current annualized return 3.2%", the model can recognize the semantic relationships between "15,000 yuan" and "account balance", and between "3.2%" and "annualized return". The generated text semantic feature vector can guide the subsequent acoustic model to automatically adjust speech stress and rhythm when broadcasting key numbers, avoiding the omission of key information. In addition, through domain-specific text fine-tuning, the model can accurately identify professional vocabulary and semantic rules in fields such as finance and medicine. For example, in the medical text "Daily intravenous injection of penicillin twice, 4 million units each time", the model can distinguish the semantic difference between "intravenous injection" and "oral administration", avoiding the problem of "incorrect broadcast of medication method" in subsequent speech generation. Furthermore, the speaker audio encoder based on the VAE architecture can extract the latent features of the speaker's timbre. Even with only 10 seconds of speaker reference audio input, it can accurately replicate the speaker's timbre, intonation, and other personalized features. For example, in the medical field, using a 10-second reference audio recording of a doctor, the encoder extracts timbre features that guide the acoustic model to generate medical advice speech with a timbre highly consistent with the doctor's own. Patients can increase their trust in the medical advice through a familiar timbre. The KL divergence loss function of VAE ensures that the speaker's timbre feature vector is evenly distributed in the latent space, avoiding overlap of timbre features from different speakers, while ensuring the consistency of features extracted from different reference audio recordings of the same speaker. For example, in the financial field, different reference audio recordings of the same customer service representative (such as voice recordings from different time periods) yield timbre feature vectors with a cosine similarity of over 0.9. The subsequently generated customer service broadcast speech has a stable timbre without significant fluctuations, meeting the requirements of key areas for voice identity recognition and stability.

[0083] S150. Input the environmental background sound features into the embedding vector, text semantic features, and speaker timbre features into the acoustic model to generate the Mel spectrum; Specifically, the acoustic model is constructed based on the Latent Diffusion Model (LDM), and the specific implementation process is as follows: Feature concatenation and dimensionality adaptation: The environmental background sound feature embedding vector (2048-dimensional), the text semantic feature vector (512-dimensional), and the speaker timbre feature vector (256-dimensional) are concatenated to obtain a comprehensive feature vector of 2816 dimensions, which is then mapped to 512 dimensions through a linear layer (to match the input dimension of the LDM model).

[0084] The diffusion process generates the Mel spectrum: the adapted integrated feature vector is used as conditional input and fed into the pre-trained LDM model. The model uses a progressively denoised diffusion process to control speech content with text semantic features, speech personalization with speaker timbre features, and scene adaptation with environmental background noise features. Finally, it generates a Mel spectrum with a dimension of 80×T (80 is the number of Mel bands, and T is the time step), which simultaneously contains speech content, personalized timbre, and scene-specific background noise information.

[0085] In one embodiment, the step of inputting the environmental background sound feature embedding vector, text semantic features, and speaker timbre features into the acoustic model to generate a Mel spectrum includes: The environmental background sound feature embedding vector, text semantic features, and speaker timbre features are input into a linear layer for mapping to obtain the mapping result; Specifically, the core function of the linear layer is to transform three types of heterogeneous features—ambient background sound feature embedding vectors, text semantic features, and speaker timbre features—into unified-dimensional features that fit the input of the diffusion model, and to strengthen the semantic associations between features through weight learning. The specific implementation steps are as follows: Input feature dimensions and format adaptation: Ambient background sound feature embedding vector: 2048-dimensional vector (generated by splicing and fusing textual semantics, acoustic style, visual semantics and modality-shared features in the absence of modality missing scenarios); Text semantic features: 512-dimensional vector (generated by context encoding of a pre-trained language model, carrying the semantic logic and key information of the text to be synthesized); Speaker timbre features: 256-dimensional vector (extracted by the speaker audio encoder of VAE architecture, representing the speaker's personalized timbre attributes).

[0086] First, the three types of features are concatenated in a fixed order: "environmental background sound feature embedding vector → text semantic features → speaker timbre features", forming a multi-source feature concatenation vector of 2048+512+256=2816 dimensions. This ensures the consistency of the feature order input to the linear layer each time, providing a stable input foundation for weight learning.

[0087] Linear layer structure design and parameter optimization: The linear layer adopts a lightweight structure of "single hidden layer + output layer", with the following specific configuration: Single hidden layer: input dimension 2816, output dimension 1024, activation function is LeakyReLU (avoids the neuron death problem of ReLU function and ensures the continuity of feature mapping). Output layer: Input dimension 1024, output dimension 512, no activation function (to ensure the linear interpretability of the output features, facilitating the subsequent diffusion model to capture feature details). During the training phase, with the optimization objective of "reconstruction loss of the Mel spectrum generated by the diffusion model", the weight parameters of the linear layer are adjusted through the backpropagation algorithm, so that the mapped 512-dimensional features simultaneously highlight the key information of the three types of input features—for example, assigning higher weights to "scene acoustic parameters" in environmental background sound features, "key information markers" in text semantic features, and "fundamental frequency features" in speaker timbre features, to achieve semantic alignment and effective fusion of multi-source features.

[0088] Mapping result output and verification: After the linear layer is processed, a 512-dimensional unified feature vector (i.e., the mapping result) is output, and the validity of the mapping result is ensured by "feature variance verification": the feature variance of the mapping result is calculated. If the variance is lower than the preset threshold (e.g., 0.1), it is determined that there is information loss in the feature mapping, and the remapping process is triggered (the linear layer weight parameters are adjusted and processed again); if the variance meets the requirements, the mapping result is passed into the diffusion model and enters the Mel spectrum generation stage.

[0089] The mapping result is input into the diffusion model, which generates a Mel spectrum that matches the mapping result through a stepwise denoising process.

[0090] Specifically, diffusion model initialization and training adaptation: The diffusion model adopts a "latent space diffusion" architecture, which consists of three core parts: an encoder (mapping audio to the latent space), a diffusion denoising network (performing denoising in the latent space), and a decoder (mapping the denoised latent features to Mel spectra). It is pre-trained and optimized based on a domain-specific dataset (over 3000 pairs of voice-text-environment feature pairings from financial and medical scenarios). Training objective: Using "L1 loss between generated Mel spectrum and real Mel spectrum" as the core loss, and combining "semantic consistency loss (ensuring that the generated spectrum matches the semantics of the text)", "timbre similarity loss (ensuring that the generated spectrum matches the speaker's timbre)" and "scene adaptation loss (ensuring that the generated spectrum matches the background sound features of the environment)" to construct a multi-task loss function, thereby optimizing the model's ability to collaboratively model multiple features; Latent space configuration: The audio is mapped to a 512-dimensional latent space via the VAE encoder (with the same dimensions as the mapping result of the linear layer output) to ensure that the diffusion process is carried out efficiently in the latent space.

[0091] The stepwise denoising process is executed as follows: The diffusion model generates the Mel spectrum through a T-step (e.g., 1000-step) denoising process, with the specific steps as follows: Step 1: Initialize the noise vector. Generate a Gaussian noise vector (following an N(0, I) distribution) in the latent space with the same dimension as the mapping result (512 dimensions) as the initial denoising input; Step 2: Iterative denoising. The mapping result and the current noise vector are input into the diffusion denoising network (using the U-Net architecture, which includes an attention module and residual connections). Based on the scene, content, and timbre information in the mapping result, the network predicts the noise components in the current noise vector and reduces the noise intensity through "noise subtraction". After each denoising step is completed, the noise vector is updated and the updated vector is passed to the next denoising step. Step 3: Denoising Termination and Latent Feature Output. After completing T-step denoising, the denoised 512-dimensional latent feature vector is output. This vector carries all the information in the mapping result (environmental features for scene adaptation, content features corresponding to the text, and timbre features corresponding to the speaker). Step 4: Mel spectrum generation. The denoised latent feature vector is input into the VAE decoder. The decoder maps the latent features into a Mel spectrum with a dimension of 80×Ttime through a deconvolution operation (80 is the number of Mel bands, and Ttime is the time step, which is determined by the length of the text to be synthesized). This spectrum is the final generated Mel spectrum, which can be directly used for subsequent speech waveform conversion.

[0092] Mel spectrum quality verification: After generating the Mel spectrum, the quality is ensured through "spectral feature verification". Semantic consistency verification: Calculate the semantic similarity between the speech content corresponding to the Mel spectrum and the text to be synthesized (using the cosine similarity algorithm). If the similarity is less than 0.8, the spectral semantic deviation is determined, and the denoising process is re-executed. Scene adaptation verification: Extract background sound features (such as noise intensity and reverberation parameters) from the Mel spectrum and compare them with the corresponding parameters in the environmental background sound feature embedding vector. If the deviation exceeds a preset threshold (such as 15%), the diffusion model parameters are readjusted and generated. Phonological matching verification: The fundamental frequency, formants, and other timbre features in the Mel spectrum are analyzed and matched with corresponding indicators in the speaker's timbre features. If the matching degree is less than 0.85, the phonological match is regenerated. The Mel spectrum that passes the verification will proceed to the next step, "speech waveform conversion".

[0093] In other words, by mapping the 2816-dimensional multi-source feature concatenation vector to a 512-dimensional unified feature through a linear layer, the input conflict caused by the dimensional differences between "environmental background sound features (2048-dimensional), text semantic features (512-dimensional), and speaker timbre features (256-dimensional)" is resolved. Simultaneously, key information is reinforced through weight learning, ensuring that the mapped features simultaneously highlight the core attributes of scene, content, and timbre. For example, in a medical scenario, the mapping result can prioritize retaining "key information from the medical order text (such as medication dosage)," "doctor's timbre features," and "low-noise features of the ward environment," providing accurate collaborative input for subsequent diffusion models. The feature collaboration efficiency is significantly improved compared to schemes without linear layer mapping. Furthermore, the linear layer adopts a lightweight architecture of "single hidden layer + output layer," with a parameter scale only 1 / 5 that of traditional multimodal fusion networks (such as Transformer fusion layers), significantly reducing computational cost while ensuring feature mapping accuracy. Furthermore, the diffusion model, through a multi-task loss function and a progressive denoising process, deeply integrates three types of features—environment, text, and timbre—into the Mel spectrum, solving the problems of "disconnect between scene and content" and "mismatch between timbre and content" in existing technologies. For example, in a financial scenario, the generated Mel spectrum can simultaneously carry "background sound features of a bank lobby (such as the spectral distribution of conversations among people)," "semantic features of transaction announcement text (such as the spectral intensity enhancement of the numerical part)," and "customer service-specific timbre features (such as a specific fundamental frequency range)." After subsequent conversion to speech, it can achieve a triple effect of "scene adaptation, accurate content, and personalized timbre," significantly improving multi-dimensional adaptability compared to the spectrum generated by traditional TTS models.

[0094] S160. Convert the Mel spectrum into the target speech waveform to obtain synthesized speech data.

[0095] Specifically, a wavelet transform-based vocoder (such as WaveNet) is used to convert the Mel spectrum to a speech waveform: Spectrum preprocessing: The generated Mel spectrum is normalized to eliminate the distribution differences between different batches of data and ensure stable vocoder input.

[0096] Waveform generation: The preprocessed Mel spectrum is input into the vocoder. The vocoder learns the mapping relationship between the Mel spectrum and the speech waveform to generate a target speech waveform with a sampling rate of 22.05kHz. This waveform is the final synthesized speech data, which can be played directly or stored in audio formats such as WAV.

[0097] In one embodiment, converting the Mel spectrum into a target speech waveform to obtain synthetic speech data includes: Use a vocoder to convert the Mel spectrum into the target speech waveform; Specifically, a WaveNet vocoder based on wavelet transform was selected and pre-trained and optimized using a dedicated speech dataset for the financial and healthcare fields (containing over 3000 high-quality speech samples, covering scenarios such as financial customer service announcements and medical prescription readings, with a uniform sampling rate of 22.05kHz). The training process aimed to minimize the mapping error between the Mel spectrum and the speech waveform, employing a joint optimization of the mean squared error (MSE) loss function and the perceptual loss function: the MSE loss ensures the numerical matching degree between the spectrum and the waveform, while the perceptual loss (based on the pre-trained speech perception model) guarantees the subjective listening quality of the generated speech, especially enhancing the ability to restore key speech details in the domain scenarios (such as clear announcements of financial figures and accurate pronunciation of medical drug dosages).

[0098] The Mel spectrum generated by the acoustic model (dimension 80×T, where 80 is the number of Mel frequency bands and T is the time step) is preprocessed for input adaptation: Dimensional calibration: If the time step T of the Mel spectrum exceeds the input limit of the vocoder due to the difference in text length (e.g., the maximum processing time for a single segment is 10 seconds), then it is segmented according to the time dimension (each segment corresponds to a short sentence with complete text semantics, such as "account balance of 15,000 yuan" in a financial broadcast) to ensure that each segment of the spectrum can be independently input into the vocoder. Spectrum normalization: Maps the numerical range of the Mel spectrum to the input interval (e.g., [-4, 4]) during vocoder pre-training. This eliminates the differences in numerical distribution of spectra from different batches through linear transformation, thus avoiding waveform distortion caused by abnormal spectrum amplitude.

[0099] Input the preprocessed Mel spectrum into the pre-trained WaveNet vocoder and perform the following conversion steps: Step 1: Feature Mapping. The vocoder maps the 80-dimensional Mel-band features into a 256-dimensional high-dimensional feature vector through a convolutional layer, providing a rich feature base for subsequent waveform generation; Step 2: Temporal modeling. Stacked causal convolutional layers and gated activation units are used to capture the temporal correlation of the Mel spectrum (such as the smooth transition of the spectrum between adjacent frames) to ensure the continuity of the generated waveform and avoid "drop-offs" or "stutters". Step 3: Waveform Output. The features after time-series modeling are converted into a mono audio waveform (i.e., target audio waveform) with a sampling rate of 22.05kHz through a linear output layer. Each spectrum segment corresponds to a complete waveform. If there is segmentation, the segmentation traces are eliminated after waveform generation using the "frame smooth splicing" technique (linear fade-in and fade-out are used in overlapping areas) to ensure the overall continuity of the speech.

[0100] The target speech waveform is subjected to loudness normalization to obtain synthesized speech data.

[0101] Specifically, the core objective of loudness normalization is to eliminate volume fluctuations in generated speech under different scenarios and modal inputs, ensuring that the synthesized speech data is stable and clearly audible in practical applications in key fields such as finance and healthcare (e.g., outdoor financial broadcasting, indoor medical consultations). The specific implementation steps are as follows: The loudness calculation model recommended by the International Organization for Standardization (ISO) (such as ISO 532-1) is used to perform frame-level loudness calculation on the target speech waveform: Frame segmentation: The speech waveform is divided into a frame sequence with a frame length of 20ms and a frame shift of 10ms, and the loudness value of each frame (unit: sone) is calculated. Baseline loudness determination: The baseline loudness range is set based on the application scenario of the field—the baseline loudness for outdoor broadcasting in the financial field is 60-70 sone (to ensure that it can be heard clearly in noisy environments), and the baseline loudness for indoor consultation in the medical field is 50-60 sone (to avoid excessive volume that may cause discomfort to patients). The baseline range can be adjusted according to the actual scenario through the system configuration interface.

[0102] Based on the calculated frame-level loudness value and the baseline loudness range, perform adaptive loudness adjustment: Global gain calculation: If the overall loudness of the speech is lower than the lower limit of the benchmark range (e.g., lower than 50 sone in medical scenarios), then calculate the global gain coefficient (e.g., gain = 50 / current average) and adjust the overall volume by linear amplification; if the overall loudness is higher than the upper limit of the benchmark range (e.g., higher than 70 sone in financial scenarios), then calculate the attenuation coefficient (e.g., attenuation = 70 / current average) to avoid auditory fatigue caused by excessive volume; Local loudness optimization: For local frames where loudness fluctuations exceed ±5sone (such as a sudden increase in loudness due to background noise in a certain frame), dynamic range compression technology is used to control the local loudness within the baseline range, while preserving the dynamic details of the speech (such as changes in the speaker's intonation), avoiding the harshness of the speech caused by a "one-size-fits-all" adjustment.

[0103] After loudness adjustment, the target speech waveform is standardized: the waveform data is converted to WAV format (uncompressed to ensure lossless sound quality), the sampling rate is kept at 22.05kHz, the bit depth is set to 16bit, and finally synthesized speech data that can be directly used is generated, which supports real-time playback or storage for later use.

[0104] The aforementioned speech synthesis method incorporates text description data, environmental audio data, and environmental image data into a multimodal background sound conditional input system. This enables the complementarity and synergy of multi-dimensional information across semantics, acoustics, and vision. Under the synergistic effect of these three elements, the generated environmental background sound feature embedding vector can deeply match the scenario requirements of key fields such as finance and healthcare. It not only covers diverse scenario types but also achieves precise matching between background sound and speech semantics, solving the problems of low scenario adaptation accuracy and semantic disconnect in existing technologies. Furthermore, by collaboratively inputting the environmental background sound feature embedding vector with text semantic features and speaker timbre features into the acoustic model, "end-to-end scenario-based speech generation" is achieved. On the one hand, the model can dynamically adjust the volume and tone of the speech based on environmental background sound features during the speech generation process, making the synthesized speech naturally match the background sound and avoiding a "disjointed" feeling. On the other hand, the model can automatically reduce the background sound intensity of corresponding time periods based on text semantic features while maintaining the scenario-based background sound effect of non-critical information segments, fundamentally avoiding the risk of speech masking.

[0105] Figure 3 This is a schematic block diagram of a speech synthesis device 300 provided in an embodiment of the present invention. Figure 3 As shown, corresponding to the above speech synthesis method, the present invention also provides a speech synthesis apparatus 300. This speech synthesis apparatus 300 includes a unit for performing the above speech synthesis method, and the apparatus can be configured in a server. Specifically, please refer to... Figure 3 The speech synthesis device 300 includes: The acquisition unit 301 is used to acquire multimodal background sound conditional input data, wherein the multimodal background sound conditional input data includes at least one of text description data, environmental audio data, and environmental image data; The detection unit 302 is used to perform modal integrity detection on the multimodal background sound conditional input data to obtain the detection result; The generation unit 303 is used to generate an environmental background sound feature embedding vector based on the detection results; The extraction unit 304 is used to acquire the text data to be synthesized and the speaker reference audio data, and to extract features from the text data to be synthesized and the speaker reference audio data respectively, so as to obtain the text semantic features and the speaker timbre features. The input generation unit 305 is used to input the environmental background sound feature embedding vector, text semantic features and speaker timbre features into the acoustic model to generate the Mel spectrum; The conversion unit 306 is used to convert the Mel spectrum into a target speech waveform to obtain synthesized speech data.

[0106] It should be noted that those skilled in the art can clearly understand that the specific implementation process of the above-mentioned speech synthesis device 300 and each unit can be referred to the corresponding description in the foregoing method embodiments. For the sake of convenience and brevity, it will not be repeated here.

[0107] The aforementioned speech synthesis device 300 can be implemented as a computer program, which can, for example... Figure 4 It runs on the computer device shown.

[0108] Please see Figure 4 , Figure 4 This is a schematic block diagram of a computer device provided in an embodiment of this application. The computer device 500 can be a server, wherein the server can be a standalone server or a server cluster composed of multiple servers.

[0109] See Figure 4 The computer device 500 includes a processor 502, a memory, and a network interface 505 connected via a system bus 501. The memory may include a non-volatile storage medium 503 and internal memory 504.

[0110] The non-volatile storage medium 503 may store an operating system 5031 and a computer program 5032. The computer program 5032 includes program instructions that, when executed, cause the processor 502 to perform a speech synthesis method.

[0111] The processor 502 provides computing and control capabilities to support the operation of the entire computer device 500.

[0112] The internal memory 504 provides an environment for the operation of the computer program 5032 in the non-volatile storage medium 503. When the computer program 5032 is executed by the processor 502, the processor 502 can execute a speech synthesis method.

[0113] This network interface 505 is used for network communication with other devices. Those skilled in the art will understand that... Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device 500 to which the present application is applied. The specific computer device 500 may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] The processor 502 is used to run a computer program 5032 stored in the memory to implement the above method. It should be understood that, in this embodiment, the processor 502 may be a Central Processing Unit (CPU).

[0115] It will be understood by those skilled in the art that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program includes program instructions and can be stored in a storage medium, which is a computer-readable storage medium. The program instructions are executed by at least one processor in the computer system to implement the process steps of the embodiments of the above methods.

[0116] Therefore, the present invention also provides a storage medium. This storage medium can be a computer-readable storage medium. The storage medium stores a computer program, wherein when executed by a processor, the computer program causes the processor to perform the methods described above.

[0117] The storage medium can be any computer-readable storage medium capable of storing program code, such as a USB flash drive, portable hard drive, read-only memory (ROM), magnetic disk, or optical disk.

[0118] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A speech synthesis method, characterized in that, include: Acquire multimodal background noise conditional input data, wherein the multimodal background noise conditional input data includes at least one of text description data, environmental audio data, and environmental image data; Modal integrity detection is performed on the multimodal background sound conditional input data to obtain the detection results; Generate an environmental background sound feature embedding vector based on the detection results; The text data to be synthesized and the speaker reference audio data are obtained, and feature extraction is performed on the text data to be synthesized and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features; The ambient background sound features, text semantic features, and speaker timbre features are input into the acoustic model to generate the Mel spectrum; The Mel spectrum is converted into the target speech waveform to obtain the synthesized speech data.

2. The speech synthesis method according to claim 1, characterized in that, The modal integrity detection of the multimodal background sound conditional input data to obtain the detection result includes: Automatically classify multimodal background sound conditional input data to separate text description data blocks, environmental audio data blocks, and environmental image data blocks; The text description data block, environmental audio data block, and environmental image data block are format-validated according to the preset modal integrity rules to generate a validation status. The detection results are generated based on the verification status.

3. The speech synthesis method according to claim 1, characterized in that, The step of generating an environmental background sound feature embedding vector based on the detection results includes: If the detection result indicates that no modality is missing, feature representations of text description data, environmental audio data, and environmental image data are extracted separately by a modality-specific encoder to obtain modality-specific features. At the same time, the text description data, environmental audio data, and environmental image data are fed into a shared encoder for unified representation learning to obtain modality-shared features. The modality-specific features and modality-shared features are then concatenated to generate an environmental background sound feature embedding vector. If the detection result indicates the presence of a missing modality, features of the existing modality data are extracted using a modality-specific encoder. These features are then input into a cross-modality mapping module to infer the semantic features of the missing modality. Subsequently, the features of the existing modality data and the semantic features of the missing modality are fed into a shared encoder for unified representation learning to obtain the unified representation learning result. Finally, the features of the existing modality data, the semantic features of the missing modality, and the unified representation learning result are concatenated to generate an environmental background sound feature embedding vector.

4. The speech synthesis method according to claim 3, characterized in that, If the detection result indicates that no modality is missing, feature representations of text description data, environmental audio data, and environmental image data are extracted separately by a modality-specific encoder to obtain modality-specific features; at the same time, text description data, environmental audio data, and environmental image data are fed into a shared encoder for unified representation learning to obtain modality-shared features. Then, modality-specific features and modality-shared features are concatenated to generate an ambient background sound feature embedding vector, including: The modality-specific encoder includes a text encoder, an audio encoder, and an image encoder, which respectively output text semantic features, acoustic style features, and visual semantic features; The shared encoder adopts a fully connected neural network structure to perform unified dimensional mapping on text description data, environmental audio data, and environmental image data in order to output modal shared features. The text semantic features, acoustic style features, visual semantic features, and modal sharing features are concatenated into vectors and then fused through a linear layer to generate an environmental background sound feature embedding vector.

5. The speech synthesis method according to claim 1, characterized in that, The process of acquiring the text data to be synthesized and the speaker's reference audio data, and extracting features from the text data to be synthesized and the speaker's reference audio data respectively to obtain text semantic features and speaker timbre features, includes: A pre-trained language model is used to extract context encoding from the text data to be synthesized, generating text semantic feature vectors. The speaker reference audio data is processed by a speaker audio encoder, which is based on a VAE architecture, to extract the latent feature representation of the speaker's timbre in order to obtain the speaker's timbre features.

6. The speech synthesis method according to claim 1, characterized in that, The step of inputting environmental background sound feature embedding vectors, text semantic features, and speaker timbre features into the acoustic model to generate a Mel spectrum includes: The environmental background sound feature embedding vector, text semantic features, and speaker timbre features are input into a linear layer for mapping to obtain the mapping result; The mapping result is input into the diffusion model, which generates a Mel spectrum that matches the mapping result through a stepwise denoising process.

7. The speech synthesis method according to claim 1, characterized in that, The process of converting the Mel spectrum into a target speech waveform to obtain synthesized speech data includes: Use a vocoder to convert the Mel spectrum into the target speech waveform; The target speech waveform is subjected to loudness normalization to obtain synthesized speech data.

8. A speech synthesis device, characterized in that, include: The acquisition unit is used to acquire multimodal background sound conditional input data, wherein the multimodal background sound conditional input data includes at least one of text description data, environmental audio data, and environmental image data; The detection unit is used to perform modal integrity detection on the multimodal background sound conditional input data to obtain the detection result; The generation unit is used to generate an environmental background sound feature embedding vector based on the detection results; The acquisition and extraction unit is used to acquire the text data to be synthesized and the speaker reference audio data, and to extract features from the text data to be synthesized and the speaker reference audio data respectively to obtain text semantic features and speaker timbre features; The input generation unit is used to input the environmental background sound feature embedding vector, text semantic features and speaker timbre features into the acoustic model to generate the Mel spectrum; The conversion unit is used to convert the Mel spectrum into the target speech waveform to obtain synthesized speech data.

9. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method as described in any one of claims 1 to 7.

10. A storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images