Voice stream generation method and device, equipment, storage medium and program product

Abstract

The invention provides a voice stream generation method and device, equipment, a storage medium and a program product, and relates to the technical field of computers. The method comprises the following steps: generating a target image feature vector based on an initial target image graph; generating a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector through a mapping model; and through a speech synthesis model, generating a speech stream based on the predicted timbre feature vector and the text information of the speech to be generated. The method can improve the matching degree of the tone and the target image.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method, apparatus, device, storage medium, and program product for generating speech streams. Background Technology

[0002] Speech synthesis technology is a technique that converts input text into natural, human-understandable speech. However, existing voice customization solutions struggle to guarantee a high degree of matching between the voice and the person. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, storage medium, and program product for generating speech streams.

[0004] In a first aspect, embodiments of this disclosure propose a speech stream generation method, comprising: generating a target image feature vector based on an initial target image image; generating a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector through a mapping model; and generating a speech stream based on the predicted timbre feature vector and the text information of the speech to be generated through a speech synthesis model.

[0005] In one possible implementation, the above-mentioned generation of the predicted timbre feature vector corresponding to the target image feature vector through the mapping model may include: when the output attributes of the mapping model are inconsistent with the input requirements of the speech synthesis model, adding a recalibration layer at the end of the mapping model to obtain a new mapping model; and generating the predicted timbre feature vector based on the target image feature vector through the new mapping model.

[0006] In one possible implementation, the above mapping model is trained based on vector sample pairs including target image feature vector samples and timbre feature vector samples.

[0007] In one possible implementation, the step of obtaining vector sample pairs includes: obtaining audio slices based on video samples; obtaining training sample pairs based on the audio slices and the corresponding video slices; wherein the training sample pairs include standard target image samples and audio slices corresponding to the standard target image samples, and the standard target image samples are related to a video frame in the video slices; and generating vector sample pairs based on the training sample pairs.

[0008] In one possible implementation, obtaining training sample pairs based on audio slices and corresponding video slices includes: detecting images of video slices to obtain detection results corresponding to the images; obtaining a first target image image that satisfies the image validity rule from the images of video slices based on the image validity rule and the detection results corresponding to the images; generating standard target image image samples corresponding to the first target image image; and obtaining training sample pairs based on the standard target image image samples and the audio slices corresponding to the video slices.

[0009] In one possible implementation, the above-mentioned generation of vector sample pairs based on training sample pairs includes: generating target image feature vector samples based on standard target image sample using an image encoding model; and generating timbre feature vector samples based on audio slices corresponding to standard target image sample using a speaker encoding model.

[0010] In one possible implementation, the above-mentioned acquisition of audio slices based on video samples includes: obtaining audio samples based on video samples; obtaining timestamps of audio activity segments based on audio samples; obtaining video slices corresponding to the timestamps based on the timestamps and video samples; and acquiring audio slices based on video slices.

[0011] In one possible implementation, the output attributes of the mapping model include at least one of the following: data distribution, dimension, mean requirement, variance, and scaling range, and the input requirements of the speech synthesis model include at least one of the following: data distribution requirement information, dimension requirement information, mean requirement information, variance requirement information, and scaling range requirement information.

[0012] In one possible implementation, the above-mentioned generation of target image feature vector based on initial target image may include: processing the initial target image based on a preset resolution and the key point coordinates of the initial target image to generate a standard target image; and generating a target image feature vector based on the standard target image.

[0013] In one possible implementation, the above-mentioned generation of target image feature vector based on standard target image image includes: obtaining a high-dimensional feature vector by forward reasoning based on standard target image image through image encoding model; and normalizing the high-dimensional feature vector through image encoding model to obtain target image feature vector.

[0014] Secondly, embodiments of this disclosure propose a speech stream generation apparatus, comprising: a first generation module, a second generation module, and a third generation module. The first generation module is configured to generate a target image feature vector based on an initial target image image. The second generation module is configured to generate a predicted timbre feature vector corresponding to the target image feature vector using a mapping model. The third generation module is configured to generate a speech stream using a speech synthesis model based on the predicted timbre feature vector and text information of the speech to be generated.

[0015] In one possible implementation, the second generation module is further configured to: add a recalibration layer at the end of the mapping model to obtain a new mapping model when the output attributes of the mapping model are inconsistent with the input requirements of the speech synthesis model; and generate a predicted timbre feature vector based on the target image feature vector using the new mapping model.

[0016] In one possible implementation, the mapping model is trained on a pair of vector samples, including target image feature vector samples and timbre feature vector samples.

[0017] In one possible implementation, the speech stream generation device may further include: a first acquisition module, a second acquisition module, and a fourth generation module. The first acquisition module is configured to acquire audio slices based on video samples. The second acquisition module is configured to obtain training sample pairs based on the audio slices and their corresponding video slices; wherein the training sample pairs include standard target image samples and audio slices corresponding to the standard target image samples, and the standard target image samples are related to a frame of video in the video slices. The fourth generation module is configured to generate vector sample pairs based on the training sample pairs.

[0018] In one possible implementation, the second acquisition module is further configured to: detect the image of the video slice to obtain a detection result corresponding to the image; based on the image validity rule and the detection result corresponding to the image, obtain a first target image image that satisfies the image validity rule from the image of the video slice; generate a standard target image image sample corresponding to the first target image image; and obtain a training sample pair based on the standard target image image sample and the audio slice corresponding to the video slice.

[0019] In one possible implementation, the fourth generation module is further configured to: generate target image feature vector samples based on standard target image samples using an image coding model; and generate timbre feature vector samples based on audio slices corresponding to standard target image samples using a speaker coding model.

[0020] In one possible implementation, the first acquisition module is further configured to: obtain an audio sample based on a video sample; obtain a timestamp of an audio activity segment based on the audio sample; obtain a video slice corresponding to the timestamp based on the timestamp and the video sample; and obtain an audio slice based on the video slice.

[0021] In one possible implementation, the output attributes of the mapping model include at least one of the following: data distribution, dimension, mean requirement, variance, and scaling range, and the input requirements of the speech synthesis model include at least one of the following: data distribution requirement information, dimension requirement information, mean requirement information, variance requirement information, and scaling range requirement information.

[0022] In one possible implementation, the first generation module is further configured to: process the initial target image based on a preset resolution and the key point coordinates of the initial target image to generate a standard target image; and generate a target image feature vector based on the standard target image.

[0023] In one possible implementation, the first generation module is further configured to: obtain a high-dimensional feature vector by forward reasoning based on a standard target image image through an image encoding model; and normalize the high-dimensional feature vector through the image encoding model to obtain the target image feature vector.

[0024] According to a third aspect of this disclosure, a chip is provided that includes a processor, which can be used to implement the methods of the first aspect and any possible implementation thereof.

[0025] According to a fourth aspect of this disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions.

[0026] The processor is configured to implement the method of the first aspect and any possible implementation thereof when executing instructions stored in memory.

[0027] According to a fifth aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, which, when executed by a processor, implement the methods of the first aspect and any possible implementation thereof.

[0028] According to a sixth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the steps of the method described in the first aspect and any possible implementation thereof.

[0029] The speech stream generation method, apparatus, chip, electronic device, computer-readable storage medium, and computer program product provided in this disclosure can improve the matching degree between voice timbre and target image.

[0030] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0031] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0032] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is an exemplary system architecture to which this disclosure can be applied; Figure 2 A flowchart of a speech stream generation method provided in this disclosure embodiment; Figure 3 A flowchart of a method for obtaining vector sample pairs provided in an embodiment of this disclosure; Figure 4 A structural block diagram of a speech stream generation device provided in an embodiment of this disclosure; Figure 5 This is a schematic diagram of the structure of an electronic device suitable for performing a speech stream generation method, provided as an embodiment of the present disclosure. Detailed Implementation

[0033] To better understand this application, various aspects of this application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are merely illustrative of exemplary embodiments of this application and are not intended to limit the scope of this application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0034] It should be understood that expressions such as "comprising," "including," "having," "containing," and / or "comprising" are open-ended rather than closed-ended expressions in this specification, indicating the presence of the stated features, elements, and / or components, but not excluding the presence of one or more other features, elements, components, and / or combinations thereof. Furthermore, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features, not just individual elements in the list. Additionally, when describing embodiments of this application, the word "may" is used to mean "one or more embodiments of this application." And the term "exemplary" is intended to refer to examples or illustrations.

[0035] Unless otherwise specified, all terms used herein (including engineering and technical terms) shall have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that, unless expressly stated herein, terms defined in common dictionaries shall be interpreted as having the meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or overly formalized meaning.

[0036] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. Furthermore, unless explicitly limited or contradicted by the context, the specific steps included in the methods described in this application are not limited to the order in which they are described, but can be performed in any order or in parallel. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] Figure 1 An exemplary system architecture 100 is shown, in which embodiments of the speech stream generation methods, apparatuses, chips, electronic devices, computer-readable storage media, and computer program products of the present disclosure can be applied.

[0038] like Figure 1 As shown, system architecture 100 may include a first device 101 and a second device 102, which can be connected via a network. The network serves as the medium for providing a communication link between the first device 101 and the second device 102. The network may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0039] The first device 101 and the second device 102 can be either hardware or software. When the first device 101 and the second device 102 are hardware, the first device 101 can be various electronic devices, including but not limited to mobile phones, computers, tablets, smart robots, smart home terminals, smart terminals, in-vehicle devices, components on various devices, etc. The second device 102 can be a cloud server, a server, etc., where the server can also be a server for a distributed system or a server combined with blockchain. A cloud server, also known as a cloud computing server or cloud host, is a host product in the cloud computing service system, designed to address the shortcomings of traditional physical hosts and Virtual Private Server (VPS) services, such as high management difficulty and weak business scalability. When the first device 101 and the second device 102 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software programs or software modules, or as a single software program or software module; no specific limitations are made here.

[0040] It should be understood that Figure 1 The number of first and second devices shown is merely illustrative. Any number of first and second devices can be used depending on implementation requirements.

[0041] The method provided in this disclosure is applicable to scenarios requiring customized voice timbres, such as 2D / 3D digital humans, virtual anchors, non-player characters (NPCs) in games, and animated characters. The specific technical implementation or deployment method of the method provided in this disclosure is not limited; for example, it can be deployed via a web platform, integrated with a software development kit (SDK), deployed privately, deployed as a plug-in edge, or deployed on a hardware platform.

[0042] Some custom timbre solutions rely on real speech as a timbre reference and lack the ability to directly generate usable timbres from images for embedding. This results in a lack of direct mapping mechanism between images and timbres, making it difficult to guarantee the consistency and matching degree between sound and image. Furthermore, it requires finding real human recordings as an intermediate step, which is time-consuming and costly.

[0043] For example, early speech synthesis often employed rule-based concatenative synthesis or statistical modeling methods based on acoustic parameters, such as Hidden Markov Model (HMM) vocoders and Gaussian Mixture Model (GMM) vocoders. Concatenation methods rely on pre-recorded, large-scale speech libraries, matching and concatenating phoneme units corresponding to the text with audio segments; statistical modeling methods generate synthesized waveforms by modeling the acoustic features of the recordings (such as fundamental frequency F0, spectral envelope, and duration). While these approaches offer strong controllability, they limit the naturalness and expressiveness of the timbre, making it difficult to support diverse timbre customization.

[0044] For example, the Tacotron series, FastSpeech series, and end-to-end text-to-speech (VITS) models based on variational inference and adversarial learning directly predict Mel spectrum from text sequences and then generate high-fidelity audio through neural vocoders (such as WaveNet and HiFi-GAN).

[0045] For example, TTS techniques that combine large-scale pre-trained speech models (such as VALL-E, NaturalSpeech 2, ChatTTS, etc.) can learn the timbre and speaking style of thousands or even tens of thousands of speakers in a unified generative model, directly controlling the timbre through speaker embedding or conditional prompts. These large models often have the ability to replicate timbre with few samples or even clone a single sentence, that is, they can generate a synthetic timbre that is highly consistent with the target speaker with only a very small amount of reference speech.

[0046] In the aforementioned multi-speaker TTS examples, voice customization typically requires capturing tens of minutes or even hours of high-quality recordings of the target speaker, followed by fine-tuning of the single-speaker model or training the speaker's independent speaker embedding in the multi-speaker model. Such approaches are costly, time-consuming, and highly dependent on recording quality, resulting in low efficiency.

[0047] In large-scale model-based voice customization solutions, large-scale model TTS leverages the pre-training capabilities of massive amounts of multi-speaker speech data to generate corresponding speaker embeddings during the inference stage using only a small amount of reference audio (even a single sentence), achieving rapid voice replication. However, this voice customization solution faces the following challenges: data acquisition is difficult, requiring additional time and organizational costs to collect high-quality speech samples for specific roles or users; recording quality issues: audio collected in non-recording studio environments often suffers from background noise, reverberation, microphone distortion, and other problems, leading to unstable speaker embeddings and degraded synthesized voices; and high costs associated with human involvement, as finding suitable human voice actors and completing recording, review, and correction requires a lengthy process.

[0048] The method provided in this disclosure does not require real human participation in voice recording. It can directly generate high-quality timbre embeddings from images that can be used for large-scale TTS models, thereby achieving "creation from nothing" timbre customization. This can significantly shorten the timbre customization cycle, reduce labor costs, and the matching process of quantifying the character image into the target image feature vector is determined by the mapping model, without the involvement of subjective judgment. This can ensure that the matching between the timbre and the target image conforms to common understanding, improve the matching degree between the timbre and the target image, and enhance the objectivity and consistency of the matching between the timbre and the character image.

[0049] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information, video data, and facial recognition-related information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0050] Please refer to Figure 2 , Figure 2 The flowchart of a speech stream generation method provided in this embodiment of the disclosure is executed by a first device 101 or a second device 102, wherein process 200 includes the following steps.

[0051] Step 201: Generate target image feature vector based on the initial target image image.

[0052] For example, the initial target image may be an initial picture that includes one or more target images. For instance, the target images may include a face, a real person, a human figure, an animal figure, a virtual person, a character illustration, and / or a cartoon character.

[0053] For example, the target image feature vector can be a feature vector related to the target image. The target image feature vector may include core visual features, such as the proportions and contour features of a human face, and the shape and texture features of an object.

[0054] For example, when the initial target image includes multiple target images, the generated target image feature vectors can also be multiple, and the target image feature vectors are related to the initial target image. Figure 1 One-to-one correspondence.

[0055] Optionally, before generating the target image feature vector, the initial target image can be preprocessed to ensure that the preprocessed initial target image meets the input requirements for subsequent feature vector extraction, thus avoiding inaccurate feature extraction due to differences in size, brightness, etc.

[0056] Optionally, the initial target image can be a local file or obtained via a Uniform Resource Locator (URL).

[0057] Optionally, the target image feature vector can be a fixed-length vector, such as a 512-dimensional or 1024-dimensional sequence of numbers. In this way, the dimension of the target image feature vector is fixed, and the length of the output vector is the same regardless of the size of the input image. Moreover, it can uniquely represent the target image, preparing for subsequent operations.

[0058] Step 202: Using a mapping model, generate a predicted timbre feature vector corresponding to the target image feature vector.

[0059] For example, the predicted timbre feature vector can be an auditory feature vector that encodes the timbre related to the target image.

[0060] For example, a mapping model generates a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector, thus binding the image and timbre. For instance, if a person's facial feature vector is input, the mapping model will output a vector that mimics that person's timbre.

[0061] For example, the predicted timbre feature vectors meet the input requirements of the speech synthesis model. For instance, the dimension, scale, distribution, and format of the predicted timbre feature vectors perfectly match the preset requirements of the speech synthesis model. Without any additional modification to the vectors themselves or the speech synthesis model, they can be directly used as input by the speech synthesis model to drive the model to generate the expected speech, without the need for additional adaptation development.

[0062] For example, if the initial target image map includes multiple target images, the number of predicted timbre feature vectors will also be multiple, and speech from multiple images can be generated subsequently.

[0063] Step 203: Generate a speech stream using a speech synthesis model based on the predicted timbre feature vector and the text information of the speech to be generated.

[0064] Exemplarily, the text information for which speech is to be generated may include the text content to be synthesized into speech, such as "The weather is nice today", or may also be a preprocessed phoneme sequence or syllable sequence, such as the phonemes corresponding to "j n ti n ti n qì bù cuò".

[0065] Optionally, preprocessing the text information may include: normalizing the input text information, for example, expanding numbers, dates, and abbreviations, such as expanding "2025" to "two thousand and twenty-five", eliminating text format ambiguities, and may also include: performing grapheme-to-phoneme (G2P) conversion or pinyinization according to the target language, and finally outputting a phoneme sequence or syllable sequence recognizable by the speech synthesis model, without the need to develop an additional front-end module. <00's content is not clear. Please provide more context or clarify the question.

[0066] Optionally, the text information may also include style parameters, which may include: speech rate adjustment, emotion setting, fundamental frequency / energy control, or prompt word guidance (if supported by the speech synthesis model).

[0067] Optionally, for the inference sampling parameters of the speech synthesis model, temperature, Top-p, and / or Top-k may adopt the default robust values of the service. If it is necessary to ensure that the synthesis results are consistent multiple times, the random seed can be fixed to achieve precise reproduction of the synthesis effect.

[0068] Optionally, the speech synthesis model may also be referred to as a Text To Speech (TTS) model.

[0069] For example, the version of the speech synthesis model during the inference process may be consistent with the training period and the calibration period.

[0070] For example, the text information may be compatible with multi-language input to meet the speech synthesis requirements of different scenarios.

[0071] Exemplarily, by combining the predicted timbre feature vectors and the text information, a speech stream of a specified timbre audible to the human ear is output through the operation of the speech synthesis model, completing voice generation based on "appearance".

[0072] Exemplarily, when the number of target images included in the initial target image is multiple, multiple speech streams can be generated through the speech synthesis model based on multiple predicted timbre feature vectors and multiple text information for which speech is to be generated. The multiple text information for which speech is to be generated respectively corresponds to different target images. In this way, the timbres of multiple characters can be quickly and batch-generated, realizing large-scale and low-latency timbre production capabilities.

[0073] For example, the predicted timbre feature vector can be injected into the speech synthesis model in any of the following ways: For instance, the direct vector conditional approach: the predicted timbre feature vector (voice_embedding_hat) is used as a speaker conditional vector and directly concatenated into the input sequence of the speech synthesis model (e.g., concatenated with the text phoneme sequence), or it can be input into the speech synthesis model as an independent additional channel. Another example is the side attention approach: the predicted timbre feature vector is first mapped to several conditional keys and values ​​through a linear layer, and then these keys and values ​​are connected to the cross-attention module as reference conditions for attention calculation. Yet another example is the pseudo-speaker token approach: the predicted timbre feature vector is first converted into 1 to k speaker feature tokens through a projection layer and concatenated at the beginning of the text / phoneme sequence as part of the model input.

[0074] For example, the paths described in Examples 1 and 2 below can be used to generate a speech stream.

[0075] Example 1 uses the predicted timbre feature vector and the text information of the speech to be generated as input. A large language model (LLM) generates discrete acoustic / codec tokens (such as EnCodec or SoundStream formats). A neural codec matching the token format is then invoked to convert the discrete tokens into a continuous speech waveform. AudioLM and / or WaveTokenizer models can be used. This approach is suitable for scenarios requiring end-to-end control and support for streaming generation, such as real-time voice interaction and live virtual human speech.

[0076] Example 2: The acoustic model uses the predicted timbre feature vector as the speaker feature modulation condition, predicts the Mel-spectrogram based on the text information of the speech to be generated, and inputs the Mel-spectrogram into a neural vocoder (such as HiFi-GAN, WaveRNN, or BigVGAN) to synthesize a time-domain speech waveform. This approach is applicable to scenarios with existing mature production chains, and each component (acoustic model, vocoder) can be independently replaced and upgraded, offering high flexibility. For example, the acoustic model can include FastSpeech, VITS, and / or NaturalSpeech models.

[0077] For example, this disclosure does not limit the specific scheme for generating speech streams based on speech synthesis models, such as the TTS backbone, vocoder, or sampling strategy. For example, the sampling rate of the generated speech stream can be 24kHz or 48kHz, the bit depth and encoding can be 16-bit Pulse Code Modulation (PCM), and the format of the speech stream can be Waveform Audio File Format (WAV), Ogg Vorbis (OGG), or MP3.

[0078] Optionally, the aforementioned execution entity may also generate metadata about the speech stream. For example, the metadata may include at least one of the following: the version of the neural codec, the version of the mapping model, the version of the speech synthesis model, inference sampling parameters, and the quality score of the speech stream.

[0079] The method provided in this disclosure only requires an initial target image image to quickly generate a matching predicted timbre feature vector. Through a speech synthesis model, a speech stream of the target image is generated based on the predicted timbre feature vector and the text information of the speech to be generated. No real person is required to participate in the speech recording, which can significantly shorten the customization cycle of timbre and reduce labor costs. Furthermore, the matching process of quantifying the character image into the target image feature vector is determined by the mapping model, without the involvement of subjective judgment. This can ensure that the matching of timbre and target image conforms to common understanding, improve the matching degree of timbre and target image, and enhance the objectivity and consistency of timbre and character image matching.

[0080] In some embodiments, step 201 above, which generates a target image feature vector based on the initial target image image, may include steps A1-A2.

[0081] Step A1: Process the initial target image based on the preset resolution and the key point coordinates of the initial target image to generate a standard target image.

[0082] For example, by setting the resolution and key point coordinates, the size, posture, and position differences of the initial target image are eliminated, and a standard target image with uniform specifications and aligned core features is generated, making subsequent feature extraction and cross-modal mapping more accurate and efficient.

[0083] Optionally, step A1 above may include: processing the initial target image image based on a preset resolution and the key point coordinates of the initial target image image using an image detection model to generate a standard target image image.

[0084] For example, the image detection model can be the RetinaFace algorithm or the YOLO-Face algorithm. The image detection model can be consistent with the image encoding model used when training the mapping model, such as having the same version.

[0085] For example, based on the detected key points, affine transformation alignment is performed, such as to achieve eye leveling. The aligned area is then cropped and scaled proportionally to a preset resolution (such as 224×224 pixels) to generate a standardized target image with uniform specifications.

[0086] Step A2: Generate target image feature vectors based on standard target image images.

[0087] For example, more accurate target image feature vectors can be generated based on standard target image maps, providing stable input for subsequent cross-modal mapping and speech synthesis.

[0088] In some embodiments, preprocessing the initial target image may include the following methods: decoding the initial target image into a red, green, and blue (RGB) three-channel image; reading the orientation parameters of the exchangeable image file format (EXIF) of the initial target image; and converting the original color gamut of the initial target image into the sRGB standard color gamut.

[0089] In some embodiments, before generating the target image feature vector based on the initial target image image in step 201 above, the method provided in this disclosure may further include: obtaining an initial target image image that conforms to the image validity rules.

[0090] For example, taking a human face as the target image, in a single input initial target image image, the coordinates of the image frame, 5 key points (such as the eyes, nose tip, mouth corners, etc.) and confidence score are simultaneously identified and output, providing basic data for subsequent judgment and alignment.

[0091] The detected images undergo compliance screening. If the detected image does not comply with the image validity rules, the process will be refused and the user will be prompted to change the image. If the detected image complies with the image validity rules, the process will proceed.

[0092] For example, image validity rules may include one or more of the following: the number of pixels on the shorter side of the image frame is greater than or equal to a size threshold, the clarity of the initial target image is greater than or equal to a clarity threshold, and the target image can be selected from multiple images.

[0093] For example, if the short side pixel of the image frame is less than the size threshold, the process will be rejected; if the clarity of the initial target image is less than the clarity threshold, the process will be rejected; and / or, when multiple images are detected, the one with the largest frame and the highest overlap with the center can be selected as the target image by default, or several images can be selected as the target image according to a preset strategy. If no target image is selected according to the preset strategy, the process will be rejected.

[0094] Taking a human face as an example, if the shorter side of the face bounding box is less than 128 pixels, or the ratio of the shorter side to the image width is less than 0.12, the image is rejected from further processing. A sharpness score is calculated quickly using the Laplacian operator variance. If the sharpness score is lower than the sharpness threshold, the image is directly rejected from further processing.

[0095] Taking a human face as an example, based on the five detected key points, an affine transformation is performed to align the faces, such as making the eyes horizontal. The aligned area is then cropped and scaled proportionally according to a preset standard resolution to generate a standardized face image with uniform specifications and pose.

[0096] Optionally, if the initial target image does not meet any of the rules in the image validity rules, such as failing to meet the size threshold, insufficient clarity, or having multiple faces and not hitting the preferred strategy, the process can be refused to proceed and the user can be prompted to change the image. Actionable suggestions can also be returned, such as suggesting that the user upload an image with a clearer face, facing the camera directly, with no obstructions to facial features, and / or a higher resolution.

[0097] Optionally, step A2 above may include: generating a target image feature vector based on a standard target image image using an image encoding model.

[0098] For example, the image encoding model can be consistent with the image encoding model used when training the mapping model. For example, keeping the versions consistent can avoid feature vector distribution drift caused by version differences, and can avoid affecting the adaptation effect of subsequent mapping models.

[0099] For example, when the target image is a face, the image encoding model can be called a face encoding model, without specifying a particular name.

[0100] In some embodiments, step A2 above, which generates a target image feature vector based on a standard target image image, may include: obtaining a high-dimensional feature vector by forward reasoning based on the standard target image image through an image encoding model; and normalizing the high-dimensional feature vector through the image encoding model to obtain the target image feature vector.

[0101] For example, a standard target image is input into the image encoding model. The image encoding model forward computes and outputs a high-dimensional feature vector (e.g., 512-dimensional). L2 normalization is performed on the high-dimensional vector to ensure uniform vector scale, generating a fixed-length target image feature vector (face_embedding), which is then used as input to the subsequent mapping model.

[0102] Optionally, after performing L2 normalization on the high-dimensional vectors, if there is a mean / variance calibration process during the training of the mapping model, the completely consistent calibration parameters need to be applied during inference to further align the vector distribution.

[0103] Optionally, the target image feature vector can be obtained and key configuration information can be recorded simultaneously, including the image encoding model version, vector normalization method, mean / variance calibration parameters (if any), and relevant threshold configurations, to ensure that the vector generation process is traceable and reproducible.

[0104] In some embodiments, step 202 above, which generates a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector using a mapping model, may include steps B1-B2.

[0105] Step B1: When the output attributes of the mapping model are inconsistent with the input requirements of the speech synthesis model, a recalibration layer is added to the end of the mapping model to obtain a new mapping model.

[0106] Step B2: Using the new mapping model, generate a predicted timbre feature vector based on the target image feature vector.

[0107] For example, when the original output attributes of the mapping model are inconsistent with the input requirements of the speech synthesis model, or if the speech synthesis model has specific requirements for the speaker conditions of the input, the model can be optimized by adding an adaptation layer, such as a recalibration layer, to ensure that the generated predicted timbre feature vector meets the input standards of the speech synthesis model. This can be directly used as the speaker condition input of the speech synthesis model, which can avoid the speech synthesis model being unable to recognize or the timbre generation being abnormal due to attribute incompatibility.

[0108] Optionally, the output attributes of the mapping model may include at least one of the following: data distribution, dimension, mean requirement, variance, and scaling range. The input requirements of the speech synthesis model may include at least one of the following: data distribution requirement information, dimension requirement information, mean requirement information, variance requirement information, and scaling range requirement information.

[0109] For example, load a pre-trained mapping model fθ, where the model type is a Multi-Layer Perceptron (MLP). The forward computation is `voice_embedding_hat = norm(fθ(face_embedding))`, where `norm` is a normalization operation, such as L2 normalization. `face_embedding` represents the target image feature vector. If the speech synthesis model has specific distribution / scale calibration requirements for the timbre feature vector, a recalibration layer is added at the end of the mapping model to align the output to the input requirements of the speech synthesis model, ensuring input compatibility.

[0110] For example, recalibration layers can include affine transformation layers or linear layers. Affine transformation layers can be used to adjust vector dimensions and align vector distributions (e.g., correcting the mean and variance), offering greater flexibility. Linear layers can be used to adjust vector dimensions (e.g., reducing from 512 dimensions to 256 dimensions) or to standardize the scale.

[0111] In some embodiments, the mapping model can be trained based on a vector sample pair including target image feature vector samples and timbre feature vector samples.

[0112] For example, during training, the input consists of a one-to-one target image feature vector sample `face_embedding` and a real-space vector `voice_embedding` of dimension Dv. The target image feature vector sample can be a real-space vector of dimension Df, and the timbre feature vector sample can be a real-space vector of dimension Dv.

[0113] For example, the mapping model can use a small multilayer perceptron (MLP), whose structure is designed according to the dimensional mapping path Df→H→H→Dv. That is, the target image feature vector sample is first mapped from the Df dimension to the intermediate dimension H. After two layers of H dimension processing, the final output is the Dv dimension vector. The output vector is further processed by L2 normalization. The optimizer is Adam or AdamW.

[0114] For example, during training, cosine similarity loss is used as the primary loss function, while InfoNCE loss or L2 loss can be selectively added as needed. The early stopping mechanism for model training uses cosine similarity or equal error rate (EER) on the validation set as the judgment metric to ensure optimal model performance.

[0115] For example, the outputs of training a mapping model include: the weight parameters fθ of the mapping network, the normalization and calibration configuration required for the model, and the version number of the encoder on which the training process depends.

[0116] The embodiments disclosed herein do not limit the specific training scheme, loss details, or network structure, and can be replaced by standard paradigms such as contrastive learning, distillation, and metric learning.

[0117] In some embodiments, obtaining vector sample pairs includes the following steps 301-303. Figure 3 The flowchart of a vector sample pair acquisition method provided in this embodiment of the disclosure is executed by a first device 101 or a second device 102, wherein process 300 includes the following steps.

[0118] Step 301: Obtain audio slices based on video samples.

[0119] For example, the source of the video samples can include video files from object storage, media libraries, or scraped sources, and the file path can be video / .

[0120] For example, video samples can be format-processed: uniformly converted to MP4 format without changing the original encoding, while extracting the main audio track of the video and exporting a WAV audio file for analysis, such as with parameters like mono and a 16kHz sampling rate. Audio slices are then obtained based on the format-processed video samples.

[0121] Step 302: Obtain training sample pairs based on audio slices and their corresponding video slices.

[0122] For example, a training sample pair may include a standard target image sample and an audio slice corresponding to the standard target image sample, wherein the standard target image sample is associated with a video frame in the video slice.

[0123] For example, taking faces as an example, frame-by-frame extraction and detection are performed on the generated video slices. For instance, frames are extracted at a frame rate of 5–10 FPS, and then face detection models such as RetinaFace, YOLO-Face, or BlazeFace are used to detect the extracted frames. If a frame contains no face, contains multiple faces, has low face clarity, or has a face size that is too small, the frame is discarded. For the frames that pass the screening, the frame with the best clarity is selected, the face region is cropped, and it is aligned to a standardized 224×224 face image through affine transformation. The standardized face image after the above standardization process is saved, and it is bound to the audio slice corresponding to the video slice to form a {face_image, audio_clip} training sample pair, and the information of the training sample pair is written into the sample list. This is only an example and does not constitute a limitation.

[0124] Step 303: Generate vector sample pairs based on the training sample pairs.

[0125] For example, taking a face as an example, face encoding can be performed: using any pre-trained face encoder such as ArcFace, FaceNet, or MobileFaceNet, forward computation is performed on the standardized face image to generate a 512-dimensional face embedding vector, which is then L2 normalized. Next, speaker encoding is performed: using a pre-trained speaker encoder such as ECAPA-TDNN, X-vector, d-vector, or WavLM-based encoder, audio slices are processed to obtain a 256-dimensional or 512-dimensional voice embedding vector, which is also L2 normalized to obtain vector sample pairs. This is merely an example and does not constitute a limitation.

[0126] Optionally, a list association can also be performed: a unique identifier sample_id is recorded for each sample, and the storage location of the target image feature vector sample and the timbre feature vector sample is recorded, and the version number and runtime parameters of the two types of encoders are marked to form a complete sample information record.

[0127] In some embodiments, step 301 above, which obtains audio slices based on video samples, includes steps C1-C4.

[0128] Step C1: Obtain audio samples based on video samples.

[0129] Step C2: Based on the audio samples, obtain the timestamps of the audio activity segments.

[0130] Step C3: Based on the timestamp and video sample, obtain the video slice corresponding to the timestamp.

[0131] Step C4: Obtain audio slices based on video slices.

[0132] Optionally, before obtaining audio samples based on video samples, the video samples can be preprocessed, such as data access and deduplication of the video samples. For example, video files can be batch-fetched from specified data sources such as Content Delivery Network (CDN), object storage, or local libraries, and deduplication can be performed through dual verification of file hash and perceptual hash to directly remove duplicate or near-duplicate invalid samples and ensure data uniqueness.

[0133] After deduplication, unified encapsulation and track extraction are performed. For example, only the video samples are lightly encapsulated, such as into an MP4 container format, without changing the original video's encoding information. At the same time, the main audio track is extracted from the video, prioritizing the mixed track containing human voices, and the original audio and video timestamps are preserved to ensure that the audio / video (A / V) synchronization is not disrupted.

[0134] Next, an audio proxy file (also known as an audio sample) for analysis can be generated based on the main audio track. For example, based on the extracted main audio track, an audio proxy file for subsequent analysis can be exported, such as a mono WAV format with parameters of 16kHz sampling rate and 16-bit PCM. Loudness normalization (e.g., aligned to -23LUFS) and lightweight denoising and clipping can be selectively performed as needed, which can improve the robustness of subsequent Voice Activity Detection (VAD) detection, and all optimization operations do not modify the original media asset file.

[0135] Optionally, metadata recording can also be performed: a unique identifier video_id is generated for each processed video sample, and key information such as the duration of the video sample, audio sampling rate, number of channels, container format, and storage location are recorded in detail to form a complete metadata archive, which serves as the basis for subsequent video segmentation and full-process traceability.

[0136] For example, running Voice Activity Detection (VAD) on a generated audio proxy file outputs a series of timestamps and corresponding confidence scores for voice activity segments, which can accurately locate valid voice intervals. The timestamps can include a start time (start_ms) and an end time (end_ms). Voice activity segments can also be called audio activity segments.

[0137] Optionally, duration and interval normalization can also be performed. For example, a minimum segment duration (e.g., ≥0.5 seconds) and a minimum silence interval (e.g., ≥0.2 seconds) can be preset; for adjacent segments, if the interval is less than the set threshold, they are merged. For segments that are too short and have a duration less than the minimum threshold, they are either discarded or merged into adjacent valid segments to ensure segment integrity.

[0138] Optionally, buffers can be added at the beginning and end boundaries of each valid speech segment, such as 100ms before and 150ms after, to prevent the loss of plosives, final sounds, or connected speech due to precise truncation, thus preserving the integrity of the speech.

[0139] After boundary processing, timeline backmapping and slice generation can be performed. For example, the timestamp information detected by VAD is backmapped to the timeline of the original video, and corresponding small video slices are generated accordingly. If the video container or encoder supports precise cropping, it is aligned near the GOP (Group of Pictures) boundary to reduce recoding; if not, local recoding is used to ensure accurate slice boundaries. Audio slices (such as WAV or Free Lossless Audio Codec (FLAC) formats) aligned with the video slice time can also be exported to provide basic materials for subsequent speech encoder operation and audio-video alignment processes.

[0140] Optionally, basic quality metrics can be calculated for each generated slice, such as root mean square (RMS) energy, peak value, simplified signal-to-noise ratio (SNR) estimate, percentage of silence, percentage of music energy, etc. Only the slice is labeled with quality tags without modifying the slice itself, which can provide reference signals for subsequent sample selection and training data cleaning.

[0141] Optionally, a slice manifest can also be generated. For example, a unique identifier, slice_id, can be assigned to each audio slice, and key information can be written into the manifest by category. The manifest can be stored in a database or a JSONL index.

[0142] For example, the manifest can contain five main categories of fields: association information, such as the video ID (video_id), audio slice ID (slice_id), and original media path; time information, such as slice start time (start_ms), end time (end_ms), slice duration (duration_ms), and pre- and post-margin times (pre_margin_ms and post_margin_ms); quality and statistical information, such as optional metrics like VAD detection confidence (vad_confidence), RMS energy (rms_dbfs), signal-to-noise ratio estimation (snr_est), and silence ratio (silence_ratio); technical parameters, such as audio sampling rate, number of channels, and audio / video encoding and container formats; and reproducibility information, such as the VAD model and version used, parameter configuration, preprocessing version, and timestamp. The manifest can also include audio slice paths and / or video slice paths. This ensures that the slice generation process is traceable and reproducible.

[0143] Optionally, storage pointers to the video slices and their corresponding audio slices can also be saved, such as object storage URLs or local paths, so that subsequent processes (such as face detection and cropping, speech coding, cross-modal pairing training, etc.) can directly access the corresponding audio and video slices through the audio slice identifier slice_id, reducing information search costs.

[0144] Optionally, each slice can be labeled with a processing status tag (such as "ready_for_face_detect" indicating that face detection is to be performed) and a quality tag (such as "low_snr", "music_heavy", "too_short", etc.). By using tags, slices can be filtered and the process can be scheduled in a regular manner, which can ensure that the next step is to process valid slices that meet the conditions.

[0145] This disclosure provides embodiments that are not limited to video preprocessing schemes or VAD technology solutions, and can use any implementation such as WebRTC-VAD, Silo-VAD, or cloud vendor audio detection APIs; parameter thresholds can be optimized according to business needs. For example, the cloud service provider's VAD / audio segment detection API can be called directly; or an accelerated version of VAD can be executed on edge nodes.

[0146] In some embodiments, step 302 above, which obtains training sample pairs based on audio slices and corresponding video slices, may include steps D1-D4.

[0147] Step D1: Detect the images of the video slices and obtain the detection results corresponding to the images.

[0148] Step D2: Based on the image validity rules and the detection results corresponding to the images, obtain the first target image that satisfies the image validity rules from the images of the video slices.

[0149] Step D3: Generate a standard target image sample corresponding to the first target image.

[0150] Step D4: Obtain training sample pairs based on standard target image samples and audio slices corresponding to video slices.

[0151] For example, an image detection model can be used to detect images in a video slice and obtain the corresponding detection results. For instance, when the target image is a face, the image detection model can be called a face detection model, without specifying a particular name.

[0152] For example, for each VAD slice of video content, frames are extracted at a fixed frame rate (e.g., 5-10 FPS). This ensures that the extracted frames cover most of the segment's time frame, thereby reducing the probability of missing fleeting target images.

[0153] Taking faces as an example, face detection can be performed by running a face detection model on each extracted image frame. Open source solutions such as RetinaFace, YOLO-Face, and BlazeFace can be used to accurately identify the face region in the image.

[0154] Record the detection results. For example, record detailed detection information for each frame of image, including the coordinates of the face detection box, face size, coordinates of key points (such as the positions of the eyes, nose tip, and corners of the mouth), and detection confidence score. Store this information in association with the timestamp of the corresponding frame for subsequent face screening, analysis, and full-process traceability.

[0155] For example, if no face is detected in any of the detection frames of a certain slice, the slice is marked as "no_face_detected" and discarded directly.

[0156] For example, screening for faces that are too small can be done by setting a specific size threshold (e.g., the smallest side of the face is less than 64 pixels, or the smallest side occupies less than 10% of the frame width). If a detected face does not meet this threshold, it is considered to be insufficiently sharp and is discarded. Image validity rules can include size thresholds.

[0157] For example, if more than one face is detected in a single frame (this is just an example and is filtered according to the image validity rule), or if the number and position of faces in different frames within the same slice vary too much, the slice is marked as "multi_face" and discarded. This can ensure a one-to-one correspondence between faces and speech in the training samples.

[0158] For example, optional screening can be performed to detect blurry or occluded images. A face sharpness test can be run, such as based on Laplacian variance or an open-source quality assessment model. If the sharpness score is below a set threshold, or if key facial features are occluded, the slice is discarded.

[0159] For the face slices that have passed the initial screening, the frames with the highest clarity and showing a frontal or slightly side profile are selected from all the frames that have detected faces as representative frames to ensure the integrity and quality of the face information.

[0160] For example, face cropping and alignment can also be performed. Precise cropping is performed based on the coordinates of the face detection box and the position of key points in the representative frame. Face alignment operations can also be performed as needed, such as adjusting the eyes to a horizontal position and uniformly scaling to a 224×224 resolution, ultimately generating a standardized face image.

[0161] The generated standardized face image (standard target image sample) is bound one-to-one with the VAD audio file corresponding to the slice to form a complete training sample pair {face_image, audio_clip}, ensuring the consistency between the face and the audio source.

[0162] For example, metadata can also be stored. The sample index records key information about the sample pair in detail, including the associated audio slice_id, face detection confidence, cropping parameters, alignment parameters, and audio file path, which can ensure the reproducibility of training data and the traceability of the entire process.

[0163] For example, a sample list can also be generated. All qualified {face_image, audio_clip} sample pairs that have been screened and processed in the early stage are organized and saved in the training sample list (manifest). The list format can be flexibly selected as comma-separated values ​​(CSV), JSON line format (JSONL), or database table, which is convenient for subsequent calling and management.

[0164] Key statistical indicators, including the total number of samples, the distribution of face resolution, the distribution of audio duration, and the coverage of source videos, can be recorded simultaneously in the sample list. These data will provide the core basis for subsequent data analysis and sampling strategy formulation for model training.

[0165] For example, all generated valid samples can be marked with the "ready_for_embedding_extraction" status.

[0166] In some embodiments, step 303 above generates vector sample pairs based on training sample pairs, including steps E1-E2.

[0167] Step E1: Generate target image feature vector samples based on standard target image sample using the image encoding model.

[0168] Taking faces as an example, standardized face images are input into the face coding model. These standardized face images have been cropped, aligned, and uniformly set to a fixed resolution (e.g., 224×224) to ensure the consistency of the input format.

[0169] For example, face encoding models can be ArcFace, FaceNet, Dlib-ResNet, MobileFaceNet, etc.

[0170] For example, taking a face as an example, the pixel values ​​of a standardized face image are normalized (e.g., mapped to the [0,1] or [-1,1] interval), which is the input requirement for the face encoding model. Next, the normalized image is input into the face encoding model, and a high-dimensional feature vector (e.g., 512-dimensional, and L2 normalized) is generated through forward inference. The high-dimensional feature vector is then standardized (ensuring the L2 norm is 1) to guarantee consistent numerical scales for different samples in subsequent cross-modal alignment training. The output of the face encoding model can be a fixed-length vector of target image feature vector samples, serving as the core representation of a person's identity and appearance features in the visual modality.

[0171] Record the storage path of the target image feature vector sample in the sample list or save it directly as a binary vector file, and associate it with the corresponding sample ID to ensure the traceability of features and original samples.

[0172] Step E2: Using the speaker coding model, generate timbre feature vector samples based on the audio slices corresponding to the standard target image samples.

[0173] For example, audio slices are input into the speaker coding model, such as slices that have been extracted by VAD speech activity detection, typically lasting 1 to 5 seconds, to ensure that they contain complete and valid speech segments.

[0174] For example, the speaker encoder can be ECAPA-TDNN, X-vector, d-vector, or a speaker encoder based on WavLM.

[0175] For example, amplitude normalization is performed on the audio slices, while ensuring that the audio sampling rate (e.g., 16kHz) and number of channels (e.g., mono) meet the input requirements of the selected model. The processed audio is then input into the speaker encoder, which extracts a fixed-length speaker embedding vector (voice_embedding), which can be 256-dimensional or 512-dimensional. L2 normalization can be performed on the extracted feature vector to ensure feature consistency and numerical stability across different samples, adapting to subsequent cross-modal training requirements. The output can be a fixed-length vector form of timbre feature vector samples (voice_embedding), specifically used as a timbre feature representation in the speech modality.

[0176] Optionally, the storage path of the timbre feature vector samples can be added to the sample list, or the vectors can be saved directly as binary files and strictly associated with the corresponding sample IDs to ensure the traceability of speech features with the original samples and facial features.

[0177] The embodiments disclosed herein do not limit the face embedding extraction algorithm and various adaptation schemes can be flexibly selected; the large language model speech synthesis (LLM TTS) architecture on which the voice embedding extraction depends is also not specifically limited, and can be either the speaker condition vector of the spectrogram route or the speaker condition vector of the codec tokens route.

[0178] For example, each selected valid training sample {face_image, audio_clip} has a pair of matching feature embeddings {face_embedding, voice_embedding}.

[0179] For example, metadata association is performed. A unique sample identifier (sample_id) is generated for each sample, and key information is centrally recorded in the manifest, including the sample-level sample_id, the associated slice_id and video_id, the face vector storage path (face_embedding_path) and the voice vector storage path (voice_embedding_path), as well as the version numbers of the face encoder and speaker encoder and relevant parameters in the inference process, ensuring that the entire process is traceable and reproducible.

[0180] For example, update the status of all samples from which bimodal embeddings have been successfully extracted to "ready_for_cross_modal_training" (ready for cross-modal training).

[0181] For example, the process of training a mapping model is as follows, taking a human face as an example.

[0182] The input vector sample pairs for training the mapping model are face embedding (face_embedding∈R^Df) and voice embedding (voice_embedding∈R^Dv), each corresponding to the same VAD slice.

[0183] During the preprocessing stage, L2 normalization is performed on the two embeddings respectively, and mean / variance normalization is added if necessary. At the same time, the version number of each encoder is recorded to ensure the consistency of data distribution during the training and inference stages.

[0184] During batch sampling, each batch assembles B positive pairs (f_i, v_i); within the same batch, B×(B-1) hard negative samples are naturally formed, namely f_i and v_j, where j≠i. The comparative learning process of the model is supported by comparing the positive pairs and hard negative samples.

[0185] The mapping network fθ can be a small feedforward multilayer perceptron (MLP) that maps face embedding (dimension Df) to speech embedding dimension (Dv).

[0186] The structural example is: Linear(Df→H)→ReLU→Linear(H→H)→ReLU→Linear (H→Dv), and the output needs to be normalized again by L2. Regarding hyperparameters, the intermediate dimension H can be 256 or 512, and the weights are initialized using the symmetric activation function Xavier or the asymmetric activation function He method. To avoid overfitting, a lightweight random dropout can be added, such as using a low dropout probability of 0.1.

[0187] Based on the solution provided in this disclosure, a simple and stable MLP can meet the needs of the scenario; if the face and voice embedding dimensions are different, dimension alignment can be achieved through the first-layer linear transformation.

[0188] The loss function uses cosine similarity loss from the regression-type similarity loss as the main loss. Choosing the simple and stable cosine similarity loss satisfies the training requirements for cross-modal feature alignment.

[0189] The training optimizer is Adam or AdamW, with the learning rate set to 1e-3 to 3e-4, and a cosine decay or step decay scheduling strategy is used. The batch size is selected from 64 to 256 depending on the GPU memory and embedding dimension.

[0190] Early stopping of the model uses the cosine similarity of the validation set or the speaker verification error rate (EER) as the core metric. Mixed precision training (AMP) can be enabled to improve training efficiency.

[0191] During training, checkpoints need to save the weights, normalized statistics, encoder version number, and loss configuration of the mapping network fθ to ensure that the training process is traceable.

[0192] The final output consists of three parts: the trained mapping network weights fθ, the normalized configuration (including Df / Dv dimension information, L2 normalization parameters, mean and variance, etc.), and the version numbers of the face encoder and speaker encoder on which it depends.

[0193] If the target TTS system has specific requirements for vector dimension or calibration method, a linear mapping or recalibration layer (such as affine transformation) can be added to the tail of fθ to make the mean / variance of the output vector consistent with the distribution during TTS training, thus adapting to different scenario requirements.

[0194] This disclosure does not limit the specific structure, loss function, or training details of the mapping network. Furthermore, this disclosure does not limit the specific implementation or vendor interface of each model or algorithm.

[0195] Based on the mapping model, usable timbre vectors can be directly obtained from faces, avoiding the need to match real human recordings with human timbre libraries, achieving timbre generation from a given image. The mapping model inference involves only one face encoding and one lightweight MLP forward pass, making it suitable for batch generation of massive numbers of characters or online interactions. The mapping model is directly reusable, providing a conditional interface and ecosystem for large-scale TTS models.

[0196] The speech rate, emotion, accent, etc. of the speech stream can be implemented by the prosody / language control module of TTS or subsequent prompt words / control vectors.

[0197] The training quality of cross-modal mapping models is related to the effectiveness of sample data cleaning and the stability of encoder operation. A simple quality weighting mechanism can also be added during the training phase of the mapping model. Specifically, samples can be assigned corresponding weights based on the slice's SNR or sharpness index, allowing high-quality samples to occupy a higher weight during training.

[0198] For example, the technical solution provided in this disclosure can eliminate the need for live recording and manual voice library matching, significantly reducing customization costs and time. Users only need to input an image to obtain usable timbre feature vectors and complete the synthesis, avoiding the high-cost links of recording studios, voice actors, and manual timbre selection. The link mainly uses face encoding + lightweight MLP mapping, embedding generation at the millisecond level and overall synthesis at the second level, compressing the weekly process of "casting - recording - fine-tuning" to a real-time response.

[0199] The technical solution provided in this disclosure improves the audio-visual matching degree and subjective acceptance by achieving "consistent prior" from vision to timbre. The mapping model learns the statistical correlation between "visual features → timbre features" in a unified and normalized embedding space. The output timbre feature vector has stable prior consistency with the human face image in dimensions such as image attributes and / or brightness, which can reduce the "audio-visual separation" caused by human subjective mismatch.

[0200] The technical solution provided in this disclosure adopts a modular, pluggable architecture, compatible with various detection / encoder / TTS backbones. Basic modules (such as VAD, face detection, encoder, vocoder, etc.) can be replaced or upgraded according to scenarios. For example, replacing different face encoders or vocoders does not require modification of the cross-modal main link and online interface, reducing coupling and maintenance costs and facilitating continuous evolution. The mapping model uses a simple MLP and a universal loss function, resulting in low replacement costs. The inventory, model, and parameters are fully versioned and traceable, making gray-scale deployments and rollbacks simple and controllable.

[0201] The technical solution provided in this disclosure can completely eliminate the need for live recording and manual selection, significantly reducing customization costs and launch cycle. It eliminates the need to organize voice actors for recording and post-processing, as well as the need for manual selection and trial dubbing in the voice library, saving a lot of manpower and studio resources; the weekly process of "casting - recording - fine-tuning" is reduced to real-time.

[0202] The technical solution provided in this disclosure provides a "visual → timbre" prior for audio-visual consistency, which improves matching accuracy and subjective acceptance. Compared with manual subjective selection, this solution can automatically output a timbre prior that is more consistent with the facial image in terms of facial attributes and / or brightness, reducing the risk of audio-visual disconnect.

[0203] The technical solution provided in this disclosure can generate timbre online in real time, meeting the low latency requirements of interactive and mass production scenarios. Compared to the hour-to-day response time of recording / manual matching, this solution provides a millisecond-to-second response time, making it suitable for online auditions, rapid comparison of multiple versions, and batch voice production of massive numbers of characters. The technical solution provided in this disclosure supports parallel streaming synthesis and sentence segmentation, with robust default parameters for both the front-end and back-end, reducing end-to-end low latency.

[0204] The technical solution provided in this disclosure offers strong scalability and automation capabilities, high throughput for single-role-multiple-text and multi-role-multiple-text scenarios, and eliminates the need for re-recording or repeated manual selection for each new role. The system can batch-generate usable voices for a large number of characters and synthesize them in parallel, significantly improving unit output. Implementation methods include: providing a caching and reuse mechanism for four core intermediate data types—face_embedding (target character feature vector), voice_embedding_hat (predicted voice feature vector), Mel spectrum (acoustic features), and CodecTokens (discrete speech tokens)—to avoid redundant computation and improve system efficiency. Representational State Transfer (REST) ​​or g Remote Procedure Call (gRPC) is used as the service communication protocol, and microservice orchestration enables collaborative work of functional modules. This architecture also supports convenient horizontal scaling, such as quickly increasing system processing capacity by adding service nodes.

[0205] The speech synthesis model in the technical solution provided in this disclosure is highly compatible with the multi-speaker TTS backbone, eliminating the need to retrain the main model, changing the TTS backbone network in production, and requiring no additional fine-tuning of the main model for each new role, thus reducing computing power and risk.

[0206] The technical solutions provided in this disclosure offer engineering stability and reproducibility. Self-checking and rollback mechanisms reduce the failure rate, ensuring stable output of usable audio even with weak images or boundary inputs, minimizing online failures and rework. Through size adjustment, clarity verification, and multi-face gating mechanisms, the offline training samples and online input facial data are ensured to be consistent in format and quality, preventing input differences from affecting mapping accuracy. When generated features or intermediate results exceed a reasonable range, recalibration (e.g., adjusting values ​​to a valid range) and cropping (e.g., removing abnormal parts) ensure data compliance. After speech synthesis, a speaker encoder calculates the similarity between the synthesized speech and the target timbre; if it fails to meet the standard, a fault-tolerance mechanism is triggered, such as rollback based on nearest-neighbor priors (calling historical high-quality results) or parameter resampling (regenerating compliant speech), ultimately ensuring output quality.

[0207] Further reference Figure 4 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a speech stream generation device, which corresponds to the above method embodiment and can be specifically applied to various electronic devices.

[0208] like Figure 4As shown, the speech stream generation device 400 of this embodiment may include: a first generation module 401, a second generation module 402, and a third generation module 403. The first generation module 401 is configured to generate a target image feature vector based on an initial target image image. The second generation module 402 is configured to generate a predicted timbre feature vector corresponding to the target image feature vector using a mapping model. The third generation module 403 is configured to generate a speech stream using a speech synthesis model based on the predicted timbre feature vector and the text information of the speech to be generated.

[0209] In this embodiment, the specific processing of the first generation module 401, the second generation module 402, and the third generation module 403 in the speech stream generation device 400 and the resulting technical effects can be referred to the relevant descriptions of the above method embodiments, and will not be repeated here.

[0210] In one possible implementation, the second generation module 402 is further configured to: add a recalibration layer at the end of the mapping model to obtain a new mapping model when the output attributes of the mapping model are inconsistent with the input requirements of the speech synthesis model; and generate a predicted timbre feature vector based on the target image feature vector through the new mapping model.

[0211] In one possible implementation, the mapping model is trained on a pair of vector samples, including target image feature vector samples and timbre feature vector samples.

[0212] In one possible implementation, the speech stream generation device 400 may further include: a first acquisition module, a second acquisition module, and a fourth generation module. The first acquisition module is configured to acquire audio slices based on video samples. The second acquisition module is configured to obtain training sample pairs based on the audio slices and their corresponding video slices; wherein the training sample pairs include standard target image samples and audio slices corresponding to the standard target image samples, and the standard target image samples are related to a frame of video in the video slices. The fourth generation module is configured to generate vector sample pairs based on the training sample pairs.

[0213] In one possible implementation, the second acquisition module is further configured to: detect the image of the video slice to obtain a detection result corresponding to the image; based on the image validity rule and the detection result corresponding to the image, obtain a first target image image that satisfies the image validity rule from the image of the video slice; generate a standard target image image sample corresponding to the first target image image; and obtain a training sample pair based on the standard target image image sample and the audio slice corresponding to the video slice.

[0214] In one possible implementation, the fourth generation module is further configured to: generate target image feature vector samples based on standard target image samples using an image coding model; and generate timbre feature vector samples based on audio slices corresponding to standard target image samples using a speaker coding model.

[0215] In one possible implementation, the first acquisition module is further configured to: obtain an audio sample based on a video sample; obtain a timestamp of an audio activity segment based on the audio sample; obtain a video slice corresponding to the timestamp based on the timestamp and the video sample; and obtain an audio slice based on the video slice.

[0216] In one possible implementation, the output attributes of the mapping model include at least one of the following: data distribution, dimension, mean requirement, variance, and scaling range, and the input requirements of the speech synthesis model include at least one of the following: data distribution requirement information, dimension requirement information, mean requirement information, variance requirement information, and scaling range requirement information.

[0217] In one possible implementation, the first generation module 401 is further configured to: process the initial target image based on a preset resolution and the key point coordinates of the initial target image to generate a standard target image; and generate a target image feature vector based on the standard target image.

[0218] In one possible implementation, the first generation module 401 is further configured to: obtain a high-dimensional feature vector by performing forward reasoning based on a standard target image image through an image encoding model; and normalize the high-dimensional feature vector through the image encoding model to obtain a target image feature vector.

[0219] This embodiment exists as a device embodiment corresponding to the above method embodiment. The technical effects brought about by the speech stream generation device provided in this embodiment can be referred to the relevant descriptions in the above method embodiments, which will not be repeated here.

[0220] This disclosure also provides a chip including a processor that can be used to implement any of the possible implementations of the methods provided in this disclosure.

[0221] This disclosure also proposes an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing instructions stored in the memory.

[0222] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the described method. The computer-readable storage medium can be volatile or non-volatile.

[0223] This disclosure also provides a computer program product that, when executed by a processor, can implement the methods described in any of the above embodiments.

[0224] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0225] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. For example... Figure 5 As shown, the electronic device 500 of this embodiment includes a processor 501 and may further include a memory 502; wherein the memory 502 is used to store computer execution instructions; and the processor 501 is used to execute the computer execution instructions stored in the memory to implement the various steps performed by the electronic device in the above embodiment. For details, please refer to the relevant descriptions in the foregoing method embodiments.

[0226] Alternatively, the memory 502 can be either standalone or integrated with the processor 501.

[0227] When the memory 502 is set up independently, the electronic device also includes a bus 503 for connecting the memory 502 and the processor 501.

[0228] It should be understood that the processor 501 described above can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly manifested as being executed by a hardware processor, or being executed by a combination of hardware and software modules within the processor.

[0229] The memory 502 may include high-speed RAM memory, and may also include non-volatile memory NVM, such as at least one disk storage device, and may also be a USB flash drive, portable hard drive, read-only memory, disk or optical disc, etc.

[0230] Bus 503 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0231] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0232] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0233] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0234] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0235] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0236] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem to address the management difficulties and weak business scalability inherent in traditional physical hosts and Virtual Private Servers (VPS) services. Servers can also be servers for distributed systems or servers integrated with blockchain technology.

[0237] According to the technical solution of this disclosure, only an initial target image image needs to be input to quickly generate a matching predicted timbre feature vector. Through a speech synthesis model, a speech stream of the target image is generated based on the predicted timbre feature vector and the text information of the speech to be generated. No real person is required to participate in the speech recording, which can significantly shorten the customization cycle of the timbre and reduce labor costs. Furthermore, the matching process of quantifying the character image into the target image feature vector is determined by the mapping model, without the involvement of subjective judgment. This can ensure that the matching of timbre and target image conforms to common understanding, improve the matching degree of timbre and target image, and enhance the objectivity and consistency of the matching of timbre and character image.

[0238] It should be understood that the various forms of processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0239] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a speech stream, comprising: Based on the initial target image, generate the target image feature vector; Using a mapping model, a predicted timbre feature vector is generated based on the target image feature vector. A speech stream is generated using a speech synthesis model based on the predicted timbre feature vector and the text information of the speech to be generated.

2. The method according to claim 1, wherein, The step of generating a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector through a mapping model includes: When the output properties of the mapping model are inconsistent with the input requirements of the speech synthesis model, a recalibration layer is added to the end of the mapping model to obtain a new mapping model. The new mapping model generates the predicted timbre feature vector based on the target image feature vector.

3. The method according to claim 1 or 2, wherein, The mapping model is obtained by training on vector sample pairs, which include target image feature vector samples and timbre feature vector samples.

4. The method according to claim 3, wherein, The steps for obtaining the vector sample pairs include: Audio slices are obtained from video samples; Based on the audio slice and the corresponding video slice, a training sample pair is obtained; wherein, the training sample pair includes a standard target image sample and an audio slice corresponding to the standard target image sample, and the standard target image sample is related to a video frame in the video slice; The vector sample pairs are generated based on the training sample pairs.

5. The method according to claim 4, wherein, The step of obtaining training sample pairs based on the audio slice and the corresponding video slice includes: The images of the video slices are detected to obtain the detection results corresponding to the images; Based on the image validity rules and the detection results corresponding to the image, a first target image image that satisfies the image validity rules is obtained from the image of the video slice; Generate a standard target image sample corresponding to the first target image; The training sample pair is obtained based on the standard target image sample and the audio slice corresponding to the video slice.

6. The method according to claim 4 or 5, wherein, The step of generating the vector sample pair based on the training sample pair includes: Based on the standard target image sample, the target image feature vector sample is generated using the image encoding model. The timbre feature vector sample is generated based on the audio slice corresponding to the standard target image sample using the speaker coding model.

7. The method according to claim 4 or 5, wherein, Audio slices are obtained from video samples, including: Audio samples are obtained based on the video samples; Based on the audio sample, the timestamps of the audio activity segments are obtained; Based on the timestamp and the video sample, obtain the video slice corresponding to the timestamp; The audio slice is obtained based on the video slice.

8. The method according to claim 2, wherein, The output attributes of the mapping model include at least one of the following: data distribution, dimension, mean requirement, variance, and scaling range. The input requirements of the speech synthesis model include at least one of the following: data distribution requirement information, dimension requirement information, mean requirement information, variance requirement information, and scaling range requirement information.

9. The method according to claim 1 or 2, wherein, The process of generating a target image feature vector based on the initial target image image includes: The initial target image is processed based on the preset resolution and the key point coordinates of the initial target image to generate a standard target image. Based on the standard target image, the target image feature vector is generated.

10. The method according to claim 9, wherein, The step of generating the target image feature vector based on the standard target image image includes: Using the image encoding model, and based on the standard target image, forward reasoning is performed to obtain a high-dimensional feature vector; The high-dimensional feature vector is normalized using the image encoding model to obtain the target image feature vector.

11. A speech stream generation apparatus, comprising: The first generation module is configured to generate a target image feature vector based on the initial target image image; The second generation module is configured to generate a predicted timbre feature vector corresponding to the target image feature vector based on the target image feature vector using a mapping model; The third generation module is configured to generate a speech stream based on the predicted timbre feature vector and the text information of the speech to be generated, using a speech synthesis model.

12. An electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to implement the method of any one of claims 1 to 10 when executing instructions stored in the memory.

13. A non-volatile computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method of any one of claims 1 to 10.

14. A computer program product comprising a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 10.

Images