Speech synthesis method and device

Abstract

The embodiment of the invention provides a speech synthesis method and device, and the method comprises the steps: obtaining a comprehensive acoustic feature corresponding to each piece of audio data, and the comprehensive acoustic feature comprises an implicit acoustic feature and an explicit acoustic feature; acquiring a clustering center corresponding to each audio clustering cluster, and calculating a target distance between each audio data and the clustering center corresponding to each audio clustering cluster; based on the target distance between each piece of audio data and the clustering center corresponding to each audio clustering cluster, obtaining a semantic cluster corresponding to each audio clustering cluster; under the condition that a target semantic cluster in the plurality of semantic clusters is determined, determining a target tone according to a clustering center of an audio clustering cluster corresponding to the semantic cluster; and inputting a target text and the target timbre into a speech synthesis model to obtain a target speech with the target timbre.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech processing, and in particular to a speech synthesis method and device. BACKGROUND

[0002] In the field of speech synthesis and timbre processing, the prior art has made certain progress. First, speaker speech data is acquired, an acoustic recognition model is constructed, the model is trained, timbre embedding feature vectors are extracted, and spectral clustering is applied for timbre clustering. In terms of acoustic model construction, a model is composed of a multi-layer time delay neural network (TDNN) layer, a statistical pooling layer, a full connection layer, and a softmax output layer. The network is trained through a multi-classification cross-entropy objective function to classify the training speakers. This method can improve the matching rate of specific human voices and the accuracy of speech recognition. However, this method mainly relies on implicit features and lacks the combination of explicit timbre features, resulting in insufficient comprehensive feature expression and difficulty in capturing subtle differences in timbre. In addition, although the clustering algorithm can process high-dimensional data, it lacks semantic interpretability and cannot dynamically introduce semantic constraints, limiting the accuracy of the clustering results and the application scenarios. SUMMARY

[0003] In view of the above problems, the embodiments of the present application provide a speech synthesis method and device that overcome the above problems or at least partially solve the above problems.

[0004] In a first aspect, the embodiments of the present application provide a speech synthesis method, which comprises:

[0005] Acquiring comprehensive acoustic features corresponding to each audio data, the comprehensive acoustic features comprising implicit acoustic features and explicit acoustic features;

[0006] Acquiring a clustering center corresponding to each audio clustering cluster and calculating a target distance between each audio data and the clustering center corresponding to each audio clustering cluster;

[0007] Based on the target distance between each audio data and the clustering center corresponding to each audio clustering cluster, acquiring a semantic cluster corresponding to each audio clustering cluster;

[0008] In a case where a target semantic cluster in the plurality of semantic clusters is determined, determining a target timbre according to the clustering center of the audio clustering cluster corresponding to the semantic cluster;

[0009] Inputting the target text and the target timbre into a speech synthesis model to obtain target speech with the target timbre.

[0010] Optionally, in the case that the comprehensive acoustic feature comprises an implicit acoustic feature, the obtaining of the comprehensive acoustic feature corresponding to each audio data comprises:

[0011] preprocessing each audio data to obtain a bottom acoustic feature corresponding to each audio data;

[0012] processing the bottom acoustic feature corresponding to each audio data in sequence through a TDNN network, a statistical pooling layer and a full connection layer to obtain an initial implicit acoustic feature corresponding to each audio data;

[0013] normalizing the initial implicit acoustic feature corresponding to each audio data to obtain an implicit acoustic feature corresponding to each audio data.

[0014] Optionally, in the case that the comprehensive acoustic feature comprises an explicit acoustic feature, the obtaining of the comprehensive acoustic feature corresponding to each audio data comprises:

[0015] obtaining an initial explicit acoustic feature corresponding to each audio data, the initial explicit acoustic feature comprising at least one of a fundamental frequency, a spectral tilt, a harmonic-to-noise ratio, a spectral flatness, a short-time energy, a short-time autocorrelation coefficient and a zero-crossing rate change rate;

[0016] normalizing the initial explicit acoustic feature corresponding to each audio data to obtain an explicit acoustic feature corresponding to each audio data.

[0017] Optionally, the obtaining of the cluster center corresponding to each audio clustering cluster comprises:

[0018] obtaining each audio clustering cluster and calculating a membership degree of each audio data belonging to each audio clustering cluster to obtain a membership matrix;

[0019] repeating the following steps until the number of iterations of updating reaches a threshold value of iterations or the change of the membership matrix before and after updating is less than a preset threshold value to stop, to obtain the cluster center corresponding to each audio clustering cluster:

[0020] calculating the cluster center corresponding to each audio clustering cluster based on the membership matrix;

[0021] calculating a target distance between each audio data and the cluster center corresponding to each audio clustering cluster;

[0022] updating the membership matrix according to the target distance between each audio data and the cluster center corresponding to each audio clustering cluster.

[0023] Optionally, the calculating of the target distance between each audio data and the cluster center corresponding to each audio clustering cluster comprises:

[0024] calculate a first distance between each audio data and each cluster center based on the implicit acoustic feature corresponding to each audio data and the implicit acoustic feature of each cluster center;

[0025] calculate a second distance between each audio data and each cluster center based on the explicit acoustic feature corresponding to each audio data and the explicit acoustic feature of each cluster center;

[0026] calculate a target distance between each audio data and each cluster center based on the first distance and the second distance between each audio data and each cluster center.

[0027] Optionally, the calculating the membership of each audio data belonging to each audio clustering cluster to obtain a membership matrix comprises:

[0028] obtain target data corresponding to each audio clustering cluster, the target data comprising at least one of the following: anchor fan data, video viewing data, interaction data, video complete playback data, audience retention data;

[0029] calculate a target data score of the target data corresponding to each audio clustering cluster based on the target data corresponding to each audio clustering cluster, the target data comprising at least one of the following: fan score, viewing volume score, interaction rate score, complete playback rate score, retention rate score, and popularity score;

[0030] calculate the membership of each audio data belonging to each audio clustering cluster according to the target data score of the target data corresponding to each audio clustering cluster to obtain a membership matrix.

[0031] Optionally, the obtaining the semantic cluster corresponding to each audio clustering cluster based on the target distance between each audio data and the cluster center corresponding to each audio clustering cluster comprises:

[0032] For each audio clustering cluster, determine audio data with a target distance to the cluster center of the audio clustering cluster less than a distance threshold as target audio data corresponding to the audio clustering cluster, and extract text in each target audio data to obtain target documents corresponding to each target audio data of the audio clustering cluster;

[0033] iteratively train a latent Dirichlet allocation model based on the target documents corresponding to the audio clustering cluster, and calculate topic consistency in the iteration process until the topic consistency meets a convergence condition to stop, to obtain document-topic probability distribution and topic-word probability distribution of the audio clustering cluster;

[0034] based on the document-topic probability distribution and the topic-word probability distribution of the audio clustering cluster, take a plurality of high-frequency words of the audio clustering cluster with a frequency greater than a preset frequency as a semantic cluster of the audio clustering cluster.

[0035] Optionally, the calculating the topic consistency in the iteration process comprises:

[0036] obtaining a topic-word count matrix of the audio clustering cluster under the current iteration, and calculating a topic-word probability distribution according to the topic-word count matrix;

[0037] determining a plurality of high-frequency words corresponding to the audio clustering cluster according to the topic-word probability distribution;

[0038] calculating the topic consistency under the current iteration according to the word probability of each high-frequency word corresponding to the audio clustering cluster and the word co-occurrence probability of each two high-frequency words.

[0039] In a second aspect, the embodiments of the present application further provide a speech synthesis device, the device comprising:

[0040] a first obtaining module, configured to obtain comprehensive acoustic features corresponding to each audio data, the comprehensive acoustic features comprising: implicit acoustic features and explicit acoustic features;

[0041] a calculating module, configured to obtain a clustering center corresponding to each audio clustering cluster, and calculate a target distance between each audio data and the clustering center corresponding to each audio clustering cluster;

[0042] a second obtaining module, configured to obtain a semantic cluster corresponding to each audio clustering cluster based on the target distance between each audio data and the clustering center corresponding to each audio clustering cluster;

[0043] a determining module, configured to, in a case where a target semantic cluster in the plurality of semantic clusters is determined, determine a target vocal color according to the clustering center of the audio clustering cluster corresponding to the semantic cluster;

[0044] a synthesizing module, configured to input a target text and the target vocal color into a speech synthesis model to obtain a target speech with the target vocal color.

[0045] In a third aspect, the embodiments of the present application further provide an electronic device, comprising a memory, a transceiver, and a processor:

[0046] the memory is configured to store a computer program; the transceiver is configured to transceive data under the control of the processor; and the processor is configured to read the computer program in the memory and execute the method of the first aspect.

[0047] In a fourth aspect, the embodiments of the present application further provide a processor-readable storage medium, the processor-readable storage medium storing a computer program, the computer program being configured to cause a processor to execute the method of the first aspect.

[0048] The embodiments described above in this application obtain comprehensive acoustic features corresponding to each audio data point. These comprehensive acoustic features include implicit and explicit acoustic features. Cluster centers corresponding to each audio cluster are obtained, and target distances between each audio data point and the corresponding cluster centers are calculated. Based on these target distances, semantic clusters corresponding to each audio cluster are obtained. When a target semantic cluster is determined from multiple semantic clusters, a target timbre is determined based on the cluster centers of the audio clusters corresponding to the semantic clusters. The target text and the target timbre are input into a speech generation model to obtain target speech with the target timbre. By combining implicit and explicit acoustic features, subtle differences in timbre can be captured. The introduction of a semantic cluster clustering algorithm increases semantic interpretability, achieving more efficient and accurate timbre clustering, providing higher-quality timbre features for speech synthesis, and meeting diverse and personalized speech synthesis needs. Attached Figure Description

[0049] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of the speech synthesis method provided in the embodiments of this application;

[0051] Figure 2 A detailed flowchart of the speech synthesis method provided in the embodiments of this application;

[0052] Figure 3 Structural block diagram of the speech synthesis device provided in the embodiments of this application

[0053] Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The technical solutions of the embodiments of this application will be clearly described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application are within the scope of protection of this application.

[0055] The terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such use of data can be interchanged where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first," "second," etc., are generally of the same class and the number of objects is not limited; for example, a first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.

[0056] The speech synthesis method provided in this application will be described in detail below with reference to the accompanying drawings, through specific embodiments and application scenarios.

[0057] Specifically, embodiments of this application provide a speech synthesis method, such as... Figure 1 and Figure 2 As shown, the specific steps may include the following:

[0058] Step 101: Obtain the comprehensive acoustic features corresponding to each audio data. The comprehensive acoustic features include implicit acoustic features and explicit acoustic features.

[0059] To automatically collect and store video-related audio data and metadata from video platforms, the following method is used:

[0060] First, the open-source command-line downloader yt-dlp library automatically crawls video resources and their front-end metadata from video websites, including the number of followers, views, likes, comments, shares, and plays. After the video is downloaded, the open-source multimedia processing engine (FastForwardMovingPicture Experts Group, FFmpeg) automatically separates the video and audio, saving the audio data in a lossless audio file format (WaveformAudioFileFormat, WAV) for subsequent timbre feature extraction and analysis. Simultaneously, yt-dlp automatically stores the video metadata in plain text data exchange format (JavaScript Object Notation, JSON) and writes the data to a database, ensuring a one-to-one correspondence between video, audio data, and metadata, facilitating subsequent correlation analysis.

[0061] For backend data such as video popularity, returning viewers, and total viewers, a web crawler was used to collect the data. The crawler simulated logging into the creator's backend and used cookies for authentication to access the platform's backend application programming interface (API) to retrieve detailed metrics such as returning viewers and total viewers. The crawled data was stored in the database along with the frontend information to ensure data integrity and consistency.

[0062] The above method combines the advantages of yt-dlp, FFmpeg, and web crawlers. It has a high degree of automation and can efficiently and stably complete the acquisition and separation of large-scale video and audio data, providing a high-quality data foundation for subsequent timbre feature extraction and data analysis.

[0063] The implicit and explicit acoustic features corresponding to each audio data point are obtained. These features are then concatenated to obtain the comprehensive acoustic features for each audio data point. This method of concatenating implicit and explicit acoustic features not only captures the deep implicit acoustic features of timbre but also preserves the physical meaning of the explicit acoustic features, providing rich and highly discriminative feature representations for subsequent timbre classification and synthesis. Concatenating implicit and explicit acoustic features enhances the model's ability to represent timbre, effectively improves the accuracy of timbre classification, and generates speech that better matches the target timbre features in speech synthesis tasks.

[0064] Step 102: Obtain the cluster center corresponding to each audio cluster and calculate the target distance between each audio data and the cluster center corresponding to each audio cluster.

[0065] Specifically, multiple audio clusters are obtained, and the cluster center corresponding to each audio cluster is calculated. The target distance between each audio data in the multiple audio data and the cluster center corresponding to each audio cluster in the multiple audio clusters is also calculated.

[0066] Step 103: Based on the target distance between each audio data and the cluster center corresponding to each audio cluster, obtain the semantic cluster corresponding to each audio cluster.

[0067] Specifically, semantic clusters enhance the semantic interpretability of audio clusters, enabling more efficient and accurate timbre clustering and providing higher-quality timbre features for speech synthesis to meet diverse and personalized speech synthesis needs.

[0068] Step 104: In the case of determining the target semantic cluster among multiple semantic clusters, determine the target timbre based on the cluster center of the audio cluster corresponding to the semantic cluster.

[0069] Specifically, semantic clusters are expressed in two dimensions: one dimension is the audio cluster, which reflects information such as the number of plays and interaction rates of the category; the other dimension is the topic semantic cluster of each audio cluster, which reflects the semantic information of the audio cluster, such as the scene to which the speech belongs and the content theme. Users can select the required audio cluster based on the information in these two dimensions and the application scenario, and directly generate the target timbre that can be used for speech synthesis in the X-vector space.

[0070] like Figure 2 As shown, there are three specific methods for generating the target timbre:

[0071] Mind-like methods

[0072] Select the audio cluster corresponding to the target semantic cluster based on the semantic cluster, obtain the cluster center of the audio cluster, and extract the implicit acoustic features from the cluster center as the target timbre for speech synthesis.

[0073] Intra-class random sampling method

[0074] Random audio data with a membership degree higher than 0.1 to the audio cluster corresponding to the target semantic cluster are selected, and their corresponding implicit acoustic features are used as the target timbre.

[0075] Mind-based weighted method

[0076] If there are multiple audio clusters corresponding to the target semantic cluster, the implicit acoustic features can be extracted by weighted summation of the multiple cluster centers corresponding to the multiple audio clusters, and used as the target timbre.

[0077] This technical solution can be applied to various downstream speech processing tasks, and it has particularly strong practical value in speech synthesis. For example, the final output X-vector that conforms to a specific scenario can be used as a speaker embedding vector and input into the Conditional Flow Matching (CFM) module in CosyVoice 2 to control the timbre of the synthesized speech.

[0078] Step 105: Input the target text and the target timbre into the speech synthesis model to obtain the target speech with the target timbre.

[0079] The target timbre is L2 normalized to ensure numerical stability and uniform distribution. Then, it is projected onto the 80-dimensional space required by the speech synthesis model through a linear mapping layer. This timbre, along with the target text, is then used as the conditional input to the Conditional Flow Matching (CFM) decoder to guide the speech synthesis model in generating target speech with the target timbre.

[0080] The embodiments described above in this application obtain comprehensive acoustic features corresponding to each audio data point. These comprehensive acoustic features include implicit and explicit acoustic features. Cluster centers corresponding to each audio cluster are obtained, and target distances between each audio data point and the corresponding cluster centers are calculated. Based on these target distances, semantic clusters corresponding to each audio cluster are obtained. When a target semantic cluster is determined from multiple semantic clusters, a target timbre is determined based on the cluster centers of the audio clusters corresponding to the semantic clusters. The target text and the target timbre are input into a speech generation model to obtain target speech with the target timbre. By combining implicit and explicit acoustic features, subtle differences in timbre can be captured. The introduction of a semantic cluster clustering algorithm increases semantic interpretability, achieving more efficient and accurate timbre clustering, providing higher-quality timbre features for speech synthesis, and meeting diverse and personalized speech synthesis needs.

[0081] As an optional specific embodiment, when the integrated acoustic features include implicit acoustic features, step 101, obtaining the integrated acoustic features corresponding to each audio data, includes:

[0082] Preprocess each audio data to obtain the underlying acoustic features corresponding to each audio data;

[0083] The underlying acoustic features corresponding to each audio data are processed sequentially through a TDNN network, a statistical pooling layer, and a fully connected layer to obtain the initial implicit acoustic features corresponding to each audio data.

[0084] The initial implicit acoustic features corresponding to each audio data are normalized to obtain the implicit acoustic features corresponding to each audio data.

[0085] In speech synthesis tasks, significant multidimensional feature differences exist between different timbres, such as pitch and brightness. However, due to the high dimensionality and nonlinear complexity of speech signals, traditional manual feature extraction methods struggle to fully characterize timbre differences and are easily affected by noise and variability. To address this issue, embodiments of this application employ a deep neural network model, X-vector, to map audio to a unified metric space, representing timbre features in the form of high-dimensional embedding vectors.

[0086] The following example uses one audio data point to illustrate the process of obtaining the implicit acoustic features corresponding to that audio data:

[0087] First, each audio data point is preprocessed by standardizing the sampling rate to 16kHz and extracting the coefficients of a 40-dimensional Vimer filter bank (FBANK) as the underlying acoustic features. The FBANK effectively characterizes the spectral properties of the audio data and exhibits robustness to background noise.

[0088] X-vector uses a multi-layer time-delay neural network (TDNN) as its encoder. TDNN can capture the temporal information of audio data and model the contextual information between adjacent frames through a sliding window processing mechanism, thereby extracting frame-level feature representations with temporal correlation.

[0089] In the feature extraction stage, after the audio data is processed by the TDNN network, it will pass through the statistical pooling layer, which calculates the mean and standard deviation of all frame-level features, thereby mapping the variable-length audio segments into fixed-dimensional representation vectors. The feature vectors after statistical pooling are then processed through two fully connected layers for segment-level feature extraction and nonlinear transformation, finally generating a 512-dimensional X-vector embedding vector, which is the initial implicit acoustic feature.

[0090] like Figure 2 As shown, since different dimensions of the initial implicit acoustic features may have different numerical magnitudes, directly calculating the distance may lead to an imbalance in feature scale, affecting the calculation results. Therefore, the initial implicit acoustic features are first L2 normalized so that the magnitude of all vectors is 1, thus focusing only on the direction information of the vectors when calculating the distance. The formula for L2 normalization is as follows:

[0091]

[0092] Where, x' i Let be the initial implicit acoustic features of the i-th audio data;

[0093] x i The normalized implicit acoustic features of the i-th audio data;

[0094] x' il This represents the initial implicit acoustic feature of the l-th dimension of the i-th audio data.

[0095] Through the above process, the normalized implicit acoustic features corresponding to each audio data are obtained.

[0096] X-vector extracts high-dimensional features through a deep neural network model. Compared to traditional hand-crafted features, it has a higher dimensionality and can more comprehensively represent timbre information, thereby improving the accuracy of timbre classification and distance calculation. Secondly, X-vector's statistical pooling layer can handle variable-length audio segments, ensuring that the model generates fixed-dimensional feature representations for different audio lengths, thus guaranteeing the stability and generalization ability of the X-vector model. Furthermore, the high-dimensional timbre vectors generated by X-vector can be used for various tasks, including timbre label classification, distance measurement (or similarity measurement), and target timbre speech synthesis, demonstrating strong versatility and scalability.

[0097] As an optional specific embodiment, when the comprehensive acoustic features include explicit acoustic features, step 101 of obtaining the comprehensive acoustic features corresponding to each audio data includes:

[0098] Obtain the initial explicit acoustic features corresponding to each audio data, wherein the initial explicit acoustic features include at least one of the following: fundamental frequency, spectral tilt, harmonic-to-noise ratio, spectral flatness, short-time energy, short-time autocorrelation coefficient, and zero-crossing rate of change.

[0099] The initial explicit acoustic features corresponding to each audio data are normalized to obtain the explicit acoustic features corresponding to each audio data.

[0100] In speech synthesis tasks, relying solely on implicit acoustic features is often insufficient to fully characterize timbre differences. Therefore, this application, in addition to obtaining implicit acoustic features, further extracts multidimensional explicit acoustic features.

[0101] like Figure 2 As shown, there are significant differences in the dimensions and numerical ranges between timbre features and video metadata. To address the feature overload problem caused by excessive differences in the numerical ranges of various features, normalization is used to map all features to a unified numerical range, ensuring that the X-vector model can utilize multi-dimensional feature information in a balanced manner. The video metadata is normalized to a range of [0, 1].

[0102] Specifically, taking one audio data point as an example, this section explains the process of obtaining the initial explicit acoustic features (10 dimensions in total) and the process of obtaining the normalized explicit acoustic features corresponding to that audio data:

[0103] Fundamental frequency (F0): Represents the fundamental vibration frequency of a sound wave, reflecting the pitch characteristic of timbre. The fundamental frequency is extracted using the autocorrelation method or the autocorrelation fundamental frequency estimation algorithm YIN, specifically through the following formula:

[0104]

[0105] Where f is the sampling rate;

[0106] ACF(t) is the autocorrelation function, where t is the delay time. Different delay times are used to calculate the corresponding ACF.

[0107] F0 is the initial base frequency corresponding to the audio data.

[0108] For frequency-related features such as the fundamental frequency, due to their large range, right-skewed distribution, and susceptibility to extreme values, the Min-Max normalization method can map a wide range of frequencies to a smooth distribution range, reducing the impact of high-frequency outliers. The calculation formula is as follows:

[0109]

[0110] Where F0 is the initial fundamental frequency corresponding to the audio data;

[0111] F 0_min The minimum value among the initial fundamental frequencies corresponding to all audio data;

[0112] F 0_max The maximum value among the initial fundamental frequencies corresponding to all audio data;

[0113] F0' is the normalized fundamental frequency.

[0114] Formants (F1 / F2 / F3): Formants are frequency bands in the speech spectrum where energy is concentrated, reflecting the brightness and resonance characteristics of the timbre. They are extracted using a linear predictive coding model and calculated using the following formula:

[0115]

[0116] Among them, P s The resonant frequency corresponding to the s-th frequency band;

[0117] f is the sampling rate;

[0118] ω s Let be the pole angular frequency corresponding to the s-th frequency band.

[0119] Using the method described above, the formant frequency corresponding to each frequency band is calculated. Since there are multiple formant frequencies, the median of these multiple formant frequencies can be used as the initial formant frequency for the entire audio data.

[0120] For frequency-related features such as resonant frequencies, due to their large range, right-skewed distribution, and susceptibility to extreme values, the Min-Max normalization method can map a wide range of frequencies to a smooth distribution interval, reducing the influence of high-frequency outliers. The calculation formula is as follows:

[0121]

[0122] Among them, P s The initial formant frequency corresponding to the audio data;

[0123] P s_min It is the minimum value among the initial formant frequencies corresponding to all audio data;

[0124] P s_max The maximum value among the initial formant frequencies corresponding to all audio data;

[0125] P s ' is the normalized resonant frequency.

[0126] Spectral slope: describes the rate at which spectral power decreases with frequency, characterizing the brightness and darkness of the timbre. It is calculated using the following formula:

[0127]

[0128] Among them, P s Let K be the resonant frequency corresponding to the s-th frequency band, and K be the number of frequency bands.

[0129] Let K be the average of the frequencies of the formant peaks.

[0130] A s The spectral amplitude corresponding to the s-th frequency band;

[0131] The mean of the amplitudes of the K spectra;

[0132] slope is the initial spectral tilt of the audio data.

[0133] The data range of spectral tilt is well-defined and contains no non-extreme outliers. Min-Max normalization can be directly used to preserve their numerical range and map them to [0, 1], avoiding unnecessary transformations. The calculation formula is as follows:

[0134]

[0135] Where slope is the initial spectral tilt of the audio data;

[0136] slope' is the normalized spectral tilt of the audio data;

[0137] slope min This is the minimum value among the initial spectral tilts corresponding to all audio data;

[0138] slope max This is the maximum value among the initial spectral tilts corresponding to all audio data.

[0139] Harmonic-to-Noise Ratio (HNR): This measures the ratio of harmonic components to noise components in audio data, reflecting the roughness and clarity of the timbre. It is calculated using the following formula:

[0140]

[0141] Wherein, HNR is the initial harmonic-to-noise ratio corresponding to the audio data;

[0142] P harmonic The power of the harmonic components of the audio data;

[0143] P noise This represents the power of the noise component in the audio data.

[0144] Harmonic-to-noise ratio (HNR) is typically expressed in decibels (dB). Since the decibel scale is logarithmic, direct normalization can lead to uneven numerical distribution. Therefore, the decibel value is first converted to a linear energy ratio to eliminate the effect of logarithmic compression, and then normalized to the [0, 1] interval using the Min-Max method. The calculation formula is as follows:

[0145]

[0146] Wherein, HNR is the initial harmonic-to-noise ratio corresponding to the audio data;

[0147] 10 HNR / 10 The initial harmonic noise ratio for converting decibels to linear energy ratios;

[0148] min(10 HNR / 10 () is the minimum value among the initial harmonic-to-noise ratios corresponding to all audio data converted to linear energy ratios;

[0149] max(10 HNR / 10 The maximum value among the initial harmonic-to-noise ratios corresponding to all audio data converted to linear energy ratios;

[0150] HNR' is the normalized harmonic-to-noise ratio corresponding to the audio data.

[0151] Short-Time Energy (STE): Represents the energy level of audio data within a short time window, characterizing changes in loudness. It is calculated using the following formula:

[0152]

[0153] Where L is the frame length of each frame in the entire speech data;

[0154] x(n, m) represents the audio data of the m-th sample point in the n-th frame;

[0155] STE(n) is the initial short-time energy corresponding to the audio data of the nth frame.

[0156] The short-time energy corresponding to each frame of audio data is calculated using the method described above. Since there are multiple short-time energies, the median of these multiple short-time energies can be used as the initial short-time energy corresponding to the entire audio data.

[0157] The range of short-time energy values ​​is typically large, and the energy differences between different audio data are significant. To avoid the influence of extreme values ​​on the normalization results, Min-Max normalization is used to reduce the impact of nonlinear amplification, while making the energy characteristics more evenly distributed in the [0, 1] interval. The calculation formula is as follows:

[0158]

[0159] Where STE(n) is the initial short-time energy corresponding to the audio data of the nth frame;

[0160] STE(n)' is the normalized short-time energy corresponding to the audio data of the nth frame;

[0161] STE(n) min It is the minimum value among the initial short-time energies corresponding to all audio data;

[0162] STE(n) max It is the maximum value among the initial short-time energies corresponding to all audio data.

[0163] Delta Zero-Crossing Rate (ΔZCR): Measures the degree of change in the zero-crossing rate of a signal between adjacent frames, reflecting the clarity and jitter characteristics of audio data. It is calculated using the following formula:

[0164]

[0165] ΔZCR=ZCR(n)-ZCR(n-1)

[0166] Where Q is the number of sampling points in the current frame;

[0167] x(m) and x(m-1) are the audio data of the current sampling point and the previous sampling point, respectively;

[0168] S(*) is an indicator function, which is 1 when the audio data signs of two adjacent sampling points are different (i.e., they cross the zero axis), and 0 otherwise;

[0169] ZCR(n) is the zero-crossing rate of the current frame, and ZCR(n-1) is the zero-crossing rate of the previous frame;

[0170] ΔZCR is the rate of change of zero-crossing rate between adjacent frames.

[0171] Using the method described above, the rate of change of the zero-crossing rate corresponding to each pair of adjacent audio data is calculated. Since there are multiple rates of change of the zero-crossing rate, the median of these multiple rates of change of the zero-crossing rate can be used as the initial rate of change of the zero-crossing rate corresponding to the entire audio data.

[0172] The data range of the zero-crossing rate of change is well-defined and contains no non-extreme outliers. Min-Max normalization can be directly used to preserve their numerical range and map them to [0, 1], avoiding unnecessary transformations. The calculation formula is as follows:

[0173]

[0174] Wherein, ΔZCR is the rate of change of the initial zero-crossing rate corresponding to the audio data;

[0175] ΔZCR' is the normalized zero-crossing rate of change corresponding to the audio data;

[0176] ΔZCR min This is the minimum value among the initial zero-crossing rate changes corresponding to all audio data;

[0177] ΔZCR max This is the maximum value among the initial zero-crossing rate changes corresponding to all audio data.

[0178] Spectral flatness measures the smoothness of the audio spectrum, reflecting the clarity of the timbre and the distribution of harmonic components in speech. It is calculated using the following formula:

[0179]

[0180] Among them, A s The spectral amplitude corresponding to the s-th frequency band;

[0181] R is the spectrum length;

[0182] K represents the number of frequency bands;

[0183] SF represents the initial spectral flatness of the audio data.

[0184] The data ranges for spectral flatness and short-time autocorrelation coefficients are well-defined and contain no non-extreme outliers. Min-Max normalization can be directly used to preserve their numerical range and map them to [0, 1], avoiding unnecessary transformations. The calculation formula is as follows:

[0185]

[0186] Wherein, SF is the initial spectral flatness corresponding to the audio data;

[0187] SF' represents the normalized spectral flatness of the audio data;

[0188] SF min This is the minimum value of the initial spectral flatness corresponding to all audio data;

[0189] SF max This represents the maximum value among the initial spectral flatness values ​​corresponding to all audio data.

[0190] Short-Time Autocorrelation Coefficient (STAC): Measures the similarity of audio signals themselves, reflecting the stability and periodicity of timbre. It is calculated using the following formula:

[0191]

[0192] Where x(m) is the audio data of the m-th sampling point of the current frame;

[0193] 'a' represents the delay amount.

[0194] Q is the number of sampling points in the current frame;

[0195] x(m+a) is the audio data of the (m+a)th sampling point;

[0196] STAC(a) is the short-time autocorrelation coefficient of the current frame;

[0197] Take multiple values ​​for 'a', calculate multiple short-time autocorrelation coefficients for the current frame, and select the median of the multiple short-time autocorrelation coefficients as the short-time autocorrelation coefficient of the current frame.

[0198] Using the method described above, the short-time autocorrelation coefficient corresponding to each frame of audio data is calculated. Since there are multiple short-time autocorrelation coefficients, the median of these coefficients can be used as the initial short-time autocorrelation coefficient for the entire audio data.

[0199] The short-time autocorrelation coefficients have a well-defined data range and no non-extreme outliers. Min-Max normalization can be used directly to maintain their numerical range and map them to [0, 1], avoiding unnecessary transformations. The calculation formula is as follows:

[0200]

[0201] Wherein, STAC(a) is the initial short-time autocorrelation coefficient corresponding to the audio data;

[0202] STAC(a)' is the normalized short-time autocorrelation coefficient corresponding to the audio data;

[0203] STAC(a) min This is the minimum value among the initial short-time autocorrelation coefficients corresponding to all audio data;

[0204] STAC(a) max This is the maximum value among the initial short-time autocorrelation coefficients corresponding to all audio data.

[0205] In the above embodiments, all explicit acoustic features are normalized to ensure that features of different scales are not dominated by features with large numerical ranges when calculating distance, thereby improving the reliability of timbre distance calculation.

[0206] In one embodiment, after obtaining the normalized implicit acoustic features and the normalized explicit acoustic features, the normalized implicit acoustic features and the normalized explicit acoustic features are concatenated to obtain the comprehensive acoustic features, as specified in the formula:

[0207] M = [X, E] norm ]

[0208] Where M represents the composite acoustic feature after splicing, X represents the normalized implicit acoustic feature, and E represents the composite acoustic feature. norm These are normalized explicit acoustic features.

[0209] Normalized implicit acoustic features can implicitly encode high-order semantic information such as speaker identity and pronunciation habits, excel at capturing long-term statistical patterns, and are sensitive to the global distribution of timbre. Normalized explicit acoustic features possess physical interpretability, are relatively stable at low signal-to-noise ratios due to the preservation of harmonic structure, and are adept at locating short-term acoustic details. Combining the two can cover multi-scale information from microscopic acoustic structure to macroscopic speaker identity: the physical constraints in explicit acoustic features can guide semantic learning, avoiding semantic drift caused by X-vector due to pure data-driven approaches; X-vector, through deep semantic modeling, can capture subtle differences, supplementing the deficiencies of physical features. The combination of the two has a synergistic effect, complementing feature representation, enhancing clustering robustness, and improving clustering accuracy.

[0210] As an optional specific embodiment, step 102, obtaining the cluster center corresponding to each audio cluster, includes:

[0211] Obtain each audio cluster and calculate the membership degree of each audio data to each audio cluster to obtain the membership degree matrix;

[0212] Repeat the following steps until the number of update iterations reaches the iteration threshold or the change in the membership matrix before and after the update is less than the preset threshold, to obtain the cluster centers corresponding to each audio cluster:

[0213] Based on the membership matrix, calculate the cluster center corresponding to each audio cluster;

[0214] Calculate the target distance between each audio data point and the cluster center corresponding to each audio cluster;

[0215] The membership matrix is ​​updated based on the target distance between each audio data point and the cluster center corresponding to each audio cluster.

[0216] Specifically, such as Figure 2 As shown, the number of audio clusters, C, is set, typically to 30 (but can be set as needed). Multiple (e.g., 15) explicit audio clusters are predefined as target audio clusters for dynamic constraint injection. For example, the 15 predefined explicit audio clusters are: High number of followers, Medium number of followers, Low number of followers, High number of views, Medium number of views, Low number of views, High interaction rate, Medium interaction rate, Low interaction rate, High completion rate, Medium completion rate, Low completion rate, High retention rate, Medium retention rate, and Low retention rate. The remaining audio clusters besides these 15 are audio clusters without explicit semantic definitions.

[0217] Initialize the membership degree of each audio data point to each audio cluster. Define the membership degree of the i-th audio data point (i = 1, 2, ..., N, where N represents the number of audio data points) to the j-th audio cluster (j = 1, 2, ..., C, where C is the number of audio clusters) as u. ij u ij The following constraints must be satisfied:

[0218] u ij ∈[0,1]

[0219]

[0220] Among them, u ij This represents the membership degree of the i-th audio data to the j-th audio cluster.

[0221] For ease of explanation, the 15 audio clusters mentioned above are assigned numerical codes from 1 to 15, and the remaining audio clusters are assigned numerical codes from 16 to C. The correspondence is shown in Table 1 below:

[0222] Table 1. Numbering Correspondence Table for Audio Clusters

[0223]

[0224]

[0225] During the initialization process, the membership degree of the audio cluster to which the audio sample belongs is partially predefined and initialized, thereby transforming known information into dynamic constraints to ensure that the cluster has a certain degree of semantic interpretability and accelerate the clustering process.

[0226] After initialization, the iterative process of fuzzy C-means clustering can be performed. The specific steps are as follows:

[0227] Cluster centers v were calculated using a weighted average. j The weights are determined by the membership degree of each audio data point to that audio cluster (i.e., the weight equals the membership degree of the audio data point to that audio cluster):

[0228]

[0229] Among them, v j This represents the cluster center corresponding to the j-th audio cluster;

[0230] Where b is the fuzziness control parameter, which can be set to 2 as needed;

[0231] N represents the number of audio data, and i represents the i-th audio data, i = 1, 2, ..., N;

[0232] C represents the number of audio clusters, and j represents the j-th audio cluster, j = 1, 2, ..., C;

[0233] u ij This represents the membership degree of the i-th audio data to the j-th audio cluster.

[0234] Calculate the target distance between each audio data point and each cluster center. Update the membership matrix based on the target distance between each audio data point and each cluster center using the following formula:

[0235]

[0236] Among them, u ij This represents the membership degree of the i-th audio data to the j-th audio cluster;

[0237] C represents the number of audio clusters, and g represents the g-th audio cluster, where g = 1, 2, ..., C;

[0238] x i This represents the i-th audio data;

[0239] D final (x i ,v j ) represents the target distance between the i-th audio data and the cluster center corresponding to the j-th audio cluster;

[0240] b is the fuzziness control parameter, which can be set to 2 as needed.

[0241] Repeat the above steps until the number of iterations in the clustering process reaches the iteration threshold or the change in the membership matrix before and after the update is less than the preset threshold, and then stop to obtain the cluster center corresponding to each audio cluster.

[0242] As an optional specific embodiment, step 102 calculates the target distance between each audio data and the cluster center corresponding to each audio cluster, including:

[0243] Based on the implicit acoustic features corresponding to each audio data and the implicit acoustic features of each cluster center, the first distance between each audio data and each cluster center is calculated.

[0244] Based on the explicit acoustic features corresponding to each audio data and the explicit acoustic features of each cluster center, the second distance between each audio data and each cluster center is calculated.

[0245] Based on the first and second distances between each audio data point and each cluster center, the target distance between each audio data point and each cluster center is calculated.

[0246] The calculation process of the target distance is illustrated using the i-th audio data and the j-th cluster center as an example:

[0247] The first distance between the i-th audio data and the j-th cluster center is calculated using the cosine distance formula as follows:

[0248]

[0249] Where, x i Let x represent the i-th audio data. il This represents the implicit acoustic feature of the l-th dimension of the i-th audio data.

[0250] v j For the j-th cluster center, v lj The implicit acoustic feature of the l-th dimension of the j-th cluster center;

[0251] D impl (x,v j ) represents the first distance between the i-th audio data and the j-th cluster center.

[0252] To measure the similarity of different audio data in the explicit feature space, Euclidean distance is selected as the metric for explicit acoustic features. Euclidean distance can intuitively reflect the absolute difference between audio data and cluster centers in the multidimensional feature space and is suitable for explicit acoustic features that have been normalized to the range [0, 1].

[0253] The second distance between the i-th audio data and the j-th cluster center is calculated using the following formula:

[0254]

[0255] Among them, D expl (x i ,v j ) represents the second distance between the i-th audio data and the j-th cluster center;

[0256] x ik Let be the explicit acoustic feature of the k-th dimension of the i-th audio data;

[0257] v jk The explicit acoustic feature of the k-th dimension for the j-th cluster center;

[0258] To comprehensively consider the contributions of explicit and implicit acoustic features, a weighted fusion strategy is used to calculate the final target distance:

[0259] D final (x i ,v j )=λD expl (x i ,v j )+(1-λ)D impl (x i ,v j )

[0260] Wherein, λ is the fusion weight coefficient, used to control the importance of implicit acoustic features and explicit acoustic features, and takes a value of [0,1].

[0261] D impl (x,v j ) represents the first distance between the i-th audio data and the j-th cluster center;

[0262] D expl (x i ,v j ) represents the second distance between the i-th audio data and the j-th cluster center;

[0263] D final (x i ,v j ) represents the target distance between the i-th audio data and the j-th cluster center.

[0264] The target distance between each audio data point and each cluster center is calculated using the method described above.

[0265] As an optional specific embodiment, the step of calculating the membership degree of each audio data belonging to each audio cluster to obtain the membership degree matrix specifically includes:

[0266] Obtain the target data corresponding to each audio cluster, wherein the target data includes at least one of the following: the broadcaster's fan data, video viewing data, interaction data, video completion data, and audience retention data;

[0267] Based on the target data corresponding to each audio cluster, the target data score corresponding to each audio cluster is calculated. The target data includes at least one of the following: fan count score, viewership score, interaction rate score, completion rate score, retention rate score, and popularity score.

[0268] Based on the target data score corresponding to each audio cluster, the membership degree of each audio data belongs to each audio cluster is calculated, and the membership degree matrix is ​​obtained.

[0269] Specifically, in order to quantify the influence and popularity of videos, normalizing and multi-dimensionally scoring video metadata can more accurately match the timbre features in specific scenarios, thereby improving the adaptability and naturalness of synthesized speech.

[0270] To comprehensively measure the popularity of videos, multi-dimensional evaluation metrics (i.e., target data) are used, including but not limited to:

[0271] Number of followers (i.e., the streamer's follower data): measures the basic size of the streamer's audience.

[0272] Views (i.e., video view data): reflects the breadth of a video's reach.

[0273] Interaction rate (i.e., interaction data): includes user interaction behaviors such as likes, comments, and shares, reflecting the audience's level of participation.

[0274] Completion rate (i.e., video completion data): measures whether viewers watch the entire video and reflects the attractiveness of the video content.

[0275] Retention rate (i.e., audience retention data): This refers to the percentage of viewers who regress to the streamer or video, and measures the long-term interest of viewers in the streamer or video.

[0276] To ensure comparability across dimensions in the comprehensive scoring model, each indicator needs to be normalized. Simultaneously, to eliminate feature overload caused by differences in feature numerical ranges, the normalization range is standardized to [0,1]:

[0277] Due to the significant differences in the number of followers among different streamers, maximum value normalization is used to prevent data bias:

[0278]

[0279] Here, "current fan count" refers to the number of fans of the current streamer, and "maximum fan count" refers to the maximum number of fans among all streamers. This method ensures that the fan count score is distributed in the range [0,1], eliminating the scoring unfairness caused by the fan count advantage of large streamers.

[0280] To prevent excessive differences in viewership from causing rating imbalances, arctangent function normalization is used:

[0281] Viewership score = arctan(expansion coefficient * total viewership)

[0282] Here, arctan() is the arctangent function, total views refer to the cumulative number of views of all videos by the current streamer, and the expansion coefficient is used to adjust the effective activation range of the total views. Generally, setting the expansion coefficient to 10000 can adjust the effective activation range to 0-100000. This is because when a video has more than 100000 views, it has often been recommended by the platform as popular and high-quality content, resulting in a geometric increase in views. If a high expansion coefficient is not set, the weight of some samples in subsequent training may be too high, thus affecting the fairness of the model. Through the combination of the arctangent function and the expansion coefficient, the view score is normalized to the [0,1) interval. This can avoid the imbalance of scores caused by some streamers having excessively high view counts when comparing streamers with different view count levels.

[0283] Interaction rate is a key indicator reflecting audience engagement, and it is calculated as follows:

[0284]

[0285] Here, likes represent the total number of likes for all the streamer's videos, comments represent the total number of comments for all the streamer's videos, and shares represent the total number of shares for all the streamer's videos. This formula calculates the average interaction frequency per viewer, and normalization ensures a fair comparison of interaction scores for streamers with different numbers of views.

[0286] Completion rate measures whether viewers watch a video in its entirety, highlighting content quality and appeal. It is calculated using normalized methods.

[0287]

[0288] Among them, the number of viewers who completed the broadcast refers to the sum of the number of viewers who completed each of the streamer's videos, while the total number of viewers refers to the number of viewers for each of the streamer's videos.

[0289] Retention rate measures audience engagement and reflects the long-term attention viewers have of the streamer. The normalized formula is:

[0290]

[0291] Returning viewers refer to the number of viewers who watched other videos by the streamer within a week. Total viewership refers to the total number of viewers who watched all of the streamer's videos. The normalized retention rate measures the streamer's long-term appeal to viewers and helps distinguish between content that becomes popular only temporarily and content that remains popular over time.

[0292] The popularity score can be calculated by summing all the normalized metrics according to their weights.

[0293] Popularity score = w1 * number of fans score + w2 * number of views score + w3 * interaction rate score + w4 * completion rate score + w5 * retention rate score

[0294] Among them, w1, w2, w3, w4, and w5 are custom weight values ​​that can be dynamically adjusted according to specific scenarios. For example, in news broadcasting scenarios, the weight of viewership and completion rate scores can be increased; in entertainment scenarios, the weight of interaction rate and retention rate scores is higher; and in brand marketing scenarios, the weight of fan count score can be increased. This flexibility allows the system to optimize the characteristics of synthesized speech according to specific needs. By dynamically adjusting the weights, the speech synthesis system can capture and enhance the key features of the target timbre, meet diverse scenario requirements, and improve user experience. This method not only improves the naturalness and applicability of synthesized speech but also provides developers with greater design space to innovate and optimize voice products.

[0295] Based on the above scores for fan count, viewership, interaction rate, completion rate, and retention rate, the initial membership degree of each audio data point to each audio cluster is calculated, resulting in the initial membership degree matrix. The specific process is as follows:

[0296] During initialization, the membership degrees of the audio clusters to which the audio data belongs are partially predefined and initialized. This transforms known information into dynamic constraints, ensuring a certain degree of semantic interpretability for the clusters and accelerating the clustering process. The membership degree initialization process is described in detail below:

[0297] For each audio data point, it is categorized into three levels—high, medium, and low—based on normalized follower count scores and view count scores. The corresponding score thresholds are: [0, 0.25] for low, [0.25, 0.75] for medium, and [0.75, 1] ​​for high. Then, the membership degree of the corresponding audio cluster is set. For example, for an audio data point i with a normalized follower count score of 0.45, this audio data point belongs to the "medium" audio cluster, so its membership degree u to the "medium" audio cluster is set. i2 Let σ be the membership degree u for audio clusters with high "fan count score" and low "fan count score". i1 u i3The score is 0. The same applies to the view count score, interaction rate score, completion rate score, and retention rate score designed for other predefined audio clusters.

[0298] The membership initialization steps for the remaining audio clusters are described below:

[0299] First, random samples are taken from the standard normal distribution N(0,1):

[0300] u ij ~N(0,1), j=16,…,C

[0301] Then, truncation is performed to ensure that the value is between [-1, 1].

[0302]

[0303] And normalize it:

[0304]

[0305] This ensures that the sum of the membership degrees of the remaining audio clusters is 1 - 15 * σ.

[0306]

[0307] Preferably, to ensure a certain pre-injection dynamic constraint strength and prevent clustering from getting trapped in local optima, σ is set to 0.04.

[0308] As an optional specific embodiment, step 103, based on the target distance between each audio data and the cluster center corresponding to each audio cluster, obtains the semantic cluster corresponding to each audio cluster, including:

[0309] For each audio cluster, audio data whose target distance to the cluster center of the audio cluster is less than a distance threshold is identified as the target audio data corresponding to the audio cluster, and the text in each target audio data is extracted to obtain the target document corresponding to each target audio data of the audio cluster.

[0310] The potential Dirichlet assignment model is iteratively trained based on the target documents corresponding to the audio clusters, and topic consistency is calculated during the iteration process until the topic consistency meets the convergence condition. This process is then stopped to obtain the document-topic probability distribution and topic-word probability distribution of the audio clusters.

[0311] Based on the document-topic probability distribution and topic-word probability distribution of the audio clusters, multiple high-frequency words with a frequency greater than a preset frequency in the audio clusters are taken as semantic clusters of the audio clusters.

[0312] Optionally, the steps described above for calculating topic consistency during the iteration process specifically include:

[0313] Obtain the topic-word count matrix of the audio clustering cluster in the current iteration, and calculate the topic-word probability distribution according to the topic-word count matrix;

[0314] Determine multiple high-frequency words corresponding to the audio clustering cluster according to the topic-word probability distribution;

[0315] Calculate the topic consistency in the current iteration according to the word probability of each high-frequency word corresponding to the audio clustering cluster and the word co-occurrence probability of every two high-frequency words.

[0316] Specifically, for each audio clustering cluster, select the target distance D final (x i ,v j ) of the audio data with a distance less than the distance threshold (which can be set as needed, preferably 0.08) in the audio clustering cluster as the target audio data corresponding to the audio clustering cluster, use an automatic speech recognition model to extract the text in the target audio data, and convert it into a piece of text, that is, the target document corresponding to the target audio data. In the above way, the target documents of the target audio data in each audio clustering cluster can be obtained, and all the target documents can form a target document set.

[0317] To improve the normalization of the target document, preprocess the above target document as follows:

[0318] Remove non-verbal symbols (such as HyperText Markup Language (HTML) tags, special characters, etc.) in the target document through regular expressions, and retain valid Chinese characters. If it involves traditional Chinese text, it needs to be converted from traditional to simplified to ensure the consistency of the corpus. For example, for a text with mixed pinyin (such as "你好(nǐhǎo)"), it is necessary to separate the semantic units and perform standardization processing. Remove redundant information (such as news signatures, advertising links, etc.) through regular matching to maintain semantic purity.

[0319] Use a hybrid strategy to optimize the word segmentation process: Combine dictionary-driven (such as the longest match in the Modern Chinese Dictionary) and statistical models (Hidden Markov Model, Conditional Random Field) to handle the problem of new word recognition. For example, use the Hidden Markov Model (HMM) to divide word boundaries according to the context probability to solve the segmentation bottleneck of out-of-vocabulary words (such as "blockchain"). At the same time, extract proper nouns (such as "XX University") through Named Entity Recognition (NER) technology to improve the accuracy of word segmentation.

[0320] Apply the stemming technology to merge cognate variants, for example, normalize the verb reduplication form "看看" to "看". Combine the domain dictionary to expand the stop word list, and filter out non-content words (such as "的", "是") and low-frequency noise words (such as: word frequency lower than the word frequency threshold of 5).

[0321] For each audio clustering cluster, iteratively train the Latent Dirichlet Allocation (LDA) topic model based on the preprocessed target documents corresponding to the audio clustering cluster, and calculate the topic coherence during the iteration process until the topic coherence meets the convergence condition and stops. Obtain the document-topic probability distribution and topic-word probability distribution of the audio clustering cluster, and based on the document-topic probability distribution and topic-word probability distribution of the audio clustering cluster, use multiple high-frequency words with frequencies greater than the preset frequency in the audio clustering cluster as the semantic cluster of the audio clustering cluster. Obtain the semantic clusters of each audio clustering cluster in the above manner. The specific steps are as follows:

[0322] Comprehensively evaluate the quality of the LDA model through perplexity and coherence score. The formula for calculating the perplexity corresponding to an audio clustering cluster (i.e., a topic) is:

[0323]

[0324] where H represents the total number of target documents under a topic;

[0325] Nd represents the total number of words in the d-th target document;

[0326] P(w|d) represents the probability that the w-th word appears in the d-th target document.

[0327] Use the Markov chain Monte Carlo method Gibbs sampling or variational inference to optimize the model parameters. The update method of the document-topic probability distribution is:

[0328]

[0329] where, represents the probability value that the d-th target document in the current iteration belongs to the k-th topic;

[0330] n dk represents the number of words assigned to the k-th topic in the d-th target document;

[0331] C represents the total number of topics, that is, the total number of audio clustering clusters;

[0332] n d represents n dk represents the total number of words in the d-th target document;

[0333] α represents the document-topic density value. α is adjusted through a grid search, and an asymmetric prior α is used to accommodate the heterogeneity of topic distribution. The Dirichlet prior parameter α is set as follows:

[0334]

[0335] The topic-word probability distribution is updated as follows:

[0336]

[0337] in, This represents the probability value of the w-th word in the k-th topic of the current iteration;

[0338] n kw This represents the number of times the w-th word appears in the k-th topic;

[0339] n k This represents the total number of words for the k-th topic;

[0340] V represents the vocabulary size, which is the total number of words obtained after deduplication from all target documents.

[0341] β represents the topic-word density value, which is adjusted through grid search. The Dirichlet prior parameter β is set to 0.001.

[0342] Random initialization: Randomly assign a topic to each word.

[0343] z∈{1, ...,C}

[0344] Statistical counting matrix n dk and n kw ;

[0345] Begin the Gibbs sampling iteration process. For each word w, perform the following operations:

[0346] Subtracting the current count, from n dk and n kw Subtract the original topic assignment count of the word from the middle.

[0347] The probability P(z=C|w,d) of sampling a new topic z:

[0348]

[0349] Where, n kw This represents the number of times the w-th word appears in the k-th topic;

[0350] n dk This represents the number of words assigned to the k-th topic in the d-th target document;

[0351] α represents the document-topic density value;

[0352] β represents the topic-word density value;

[0353] C represents the total number of topics, and j takes values ​​from 1 to C;

[0354] V represents the vocabulary size, with values ​​ranging from 1 to V.

[0355] New topics z are extracted based on the probability distribution, and the counting matrix n is updated. kw and n dk For example, if a word has a probability of 0.4 belonging to topic A and a probability of 0.6 belonging to topic B, this probability distribution indicates that the word belongs to topic B. If each word is sampled 10 times, and topic A is sampled 8 times, then the word is determined to belong to topic A. By sampling topics for each word in this way, the number of words assigned to the k-th topic in the d-th target document and the frequency of the w-th word in the k-th topic are updated.

[0356] Select M high-frequency words for each topic based on the topic-word probability distribution:

[0357]

[0358] Among them, S k This represents the set of M high-frequency words under the k-th topic;

[0359] Represents the topic-word probability distribution;

[0360] w represents the w-th word.

[0361] For the set S of M high-frequency words under the k-th topic k To calculate topic consistency, it can be measured using the pointwise mutual information (PMI) of top term pairs. The topic PMI index (i.e., PMI score) is monitored at regular intervals (e.g., 100 steps).

[0362]

[0363] Wherein, PMI(k) represents topic consistency under the k-th topic;

[0364] M represents set S k The number of high-frequency words in the text, where i ranges from 1 to M-1 and j ranges from 2 to M;

[0365] P(w i w j ) indicates w i w j The co-occurrence probability under the k-th topic;

[0366] P(w i ) indicates w i The probability of appearing under the k-th topic, P(w) j ) indicates w j The probability of occurrence under the k-th topic.

[0367] Calculate the word probability P(w) i ) and P(w j The formula is as follows:

[0368]

[0369] Where M represents the number of high-frequency words, and r takes values ​​from 1 to M;

[0370] P(w i w j ) indicates w i w j Co-occurrence probability under the current topic;

[0371] P(w i ) indicates w i The probability of occurrence under the current topic, P(w) j ) indicates w j The probability of it appearing under the current topic;

[0372] count(w i ) indicates w i Number of times it appears under the current topic;

[0373] count(w j ) indicates w j Number of times it appears under the current topic;

[0374] count(r) represents the number of times the r-th high-frequency word appears in the current topic.

[0375] Calculate the co-occurrence probability P(w) i w j The formula is as follows:

[0376]

[0377] Among them, P(w i w j ) indicates w i w j Co-occurrence probability under the current topic;

[0378] co_count(w i w j ) indicates w i w jThe number of times they appear together under the current topic;

[0379] W m w n Each represents any two high-frequency words;

[0380] co_count(w m w n ) indicates w m w n The number of times they appear together under the current topic.

[0381] The iteration stops when the PMI score stabilizes (e.g., a PMI score > 0.4 indicates a valid topic), and the results are output, including the document-topic probability distribution and the topic-word probability distribution.

[0382]

[0383] Where, θ dk This represents the probability that the d-th target document belongs to the k-th topic;

[0384] n dk This represents the number of words assigned to the k-th topic in the d-th target document;

[0385] C represents the total number of topics, i.e. the total number of audio clusters, and f takes values ​​from 1 to C;

[0386] n df This represents the total number of words in the d-th target document under the f-th topic;

[0387] α represents the document-topic density value;

[0388] This represents the probability value of the w-th word in the k-th topic;

[0389] n kw This represents the number of times the w-th word appears in the k-th topic;

[0390] n kv This represents the number of times the v-th word appears in the k-th topic;

[0391] V indicates the size of the vocabulary;

[0392] β represents the topic-word density value.

[0393] Based on the document-topic probability distribution and the topic-word probability distribution, M high-probability words (i.e., high-frequency words with a frequency greater than a preset frequency) are obtained under each topic, and the M high-frequency words are used to form the topic semantic cluster of that topic.

[0394] The above embodiments, by performing automatic speech recognition on audio data in audio clusters where the target distance is less than a distance threshold to extract text content, and using LDA topic model for semantic mining, can obtain the semantic topic of each audio cluster, providing users with a more intuitive and interpretable basis for timbre selection, and improving the accuracy and personalization of timbre selection.

[0395] In summary, the embodiments described above in this application normalize implicit and explicit features to eliminate the impact of dimensional differences on distance calculation. By combining explicit and implicit acoustic features, introducing a clustering algorithm with dynamic semantic constraints, and automating the collection and processing of large-scale video and audio data through web crawlers, more efficient and accurate timbre clustering and selection are achieved, providing higher-quality timbre features for speech synthesis and meeting diverse and personalized speech synthesis needs. Dynamic semantic constraints are introduced during the clustering process to improve the semantic interpretability of the clusters. After clustering, automatic speech recognition is performed on audio data with high membership in the clustering results to extract text content, and semantic mining is performed using an LDA model to obtain the semantic theme of each audio cluster. Finally, through the core method, intra-cluster random sampling method, and core weighting method, users can independently select the target timbre based on the cluster semantics and application scenario. Clustering controls the mixing ratio of different timbre prototypes through membership weights, achieving gradual rather than abrupt timbre changes, allowing users to continuously adjust the timbre according to their preferences during use.

[0396] The speech synthesis method provided by the embodiments of this application has been described above. The speech synthesis device provided by the embodiments of this application will be described below with reference to the accompanying drawings.

[0397] like Figure 3 As shown in the figure, this application embodiment also provides a speech synthesis device 300, the device comprising:

[0398] The first acquisition module 301 is used to acquire the comprehensive acoustic features corresponding to each audio data, wherein the comprehensive acoustic features include: implicit acoustic features and explicit acoustic features.

[0399] The calculation module 302 is used to obtain the cluster center corresponding to each audio cluster and calculate the target distance between each audio data and the cluster center corresponding to each audio cluster.

[0400] The second acquisition module 303 is used to acquire the semantic clusters corresponding to each audio cluster based on the target distance between each audio data and the cluster center corresponding to each audio cluster.

[0401] The determining module 304 is used to determine the target timbre based on the cluster center of the audio clustering cluster corresponding to the semantic cluster when a target semantic cluster is determined among multiple semantic clusters;

[0402] The synthesis module 305 is used to input the target text and the target timbre into the speech synthesis model to obtain target speech with the target timbre.

[0403] Optionally, when the comprehensive acoustic features include implicit acoustic features, the first acquisition module 301 is specifically used for:

[0404] Preprocess each audio data to obtain the underlying acoustic features corresponding to each audio data;

[0405] The underlying acoustic features corresponding to each audio data are processed sequentially through a TDNN network, a statistical pooling layer, and a fully connected layer to obtain the initial implicit acoustic features corresponding to each audio data.

[0406] The initial implicit acoustic features corresponding to each audio data are normalized to obtain the implicit acoustic features corresponding to each audio data.

[0407] Optionally, when the comprehensive acoustic features include explicit acoustic features, the first acquisition module 301 is specifically used for:

[0408] Obtain the initial explicit acoustic features corresponding to each audio data, wherein the initial explicit acoustic features include at least one of the following: fundamental frequency, spectral tilt, harmonic-to-noise ratio, spectral flatness, short-time energy, short-time autocorrelation coefficient, and zero-crossing rate of change.

[0409] The initial explicit acoustic features corresponding to each audio data are normalized to obtain the explicit acoustic features corresponding to each audio data.

[0410] Optionally, when obtaining the cluster centers corresponding to each audio cluster, the calculation module 302 is specifically used for:

[0411] Obtain each audio cluster and calculate the membership degree of each audio data to each audio cluster to obtain the membership degree matrix;

[0412] Repeat the following steps until the number of update iterations reaches the iteration threshold or the change in the membership matrix before and after the update is less than the preset threshold, to obtain the cluster centers corresponding to each audio cluster:

[0413] Based on the membership matrix, calculate the cluster center corresponding to each audio cluster;

[0414] Calculate the target distance between each audio data point and the cluster center corresponding to each audio cluster;

[0415] The membership matrix is ​​updated based on the target distance between each audio data point and the cluster center corresponding to each audio cluster.

[0416] Optionally, when calculating the target distance between each audio data point and the cluster center corresponding to each audio cluster, the calculation module 302 is specifically used for:

[0417] Based on the implicit acoustic features corresponding to each audio data and the implicit acoustic features of each cluster center, the first distance between each audio data and each cluster center is calculated.

[0418] Based on the explicit acoustic features corresponding to each audio data and the explicit acoustic features of each cluster center, the second distance between each audio data and each cluster center is calculated.

[0419] Based on the first and second distances between each audio data point and each cluster center, the target distance between each audio data point and each cluster center is calculated.

[0420] Optionally, when calculating the membership degree of each audio data belonging to each audio cluster and obtaining the membership degree matrix, the calculation module 302 is specifically used for:

[0421] Obtain the target data corresponding to each audio cluster, wherein the target data includes at least one of the following: the broadcaster's fan data, video viewing data, interaction data, video completion data, and audience retention data;

[0422] Based on the target data corresponding to each audio cluster, the target data score corresponding to each audio cluster is calculated. The target data includes at least one of the following: fan count score, viewership score, interaction rate score, completion rate score, retention rate score, and popularity score.

[0423] Based on the target data score corresponding to each audio cluster, the membership degree of each audio data belongs to each audio cluster is calculated, and the membership degree matrix is ​​obtained.

[0424] Optionally, the second acquisition module 303 is specifically used for:

[0425] For each audio cluster, audio data whose target distance to the cluster center of the audio cluster is less than a distance threshold is identified as the target audio data corresponding to the audio cluster, and the text in each target audio data is extracted to obtain the target document corresponding to each target audio data of the audio cluster.

[0426] The potential Dirichlet assignment model is iteratively trained based on the target documents corresponding to the audio clusters, and topic consistency is calculated during the iteration process until the topic consistency meets the convergence condition. This process is then stopped to obtain the document-topic probability distribution and topic-word probability distribution of the audio clusters.

[0427] Based on the document-topic probability distribution and topic-word probability distribution of the audio clusters, multiple high-frequency words with a frequency greater than a preset frequency in the audio clusters are taken as semantic clusters of the audio clusters.

[0428] Optionally, when calculating topic consistency during the iteration process, the second acquisition module 303 is specifically used for:

[0429] Obtain the topic-word count matrix of the audio cluster in the current iteration, and calculate the topic-word probability distribution based on the topic-word count matrix;

[0430] Based on the topic-word probability distribution, determine multiple high-frequency words corresponding to the audio clusters;

[0431] The topic consistency in the current iteration is calculated based on the word probability of each high-frequency word corresponding to the audio cluster and the word co-occurrence probability of every two high-frequency words.

[0432] It should be noted that the speech synthesis apparatus provided in this application embodiment can implement all the method steps implemented in the speech synthesis method embodiment and can achieve the same technical effect. Therefore, the parts and beneficial effects that are the same as those in the method embodiment will not be described in detail here.

[0433] It should be noted that the division of units in the embodiments of this application is illustrative and only represents one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units.

[0434] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a processor-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0435] like Figure 4 As shown, embodiments of this application also provide an electronic device, including a memory 620, a transceiver 610, and a processor 600:

[0436] Memory 620 is used to store computer programs;

[0437] Transceiver 610 is used to send and receive data under the control of the processor;

[0438] The processor 600 is configured to read a computer program from a memory and execute the steps of the speech synthesis method as described in any of the above embodiments.

[0439] Among them, Figure 4 In this context, the bus architecture can include any number of interconnected buses and bridges, specifically linking various circuits together, represented by one or more processors (processor 600) and memory (memory 620). The bus architecture can also link together various other circuits such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. The bus interface provides an interface. The transceiver 610 can be multiple elements, including transmitters and receivers, providing a unit for communicating with various other devices over transmission media, including wireless channels, wired channels, optical fibers, etc. The processor 600 is responsible for managing the bus architecture and general processing, and the memory 620 can store data used by the processor 600 during operation.

[0440] The processor 600 can be a central processing unit (CPU), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or a complex programmable logic device (CPLD). The processor can also adopt a multi-core architecture.

[0441] The processor executes the speech synthesis method provided in this application embodiment by calling a computer program stored in memory, according to the obtained executable instructions. The processor and memory can also be physically separated.

[0442] It should be noted that the electronic device provided in this application embodiment can implement all the method steps implemented in the above speech synthesis method embodiment and can achieve the same technical effect. Here, the parts that are the same as those in the method embodiment and the beneficial effects will not be described in detail.

[0443] Embodiments of this application also provide a processor-readable storage medium storing a computer program for causing the processor to execute the above-described speech synthesis method.

[0444] The processor-readable storage medium can be any available medium or data storage device that the processor can access, including but not limited to magnetic memory (e.g., floppy disk, hard disk, magnetic tape, magneto-optical disk (MO)), optical memory (e.g., CD, DVD, BD, HVD), and semiconductor memory (e.g., ROM, EPROM, EEPROM, non-volatile memory (NAND FLASH), solid-state drive (SSD)).

[0445] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0446] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-executable instructions. These computer-executable instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0447] These processor-executable instructions may also be stored in a processor-readable memory that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the processor-readable memory produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0448] These processors can execute instructions that can also be loaded onto a computer or other programmable data processing device, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0449] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the comprehensive acoustic features corresponding to each audio data, wherein the comprehensive acoustic features include: implicit acoustic features and explicit acoustic features; Obtain the cluster center corresponding to each audio cluster, and calculate the target distance between each audio data and the cluster center corresponding to each audio cluster; Based on the target distance between each audio data and the cluster center corresponding to each audio cluster, the semantic cluster corresponding to each audio cluster is obtained; In the case of determining the target semantic cluster among multiple semantic clusters, the target timbre is determined based on the cluster center of the audio cluster corresponding to the semantic cluster; The target text and the target timbre are input into the speech synthesis model to obtain target speech with the target timbre.

2. The method according to claim 1, characterized in that, When the integrated acoustic features include implicit acoustic features, the step of obtaining the integrated acoustic features corresponding to each audio data includes: Preprocess each audio data to obtain the underlying acoustic features corresponding to each audio data; The underlying acoustic features corresponding to each audio data are processed sequentially through a TDNN network, a statistical pooling layer, and a fully connected layer to obtain the initial implicit acoustic features corresponding to each audio data. The initial implicit acoustic features corresponding to each audio data are normalized to obtain the implicit acoustic features corresponding to each audio data.

3. The method according to claim 1, characterized in that, When the integrated acoustic features include explicit acoustic features, the step of acquiring the integrated acoustic features corresponding to each audio data includes: Obtain the initial explicit acoustic features corresponding to each audio data, wherein the initial explicit acoustic features include at least one of the following: fundamental frequency, spectral tilt, harmonic-to-noise ratio, spectral flatness, short-time energy, short-time autocorrelation coefficient, and zero-crossing rate of change. The initial explicit acoustic features corresponding to each audio data are normalized to obtain the explicit acoustic features corresponding to each audio data.

4. The method according to claim 1, characterized in that, The step of obtaining the cluster centers corresponding to each audio cluster includes: Obtain each audio cluster and calculate the membership degree of each audio data to each audio cluster to obtain the membership degree matrix; Repeat the following steps until the number of update iterations reaches the iteration threshold or the change in the membership matrix before and after the update is less than the preset threshold, to obtain the cluster centers corresponding to each audio cluster: Based on the membership matrix, calculate the cluster center corresponding to each audio cluster; Calculate the target distance between each audio data point and the cluster center corresponding to each audio cluster; The membership matrix is ​​updated based on the target distance between each audio data point and the cluster center corresponding to each audio cluster.

5. The method according to claim 4, characterized in that, The calculation of the target distance between each audio data point and the cluster center corresponding to each audio cluster includes: Based on the implicit acoustic features corresponding to each audio data and the implicit acoustic features of each cluster center, the first distance between each audio data and each cluster center is calculated. Based on the explicit acoustic features corresponding to each audio data and the explicit acoustic features of each cluster center, the second distance between each audio data and each cluster center is calculated. Based on the first and second distances between each audio data point and each cluster center, the target distance between each audio data point and each cluster center is calculated.

6. The method according to claim 4, characterized in that, The calculation of the membership degree of each audio data point to each audio cluster, resulting in a membership degree matrix, includes: Obtain the target data corresponding to each audio cluster, wherein the target data includes at least one of the following: the broadcaster's fan data, video viewing data, interaction data, video completion data, and audience retention data; Based on the target data corresponding to each audio cluster, the target data score corresponding to each audio cluster is calculated. The target data includes at least one of the following: fan count score, viewership score, interaction rate score, completion rate score, retention rate score, and popularity score. Based on the target data score corresponding to each audio cluster, the membership degree of each audio data belongs to each audio cluster is calculated, and the membership degree matrix is ​​obtained.

7. The method according to claim 1, characterized in that, The step of obtaining the semantic clusters corresponding to each audio cluster based on the target distance between each audio data and the cluster center corresponding to each audio cluster includes: For each audio cluster, audio data whose target distance to the cluster center of the audio cluster is less than a distance threshold is identified as the target audio data corresponding to the audio cluster, and the text in each target audio data is extracted to obtain the target document corresponding to each target audio data of the audio cluster. The potential Dirichlet assignment model is iteratively trained based on the target documents corresponding to the audio clusters, and topic consistency is calculated during the iteration process until the topic consistency meets the convergence condition. This process is then stopped to obtain the document-topic probability distribution and topic-word probability distribution of the audio clusters. Based on the document-topic probability distribution and topic-word probability distribution of the audio clusters, multiple high-frequency words with a frequency greater than a preset frequency in the audio clusters are taken as semantic clusters of the audio clusters.

8. The method according to claim 7, characterized in that, The calculation of topic consistency during the iteration process includes: Obtain the topic-word count matrix of the audio cluster in the current iteration, and calculate the topic-word probability distribution based on the topic-word count matrix; Based on the topic-word probability distribution, determine multiple high-frequency words corresponding to the audio clusters; The topic consistency in the current iteration is calculated based on the word probability of each high-frequency word corresponding to the audio cluster and the word co-occurrence probability of every two high-frequency words.

9. A speech synthesis device, characterized in that, The device includes: The first acquisition module is used to acquire the comprehensive acoustic features corresponding to each audio data, the comprehensive acoustic features including: implicit acoustic features and explicit acoustic features; The calculation module is used to obtain the cluster center corresponding to each audio cluster and calculate the target distance between each audio data and the cluster center corresponding to each audio cluster. The second acquisition module is used to acquire the semantic clusters corresponding to each audio cluster based on the target distance between each audio data and the cluster center corresponding to each audio cluster. The determination module is used to determine the target timbre based on the cluster center of the audio cluster corresponding to the semantic cluster when a target semantic cluster is determined among multiple semantic clusters; The synthesis module is used to input the target text and the target timbre into the speech synthesis model to obtain target speech with the target timbre.

10. An electronic device, characterized in that, Includes memory, transceiver, and processor: Memory, used to store computer programs; Transceiver, used to send and receive data under the control of the processor; A processor for reading a computer program from the memory and executing the speech synthesis method as described in any one of claims 1 to 8.

11. A processor-readable storage medium, characterized in that, The processor-readable storage medium stores a computer program for causing the processor to perform the speech synthesis method as described in any one of claims 1 to 8.

Images