Voice editing processing method and device and electronic equipment
By constructing the latent semantic direction in the h-space of the diffusion model and determining the latent semantic direction using supervised or unsupervised methods, the problems of existing speech generation models being unable to capture text sentiment and high-cost editing are solved, thus achieving flexible and efficient speech editing.
Patent Information
- Application Number
- CN202510845495.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-23
- Publication Date
- 2025-10-17
AI Technical Summary
Existing speech generation models struggle to accurately capture the emotional tone, contextual intent, or implicit information in text, resulting in synthesized speech lacking the nuanced layers of natural human speech, and the speech editing process is costly.
By constructing the latent semantic directions in the h-space of the diffusion model, and determining the latent semantic directions using supervised or unsupervised methods, the diffusion processing is performed on the speech to be edited based on the initial speech feature attributes, thereby achieving the editing of the target speech feature attributes.
Flexible voice editing is achieved without modifying the model, reducing the cost of voice editing and improving editing efficiency.
Smart Images
Figure CN120808751A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of speech editing, and in particular to a speech editing processing method and device and electronic equipment. BACKGROUND
[0002] In the field of speech synthesis and editing, with the rapid development of artificial intelligence technology, the demand for naturalness, expressiveness and controllability of generated speech is increasing.
[0003] Although traditional speech generation models can generate speech based on text input, their deep understanding of text semantics has not been fully explored. For example, the model is difficult to accurately capture the emotional tendency, contextual intention or implicit information in the text, resulting in a lack of natural fine-grained level in the tone, pause and emotional expression of the synthesized speech. In addition, in the process of speech editing, the current mainstream methods (such as PromptTTS and NaturalSpeech 2) attempt to control the style of speech through a prompt mechanism, but the control dimension is still single, and it also needs a large amount of labeled data (such as artificially labeled acoustic feature labels) or relies on model fine-tuning, which increases the cost of speech editing.
[0004] Therefore, finding a flexible speech editing processing method that can reduce the cost of speech editing has become a current research hotspot. SUMMARY
[0005] The present application provides a speech editing processing method, device and electronic equipment, which realizes flexible speech editing and reduces the cost of speech editing.
[0006] The present application provides a speech editing processing method, which comprises: obtaining a speech to be edited, and obtaining initial speech feature attributes corresponding to the speech to be edited; determining the latent semantic direction of the h space in the diffusion model in a supervised manner and / or an unsupervised manner; in the process of diffusion processing of the speech to be edited based on the h space in the diffusion model, based on the initial speech feature attributes, the speech to be edited is edited along the latent semantic direction to obtain a target speech with target speech feature attributes.
[0007] According to the speech editing processing method provided by the application, the potential semantic direction of the h space is determined in the following supervised manner: a speech sample pair is obtained, wherein the speech sample pair includes a first speech sample and a second speech sample, and the first speech sample and the second speech sample are speech samples with opposite attribute characteristics in the same attribute characteristic dimension; attribute characteristic extraction is performed on the speech sample pair respectively to obtain a first speech sample attribute characteristic of the first speech sample and a second speech sample attribute characteristic of the second speech sample; a first latent code corresponding to the first speech sample attribute characteristic is obtained based on the first speech sample attribute characteristic, and a second latent code corresponding to the second speech sample attribute characteristic is obtained based on the second speech sample attribute characteristic; a semantic characteristic change vector of the speech sample pair in the attribute characteristic dimension is obtained based on the first latent code and the second latent code; and the semantic characteristic change vector is taken as the potential semantic direction of the h space.
[0008] According to the speech editing processing method provided by the application, the speech sample pair is a plurality of speech sample pairs, the corresponding first latent code is a plurality of first latent codes, and the second latent code is a plurality of second latent codes; and the semantic characteristic change vector of the speech sample pair in the attribute characteristic dimension is obtained based on the first latent code and the second latent code, specifically including: taking the average value of the difference values of the plurality of first latent codes and the plurality of second latent codes corresponding to the plurality of first latent codes as the semantic characteristic change vector of the speech sample pair in the attribute characteristic dimension.
[0009] According to the speech editing processing method provided by the application, the potential semantic direction of the h space is determined in the following unsupervised manner: a speech sample is obtained; a bottleneck feature corresponding to the speech sample is obtained based on the speech sample; principal component feature extraction is performed on the bottleneck feature based on principal component analysis to obtain a main bottleneck speech feature, and the potential semantic direction of the h space is determined based on the main bottleneck speech feature, wherein the feature contribution degree of the main bottleneck speech feature is greater than a feature contribution degree threshold.
[0010] According to the speech editing processing method provided by the application, the potential semantic direction of the h space is determined based on the main bottleneck speech feature, specifically including: the connection of the main bottleneck speech feature across time steps is determined based on the main bottleneck speech feature at each time step; and the potential semantic direction of the h space is determined based on the connection of the main bottleneck speech feature across each time step.
[0011] According to the speech editing processing method provided by the application, before the speech editing processing is performed on the speech to be edited along the potential semantic direction based on the initial speech feature attribute to obtain the target speech with the target speech feature attribute, the method further comprises: determining an adjustment parameter, wherein the adjustment parameter is used to control the editing intensity of the speech editing processing performed on the speech to be edited along the potential semantic direction; and the speech editing processing is performed on the speech to be edited along the potential semantic direction based on the initial speech feature attribute to obtain the target speech with the target speech feature attribute, specifically comprising: performing the speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attribute and the adjustment parameter to obtain the target speech with the target speech feature attribute.
[0012] According to the speech editing processing method provided by the application, the attribute feature dimension comprises at least a phoneme attribute feature dimension, an acoustic attribute feature dimension and an emotion attribute feature dimension.
[0013] The application further provides a speech editing processing device, which comprises: an acquisition module configured to acquire a speech to be edited and acquire an initial speech feature attribute corresponding to the speech to be edited; a determination module configured to determine a potential semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; and an editing module configured to perform speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attribute in a process of performing diffusion processing on the speech to be edited based on the h space in the diffusion model to obtain a target speech with a target speech feature attribute.
[0014] The application further provides an electronic device comprising a memory, a processor and a computer program stored in the memory and capable of being run on the processor, wherein the processor implements the speech editing processing method according to any one of the above-mentioned speech editing processing methods when the computer program is run.
[0015] The application further provides a non-transitory computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the speech editing processing method according to any one of the above-mentioned speech editing processing methods.
[0016] The application further provides a computer program product comprising a computer program, wherein the computer program is executed by a processor to implement the speech editing processing method according to any one of the above-mentioned speech editing processing methods.
[0017] The speech editing processing method, device and electronic equipment provided by the application include the following steps: obtaining a speech to be edited and an initial speech feature attribute corresponding to the speech to be edited; determining a latent semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; in a process of performing diffusion processing on the speech to be edited based on the h space in the diffusion model, performing speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech feature attribute to obtain a target speech with a target speech feature attribute. The speech editing cost is reduced and the speech editing efficiency is improved by flexibly performing speech editing processing on the speech to be edited through the latent semantic direction without modifying the diffusion model. BRIEF DESCRIPTION OF DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the application, and other drawings can be obtained by those skilled in the art without creative effort based on these drawings.
[0019] Figure 1 is a flowchart of the speech editing processing method provided by the application.
[0020] Figure 2 is a flowchart of the method for determining the latent semantic direction of the h space in a supervised manner.
[0021] Figure 3 is a flowchart of the method for determining the latent semantic direction of the h space in an unsupervised manner.
[0022] Figure 4 is a flowchart of the method for performing speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech feature attribute to obtain a target speech with the target speech feature attribute.
[0023] Figure 5 is a structural diagram of the speech editing processing device provided by the application.
[0024] Figure 6 is a structural diagram of the electronic equipment provided by the application. DETAILED DESCRIPTION
[0025] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.
[0026] The speech editing processing method provided by the present application can define the latent semantic direction in a supervised or unsupervised manner, and then apply the direction to the latent space in the process of generating a new speech sample (corresponding to a target speech with target speech feature attributes), so as to edit the corresponding speech attributes. The method does not require additional data or training, and can be applied to any frozen diffusion-based text-to-speech model (also referred to as TTS model) containing a bottleneck.
[0027] Figure 1 FIG. 1 is a flowchart of the speech editing processing method provided by the present application.
[0028] The speech editing processing method provided by the present application will be described below in conjunction with Figure 1 The process of the speech editing processing method provided by the present application will be described below.
[0029] In an exemplary embodiment of the present application, the speech editing processing method can include steps 110 to 130, which will be described below. Figure 1 As can be seen, the speech editing processing method can include steps 110 to 130, which will be described below.
[0030] In step 110, the speech to be edited is obtained, and the initial speech feature attributes corresponding to the speech to be edited are obtained.
[0031] In an embodiment, the speech to be edited can be obtained. In another example, the initial speech feature attributes corresponding to the speech to be edited can be obtained. In the application process, a speech feature extraction tool can be used to extract the initial speech feature attributes, which can include, for example, fundamental frequency F0, mel spectrum features, speech rate parameters, and voiceprint features.
[0032] In step 120, the latent semantic direction of the h space in the diffusion model is determined in a supervised manner and / or an unsupervised manner.
[0033] In step 130, in the process of diffusing the speech to be edited based on the h space in the diffusion model, the speech to be edited is edited along the latent semantic direction based on the initial speech feature attributes, and a target speech with target speech feature attributes is obtained.
[0034] In yet another embodiment, the h-space in the diffusion model can be a feature space formed by concatenating the activation values of the diffusion model at different processing stages of the bottleneck layer along the channel dimension. Its core role is to enhance the understanding of the data distribution of the model by integrating multi-level feature information, thereby improving the denoising and generation capabilities. Semantic directions refer to the strong correlation between certain specific vector directions in the latent space (Latent Space) or feature space (such as h-space) of a generative model (such as GAN, diffusion model) and the semantic attributes (such as object category, emotion, style, etc.) of the generated content. By moving the input features along these directions, controllable editing of the generated results can be achieved.
[0035] In one embodiment, the latent semantic directions of the h-space in the diffusion model can be determined in a supervised manner and / or an unsupervised manner. Further, in the process of diffusion processing of the speech to be edited based on the h-space in the diffusion model, the speech editing processing of the speech to be edited can be performed along the latent semantic directions based on the initial speech feature attributes, so that the target speech with the target speech feature attributes can be obtained.
[0036] In yet another embodiment, the target speech with the target speech feature attributes can be obtained using the following formula (1): ≡ = + . (1) wherein, is the original speech feature (corresponding to the initial speech feature attribute), controls the editing intensity (corresponding to the adjustment parameter below), and the addition and scaling are element-wise operations. In the generation process, replacing with makes the synthesized speech have acoustic properties related to the selected editing direction, i.e., the obtained represents the target speech with the target speech feature attributes, and the proposed latent space editing method maintains and even enhances the acoustic quality of the generated samples. represents the latent semantic direction. In this embodiment, by constructing the latent semantic direction of the h-space, the control of the speech attributes without modifying the model is achieved.
[0037] The speech editing processing method provided by the application comprises the following steps: obtaining a speech to be edited and an initial speech feature attribute corresponding to the speech to be edited; determining a potential semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; in the process of diffusion processing of the speech to be edited based on the h space in the diffusion model, performing speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attribute, so as to obtain a target speech with a target speech feature attribute. The speech editing cost is reduced and the speech editing efficiency is improved by flexibly performing speech editing processing on the speech to be edited through the potential semantic direction without modifying the diffusion model.
[0038] Figure 2 FIG. 1 is a flowchart of a process of determining a potential semantic direction of an h space in a supervised manner provided by the application.
[0039] The following will be described in combination with Figure 2 The process of determining a potential semantic direction of an h space in a supervised manner provided by the application will be described.
[0040] In an exemplary embodiment of the application, the process of determining a potential semantic direction of an h space in a supervised manner provided by the application will be described in combination with Figure 2 It can be known that the process of determining a potential semantic direction of an h space in a supervised manner can comprise steps 210 to 250, which will be introduced respectively.
[0041] In step 210, a speech sample pair is obtained, wherein the speech sample pair comprises a first speech sample and a second speech sample, and the first speech sample and the second speech sample are speech samples with opposite attribute features in the same attribute feature dimension.
[0042] In step 220, attribute feature extraction is performed on the speech sample pair respectively, so as to obtain a first speech sample attribute feature of the first speech sample and a second speech sample attribute feature of the second speech sample.
[0043] In an embodiment, m groups of speech sample pairs can be selected from a speech emotion database, and each group can comprise a first speech sample and a second speech sample. In an example, the first speech sample can be a Mandarin speech segment expressing an "angry" emotion (such as a fundamental frequency F0=180Hz and a speech rate=4 syllables per second); the second speech sample can be a speech segment expressing an "unangry" emotion (F0=120Hz and a speech rate=3 syllables per second) expressed by the same speaker; it can be understood that the first speech sample and the second speech sample are speech samples with opposite attribute features in the same attribute feature dimension.
[0044] In yet another embodiment, the attribute feature extraction can be performed on the speech sample pairs respectively, to obtain the first speech sample attribute feature of the first speech sample, for example, a 128-dimensional vector containing F0, speech rate, energy distribution and other parameters, and to obtain the second speech sample attribute feature of the second speech sample, for example, also a same-dimension vector (128-dimensional vector), in which the emotion-related parameters show opposite trends.
[0045] In step 230, based on the first speech sample attribute feature, a first latent code corresponding to the first speech sample attribute feature is obtained, and based on the second speech sample attribute feature, a second latent code corresponding to the second speech sample attribute feature is obtained.
[0046] In yet another embodiment, a pre-trained TTS model and a specific text prompt can be given to generate samples , the features of which are with or without the required attributes, and their matching latent codes are represented. Wherein, the first latent code can be represented; the second latent code can be represented; the first speech sample can be represented; the second speech sample can be represented.
[0047] In step 240, based on the first latent code and the second latent code, a semantic feature change vector of the speech sample pair in the attribute feature dimension is obtained.
[0048] In step 250, the semantic feature change vector is taken as the latent semantic direction of the h space.
[0049] In yet another embodiment, the semantic change vector can be calculated. By analyzing the semantic features of different samples, the semantic change vector is calculated, that is, the latent semantic direction of the h space is obtained , for example, Through this embodiment, the latent semantic direction of the h space can be obtained without modifying the model, and further the speech attribute control is laid a foundation for obtaining the target speech with target speech feature attributes.
[0050] In yet another exemplary embodiment of the present application, the speech sample pair can be a plurality of speech sample pairs, and correspondingly, the first latent code can be a plurality of first latent codes, and the second latent code can be a plurality of second latent codes. Wherein, based on the first latent code and the second latent code, the semantic feature change vector of the speech sample pair in the attribute feature dimension can be realized in the following way: The average value of the difference between the plurality of first latent codes and the plurality of second latent codes corresponding to the plurality of first latent codes is taken as the semantic feature change vector of the speech sample pair in the attribute feature dimension.
[0051] In an embodiment, the semantic feature change vector can be expressed by the following formula (2): (2) Wherein, represents the semantic feature change vector; m represents the logarithm of the first latent code and the second latent code; may represent the first latent code; may represent the second latent code.
[0052] In an embodiment, 2000 Chinese speech samples covering different emotional states (30% happy / sad / angry, 40% neutral) can be collected first, and multi-dimensional labeling is performed, including phoneme boundary, emotion label (discrete label + continuous arousal value) and prosody feature (F0 curve, energy envelope). Based on these labeled data, three types of core semantic features are extracted: 256-dimensional linguistic features (phoneme one-hot encoding and syllable stress markers), 128-dimensional acoustic features (PCA dimensionality-reduced mel spectrograms and DTW-aligned trajectories) and 64-dimensional emotional features (wav2vec 2.0 bottleneck layer features combined with arousal value polynomial coefficients), and finally a 2000x448-dimensional feature matrix is constructed. In this semantic space, we realize various fine-grained speech attribute editing: adjusting emotional intensity through linear interpolation of the emotional feature subspace ( ), controlling speech rate by time scaling the phoneme boundary feature ( ), and adjusting pitch by superimposing the target curve in the acoustic subspace. To ensure feature decoupling, the system uses adversarial training to constrain the orthogonality of different subspaces, and uses the feature matrix as a conditional input to diffuse the cross-attention layer of the model (CFG coefficient = 1.2), and finally verifies the explainability of each feature dimension through SHAP value analysis.
[0053] In another embodiment, the attribute feature dimension can at least include a phoneme attribute feature dimension, an acoustic attribute feature dimension, and an emotional attribute feature dimension. In this embodiment, by constructing a multi-dimensional semantic space containing phoneme, acoustic and emotional features, speech attribute control without modifying the model is realized. This method can adjust emotional intensity, speech rate and pitch and other parameters through simple mathematical operations, and all operations can be completed through forward propagation.
[0054] Figure 3 is a flowchart of the process of determining the potential semantic direction of h space in an unsupervised manner provided by the present application.
[0055] The process of determining the potential semantic direction of h space in an unsupervised manner will be described below. Figure 3 .
[0056] In an exemplary embodiment of the present invention, Figure 3 It can be seen that determining the latent semantic direction of the h-space in an unsupervised manner may include steps 310 to 330, and each step will be described below.
[0057] In step 310, a speech sample is obtained.
[0058] In step 320 , based on the speech sample, a bottleneck feature corresponding to the speech sample is obtained.
[0059] In step 330, principal component feature extraction is performed on the bottleneck feature based on principal component analysis to obtain the main bottleneck speech feature, and the latent semantic direction of the h-space is determined based on the main bottleneck speech feature, wherein the feature contribution of the main bottleneck speech feature is greater than the feature contribution threshold.
[0060] In one embodiment, a speech sample may be obtained. In one example, a feature matrix may be constructed for a given text input. , and generate a speech sample. Further, based on the speech sample, the bottleneck feature corresponding to the speech sample can be extracted. , where each time step .
[0061] In another embodiment, the bottleneck features can be extracted based on principal component analysis to obtain the main bottleneck speech features, and the potential semantic direction of the h space can be determined based on the main bottleneck speech features. In one example, the semantic change vector can be calculated using the features obtained by PCA (corresponding to the potential semantic direction of h space), as shown in formula (3): v = PC1(X) (3) In another exemplary embodiment of the present invention, determining the potential semantic direction of the h-space based on the main bottleneck speech features can be achieved in the following manner: Based on the main bottleneck speech features at each time step, determining the connection of the main bottleneck speech features across time steps; The latent semantic direction in h-space is determined based on the concatenation of the main bottleneck speech features across time steps.
[0062] In one embodiment, PCA (Principal Component Analysis) is applied to each time step, and the editing direction (corresponding to the latent semantic direction at the j-th time step) can be converted into Defined as the time step In other words, it can be based on the connection of the main bottleneck speech features across each time step. , determine the latent semantic direction of h space .
[0063] Figure 4 is a flowchart of a process of speech editing processing of the speech to be edited along the latent semantic direction based on the initial speech feature attribute, to obtain the target speech with the target speech feature attribute.
[0064] The following will be described in combination with Figure 4 The process of speech editing processing of the speech to be edited along the latent semantic direction based on the initial speech feature attribute, to obtain the target speech with the target speech feature attribute provided by the application will be described.
[0065] In an exemplary embodiment of the application, in combination with Figure 4 It can be seen that the speech editing processing of the speech to be edited along the latent semantic direction based on the initial speech feature attribute, to obtain the target speech with the target speech feature attribute can include steps 410 and 420, which will be described respectively.
[0066] In step 410, an adjustment parameter is determined, wherein the adjustment parameter is used to control the editing intensity of the speech editing processing of the speech to be edited along the latent semantic direction; In step 420, the speech editing processing of the speech to be edited along the latent semantic direction based on the initial speech feature attribute and the adjustment parameter, to obtain the target speech with the target speech feature attribute.
[0067] It should be noted that the target speech with the target speech feature attribute can be realized by the aforementioned formula (1). In the application process, the adjustment parameter can be determined, wherein the adjustment parameter can linearly control the editing intensity of the speech editing processing of the speech to be edited along the latent semantic direction, so that the control of the attribute of the speech to be edited can be conveniently realized.
[0068] In yet another embodiment, the editing intensity of the speech editing processing of the speech to be edited along the latent semantic direction can be controlled based on the adjustment parameter, for example, the editing intensity can be obtained by multiplying the adjustment parameter by the latent semantic direction . Further, based on the initial speech feature attribute and the editing intensity, that is + . , the target speech with the target speech feature attribute is obtained. The application realizes the speech attribute control without modifying the model by constructing a multi-dimensional semantic space (i.e. the latent semantic direction) containing phoneme, acoustic and emotional features. This method can adjust the parameters such as emotional intensity, speech rate and tone by using simple mathematical operations, and all operations can be completed by only forward propagation.
[0069] According to the foregoing description, the speech editing processing method provided by the application realizes speech attribute control without modifying the model by constructing a multi-dimensional semantic space containing phonemes, acoustic characteristics and emotional characteristics. The method can adjust parameters such as emotional intensity, speech rate and pitch by using simple mathematical operations, and all operations can be completed by only forward propagation; in terms of sound quality maintenance, the trajectory of DTW alignment and the reduced Mel spectrum features can be used to ensure the independence of each feature through adversarial training. In addition, the feature decoupling design supports the combination of multiple editing operations, and can adapt to various pre-trained TTS models. Users only need to provide labeled data to use, without additional training, which provides a practical solution for the commercialization of speech synthesis.
[0070] The speech editing processing device provided by the application will be described below. The speech editing processing device described below can be referred to in correspondence with the speech editing processing method described above.
[0071] Figure 5 FIG. 1 is a structural schematic diagram of the speech editing processing device provided by the application.
[0072] The speech editing processing device provided by the application will be described below. Figure 5 The structure of the speech editing processing device provided by the application will be described below.
[0073] In an exemplary embodiment of the application, the speech editing processing device provided by the application will be described below. Figure 5 As can be known, the speech editing processing device can include an acquisition module 510, a determination module 520 and an editing module 530, which will be introduced respectively.
[0074] The acquisition module 510 can be configured to acquire a speech to be edited and acquire initial speech characteristic attributes corresponding to the speech to be edited. The determination module 520 can be configured to determine the latent semantic direction of the h space in the diffusion model in a supervised manner and / or an unsupervised manner. The editing module 530 can be configured to, in the process of diffusing the speech to be edited based on the h space in the diffusion model, perform speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech characteristic attributes, to obtain a target speech with target speech characteristic attributes.
[0075] In an exemplary embodiment of the application, the determination module 520 can determine the latent semantic direction of the h space in a supervised manner in the following manner: Obtain a speech sample pair, wherein the speech sample pair includes a first speech sample and a second speech sample, and the first speech sample and the second speech sample are speech samples with opposite attribute characteristics in the same attribute characteristic dimension. attribute feature extraction is performed on the speech sample pairs respectively to obtain first speech sample attribute features of the first speech sample and second speech sample attribute features of the second speech sample; based on the first speech sample attribute features, a first latent code corresponding to the first speech sample attribute features is obtained, and based on the second speech sample attribute features, a second latent code corresponding to the second speech sample attribute features is obtained; based on the first latent code and the second latent code, a semantic feature change vector of the speech sample pair in the attribute feature dimension is obtained; the semantic feature change vector is taken as a latent semantic direction of the h space.
[0076] In an exemplary embodiment of the present application, the speech sample pair is a plurality of speech sample pairs, and correspondingly, the first latent code is a plurality of first latent codes, and the second latent code is a plurality of second latent codes; The determining module 520 can be implemented in the following manner to obtain the semantic feature change vector of the speech sample pair in the attribute feature dimension based on the first latent code and the second latent code: The average value of the difference values of the plurality of first latent codes and the plurality of second latent codes corresponding to the plurality of first latent codes is taken as the semantic feature change vector of the speech sample pair in the attribute feature dimension.
[0077] In an exemplary embodiment of the present application, the determining module 520 can be implemented in the following manner to determine the latent semantic direction of the h space in an unsupervised manner: obtain a speech sample; based on the speech sample, a bottleneck feature corresponding to the speech sample is obtained; based on principal component analysis, principal component feature extraction is performed on the bottleneck feature to obtain a main bottleneck speech feature, and based on the main bottleneck speech feature, the latent semantic direction of the h space is determined, wherein the feature contribution degree of the main bottleneck speech feature is greater than a feature contribution degree threshold.
[0078] In an exemplary embodiment of the present application, the determining module 520 can be implemented in the following manner to determine the latent semantic direction of the h space based on the main bottleneck speech feature: based on the main bottleneck speech feature at each time step, a connection of the main bottleneck speech feature across time steps is determined; based on the connection of the main bottleneck speech feature across each time step, the latent semantic direction of the h space is determined.
[0079] In an exemplary embodiment of the present application, the editing module 530 can be further configured to: determine an adjustment parameter, wherein the adjustment parameter is used to control an editing intensity of speech editing processing on the speech to be edited along the latent semantic direction; The editing module 530 can adopt the following manner to perform speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech feature attribute, so as to obtain the target speech with the target speech feature attribute: perform speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech feature attribute and the adjustment parameter, so as to obtain the target speech with the target speech feature attribute.
[0080] In an exemplary embodiment of the present application, the attribute feature dimension at least includes a phoneme attribute feature dimension, an acoustic attribute feature dimension and an emotion attribute feature dimension.
[0081] Figure 6 An example of an entity structure diagram of an electronic device is shown in FIG. 6. Figure 6 As shown, the electronic device can include a processor 610, a communications interface 620, a memory 630 and a communications bus 640, wherein the processor 610, the communications interface 620 and the memory 630 complete mutual communication through the communications bus 640. The processor 610 can invoke a logical instruction in the memory 630 to execute a speech editing processing method, which includes: obtaining speech to be edited, and obtaining initial speech feature attributes corresponding to the speech to be edited; determining a latent semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; in a process of performing diffusion processing on the speech to be edited based on the h space in the diffusion model, performing speech editing processing on the speech to be edited along the latent semantic direction based on the initial speech feature attributes, so as to obtain target speech with target speech feature attributes.
[0082] In addition, the logic instructions in the memory 630 described above can be implemented in the form of software functional units and sold or used as independent products, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes several instructions for making a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.
[0083] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the voice editing processing method provided by the above-mentioned methods. The method comprises: obtaining a to-be-edited voice, and obtaining an initial voice feature attribute corresponding to the to-be-edited voice; determining a potential semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; in the process of diffusion processing of the to-be-edited voice based on the h space in the diffusion model, performing voice editing processing on the to-be-edited voice along the potential semantic direction based on the initial voice feature attribute, to obtain a target voice with a target voice feature attribute.
[0084] In another aspect, the present application also provides a computer program product, which comprises a computer program, the computer program can be stored on a non-transitory computer readable storage medium, and the computer program is executed by a processor, so that the computer can execute the voice editing processing method provided by the above-mentioned methods. The method comprises: obtaining a to-be-edited voice, and obtaining an initial voice feature attribute corresponding to the to-be-edited voice; determining a potential semantic direction of an h space in a diffusion model in a supervised manner and / or an unsupervised manner; in the process of diffusion processing of the to-be-edited voice based on the h space in the diffusion model, performing voice editing processing on the to-be-edited voice along the potential semantic direction based on the initial voice feature attribute, to obtain a target voice with a target voice feature attribute.
[0085] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.
[0086] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.
[0087] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A voice editing processing method, characterized in that: The method comprises: Acquire the speech to be edited, and acquire the initial speech feature attributes corresponding to the speech to be edited; Determine the latent semantic direction of the h-space in the diffusion model using a supervised or unsupervised approach; In the process of performing diffusion processing on the speech to be edited based on the h-space in the diffusion model, speech editing processing is performed on the speech to be edited along the latent semantic direction based on the initial speech feature attributes to obtain a target speech with target speech feature attributes.
2. The voice editing processing method according to claim 1, characterized in that: The latent semantic direction of the h-space is determined in the following supervised manner: Acquire a speech sample pair, wherein the speech sample pair includes a first speech sample and a second speech sample, and the first speech sample and the second speech sample are speech samples having opposite attribute features under the same attribute feature dimension; Extracting attribute features from the voice sample pairs respectively to obtain a first voice sample attribute feature of the first voice sample and a second voice sample attribute feature of the second voice sample; Based on the first speech sample attribute feature, obtaining a first latent code corresponding to the first speech sample attribute feature, and based on the second speech sample attribute feature, obtaining a second latent code corresponding to the second speech sample attribute feature; Based on the first latent code and the second latent code, obtaining a semantic feature change vector of the speech sample pair in the attribute feature dimension; The semantic feature change vector is used as the potential semantic direction of the h space.
3. The voice editing processing method according to claim 2, characterized in that: The speech sample pair is a plurality of speech sample pairs, and correspondingly, the first latent code is a plurality of first latent codes, and the second latent code is a plurality of second latent codes; The obtaining, based on the first latent code and the second latent code, a semantic feature change vector of the speech sample pair in the attribute feature dimension specifically includes: An average value of differences between the plurality of first latent codes and the plurality of second latent codes corresponding to the plurality of first latent codes is used as a semantic feature change vector of the speech sample pair in the attribute feature dimension.
4. The voice editing method according to claim 1, wherein: The latent semantic direction of the h-space is determined in the following unsupervised manner: Get a voice sample; Based on the speech sample, obtaining a bottleneck feature corresponding to the speech sample; Based on principal component analysis, principal component feature extraction is performed on the bottleneck feature to obtain a main bottleneck speech feature, and based on the main bottleneck speech feature, a latent semantic direction of the h-space is determined, wherein a feature contribution of the main bottleneck speech feature is greater than a feature contribution threshold.
5. The voice editing processing method according to claim 4, characterized in that: Determining the potential semantic direction of the h-space based on the main bottleneck speech features specifically includes: Determining, based on the main bottleneck speech features at each time step, a connection of the main bottleneck speech features across time steps; Based on the connection of the main bottleneck speech features across time steps, the latent semantic direction of the h-space is determined.
6. The voice editing method according to any one of claims 1 to 5, characterized in that: Before performing speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attributes to obtain a target speech having the target speech feature attributes, the method further includes: determining an adjustment parameter, wherein the adjustment parameter is used to control the editing intensity of the speech editing process performed on the speech to be edited along the potential semantic direction; The step of performing speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attributes to obtain a target speech having the target speech feature attributes specifically includes: Based on the initial speech feature attributes and the adjustment parameters, speech editing processing is performed on the speech to be edited along the potential semantic direction to obtain a target speech with the target speech feature attributes.
7. The voice editing method according to any one of claims 2 or 3, characterized in that: The attribute feature dimensions include at least a phoneme attribute feature dimension, an acoustic attribute feature dimension, and an emotional attribute feature dimension.
8. A voice editing and processing device, characterized in that: The device comprises: An acquisition module, configured to acquire the speech to be edited and to acquire initial speech feature attributes corresponding to the speech to be edited; A determination module for determining the latent semantic direction of the h-space in the diffusion model using a supervised and / or unsupervised method; The editing module is used to perform speech editing processing on the speech to be edited along the potential semantic direction based on the initial speech feature attributes during the process of performing diffusion processing on the speech to be edited based on the h-space in the diffusion model, so as to obtain a target speech with target speech feature attributes.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the speech editing method according to any one of claims 1 to 7 is implemented.
10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech editing method according to any one of claims 1 to 7 is implemented.