Voice data processing method and device, electronic equipment and readable medium

Abstract

The invention provides a voice data processing method and apparatus, an electronic device and a readable medium. The method comprises the steps of obtaining source voice data of a source speaker and speaker information of a target speaker; carrying out feature extraction on the source voice data to obtain a posterior probability vector containing K voice units belonging to voice frames in the source voice data; according to the content expressions corresponding to the K voice units in a semantic dictionary and the posterior probability vector, content re-expressions of the voice frames in the source voice data are determined, and the semantic dictionary comprises the content expressions corresponding to the K voice units; the content expression is obtained by performing statistical calculation according to semantic expressions and posterior probabilities in voice data from at least two speakers; and performing voice conversion according to the content re-expression of the voice frame in the source voice data and the speaker information to obtain target voice data of the target speaker. According to the method, tone leakage in the converted audio can be reduced, and the sound similarity of the converted audio is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method, device, electronic device and readable medium for processing voice data. Background Art

[0002] With the development of computer technology, voice conversion technology has developed rapidly. The task of voice conversion (VC) includes converting the semantic representation of the source speech into the speech of the target speaker while preserving the language content.

[0003] In the related art, the speech conversion service extracts the semantic content representation in the training speech from the trained model, and then converts the speech audio of the target speaker based on the content representation.

[0004] However, in such schemes, since the training data of the training model will contain information such as semantics, timbre and rhythm at the same time, the semantic content representation will also contain timbre information. As a result, the timbre information in the semantic content representation causes the converted speech audio to retain part of the timbre of the source speaker, thereby affecting the similarity between the sound of the converted audio and the sound of the target speaker. Summary of the invention

[0005] Based on the above technical problems, the present application provides a method, device, electronic device and readable medium for processing voice data to reduce the timbre leaked into the final generated target voice data and improve the similarity between the sound of the converted audio and the voice of the target speaker.

[0006] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0007] According to one aspect of an embodiment of the present application, a method for processing voice data is provided, including:

[0008] Acquire source speech data of a source speaker and speaker information of a target speaker;

[0009] Performing feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice units include at least one of phonemes and soft voice units, and K is an integer greater than 1;

[0010] Determine the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expressions and the posterior probability in the speech data from at least two speakers;

[0011] The target speech data of the target speaker is obtained by performing speech conversion according to the content re-expression of the speech frame in the source speech data and the speaker information.

[0012] According to one aspect of an embodiment of the present application, there is provided a device for processing voice data, including:

[0013] An information acquisition module configured to acquire source speech data of a source speaker and speaker information of a target speaker;

[0014] A feature extraction module is configured to perform feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice units include at least one of phonemes and soft voice units, and K is an integer greater than 1;

[0015] an expression conversion module, configured to determine the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expressions and the posterior probability in the speech data from at least two speakers;

[0016] The voice conversion module is configured to perform voice conversion according to the content re-expression of the voice frame in the source voice data and the speaker information to obtain the target voice data of the target speaker.

[0017] In some embodiments of the present application, based on the above technical solution, the semantic dictionary includes a global dictionary, and the speech conversion module is further configured to: obtain the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, and the dictionary audio set contains speech data from at least two speakers; perform statistics on the posterior probability that the extracted speech frame belongs to the Kth speech unit to obtain a first statistic of the Kth speech unit; obtain the semantic expression of the Kth speech unit according to the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the first statistic; determine the global dictionary according to the semantic expression of the Kth speech unit and the semantic expressions of other K-1 speech units among the K speech units.

[0018] In some embodiments of the present application, based on the above technical solution, the semantic dictionary includes a global dictionary, and the speech conversion module is specifically configured to: input the source speech data into a pre-trained first feature extraction model, the first feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K phonemes respectively; obtain the output result of the bottleneck layer in the first feature extraction model as the posterior probability vector of the source speech data.

[0019] In some embodiments of the present application, based on the above technical solution, the speech conversion module is specifically configured to: input the source speech data into a pre-trained second feature extraction model, the second feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K soft speech units respectively; and use the output result of the specified converter layer in the second feature extraction model as the posterior probability vector of the source speech data.

[0020] In some embodiments of the present application, based on the above technical solution, the content re-expression of the source voice data is generated based on the global semantic dictionary, and the speaker information includes the prompt voice data of the target speaker; the voice conversion module is specifically configured to: based on the global semantic dictionary, obtain the content re-expression of the prompt voice data; input the content re-expression of the voice frame in the source voice data, the content re-expression of the prompt voice data, and the encoding tokens of the source voice data and the prompt voice into the first voice generation model to generate target decoder tokens; perform audio decoding according to the target decoder tokens to obtain the target voice data of the target speaker.

[0021] In some embodiments of the present application, based on the above technical solution, the semantic dictionary also includes a speaker dictionary, and the speech conversion module is further configured to: respectively perform statistics on the posterior probability that the speech frame in the speech data of each speaker belongs to the Kth speech unit, and obtain a second statistic corresponding to the Kth speech unit for each speaker; for each speaker, obtain the semantic expression corresponding to the Kth speech unit for each speaker based on the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the second statistic; determine the speaker dictionary corresponding to each speaker based on the semantic expression of the Kth speech unit corresponding to each speaker and the semantic expression of the other K-1 speech units among the K speech units corresponding to each speaker.

[0022] In some embodiments of the present application, based on the above technical solution, the expression conversion module is specifically configured to: respectively multiply the content expressions corresponding to the K speech units in the global semantic dictionary and the speaker dictionary by the posterior probability vector to obtain a global content re-expression and a speaker content re-expression; calculate the weighted sum of the global content re-expression and the speaker content re-expression to obtain the content re-expression of the speech frame in the source speech data;

[0023] The speech conversion module is specifically configured to: re-express the content of the speech frame in the source speech data and input the speaker identification of the target speaker in the speaker information into the second speech generation model for speech conversion to obtain the target speech data of the target speaker.

[0024] In some embodiments of the present application, based on the above technical solution, the expression conversion module is specifically configured to: obtain the source content expression of the source voice data; calculate the weighted sum of the source content expression, the global content re-expression and the speaker content re-expression to obtain the content re-expression of the voice frame in the source voice data.

[0025] In some embodiments of the present application, based on the above technical solution, the speech conversion module is specifically configured to: use the content re-expression of the speech frame in the source speech data and the embedded expression of the speaker information as conditions, and the specified spectrum containing noise Figure 1 The third speech generation model is input for spectrum prediction to obtain a result spectrum diagram, wherein the third speech generation model contains embedded information of time step information and speaker information; the result spectrum diagram is voice-converted by a pre-trained vocoder to obtain target speech data of the target speaker.

[0026] In some embodiments of the present application, based on the above technical solution, the speech conversion module is further configured to: perform feature extraction on the training speech data based on the semantic dictionary to obtain content re-representation of the training speech data; splice the content re-representation and a specified spectrum graph containing noise into training data, wherein noise is added to the specified spectrum graph according to the time step; train the third model to be trained using the training data to obtain the third speech generation model, wherein the third model to be trained includes a feature linear modulation layer, a convolution layer and a diffusion transformer layer, the feature linear modulation layer includes information about the time step, and the convolution layer and the diffusion transformer layer include speaker embedding information of the speaker of the training speech data.

[0027] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute a method for processing voice data as in the above technical solution by executing the executable instructions.

[0028] According to one aspect of an embodiment of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method for processing voice data in the above technical solution is implemented.

[0029] According to one aspect of the embodiments of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for processing voice data provided in the above-mentioned various optional implementations.

[0030] In an embodiment of the present application, a speech conversion device obtains source speech data of a source speaker and speaker information of a target speaker, then performs feature extraction on the source speech data to obtain a posterior probability vector containing speech frames in the source speech data belonging to K speech units, wherein the speech unit contains at least one of a phoneme and a soft speech unit, and K is an integer greater than 1; determines a content re-expression of the speech frames in the source speech data according to the content expressions corresponding to the K speech units in a semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation of semantic expressions and posterior probabilities in speech data from at least two speakers; then performs speech conversion according to the content re-expression of the speech frames in the source speech data and the speaker information to obtain target speech data of the target speaker. In this way, the source voice data is converted through a semantic dictionary, and the semantic dictionary is constructed by weighted combination of semantic expressions of different speakers, so as to obtain speaker-independent content re-expression, thereby reducing the timbre leaked into the final generated target voice data and improving the similarity between the sound of the converted audio and the sound of the target speaker.

[0031] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 The voice data processing method of the embodiment of the present application is applied to the system structure of the cloud host platform.

[0034] Figure 2 The present invention is a flowchart of a method for processing voice data according to an embodiment of the present application.

[0035] Figure 3 The present invention is a flowchart of a method for processing voice data according to an embodiment of the present application.

[0036] Figure 4 Schematic diagram of the structure of the PPG extractor in the embodiment of the present application.

[0037] Figure 5 Schematic diagram of the structure of the Hubert extractor in the embodiment of the present application.

[0038] Figure 6 It is a schematic diagram of the conversion process based on the large language model in the embodiment of the present application.

[0039] Figure 7 This is a schematic flowchart of using the second speech generation model for speech conversion in an embodiment of the present application.

[0040] Figure 8 Schematic diagram of the structure of the diffusion model in the embodiment of the present application.

[0041] Fig. 9 The block diagram schematically shows the composition of the speech data processing device in the embodiment of the present application.

[0042] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown. DETAILED DESCRIPTION

[0043] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of the example embodiments to those skilled in the art.

[0044] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve the predetermined function, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement at least one module or unit. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0046] The block diagrams shown in the accompanying drawings are only functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in at least one hardware module or integrated circuit, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0048] It should be understood that the scheme of the present application can be applied in a voice conversion system, specifically in the voice conversion service provided by the voice conversion system. Specifically, the task of voice conversion includes converting the semantic representation of the source voice into the voice of the target speaker while retaining the language content. Specifically, it usually includes converting the source voice into the voice of the target speaker, or directly generating the voice of the target speaker based on text information. The scheme of the present application uses semantic representations from different speakers to build an offline global semantic dictionary. Each entry in the dictionary is a stable and timbre-free representation of a certain phoneme class. In the task of voice conversion, each content feature from the source speaker can be re-expressed as a weighted combination of entries in the semantic dictionary. The resulting new content representation contains stable semantic information and retains contextual information. At the same time, for one-to-many or many-to-many voice conversion tasks, it is feasible to construct a semantic dictionary dependent on the speaker for a single target speaker, and the resulting new content representation will contain target timbre information that is conducive to the conversion task. With the development of computer technology, voice conversion technology has developed rapidly. The task of voice conversion (voice conversion, VC) includes converting the semantic representation of the source voice into the voice of the target speaker while retaining the language content. In the related art, the speech conversion service extracts the semantic content representation of the training speech from the trained model, and then converts the speech audio of the target speaker based on the content representation. However, in such solutions, since the training data of the training model contains information such as semantics, timbre and rhythm at the same time, the semantic content representation also contains timbre information, which causes the timbre information in the semantic content representation to retain part of the timbre of the source speaker in the converted speech audio, thereby affecting the similarity between the sound of the converted audio and the sound of the target speaker.

[0049] Based on this, the technical solution of the embodiment of the present application proposes a solution for processing voice data. Figure 1According to the method for processing voice data of the embodiment of the present application, the voice conversion system may mainly include a terminal device 110, a network 120 and a server 130. The terminal device 110 may include a smart phone, a tablet computer, a laptop computer, an intelligent voice interaction device, a smart home appliance, a vehicle terminal, an aircraft, etc. The server 130 may be a server that provides various services, which may be an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. The network 120 may be a communication medium of various connection types that can provide a communication link between the terminal device 110 and the server 130, such as a wired communication link or a wireless communication link.

[0050] According to the implementation needs, the system architecture in the embodiment of the present application can have any number of playback terminals, networks and servers. For example, server 130 can be a server group composed of multiple server devices. In addition, the technical solution provided in the embodiment of the present application can be applied to terminal device 110, can also be applied to server 130, or can be implemented by terminal device 110 and server 130 together, and the present application does not make special restrictions on this.

[0051] like Figure 1 As shown, the speech conversion system in the present application can be deployed in the server 130. The server 130 obtains the source speech data of the source speaker and the speaker information of the target speaker from the terminal device 110. Then the server 130 performs feature extraction on the source speech data to obtain a posterior probability vector containing the speech frame in the source speech data belonging to K speech units, wherein the speech unit contains at least one of a phoneme and a soft speech unit, and K is an integer greater than 1. The server 130 then determines the content re-expression of the speech frame in the source speech data according to the content expression corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expression corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expression and the posterior probability in the speech data from at least two speakers. Finally, the server 130 performs speech conversion according to the content re-expression of the speech frame in the source speech data and the speaker information to obtain the target speech data of the target speaker.

[0052] The implementation details of the technical solution of the embodiment of the present application are described in detail below: Figure 2A flowchart of a method for processing voice data according to an embodiment of the present application is shown. The method for processing voice data can be executed by a device with a computing and processing function, such as a server or a terminal device. The following describes the method for processing voice data in the present application from the perspective of a voice conversion system on a server. Figure 2 As shown, the method for processing voice data at least includes steps S210 to S240, which are described in detail as follows:

[0053] Step S210, acquiring source speech data of a source speaker and speaker information of a target speaker.

[0054] The speech conversion device can obtain the source speech data of the source speaker and the speaker information of the target speaker. The speaker information is usually the speaker identification of the target speaker. Specifically, the speech conversion device can encode each speaker that it can convert, and determine the speaker to be converted into which tone through the speaker identification. The source speech data is the speech audio data of the source speaker, such as the speech or song of the source speaker.

[0055] Step S220, performing feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice unit includes at least one of a phoneme and a soft voice unit, and K is an integer greater than 1.

[0056] The speech conversion device will extract features from the source speech data. The specific feature extraction method will be carried out using a special feature extractor. The feature extractor can be a pre-trained machine learning model, such as a supervised semantic posterior graph model (phoneme-posteriorgram, PPG) or a self-supervised machine learning model, such as a Hubert model. The speech conversion device will obtain the corresponding content expression according to the source speech data through these models. The acquisition of content expression is usually in units of speech frames. The source speech data will contain multiple speech frames, and each speech frame will obtain the corresponding content expression through the feature extractor. The speech conversion device will also obtain the posterior probability vector that the speech frame belongs to K speech units. It should be emphasized that although the posterior probability vector is used as an example in this application, in the specific implementation, other forms of probability data can also be used, such as a priori probability, likelihood probability, conditional probability, total probability or predicted probability. In addition, depending on the specific implementation, a data form of a form can also be used, such as a probability matrix or tensor. Specifically, the posterior probability vector will contain the posterior probability that the speech frame belongs to each speech unit. The number of speech units depends on the specific type of speech unit. A speech unit is usually one of a phoneme or a soft speech unit. Both phonemes and soft speech units are the smallest processing units in audio processing. Phonemes are usually determined by summarizing the pronunciation rules of different languages, while soft speech units are obtained after segmentation based on specific speech data. The speech conversion device will predetermine the number of speech units, for example, in the process of generating a semantic dictionary. Assuming that the source speech data contains Q speech frames, the speech conversion device will determine the probability that the Q speech frames belong to K speech units, thereby obtaining a posterior probability vector. Depending on the specific implementation method, the speech conversion device can determine the probability of each speech frame belonging to K speech units separately, or it can only determine the semantic unit to which each speech frame is most likely to belong and the posterior probability of belonging to the speech unit.

[0057] Step S230, determining the content re-expression of the speech frame in the source speech data based on the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation of the semantic expressions and the posterior probability in the speech data from at least two speakers.

[0058] Specifically, the speech conversion device obtains the content re-expression of the speech frame in the source speech data according to the mapping relationship between the content expression corresponding to the K speech units in the semantic dictionary and the posterior probability vector. The mapping relationship can be a suitable data calculation relationship, such as addition, multiplication or weighted calculation. The semantic dictionary contains the content expression corresponding to the K speech units. The content expression in the semantic dictionary is obtained by the speech conversion device in advance by weighted merging according to the speech data of at least two speakers. Therefore, the content expression has nothing to do with the timbre of the specific speaker and is only related to the semantic content. It can be understood that the semantic dictionary can be a row of data containing K column elements, each of which corresponds to the content expression of a semantic unit. The posterior probability vector is a column of data containing K row elements, each of which corresponds to the posterior probability between a speech frame and a semantic unit. The product of the two obtains the content expression based on the statistics of the semantic dictionary and the content re-expression of a speech frame in the source speech data. The content re-expression of each speech frame in the source speech data can be integrated to obtain the content re-expression of the source speech data. It can be understood that since the expression in the semantic dictionary is independent of the timbre, the content re-expression of the source speech data is also independent of the timbre and is only related to the semantics. In some embodiments, phonemes and soft speech units may be used simultaneously, for example, extracted and calculated separately, semantic re-expressions may be obtained separately, and the results may be fused at last, or the posterior probabilities of the two may be combined and calculated with the semantic dictionary.

[0059] Step S240, performing voice conversion according to the content re-expression of the voice frame in the source voice data and the speaker information to obtain the target voice data of the target speaker.

[0060] The speech conversion device can convert the content re-expression into speech through a pre-trained converter. The converter receives the speaker information as input to determine the timbre of the target speaker to be converted. The converter can use a special machine learning model, such as a variational reasoning and adversarial learning model, a large language model, or a vocoder in a diffusion model.

[0061] In an embodiment of the present application, a speech conversion device obtains source speech data of a source speaker and speaker information of a target speaker, then performs feature extraction on the source speech data to obtain a posterior probability vector containing speech frames in the source speech data belonging to K speech units, wherein the speech unit contains at least one of a phoneme and a soft speech unit, and K is an integer greater than 1; determines a content re-expression of the speech frames in the source speech data according to the content expressions corresponding to the K speech units in a semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation of semantic expressions and posterior probabilities in speech data from at least two speakers; then performs speech conversion according to the content re-expression of the speech frames in the source speech data and the speaker information to obtain target speech data of the target speaker. In this way, the source voice data is converted through a semantic dictionary, and the semantic dictionary is constructed by weighted combination of semantic expressions of different speakers, so as to obtain speaker-independent content re-expression, thereby reducing the timbre leaked into the final generated target voice data and improving the similarity between the sound of the converted audio and the sound of the target speaker.

[0062] In the embodiments of the present application, it is also proposed to Figure 2 Other embodiments that refine the technical solution of the embodiment shown in the figure are as follows Figure 3 As shown, the semantic dictionary includes a global dictionary. In a method for processing speech data in an embodiment of the present application, the following steps may be included:

[0063] Step S310, obtaining the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, wherein the dictionary audio set contains speech data from at least two speakers;

[0064] Step S320, performing statistics on the posterior probability that the extracted speech frame belongs to the K-th speech unit to obtain a first statistic of the K-th speech unit;

[0065] Step S330, obtaining the semantic expression of the Kth speech unit according to the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the first statistic;

[0066] Step S340, determining the global dictionary according to the semantic expression of the Kth speech unit and the semantic expressions of other K-1 speech units in the K speech units.

[0067] In this embodiment, the dictionary audio set is a pre-selected training speech set, which contains speech data of multiple speakers. Depending on the conversion target, these speech data can come from people of different genders or in different languages. The speech conversion device first obtains the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit. It can be understood that the semantic expression of each speech frame can be extracted in the previous processing of other speech units, or it can be obtained from the dictionary audio set by an extractor. The method of extracting the semantic expression and the posterior probability is the same as the method used in the speech conversion process in the above embodiment, and the speech unit is also the same. That is, if the extracted speech unit is a phoneme, the speech unit in the semantic dictionary is also a phoneme. Taking phonemes as an example, let X be i，j，t ∈R d×1 , where d is the dimension of semantic expression, i = 1…S, j = 1…N i , t=1…T ij , is the semantic expression of the tth frame of the jth speech data of the ith speaker. Let represents the posterior probability that the current frame belongs to the kth phoneme class. Then the first statistic n k It can be calculated as follows:

[0068]

[0069] The semantic expression m of the Kth phonetic unit k The calculation method is as follows:

[0070]

[0071] Finally, the global dictionary M g It can be expressed as follows:

[0072] M g =[m1,m2,…m K ]

[0073] where K is the total number of phoneme classes or discrete speech units, m1 to m k－1 is the semantic expression of the other K-1 speech units, which can be calculated by the speech conversion device in the previous process. In the above manner, the semantic expression of the speech frame of the speech data from different speakers corresponding to a speech unit is comprehensively calculated in the calculation process to obtain the corresponding semantic expression, so as to be combined into a global dictionary, so that the semantic expression in the global dictionary is less affected by the timbre of a single speaker, which is conducive to improving the generalization ability of the semantic expression in the global dictionary.

[0074] Step S350, obtaining source speech data of a source speaker and speaker information of a target speaker.

[0075] Optionally, the implementation details of step S350 are the same as Figure 2 The step S210 shown in FIG. 1 is the same as that in FIG. 1 and will not be described in detail.

[0076] Step S360, performing feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice unit includes at least one of a phoneme and a soft voice unit, and K is an integer greater than 1.

[0077] Optionally, the implementation details of step S360 are the same as Figure 2 The step S220 shown in FIG. 1 is consistent with the step S220 shown in FIG. 1 and will not be

[0078] Step S370, determining the content re-expression of the speech frame in the source speech data based on the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation of the semantic expressions and the posterior probability in the speech data from at least two speakers.

[0079] Optionally, the implementation details of step S370 are the same as Figure 2 The step S230 shown in FIG. 1 is consistent with the step S230 shown in FIG. 1 and will not be

[0080] Step S380, performing voice conversion based on the content re-expression of the voice frame in the source voice data and the speaker information to obtain the target voice data of the target speaker.

[0081] Optionally, the implementation details of step S380 are the same as Figure 2 The step S240 shown in FIG. 1 is consistent with the step S240 shown in FIG. 1 and will not be

[0082] In an embodiment of the present application, a speech conversion device obtains source speech data of a source speaker and speaker information of a target speaker, then performs feature extraction on the source speech data to obtain a posterior probability vector containing speech frames in the source speech data belonging to K speech units, wherein the speech unit contains at least one of a phoneme and a soft speech unit, and K is an integer greater than 1; determines a content re-expression of the speech frames in the source speech data according to the content expressions corresponding to the K speech units in a semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation of semantic expressions and posterior probabilities in speech data from at least two speakers; then performs speech conversion according to the content re-expression of the speech frames in the source speech data and the speaker information to obtain target speech data of the target speaker. In this way, the source voice data is converted through a semantic dictionary, and the semantic dictionary is constructed by weighted combination of semantic expressions of different speakers, so as to obtain speaker-independent content re-expression, thereby reducing the timbre leaked into the final generated target voice data and improving the similarity between the sound of the converted audio and the sound of the target speaker.

[0083] In some embodiments of the present application, based on the technical solution in the present application, in the process of obtaining the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, the speech conversion device will input the source speech data into a pre-trained first feature extraction model, and the first feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K phonemes respectively, and then obtain the output result of the bottleneck layer in the first feature extraction model as the posterior probability vector of the source speech data. In this embodiment, the first feature extraction model can be implemented using PPG. Specifically, please refer to Figure 4 , Figure 4 Schematic diagram of the structure of the PPG extractor in the embodiment of the present application. Figure 4 As shown, the PPG extractor is trained using cross entropy and center loss. The speech conversion device obtains the corresponding semantic expression from the output of the bottleneck layer of the PPG structure, and obtains the corresponding posterior probability from the output of the PPG.

[0084] In some embodiments of the present application, based on the technical solution in the present application, in the process of obtaining the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, the speech conversion device will input the source speech data into a pre-trained second feature extraction model, and the second feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K soft speech units respectively, and then use the output result of the specified converter layer in the second feature extraction model as the posterior probability vector of the source speech data. In this embodiment, the first feature extraction model can be implemented using the Hubert model. Specifically, please refer to Figure 5 , Figure 5 Schematic diagram of the structure of the Hubert extractor in the embodiment of the present application. Figure 5 As shown, in this embodiment, two fully connected layers are added after the backbone of the Hubert model. The speech conversion device uses the converter of the specified number of layers of the Hubert model to extract semantic features, for example, using the seventh layer converter to obtain semantic features. For the acquired semantic features, the present application adopts a clustering method to extract the semantic representation corresponding to the soft speech unit from the speech of different speakers.

[0085] In some embodiments of the present application, based on the technical solution in the present application, the content re-expression of the source voice data is generated based on the global semantic dictionary, and the speaker information includes the prompt voice data of the target speaker; in the process of performing voice conversion according to the content re-expression of the voice frame in the source voice data and the speaker information to obtain the target voice data of the target speaker, the voice conversion device will obtain the content re-expression of the prompt voice data based on the global semantic dictionary, and then input the content re-expression of the voice frame in the source voice data, the content re-expression of the prompt voice data, and the encoding tokens of the source voice data and the prompt voice into the first voice generation model to generate the target decoder token, and finally perform audio decoding according to the target decoder token to obtain the target voice data of the target speaker. In the embodiment of the present application, a large language model (LLM) is used as a model for performing voice conversion in the voice conversion device. Specifically, the first voice generation model is implemented by the LLM model. The model receives the source voice data and a prompt voice data as input. The prompt voice data is usually data related to the target speaker. In some embodiments, the prompt voice data is the target speaker's voice information of a portion of the source voice data, for example, the target speaker's audio of the first three seconds of the source voice data. In this embodiment, the voice conversion device converts the content expressions of the source voice information and the prompt information into content expressions based on the global dictionary. Figure 6 , Figure 6 Schematic diagram of the conversion process based on the large language model in the embodiment of the present application. Figure 6 As shown in the figure, the semantic expression TICR based on the global dictionary will be input into the large language model together with the prompt voice prompt and the encoding token of the source voice data source, so as to obtain the output decoding token. The large language model in the figure adopts the Neural Codec Language Model. The decoded token is then transcoded by the decoder to obtain the output audio content.

[0086] In some embodiments of the present application, based on the technical solution in the present application, the semantic dictionary also includes a speaker dictionary, and the speech conversion device will respectively count the posterior probability that the speech frame in the speech data of each speaker belongs to the Kth speech unit, and obtain the second statistic corresponding to the Kth speech unit for each speaker, and then for each speaker, according to the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the second statistic, obtain the semantic expression corresponding to the Kth speech unit for each speaker, and finally determine the speaker dictionary corresponding to each speaker according to the semantic expression of the Kth speech unit corresponding to each speaker and the semantic expression of the other K-1 speech units in the K speech units corresponding to each speaker. In this embodiment, the speech conversion device will also generate a speaker dictionary corresponding to the speaker. Specifically, the generation method of the speaker dictionary is similar to that of the global dictionary, except that the statistical and calculation processes are organized according to the speaker. Specifically, the second statistic n ik It can be calculated as follows:

[0087]

[0088] The Kth phonetic unit corresponds to the semantic expression m of each speaker. ik The calculation method is as follows:

[0089]

[0090] Finally, the Pronouncer Dictionary M S It can be expressed as follows:

[0091] M s =[m i1 , m i2 ,...m iK ]

[0092] In some embodiments of the present application, based on the technical solution in the present application, in the process of determining the content re-expression of the speech frame in the source speech data according to the content expression corresponding to the K speech units in the semantic dictionary and the posterior probability vector, the speech conversion device will respectively multiply the content expression corresponding to the K speech units in the global semantic dictionary and the speaker dictionary and the posterior probability vector to obtain the global content re-expression and the speaker content re-expression, calculate the weighted sum of the global content re-expression and the speaker content re-expression, and obtain the content re-expression of the speech frame in the source speech data.

[0093] Specifically, the global content re-expression and the speaker content re-expression can be determined by referring to the following method:

[0094]

[0095] in, Re-expression of global content, Speaker's content re-expression, M g is the global dictionary, M S For the speaker dictionary, P i，j，t ∈R K×1 is the posterior probability vector.

[0096] In the process of performing speech conversion based on the content re-expression of the speech frame in the source speech data and the speaker information to obtain the target speech data of the target speaker, the speech conversion device will input the content re-expression of the speech frame in the source speech data and the speaker identification of the target speaker in the speaker information into the second speech generation model for speech conversion to obtain the target speech data of the target speaker. In this embodiment, the variational inference with adversarial learning for end-to-end Text-to-Speech (VITS) model is used to perform speech conversion. For details, please refer to Figure 7 , Figure 7 FIG. 1 is a schematic flow chart of using the second speech generation model to perform speech conversion in an embodiment of the present application. Figure 7 As shown in the figure, after the input source speech data is extracted, it is calculated with the global dictionary and the speaker dictionary respectively, so as to obtain the global content re-expression that is not related to semantics and the speaker content re-expression that is related to the speaker. Then the global content re-expression and the speaker content re-expression are weighted and summed, and the result is input into the VITS model for speech conversion to obtain the target speech data of the target speaker.

[0097] In some embodiments of the present application, based on the technical solution in the present application, in the process of calculating the weighted sum of the global content re-expression and the speaker content re-expression to obtain the content re-expression of the speech frame in the source speech data, the speech conversion device will obtain the source content expression of the source speech data. Then, the weighted sum of the source content expression, the global content re-expression and the speaker content re-expression is calculated to obtain the content re-expression of the speech frame in the source speech data. In this embodiment, the source content expression of the source speech data is further added to the input data of the VITS model. Figure 7 As shown in the dotted part in , the speech conversion device performs weighted summation on the source content expression, the global content re-expression and the speaker content re-expression, and inputs the result into the VITS model for speech conversion to obtain the target speech data of the target speaker.

[0098] In some embodiments of the present application, based on the technical solution of the present application, in the process of performing speech conversion according to the content re-expression of the speech frame in the source speech data and the speaker information to obtain the target speech data of the target speaker, the speech conversion device will use the content re-expression of the speech frame in the source speech data and the embedded expression of the speaker information as conditions, and the specified spectrum containing noise Figure 1 A third speech generation model is input for spectrum prediction to obtain a result spectrogram, wherein the third speech generation model contains embedded information of time step information and speaker information, and then the result spectrogram is speech-converted by a pre-trained vocoder to obtain the target speech data of the target speaker.

[0099] In some embodiments of the present application, based on the technical solution in the present application, the speech conversion device will also perform feature extraction on the training speech data based on the semantic dictionary to obtain the content re-representation of the training speech data, and then splice the content re-representation with the specified spectrum graph containing noise into training data, wherein noise is added to the specified spectrum graph according to the time step, and then the third model to be trained is trained with the training data to obtain the third speech generation model, wherein the third model to be trained includes a feature linear modulation layer, a convolution layer and a diffusion transformer layer, the feature linear modulation layer includes the information of the time step, and the convolution layer and the diffusion transformer layer include the speaker embedding information of the speaker of the training speech data.

[0100] In this embodiment, the voice conversion process is performed through a diffusion model. For details, see Figure 8 , Figure 8 Schematic diagram of the structure of the diffusion model in the embodiment of the present application. Figure 8As shown, the speech conversion device will use the content re-expression and the embedded expression of the speaker information as conditions, and the specified spectrum containing noise Figure 1 The content is re-expressed and the content is input into the third speech generation model for spectrum prediction. Specifically, the specified spectrogram can be a Mel-spectrogram, and the content re-expression and the Mel-spectrogram are combined and input into the third speech generation model. The third speech generation model includes a feature linear modulation layer, a convolution layer, and a diffusion transformer layer. Among them, the time step used in the diffusion process is embedded in the feature linear modulation layer, and the speaker's identifier is embedded in the convolution layer and the diffusion transformer layer. During the training process, the third speech generation model outputs a predicted Mel-spectrogram, and in the actual conversion process, the result spectrogram is output. The result spectrogram is converted into the target speech data of the target speaker by a pre-trained vocoder.

[0101] It should be noted that although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0102] The following introduces the device implementation of the present application, which can be used to execute the voice data processing method in the above-mentioned embodiment of the present application. Fig. 9 The block diagram schematically shows the composition of the apparatus for processing speech data in the embodiment of the present application. Fig. 9 As shown, the speech data processing device 900 may mainly include:

[0103] An information acquisition module 910 is configured to acquire source speech data of a source speaker and speaker information of a target speaker;

[0104] A feature extraction module 920 is configured to perform feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice units include at least one of phonemes and soft voice units, and K is an integer greater than 1;

[0105] The expression conversion module 930 is configured to determine the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expressions and the posterior probability in the speech data from at least two speakers;

[0106] The speech conversion module 940 is configured to perform speech conversion according to the content re-expression of the speech frame in the source speech data and the speaker information to obtain the target speech data of the target speaker.

[0107] In some embodiments of the present application, based on the above technical solution, the semantic dictionary includes a global dictionary, and the speech conversion module 940 is further configured to: obtain the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, and the dictionary audio set contains speech data from at least two speakers; perform statistics on the posterior probability that the extracted speech frame belongs to the Kth speech unit to obtain a first statistic of the Kth speech unit; obtain the semantic expression of the Kth speech unit according to the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the first statistic; determine the global dictionary according to the semantic expression of the Kth speech unit and the semantic expressions of the other K-1 speech units among the K speech units.

[0108] In some embodiments of the present application, based on the above technical solution, the semantic dictionary includes a global dictionary, and the speech conversion module 940 is specifically configured to: input the source speech data into a pre-trained first feature extraction model, the first feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K phonemes respectively; obtain the output result of the bottleneck layer in the first feature extraction model as the posterior probability vector of the source speech data.

[0109] In some embodiments of the present application, based on the above technical solution, the speech conversion module 940 is specifically configured as: inputting the source speech data into a pre-trained second feature extraction model, the second feature extraction model being used to calculate the posterior probability that each speech frame in the source speech data belongs to the K soft speech units respectively; and using the output result of the specified converter layer in the second feature extraction model as the posterior probability vector of the source speech data.

[0110] In some embodiments of the present application, based on the above technical solution, the content re-expression of the source voice data is generated based on the global semantic dictionary, and the speaker information includes the prompt voice data of the target speaker; the voice conversion module 940 is specifically configured to: based on the global semantic dictionary, obtain the content re-expression of the prompt voice data; input the content re-expression of the voice frame in the source voice data, the content re-expression of the prompt voice data, and the encoding tokens of the source voice data and the prompt voice into the first voice generation model to generate a target decoder token; perform audio decoding according to the target decoder token to obtain the target voice data of the target speaker.

[0111] In some embodiments of the present application, based on the above technical solution, the semantic dictionary also includes a speaker dictionary, and the speech conversion module 940 is further configured to: respectively perform statistics on the posterior probability that the speech frame in the speech data of each speaker belongs to the Kth speech unit, and obtain a second statistic corresponding to the Kth speech unit for each speaker; for each speaker, obtain the semantic expression corresponding to the Kth speech unit for each speaker based on the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the second statistic; determine the speaker dictionary corresponding to each speaker based on the semantic expression of the Kth speech unit corresponding to each speaker and the semantic expression of the other K-1 speech units among the K speech units corresponding to each speaker.

[0112] In some embodiments of the present application, based on the above technical solution, the expression conversion module 930 is specifically configured to: respectively multiply the content expressions corresponding to the K speech units in the global semantic dictionary and the speaker dictionary by the posterior probability vector to obtain a global content re-expression and a speaker content re-expression; calculate the weighted sum of the global content re-expression and the speaker content re-expression to obtain a content re-expression of the speech frame in the source speech data;

[0113] The speech conversion module 940 is specifically configured to: re-express the content of the speech frame in the source speech data and input the speaker identification of the target speaker in the speaker information into the second speech generation model for speech conversion to obtain the target speech data of the target speaker.

[0114] In some embodiments of the present application, based on the above technical solution, the expression conversion module 930 is specifically configured to: obtain the source content expression of the source voice data; calculate the weighted sum of the source content expression, the global content re-expression and the speaker content re-expression to obtain the content re-expression of the voice frame in the source voice data.

[0115] In some embodiments of the present application, based on the above technical solution, the speech conversion module 940 is specifically configured to: use the content re-expression of the speech frame in the source speech data and the embedded expression of the speaker information as conditions, and the specified spectrum containing noise Figure 1 The third speech generation model is input for spectrum prediction to obtain a result spectrum diagram, wherein the third speech generation model contains embedded information of time step information and speaker information; the result spectrum diagram is voice-converted by a pre-trained vocoder to obtain target speech data of the target speaker.

[0116] In some embodiments of the present application, based on the above technical solution, the speech conversion module 940 is also configured to: perform feature extraction on the training speech data based on the semantic dictionary to obtain content re-representation of the training speech data; splice the content re-representation and a specified spectrum graph containing noise into training data, wherein noise is added to the specified spectrum graph according to the time step; train the third model to be trained with the training data to obtain the third speech generation model, wherein the third model to be trained includes a feature linear modulation layer, a convolution layer and a diffusion transformer layer, the feature linear modulation layer includes information about the time step, and the convolution layer and the diffusion transformer layer include speaker embedding information of the speaker of the training speech data.

[0117] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0118] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0119] It should be noted that Fig.10 The computer system 1000 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0120] like Fig.10 As shown, the computer system 1000 includes a central processing unit (CPU) 1001, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1002 or the program loaded from the storage part 1008 to the random access memory (RAM) 1003. Various programs and data required for system operation are also stored in the RAM 1003. The CPU 1001, the ROM 1002 and the RAM 1003 are connected to each other through a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0121] The following components are connected to the I / O interface 1005: an input section 1006 including a keyboard, a mouse, etc.; an output section 1007 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1008 including a hard disk, etc.; and a communication section 1009 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1009 performs communication processing via a network such as the Internet. A drive 1010 is also connected to the I / O interface 1005 as needed. A removable medium 1011, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1010 as needed so that a computer program read therefrom is installed into the storage section 1008 as needed.

[0122] In particular, according to an embodiment of the present application, the process described in each method flow chart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flow chart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part 1009, and / or installed from a removable medium 1011. When the computer program is executed by a central processing unit (CPU) 1001, various functions defined in the system of the present application are executed.

[0123] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CompactDisc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0124] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0125] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0126] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation method of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation method of the present application.

[0127] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0128] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for processing speech data, characterized in that: include: Acquire source speech data of a source speaker and speaker information of a target speaker; Performing feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice units include at least one of phonemes and soft voice units, and K is an integer greater than 1; Determine the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expressions and the posterior probability in the speech data from at least two speakers; The target speech data of the target speaker is obtained by performing speech conversion according to the content re-expression of the speech frame in the source speech data and the speaker information.

2. The processing method according to claim 1, characterized in that: The semantic dictionary includes a global dictionary, and the method further includes: Obtaining the semantic expression of each speech frame in each dictionary speech data of a dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit, wherein the dictionary audio set contains speech data from at least two speakers; Performing statistics on the posterior probability that the extracted speech frame belongs to the K-th speech unit to obtain a first statistic of the K-th speech unit; Obtaining the semantic expression of the Kth speech unit according to a statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the Kth speech unit and the first statistic; The global dictionary is determined according to the semantic expression of the Kth speech unit and the semantic expressions of other K-1 speech units among the K speech units.

3. The processing method according to claim 2, characterized in that: The step of obtaining the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit includes: Inputting the source speech data into a pre-trained first feature extraction model, wherein the first feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K phonemes; An output result of the bottleneck layer in the first feature extraction model is obtained as the posterior probability vector of the source speech data.

4. The processing method according to claim 2, characterized in that: The step of obtaining the semantic expression of each speech frame in each dictionary speech data of the dictionary audio set and the posterior probability that each speech frame belongs to the Kth speech unit includes: Inputting the source speech data into a pre-trained second feature extraction model, wherein the second feature extraction model is used to calculate the posterior probability that each speech frame in the source speech data belongs to the K soft speech units; The output result of the specified converter layer in the second feature extraction model is used as the posterior probability vector of the source speech data.

5. The processing method according to claim 2, characterized in that: The content re-expression of the source speech data is generated based on the global semantic dictionary, and the speaker information includes the prompt speech data of the target speaker; The re-expression of the content of the speech frame in the source speech data and the speech conversion according to the speaker information to obtain the target speech data of the target speaker include: Based on the global semantic dictionary, obtaining a content re-expression of the prompt voice data; Inputting the content re-expression of the speech frame in the source speech data, the content re-expression of the prompt speech data, and the encoding tokens of the source speech data and the prompt speech into a first speech generation model to generate target decoder tokens; Audio decoding is performed according to the target decoder token to obtain target speech data of the target speaker.

6. The processing method according to claim 2, characterized in that: The semantic dictionary also includes a speaker dictionary, and the method further includes: The posterior probability that the speech frame in the speech data of each speaker belongs to the K-th speech unit is respectively counted to obtain a second statistic corresponding to the K-th speech unit for each speaker; For each speaker, according to the statistical result of the product of the semantic expression of each speech frame and the posterior probability that each speech frame belongs to the K-th speech unit and the second statistic, the semantic expression corresponding to the K-th speech unit corresponding to each speaker is obtained; According to the semantic expression of the Kth speech unit corresponding to each speaker and the semantic expression of other K-1 speech units in the K speech units corresponding to each speaker, a speaker dictionary corresponding to each speaker is determined.

7. The processing method according to claim 6, characterized in that: The step of determining the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector comprises: Multiplying the content expressions corresponding to the K speech units in the global semantic dictionary and the speaker dictionary by the posterior probability vector respectively to obtain a global content re-expression and a speaker content re-expression; Calculating a weighted sum of the global content re-expression and the speaker content re-expression to obtain a content re-expression of a speech frame in the source speech data; The re-expression of the content of the speech frame in the source speech data and the speech conversion according to the speaker information to obtain the target speech data of the target speaker include: The content of the speech frame in the source speech data is re-expressed and the speaker identification of the target speaker in the speaker information is input into the second speech generation model for speech conversion to obtain the target speech data of the target speaker.

8. The processing method according to claim 7, characterized in that: Calculating a weighted sum of the global content re-expression and the speaker content re-expression to obtain a content re-expression of a speech frame in the source speech data comprises: Acquire a source content expression of the source voice data; A weighted sum of the source content expression, the global content re-expression and the speaker content re-expression is calculated to obtain a content re-expression of a speech frame in the source speech data.

9. The processing method according to claim 6, characterized in that: The re-expression of the content of the speech frame in the source speech data and the speech conversion according to the speaker information to obtain the target speech data of the target speaker include: Using the content re-expression of the speech frame in the source speech data and the embedded expression of the speaker information as conditions, together with the specified spectrogram containing noise, input into a third speech generation model for spectrum prediction to obtain a result spectrogram, wherein the third speech generation model contains embedded information of time step information and speaker information; The resultant spectrogram is subjected to speech conversion by a pre-trained vocoder to obtain target speech data of the target speaker.

10. The processing method according to claim 9, characterized in that: The method further comprises: Extracting features from the training speech data based on the semantic dictionary to obtain content re-representation of the training speech data; splicing the content re-representation and a specified spectrogram containing noise into training data, wherein the specified spectrogram is added with noise according to a time step; The third model to be trained is trained using the training data to obtain the third speech generation model, wherein the third model to be trained comprises a feature linear modulation layer, a convolution layer and a diffusion transformer layer, the feature linear modulation layer comprises information of the time step, and the convolution layer and the diffusion transformer layer comprise speaker embedding information of the speaker of the training speech data.

11. A device for processing speech data, characterized in that: include: An information acquisition module configured to acquire source speech data of a source speaker and speaker information of a target speaker; A feature extraction module is configured to perform feature extraction on the source voice data to obtain a posterior probability vector containing K voice units that the voice frames in the source voice data belong to, wherein the voice units include at least one of phonemes and soft voice units, and K is an integer greater than 1; an expression conversion module, configured to determine the content re-expression of the speech frame in the source speech data according to the content expressions corresponding to the K speech units in the semantic dictionary and the posterior probability vector, wherein the semantic dictionary contains the content expressions corresponding to the K speech units, and the content expression of each speech unit in the semantic dictionary is obtained after statistical calculation based on the semantic expressions and the posterior probability in the speech data from at least two speakers; The voice conversion module is configured to perform voice conversion according to the content re-expression of the voice frame in the source voice data and the speaker information to obtain the target voice data of the target speaker.

12. An electronic device, characterized in that: include: processor; A memory, configured to store executable instructions of the processor; The processor is configured to execute the method for processing speech data according to any one of claims 1 to 10 by executing the executable instructions.

13. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for processing speech data according to any one of claims 1 to 10 is implemented.

14. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method for processing speech data as described in any one of claims 1 to 10.

Images