Voice conversion generation method and device, computer equipment and storage medium

Abstract

The invention belongs to the field of artificial intelligence, medical treatment and finance, and relates to a voice conversion generation method, which comprises the following steps: inputting a target accent sample set into a pre-constructed initial conversion model for training to obtain a pre-trained conversion model, obtaining a parallel voice data set containing a corresponding relationship between a source accent sample and the target accent sample, and generating a voice data set; fine tuning the conversion model to obtain an accent conversion model, training an initial generation model by using a target accent sample set to obtain a voice generation model, integrating the accent conversion model and the voice generation model to obtain a voice accent conversion generation model, and inputting the to-be-converted source accent voice into the voice accent conversion generation model. And generating the target accent voice. The invention further provides a voice conversion generation device, computer equipment and a storage medium. In addition, the invention also relates to a block chain technology, and the to-be-converted source accent voice can be stored in a block chain. According to the invention, dependence on parallel voice data is significantly reduced, and accent conversion and generation quality is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence, digital medicine and financial technology, and in particular to a speech conversion generation method, apparatus, computer equipment and storage medium. Background Art

[0002] With the rapid development of science and technology, the accuracy and efficiency of voice technology have become crucial factors in applications in the medical and financial fields, especially in telemedicine services and financial voice interaction systems. Accent Conversion (AC), as one of the key technologies to improve the voice interaction experience, faces the challenge of low resources in parallel data processing. This challenge is mainly reflected in the fact that accent conversion not only requires accurate conversion of the acoustic features of different pronunciation units, but also requires synchronous adjustment of prosodic patterns such as intonation, rhythm, and stress to adapt to the target accent. However, in practical applications, especially when processing diverse accent data from medical consultation hotlines or financial trading platforms, the system is often limited by the scarcity of computing resources and labeled data, and it is difficult to complete the accent conversion task efficiently and accurately. Summary of the invention

[0003] The purpose of the embodiments of the present application is to propose a speech conversion generation method, apparatus, computer device and storage medium to solve the technical problem in the existing accent conversion technology that the parallel speech resources are limited, resulting in low accent conversion efficiency and poor accuracy.

[0004] In order to solve the above technical problems, the present application embodiment provides a voice conversion generation method, which adopts the following technical solution:

[0005] Acquire a target accent sample set, and input the target accent sample set into a pre-built initial conversion model, wherein the initial conversion model includes a content encoder and a conversion decoder;

[0006] Extracting discrete semantic tags of each target accent sample in the target accent sample set by the content encoder to obtain a sample accent semantic tag sequence;

[0007] Damaging the sample accent semantic marker sequence in a marker damage manner to obtain a damaged accent semantic marker sequence;

[0008] Using the damaged accent semantic marker sequence, training the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model;

[0009] Acquire a parallel speech data set containing a correspondence between a source accent sample and a target accent sample, input the parallel speech data set into the conversion model for fine-tuning, and obtain a fine-tuned accent conversion model;

[0010] Inputting the target accent sample set into a pre-built initial generative model, wherein the initial generative model includes a codec and a generative decoder;

[0011] Extracting the acoustic marker of each target accent sample in the target accent sample set through the codec to obtain a style cue marker sequence;

[0012] Using the sample accent semantic marker sequence and the style cue marker sequence, training the generative decoder to reconstruct the corresponding target accent sample, and obtaining a trained speech generation model;

[0013] Integrating the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model;

[0014] The source accent speech to be converted is obtained, and the source accent speech to be converted is input into the speech accent conversion generation model to generate the target accent speech.

[0015] In order to solve the above technical problems, the embodiment of the present application also provides a speech conversion generation device, which adopts the following technical solution:

[0016] An acquisition module, used for acquiring a target accent sample set, and inputting the target accent sample set into a pre-built initial conversion model, wherein the initial conversion model includes a content encoder and a conversion decoder;

[0017] A semantic tag module, configured to extract discrete semantic tags of each target accent sample in the target accent sample set through the content encoder to obtain a sample accent semantic tag sequence;

[0018] A marking damage module, used for damaging the sample accent semantic marking sequence by a marking damage method to obtain a damaged accent semantic marking sequence;

[0019] A conversion module, used to adopt the damaged accent semantic marker sequence to train the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model;

[0020] A fine-tuning module, used to obtain a parallel speech data set containing a correspondence between a source accent sample and a target accent sample, and input the parallel speech data set into the conversion model for fine-tuning to obtain a fine-tuned accent conversion model;

[0021] An input module, used for inputting the target accent sample set into a pre-built initial generation model, wherein the initial generation model includes a codec and a generation decoder;

[0022] an acoustic tag module, configured to extract an acoustic tag of each target accent sample in the target accent sample set through the codec to obtain a style cue tag sequence;

[0023] A generation module, used to train the generation decoder to reconstruct the corresponding target accent sample using the sample accent semantic tag sequence and the style cue tag sequence, so as to obtain a trained speech generation model;

[0024] An integration module, used for integrating the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model;

[0025] The conversion generation module is used to obtain the source accent speech to be converted, input the source accent speech to be converted into the speech accent conversion generation model, and generate the target accent speech.

[0026] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0027] The computer device comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the voice conversion generation method as described above when executing the computer-readable instructions.

[0028] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0029] The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the voice conversion generation method described above are implemented.

[0030] Compared with the prior art, this application has the following beneficial effects:

[0031] The present application provides a speech conversion generation method. Through a two-stage generation framework, the first stage performs conversion at the semantic tag level, and the second stage synthesizes speech based on the converted semantic tags, decoupling accent conversion into two processes: conversion and speech generation. By using semantic tags as a bridge for conversion, the speech generation stage can make full use of a large amount of target accent speech data, thereby significantly reducing the conversion stage's dependence on parallel speech data. In addition, the introduction of language pre-training technology further reduces the demand for parallel speech data, improves the quality of accent conversion, and can retain the speaker's voice style characteristics while accurately converting to the target accent, providing an effective solution for cross-language and cross-dialect speech interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the scheme in the present application, a brief introduction is given below to the drawings required for use in the description of the embodiments of the present application. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0033] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0034] Figure 2 is a flow chart of an embodiment of a speech conversion generation method according to the present application;

[0035] Figure 3 is a structural schematic diagram of an embodiment of a speech conversion generation device according to the present application;

[0036] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0037] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of the present application; the terms used in the specification of the application herein are only for the purpose of describing specific embodiments and are not intended to limit the present application; the terms "including" and "having" and any variations thereof in the specification and claims of the present application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of the present application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0038] Reference to "embodiments" herein means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0039] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0040] like Figure 1As shown, the system architecture 100 may include a terminal device 101, a network 102 and a server 103. The terminal device 101 may be a laptop 1011, a tablet computer 1012 or a mobile phone 1013. The network 102 is used to provide a medium for a communication link between the terminal device 101 and the server 103. The network 102 may include various connection types, such as wired, wireless communication links or optical fiber cables.

[0041] The user can use the terminal device 101 to interact with the server 103 through the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0042] The terminal device 101 can be any electronic device with a display screen and supporting web browsing. In addition to a laptop computer 1011, a tablet computer 1012 or a mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV), a laptop computer, a desktop computer, etc.

[0043] The server 103 may be a server that provides various services, such as a background server that provides support for a web page displayed on the terminal device 101 .

[0044] It should be noted that the voice conversion generation method provided in the embodiment of the present application is generally executed by a server / terminal device, and accordingly, the voice conversion generation device is generally arranged in the server / terminal device.

[0045] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0046] Continue to refer Figure 2 , shows a flow chart of an embodiment of a speech conversion generation method according to the present application, comprising the following steps:

[0047] Step S201 , obtaining a target accent sample set, and inputting the target accent sample set into a pre-built initial conversion model, where the initial conversion model includes a content encoder and a conversion decoder.

[0048] The target accent sample set is formed by directly collecting the user's voice data, or it can be obtained from a public corpus or a data service platform that specializes in providing voice data sets, without limitation. The target accent sample set contains a large number of target accent samples, where the target accent refers to the accent that the original accent needs to be converted into during the accent conversion process and meets specific requirements or standards, including but not limited to standard Mandarin, a certain dialect, or the official language accent of other countries.

[0049] The choice of target accent depends on the specific scenario and needs. For example, in the medical field, it may be necessary to convert the patient's accent into an accent familiar to the doctor to improve communication efficiency; in the financial field, it may be necessary to convert the customer's accent into a standard accent that the system can recognize more accurately.

[0050] In this embodiment, the electronic device (eg Figure 1 The server / terminal device shown in the figure can receive the acquired target accent sample set via a wired connection or a wireless connection. It should be noted that the above wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future.

[0051] In this embodiment, the target accent sample set is denoised to obtain a denoised target accent sample set, and the denoised target accent sample set is input into a pre-built initial conversion model, which includes a content encoder and a conversion decoder. The target accent samples in the target accent sample set are semantically labeled in a discrete semantic label space by the content encoder, and the semantic labels are converted into accents by the conversion decoder.

[0052] For example, spectral subtraction is used for denoising, the noise power spectrum density is estimated, the signal-to-noise ratio threshold is set to 10 dB, and the estimated noise power spectrum is subtracted from the noisy speech power spectrum to obtain the denoised speech.

[0053] Step S202: extracting discrete semantic tags of each target accent sample in the target accent sample set through a content encoder to obtain a sample accent semantic tag sequence.

[0054] In this embodiment, a self-supervised learning mechanism is used to train the initial conversion model. In order to deal with the shortage of parallel accent data, the initial conversion model is pre-trained using the target accent sample set. Then, a small amount of weak parallel accent data is used to fine-tune the pre-trained conversion model so that the conversion model learns the conversion relationship from the source accent to the target accent.

[0055] In some alternative implementation manners, the content encoder adopts a pre-trained self-supervised speech representation model HuBERT, which includes a feature extractor and a plurality of content encoders connected together; the feature extractor extracts features from the target accent sample to obtain an accent feature representation, and the plurality of content encoders perform clustering and prediction on the accent feature representation to obtain a sample accent semantic tag sequence. Among them, the content encoder is a discrete content encoder, which divides the accent feature representation into a series of discrete semantic tags through a clustering algorithm, that is, clusters the accent feature representation through the K-means clustering algorithm to obtain a sample accent semantic tag sequence. During the clustering process, by calculating the Euclidean distance between the accent feature representations, similar accent feature representations are aggregated into the same category, and the number of clustering categories can be adjusted according to the scale of the corpus and the degree of accent difference.

[0056] Step S203, corrupt the sample accent semantic tag sequence through a tag corruption method to obtain a corrupted accent semantic tag sequence.

[0057] In this embodiment, the task of the conversion model aims to construct a probability space of discrete semantic tags in the target accent domain, so that the semantic tags of the target accent can be generated in a closed-loop manner according to the context of their previous tags in the target accent domain. In this task, the conversion model is trained in a self-supervised manner, that is, given a corrupted accent semantic tag sequence to generate the original sample accent semantic tag sequence Y={y0,…,y t}}, t<T, the formula is as follows:

[0058]

[0059] The tag corruption methods include tag masking, tag deletion, and tag filling, etc. In this embodiment, the tag filling method is adopted to corrupt the sample accent semantic tag sequence to obtain a corrupted accent semantic tag sequence.

[0060] Among them, tag filling means filling sequences with inconsistent lengths to the same length, usually by adding special padding tags at the end or beginning of the sequence, and these padding tags will not affect the prediction of the model. However, during the training process of the conversion model, padding tags can be added at any position in the sample accent semantic tag sequence to let the model predict the features before being filled, so as to learn the context information and high-level features of the speech accent.

[0061] Step S204, adopt the corrupted accent semantic tag sequence to train the conversion decoder to reconstruct the sample accent semantic tag sequence to obtain a pre-trained conversion model.

[0062] Specifically, the damaged accent semantic marker sequence is input into the conversion decoder for autoregressive generation to obtain a predicted accent semantic marker sequence; the cross entropy loss function between the predicted accent semantic marker sequence and the sample accent semantic marker sequence is calculated to obtain a first loss value; the model parameters of the content encoder and the conversion decoder are adjusted based on the first loss value, and iterative training is continued until the iteration stop condition is met to obtain a pre-trained conversion model.

[0063] The conversion decoder is pre-trained using the Transformer structure generated by autoregression to establish a mapping relationship between the damaged accent semantic marker sequence and the sample accent semantic marker sequence. In a specific example, the Transformer structure includes an input layer, an encoder part and a decoder part. The encoder part is composed of a plurality of encoder layers stacked together, and the decoder part is also composed of a plurality of decoder layers stacked together. The semantic and contextual features of the damaged accent semantic marker sequence are extracted through the multi-head attention mechanism of each encoder layer to obtain the accent semantic coding features, and the accent semantic coding features are input into the decoder part. The long-distance dependency between the accent semantic coding features is captured through the multi-head attention mechanism and the cross-attention mechanism of each encoder to obtain the predicted accent semantic marker sequence.

[0064] The cross entropy loss function is used to measure the difference between the probability distribution of the predicted accent semantic marker sequence generated by the model and the probability distribution of the real sample accent semantic marker sequence. The first loss value between the predicted accent semantic marker sequence and the sample accent semantic marker sequence is calculated by the cross entropy loss function, and the model parameters of the content encoder and the conversion decoder are adjusted according to the first loss value using the Adam optimizer. The adjusted model is iteratively trained until the iteration stop condition is met, that is, the number of iterations reaches the preset number or the first loss value does not change significantly, and the pre-trained conversion model is obtained.

[0065] By pre-training the conversion model with damaged accent semantic marker sequences, the conversion model can learn the high-level feature representation and contextual information of the speech and the accent, improve the model transfer performance and enhance the model's understanding of complex structures. At the same time, there is no need to use parallel corpus, which improves the generalization ability of the model.

[0066] Step S205, obtaining a parallel speech data set including the correspondence between the source accent sample and the target accent sample, inputting the parallel speech data set into the conversion model for fine-tuning, and obtaining a fine-tuned accent conversion model.

[0067] Since some phonemes in the source accent need to be converted to target accent phonemes, it is necessary to learn the mapping between source accent phonemes and target accent phonemes. Specifically, a pre-trained conversion model is fine-tuned under the semantic labeling of the source accent using a small amount of weakly parallel accent data.

[0068] Specifically, a parallel speech dataset is input into a conversion model, and the speech feature vector sequences of the source accent samples and the target accent samples in the parallel speech dataset are respectively extracted through the content encoder of the conversion model; the speech feature vector sequences are clustered through a clustering algorithm to obtain a source accent marker sequence and a target accent marker sequence; the source accent marker sequence is input into a conversion decoder of the conversion model to obtain a corresponding predicted accent marker sequence; the cross entropy loss function between the predicted accent marker sequence and the target accent marker sequence is calculated to obtain a second loss value; the model parameters of the conversion decoder are fine-tuned based on the second loss value, and the iterative training is continued until the iteration stop condition is met to obtain a fine-tuned accent conversion model.

[0069] For the source accent samples and the target accent samples in the parallel speech dataset, the speech feature vector sequences of the source accent speech and the target accent speech are extracted respectively through the content encoder of the pre-trained conversion model to obtain the source accent speech feature vector sequence and the target accent speech feature vector sequence; the extracted source accent speech feature vector sequence and the target accent speech feature vector sequence are aligned to ensure that the lengths of the two sequences are consistent, which is convenient for subsequent feature mapping and conversion; the source accent speech feature vector sequence and the target accent speech feature vector sequence are clustered respectively through the K-means clustering algorithm to obtain the source accent marker sequence and the target accent marker sequence; the conversion decoder is trained according to the source accent marker sequence and the target accent marker sequence to learn the conversion relationship of mapping the source accent marker sequence to the target accent marker sequence, and the predicted accent marker sequence is obtained after the conversion decoder converts the source accent marker sequence.

[0070] The second loss value between the predicted accent mark sequence and the target accent mark sequence is calculated by the cross entropy loss function. The Adam optimizer is used to fine-tune the model parameters of the conversion decoder according to the second loss value. The fine-tuned model is then iteratively trained until the iteration stop condition is met to obtain the final fine-tuned accent conversion model.

[0071] By fine-tuning the conversion model using a parallel speech dataset, parallel data is reduced and the adaptability of the model is improved, while the characteristics of the source accent and the target accent can be accurately captured, thereby improving the quality of accent conversion.

[0072] Step S206: input the target accent sample set into a pre-built initial generation model, wherein the initial generation model includes a codec and a generation decoder.

[0073] In this embodiment, the accent conversion process is divided into two stages. The first stage implements semantic tagging through the accent conversion model, and the second stage performs speech synthesis of the target accent through the speech generation model.

[0074] The target accent sample set is obtained from a large target accent speech corpus, and an initial generation model based on TF-Codec is pre-built. The initial generation model iteratively generates TF-Codec acoustic tags based on the converted target accent semantic tags through single-stage causal speech generation.

[0075] In this embodiment, the initial generative model includes a codec and a generative decoder. The codec extracts acoustic markers representing the speaker's style from a target accent sample set, and the generative decoder fuses the sample accent semantic marker sequence and acoustic markers to reconstruct the target accent speech.

[0076] Step S207: extracting the acoustic marker of each target accent sample in the target accent sample set through a codec to obtain a style cue marker sequence.

[0077] The codec uses the pre-trained causal speech neural codec TF-Codec, referred to as TF-Codec codec. The TF-Codec codec extracts the acoustic markers of each frame in the target accent sample. The TF-Codec codec can decompose speech into subspaces such as content, prosody, timbre, and audio details, achieving efficient attribute decoupling.

[0078] In this embodiment, the prediction loop is removed and a 6 kbps non-predictive model is used for efficient acoustic modeling to obtain high-quality output.

[0079] In some optional implementations, the step of extracting the acoustic marker of each target accent sample in the target accent sample set by a codec to obtain a style cue marker sequence includes:

[0080] Inputting the target accent sample set into a codec, wherein the codec comprises an input layer, a style encoder, a style decoder, and an output layer;

[0081] Preprocessing the target accent sample set through the input layer to obtain a purified speech signal, and converting the purified speech signal according to a preset format to obtain an input speech signal;

[0082] The acoustic features of the input speech signal are extracted through the style encoder to obtain an acoustic feature representation vector;

[0083] Input the acoustic feature representation vector into the style decoder for decoding to generate the corresponding style acoustic feature vector;

[0084] The style acoustic feature vector is post-processed through the output layer, and the post-processed style acoustic feature vector is mapped to a preset style acoustic tag space to obtain a final style cue tag sequence.

[0085] The target accent sample set is preprocessed through the input layer, including noise reduction, endpoint detection, and removal of silence segments. Specifically, spectral subtraction is used for noise reduction, the noise power spectrum density is estimated, the signal-to-noise ratio threshold is set, and the estimated noise power spectrum is subtracted from the noisy speech power spectrum to obtain the noise-reduced speech. Then, an energy-based endpoint detection algorithm is used to set the energy threshold, detect the start and end points of the noise-reduced speech, and remove the silence segments according to the start and end points of the speech to obtain a purified speech signal.

[0086] In this embodiment, the TF-Codec codec accepts a time-frequency spectrum compressed with a 16kHz amplitude as input, a window length of 20 milliseconds, and a step length of 5 milliseconds, and the preset format is a time-frequency spectrum compressed with a 16kHz amplitude, a window length of 20 milliseconds, and a step length of 5 milliseconds. The cleaned speech signal is converted into an input speech signal that can be processed by the model according to the preset format.

[0087] The input speech signal is fed into a style encoder to capture the short-term and long-term temporal dependencies between input frames in a causal manner, resulting in an acoustic feature representation vector.

[0088] In some optional implementations of this embodiment, the style encoder includes multiple 2D causal convolutional layers, a temporal convolutional network, and a gated recurrent unit block, and the step of extracting acoustic features from the input speech signal through the style encoder to obtain an acoustic feature representation vector includes:

[0089] The input speech signal is convolved through multiple 2D causal convolutional layers to obtain acoustic convolution features;

[0090] The acoustic convolution features are input into the temporal convolutional network for causal convolution and dilated convolution to obtain acoustic time series features.

[0091] The acoustic time series features are input into the gated recurrent unit block to calculate the hidden state and output the acoustic feature representation vector.

[0092] The number of 2D causal convolution layers is selected according to actual conditions. A 2D convolution operation is performed on the input speech signal through multiple 2D causal convolution layers to extract the time domain and frequency domain features of the speech signal to obtain acoustic convolution features.

[0093] The temporal convolutional network consists of multiple convolutional layers, each of which uses causal convolution and dilated convolution to extract temporal features. In causal convolution, when the convolution kernel slides on the input sequence, it can only see the elements at the current position and before, that is, the output of the current time step only depends on the past time steps; dilated convolution expands the receptive field of the convolution kernel by introducing a dilation rate parameter, thereby increasing the range of contextual information that can be captured while keeping the size of the convolution kernel unchanged, that is, it can capture long-distance temporal dependencies. When processing the input acoustic convolution features, the temporal convolutional network will perform convolution operations on the acoustic convolution features layer by layer, extract temporal features at different levels, and output the final acoustic temporal features.

[0094] The Gated Recurrent Unit (GRU) block includes a reset gate and an update gate. The reset gate determines how much information of the previous hidden state needs to be forgotten; the update gate determines how much information of the current hidden state needs to be retained. The hidden state of the acoustic time series feature is calculated through the GRU block, and the final hidden state is output as the acoustic feature representation vector.

[0095] In some optional implementations, in the temporal convolutional network, after each convolutional layer, residual connections and layer normalization operations are performed to accelerate the convergence speed and training stability of the network while avoiding the gradient vanishing problem.

[0096] By extracting acoustic features from input speech signals through a style encoder, we can improve efficient parallel processing capabilities and long-term dependency capture capabilities, reduce resource consumption, and more accurately capture acoustic features, thereby improving the quality of accent conversion.

[0097] In some optional implementations of this embodiment, the style decoder includes an embedding layer, multiple vector quantizers and a decoding layer, and the step of inputting the acoustic feature representation vector into the style decoder for decoding to generate a corresponding style acoustic feature vector includes:

[0098] The acoustic feature representation vector is transformed into a vector through the embedding layer to obtain an acoustic embedding vector;

[0099] Inputting the acoustic embedding vector into a plurality of vector quantizers for group quantization, and obtaining acoustic embedding group vectors having the same number as the vector quantizers;

[0100] The acoustic embedding group vector is input into the decoding layer for attention mechanism calculation to generate the style acoustic feature vector.

[0101] The acoustic feature representation vector is converted into an acoustic embedding vector through the embedding layer. The acoustic embedding vector is divided into K groups by group quantization. Each group is quantized by a vector quantizer with 1024 codewords to obtain an acoustic embedding group vector. The acoustic embedding group vector is input into the decoding layer. The decoding layer introduces the attention mechanism and uses the corresponding decoding algorithm to decode the acoustic embedding group vector to obtain a style acoustic feature vector that represents the speaker's style. The speaker's style includes pronunciation habits, intonation patterns, etc.

[0102] Grouping quantization and decoding through the style decoder can better capture the data characteristics of different numerical ranges and improve the model accuracy. At the same time, it can achieve efficient compression of the model and accelerate reasoning.

[0103] In this embodiment, the style acoustic feature vector is post-processed through the output layer. Specifically, the style acoustic feature vector is normalized using the L2 norm, and the normalized vector is mapped to a style acoustic tag space of a preset dimension, and a final style cue tag sequence is output.

[0104] By extracting style cue marker sequences through the codec, it is possible to more accurately extract style features in speech, thereby achieving higher quality accent conversion, more flexibly adapting to the accent characteristics of different users, and improving the generalization ability of the model.

[0105] Step S208, using the sample accent semantic marker sequence and the style cue marker sequence, training the generated decoder to reconstruct the corresponding target accent sample, and obtaining a trained speech generation model.

[0106] Specifically, the sample accent semantic marker sequence and style cue marker sequence are decoded by a generative decoder to obtain predicted target accent speech; according to a preset loss function, a third loss value between the predicted target accent speech and the corresponding target accent sample is calculated; based on the third loss value, the model parameters of the generative decoder are adjusted, and iterative training continues until the iteration stop condition is met.

[0107] In this embodiment, the generative decoder is a single-stage causal speech generation model, which adopts a Transformer structure generated by autoregression. In a specific example, the Transformer structure includes an input layer, an encoder part, a decoder part and an output layer. The encoder part includes a plurality of encoder layers stacked together, and the decoder part is also a plurality of decoder layers stacked together. The sample accent semantic marker sequence and the style cue marker sequence are spliced ​​through the input layer to obtain a spliced ​​semantic marker sequence; the semantic and contextual features of the spliced ​​semantic marker sequence are extracted through the multi-head attention mechanism of each encoder layer to obtain the target accent style coding features, and the target accent style coding features are input into the decoder part. The long-distance dependency between the target accent style coding features is captured through the multi-head attention mechanism and the cross-attention mechanism of each encoder to obtain the predicted target accent speech.

[0108] The reconstruction loss between the predicted target accent speech and the corresponding target accent sample is calculated, and the preset loss function adopts L1 loss to obtain a third loss value; through the back propagation algorithm, the model parameters of the generated decoder are optimized and updated using the calculated third loss value to obtain an updated model, and iterative training is continued until the iteration stop condition is met, that is, the number of iterations reaches the preset number of times or the first loss value does not change significantly, and a pre-trained generated decoder is obtained.

[0109] Among them, the L1 loss calculation formula is as follows:

[0110]

[0111] Where n represents the number of target accent samples in the target accent sample set; y i represents the i-th target accent sample in the target accent sample set; y' i represents the predicted target accent speech corresponding to the i-th target accent sample in the initial reconstructed speech dataset.

[0112] By training the generative decoder, the naturalness and fluency of the converted speech are improved, achieving better speech quality at lower complexity.

[0113] Step S209, integrating the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model.

[0114] The trained accent conversion model and speech generation model are connected and integrated, where the output of the accent conversion model is connected to the codec of the speech generation model to form a complete speech accent conversion generation model.

[0115] Step S210, obtaining the source accent speech to be converted, inputting the source accent speech to be converted into the speech accent conversion generation model, and generating the target accent speech.

[0116] In this embodiment, a source accent speech input to be converted is obtained into a speech accent conversion generation model, and discrete semantic tags are performed on the source accent speech to be converted through a content encoder to obtain source semantic features; the source semantic features are input into a conversion decoder for target accent feature mapping and conversion to obtain target semantic features; the target semantic features are input into a speech generation model; speech of a preset time length before the source accent speech to be converted is extracted to obtain a target style speech; the target style speech is input into a codec for acoustic tag extraction to obtain style cue features; the target semantic features and style cue features are concatenated to obtain concatenated features; the concatenated features are input into a generation decoder for autoregressive generation to obtain the target accent speech.

[0117] It should be emphasized that in order to further ensure the privacy and security of the source accent voice to be converted, the above-mentioned source accent voice to be converted can also be stored in a node of a blockchain.

[0118] The blockchain referred to in this application is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, encryption algorithm, etc. Blockchain is essentially a decentralized database, a string of data blocks generated by cryptographic methods. Each data block contains a batch of network transaction information, which is used to verify the validity of its information (anti-counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, platform product service layer, and application service layer.

[0119] The present application adopts a two-stage generation framework, including a conversion model and a generation model. In the first stage, the conversion model is used to perform conversion at the semantic tag level, and the generation model in the second stage synthesizes speech based on the converted semantic tags, decoupling accent conversion into two processes: conversion and speech generation. By using semantic tags as a bridge for conversion, the speech generation stage can make full use of a large amount of target accent speech data, thereby significantly reducing the conversion stage's dependence on parallel speech data. In addition, the introduction of language pre-training technology further reduces the demand for parallel speech data, improves the quality of accent conversion, and can retain the speaker's voice style characteristics while accurately converting to the target accent, providing an effective solution for cross-language and cross-dialect speech interaction.

[0120] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0121] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0122] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through computer-readable instructions, and the computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the aforementioned storage medium can be a non-volatile storage medium such as a disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0123] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0124] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech conversion generation device, and the device embodiment is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0125] like Figure 3 As shown, the speech conversion generation device 300 described in this embodiment includes: an acquisition module 301, a semantic tag module 302, a tag damage module 303, a conversion module 304, a fine-tuning module 305, an input module 306, an acoustic tag module 307, a generation module 308, an integration module 309 and a conversion generation module 310. Among them:

[0126] The acquisition module 301 is used to acquire a target accent sample set, and input the target accent sample set into a pre-built initial conversion model, wherein the initial conversion model includes a content encoder and a conversion decoder;

[0127] The semantic tag module 302 is used to extract the discrete semantic tag of each target accent sample in the target accent sample set through the content encoder to obtain a sample accent semantic tag sequence;

[0128] The mark damage module 303 is used to damage the sample accent semantic mark sequence by a mark damage method to obtain a damaged accent semantic mark sequence;

[0129] The conversion module 304 is used to use the damaged accent semantic marker sequence to train the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model;

[0130] The fine-tuning module 305 is used to obtain a parallel speech data set containing a correspondence between a source accent sample and a target accent sample, and input the parallel speech data set into the conversion model for fine-tuning to obtain a fine-tuned accent conversion model;

[0131] The input module 306 is used to input the target accent sample set into a pre-built initial generation model, wherein the initial generation model includes a codec and a generation decoder;

[0132] The acoustic labeling module 307 is used to extract the acoustic label of each target accent sample in the target accent sample set through the codec to obtain a style cue label sequence;

[0133] The generation module 308 is used to use the sample accent semantic marker sequence and the style cue marker sequence to train the generation decoder to reconstruct the corresponding target accent sample, thereby obtaining a trained speech generation model;

[0134] The integration module 309 is used to integrate the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model;

[0135] The conversion generation module 310 is used to obtain the source accent speech to be converted, input the source accent speech to be converted into the speech accent conversion generation model, and generate the target accent speech.

[0136] It should be emphasized that in order to further ensure the privacy and security of the source accent voice to be converted, the above-mentioned source accent voice to be converted can also be stored in a node of a blockchain.

[0137] Based on the above-mentioned speech conversion generation device 300, a two-stage generation framework is adopted, including a conversion model and a generation model. In the first stage, the conversion model is used to perform conversion at the semantic tag level, and the generation model in the second stage synthesizes speech based on the converted semantic tags, decoupling the accent conversion into two processes: conversion and speech generation. By using semantic tags as a bridge for conversion, the speech generation stage can make full use of a large amount of target accent speech data, thereby significantly reducing the dependence of the conversion stage on parallel speech data. In addition, the introduction of language pre-training technology can further reduce the demand for parallel speech data, improve the quality of accent conversion, and retain the speaker's voice style characteristics while accurately converting them into the target accent, providing an effective solution for cross-language and cross-dialect speech interaction.

[0138] In some optional implementations of this embodiment, the conversion module 304 is further configured to:

[0139] Inputting the damaged accent semantic marker sequence into the conversion decoder for autoregressive generation to obtain a predicted accent semantic marker sequence;

[0140] Calculating a cross entropy loss function between the predicted accent semantic marker sequence and the sample accent semantic marker sequence to obtain a first loss value;

[0141] The model parameters of the content encoder and the conversion decoder are adjusted based on the first loss value, and the iterative training is continued until an iteration stop condition is met, thereby obtaining a pre-trained conversion model.

[0142] By pre-training the conversion model with damaged accent semantic marker sequences, the conversion model can learn the high-level feature representation and contextual information of the speech and the accent, improve the model transfer performance and enhance the model's understanding of complex structures. At the same time, there is no need to use parallel corpus, which improves the generalization ability of the model.

[0143] In some optional implementations of this embodiment, the fine-tuning module 305 is further configured to:

[0144] Extracting speech feature vector sequences of source accent samples and target accent samples in the parallel speech dataset respectively through the content encoder of the conversion model;

[0145] Clustering the speech feature vector sequence by a clustering algorithm to obtain a source accent mark sequence and a target accent mark sequence;

[0146] Inputting the source accent mark sequence into a conversion decoder of the conversion model to obtain a corresponding predicted accent mark sequence;

[0147] calculating a cross entropy loss function between the predicted accent mark sequence and the target accent mark sequence to obtain a second loss value;

[0148] The model parameters of the conversion decoder are fine-tuned based on the second loss value, and the iterative training is continued until an iteration stop condition is met to obtain a fine-tuned accent conversion model.

[0149] By fine-tuning the conversion model using a parallel speech dataset, parallel data is reduced and the adaptability of the model is improved, while the characteristics of the source accent and the target accent can be accurately captured, thereby improving the quality of accent conversion.

[0150] In some optional implementations of this embodiment, the acoustic marking module 307 includes:

[0151] An input submodule, configured to input the target accent sample set into the codec, wherein the codec comprises an input layer, a style encoder, a style decoder and an output layer;

[0152] A preprocessing submodule, configured to preprocess the target accent sample set through the input layer to obtain a purified speech signal, and convert the purified speech signal according to a preset format to obtain an input speech signal;

[0153] An encoding submodule, configured to extract acoustic features from the input speech signal through the style encoder to obtain an acoustic feature representation vector;

[0154] A decoding submodule, used for inputting the acoustic feature representation vector into the style decoder for decoding to generate a corresponding style acoustic feature vector;

[0155] The post-processing submodule is used to post-process the style acoustic feature vector through the output layer, and map the post-processed style acoustic feature vector to a preset style acoustic tag space to obtain a final style prompt tag sequence.

[0156] By extracting style cue marker sequences through the codec, it is possible to more accurately extract style features in speech, thereby achieving higher quality accent conversion, more flexibly adapting to the accent characteristics of different users, and improving the generalization ability of the model.

[0157] In some optional implementations, the style encoder includes a plurality of 2D causal convolutional layers, a temporal convolutional network, and a gated recurrent unit block, and the encoding submodule is further used to:

[0158] Performing a convolution operation on the input speech signal through the multiple 2D causal convolution layers to obtain acoustic convolution features;

[0159] Inputting the acoustic convolutional features into the temporal convolutional network for causal convolution and dilated convolution to obtain acoustic time series features;

[0160] The acoustic time series feature is input into the gated recurrent unit block to calculate the hidden state, and an acoustic feature representation vector is output.

[0161] By extracting acoustic features from input speech signals through a style encoder, we can improve efficient parallel processing capabilities and long-term dependency capture capabilities, reduce resource consumption, and more accurately capture acoustic features, thereby improving the quality of accent conversion.

[0162] In some optional implementations, the style decoder includes an embedding layer and a plurality of vector quantizers and a decoding layer, and the decoding submodule is further used for:

[0163] Performing vector conversion on the acoustic feature representation vector through the embedding layer to obtain an acoustic embedding vector;

[0164] Inputting the acoustic embedding vector into the multiple vector quantizers for group quantization to obtain the same number of acoustic embedding group vectors as the number of the vector quantizers;

[0165] The acoustic embedding group vector is input into the decoding layer for attention mechanism calculation to generate a style acoustic feature vector.

[0166] Grouping quantization and decoding through the style decoder can better capture the data characteristics of different numerical ranges and improve the model accuracy. At the same time, it can achieve efficient compression of the model and accelerate reasoning.

[0167] In some optional implementations, the generating module 308 is further configured to:

[0168] Decoding the sample accent semantic marker sequence and the style cue marker sequence by the generative decoder to obtain a predicted target accent speech;

[0169] Calculating a third loss value between the predicted target accent speech and the corresponding target accent sample according to a preset loss function;

[0170] The model parameters of the generated decoder are adjusted based on the third loss value, and the iterative training is continued until an iteration stopping condition is met.

[0171] By training the generative decoder, the naturalness and fluency of the converted speech are improved, achieving better speech quality at lower complexity.

[0172] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4This is a basic structural block diagram of the computer device in this embodiment.

[0173] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected and communicated through a system bus. It should be noted that the figure only shows a computer device 4 having a memory 41, a processor 42, and a network interface 43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital processor (DSP), an embedded device, etc.

[0174] The computer device may be a computing device such as a desktop computer, a notebook, a PDA, a cloud server, etc. The computer device may interact with a user through a keyboard, a mouse, a remote controller, a touch pad, or a voice control device.

[0175] The memory 41 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as a hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (FlashCard), etc. equipped on the computer device 4. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions of the speech conversion generation method, etc. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0176] The processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to run computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for running the speech conversion generation method.

[0177] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0178] Through a two-stage generation framework, including a conversion model and a generation model, the first stage uses the conversion model to perform conversion at the semantic tag level, and the second stage's generation model synthesizes speech based on the converted semantic tags, decoupling accent conversion into two processes: conversion and speech generation. By using semantic tags as a bridge for conversion, the speech generation stage can make full use of a large amount of target accent speech data, thereby significantly reducing the conversion stage's dependence on parallel speech data. In addition, the introduction of language pre-training technology can further reduce the demand for parallel speech data, improve the quality of accent conversion, and retain the speaker's voice style characteristics while accurately converting to the target accent.

[0179] The present application also provides another implementation, namely, providing a computer-readable storage medium, wherein the computer-readable storage medium stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the speech conversion generation method as described above.

[0180] Through a two-stage generation framework, including a conversion model and a generation model, the first stage uses the conversion model to perform conversion at the semantic tag level, and the second stage's generation model synthesizes speech based on the converted semantic tags, decoupling accent conversion into two processes: conversion and speech generation. By using semantic tags as a bridge for conversion, the speech generation stage can make full use of a large amount of target accent speech data, thereby significantly reducing the conversion stage's dependence on parallel speech data. In addition, the introduction of language pre-training technology can further reduce the demand for parallel speech data, improve the quality of accent conversion, and retain the speaker's voice style characteristics while accurately converting to the target accent.

[0181] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0182] Obviously, the embodiments described above are only some embodiments of the present application, rather than all embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application is described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions recorded in the aforementioned specific implementation methods, or to perform equivalent replacement of some of the technical features therein. Any equivalent structure made using the contents of the specification and drawings of this application, directly or indirectly used in other related technical fields, is similarly within the scope of patent protection of this application.

Claims

1. A speech conversion generation method, characterized in that: The steps include: Acquire a target accent sample set, and input the target accent sample set into a pre-built initial conversion model, wherein the initial conversion model includes a content encoder and a conversion decoder; Extracting discrete semantic tags of each target accent sample in the target accent sample set by the content encoder to obtain a sample accent semantic tag sequence; Damaging the sample accent semantic marker sequence in a marker damage manner to obtain a damaged accent semantic marker sequence; Using the damaged accent semantic marker sequence, training the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model; Acquire a parallel speech data set containing a correspondence between a source accent sample and a target accent sample, input the parallel speech data set into the conversion model for fine-tuning, and obtain a fine-tuned accent conversion model; Inputting the target accent sample set into a pre-built initial generative model, wherein the initial generative model includes a codec and a generative decoder; Extracting the acoustic marker of each target accent sample in the target accent sample set through the codec to obtain a style cue marker sequence; Using the sample accent semantic marker sequence and the style cue marker sequence, training the generative decoder to reconstruct the corresponding target accent sample, and obtaining a trained speech generation model; Integrating the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model; The source accent speech to be converted is obtained, and the source accent speech to be converted is input into the speech accent conversion generation model to generate the target accent speech.

2. The speech conversion generation method according to claim 1, characterized in that: The step of adopting the damaged accent semantic marker sequence to train the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model comprises: Inputting the damaged accent semantic marker sequence into the conversion decoder for autoregressive generation to obtain a predicted accent semantic marker sequence; Calculating a cross entropy loss function between the predicted accent semantic marker sequence and the sample accent semantic marker sequence to obtain a first loss value; The model parameters of the content encoder and the conversion decoder are adjusted based on the first loss value, and the iterative training is continued until an iteration stop condition is met, thereby obtaining a pre-trained conversion model.

3. The speech conversion generation method according to claim 1, characterized in that: The step of inputting the parallel speech data set into the conversion model for fine-tuning to obtain a fine-tuned accent conversion model comprises: Extracting speech feature vector sequences of source accent samples and target accent samples in the parallel speech dataset respectively through the content encoder of the conversion model; Clustering the speech feature vector sequence by a clustering algorithm to obtain a source accent mark sequence and a target accent mark sequence; Inputting the source accent mark sequence into a conversion decoder of the conversion model to obtain a corresponding predicted accent mark sequence; calculating a cross entropy loss function between the predicted accent mark sequence and the target accent mark sequence to obtain a second loss value; The model parameters of the conversion decoder are fine-tuned based on the second loss value, and the iterative training is continued until an iteration stop condition is met to obtain a fine-tuned accent conversion model.

4. The speech conversion generation method according to claim 1, characterized in that: The step of extracting the acoustic marker of each target accent sample in the target accent sample set by the codec to obtain a style cue marker sequence comprises: Inputting the target accent sample set into the codec, wherein the codec comprises an input layer, a style encoder, a style decoder, and an output layer; Preprocessing the target accent sample set through the input layer to obtain a purified speech signal, and converting the purified speech signal according to a preset format to obtain an input speech signal; Extracting acoustic features from the input speech signal through the style encoder to obtain an acoustic feature representation vector; Inputting the acoustic feature representation vector into the style decoder for decoding to generate a corresponding style acoustic feature vector; The style acoustic feature vector is post-processed through the output layer, and the post-processed style acoustic feature vector is mapped into a preset style acoustic tag space to obtain a final style prompt tag sequence.

5. The speech conversion generation method according to claim 4, characterized in that: The style encoder includes a plurality of 2D causal convolutional layers, a temporal convolutional network and a gated recurrent unit block. The step of extracting acoustic features from the input speech signal through the style encoder to obtain an acoustic feature representation vector includes: Performing a convolution operation on the input speech signal through the multiple 2D causal convolution layers to obtain acoustic convolution features; Inputting the acoustic convolutional features into the temporal convolutional network for causal convolution and dilated convolution to obtain acoustic time series features; The acoustic time series feature is input into the gated recurrent unit block to calculate the hidden state, and an acoustic feature representation vector is output.

6. The speech conversion generation method according to claim 4, characterized in that: The style decoder includes an embedding layer and a plurality of vector quantizers and a decoding layer. The step of inputting the acoustic feature representation vector into the style decoder for decoding to generate a corresponding style acoustic feature vector includes: Performing vector conversion on the acoustic feature representation vector through the embedding layer to obtain an acoustic embedding vector; Inputting the acoustic embedding vector into the multiple vector quantizers for group quantization to obtain the same number of acoustic embedding group vectors as the number of the vector quantizers; The acoustic embedding group vector is input into the decoding layer for attention mechanism calculation to generate a style acoustic feature vector.

7. The speech conversion generation method according to claim 1, characterized in that: The step of using the sample accent semantic marker sequence and the style cue marker sequence to train the generative decoder to reconstruct the corresponding target accent sample comprises: Decoding the sample accent semantic marker sequence and the style cue marker sequence by the generative decoder to obtain a predicted target accent speech; Calculating a third loss value between the predicted target accent speech and the corresponding target accent sample according to a preset loss function; The model parameters of the generated decoder are adjusted based on the third loss value, and the iterative training is continued until an iteration stopping condition is met.

8. A speech conversion generation device, characterized in that: include: An acquisition module, used for acquiring a target accent sample set, and inputting the target accent sample set into a pre-built initial conversion model, wherein the initial conversion model includes a content encoder and a conversion decoder; A semantic tag module, configured to extract discrete semantic tags of each target accent sample in the target accent sample set through the content encoder to obtain a sample accent semantic tag sequence; A marking damage module, used for damaging the sample accent semantic marking sequence by a marking damage method to obtain a damaged accent semantic marking sequence; A conversion module, used to adopt the damaged accent semantic marker sequence to train the conversion decoder to reconstruct the sample accent semantic marker sequence to obtain a pre-trained conversion model; A fine-tuning module, used to obtain a parallel speech data set containing a correspondence between a source accent sample and a target accent sample, and input the parallel speech data set into the conversion model for fine-tuning to obtain a fine-tuned accent conversion model; An input module, used for inputting the target accent sample set into a pre-built initial generation model, wherein the initial generation model includes a codec and a generation decoder; an acoustic tag module, configured to extract an acoustic tag of each target accent sample in the target accent sample set through the codec to obtain a style cue tag sequence; A generation module, used to train the generation decoder to reconstruct the corresponding target accent sample using the sample accent semantic tag sequence and the style cue tag sequence, so as to obtain a trained speech generation model; An integration module, used for integrating the accent conversion model and the speech generation model to obtain a final speech accent conversion generation model; The conversion generation module is used to obtain the source accent speech to be converted, input the source accent speech to be converted into the speech accent conversion generation model, and generate the target accent speech.

9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores computer-readable instructions, and the processor implements the steps of the speech conversion generation method according to any one of claims 1 to 7 when executing the computer-readable instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, the steps of the speech conversion generation method according to any one of claims 1 to 7 are implemented.

Images