Model updating method and device, voice conversion method, equipment and storage medium

Abstract

This application provides a model update method and apparatus, a speech conversion method, device, and storage medium, belonging to the field of financial technology. The method includes: acquiring sample speech data; inputting the sample speech data into a neural network model; encoding the sample speech data through an encoding network to obtain an initial audio feature vector; indexing the initial audio feature vector based on a preset codebook to obtain an audio frame index; extracting phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; performing speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector; decoding the sample audio embedding vector and the speech style embedding vector through a decoding network to obtain synthesized speech data; and updating the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech conversion model. This application can improve the accuracy of the model in speech conversion.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a model update method and apparatus, a speech conversion method, device and storage medium. Background Technology

[0002] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0003] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue used by these chatbots is often generated through speech-to-text conversion.

[0004] Speech conversion typically refers to changing the speaking style of a chatbot from one speaker to another without altering the speech content. However, current speech conversion methods using neural network models often fail to adequately learn the actual speech content and the speaker's stylistic features, resulting in poor accuracy. Therefore, improving the accuracy of speech conversion models has become a pressing technical problem. Summary of the Invention

[0005] The main objective of this application is to provide a training method and apparatus, electronic device and storage medium for a speech conversion model, which aims to improve the accuracy of the model in speech conversion.

[0006] To achieve the above objectives, a first aspect of this application proposes a model update method, the method comprising:

[0007] Obtain sample speech data from the speaker;

[0008] The sample speech data is input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0009] The sample speech data is encoded using the encoding network to obtain an initial audio feature vector;

[0010] Based on a preset codebook, the initial audio feature vector is indexed to obtain an audio frame index, and based on the audio frame index, phoneme features are extracted from the initial audio feature vector to obtain an initial phoneme feature vector.

[0011] Speech alignment is performed on the initial phoneme feature vector to obtain the sample audio embedding vector;

[0012] The decoding network decodes the sample audio embedding vector and the pre-acquired speech style embedding vector to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaker.

[0013] The parameters of the neural network model are updated based on the synthesized speech data and the sample speech data to obtain a speech conversion model.

[0014] In some embodiments, the step of indexing the initial audio feature vector based on a preset codebook to obtain an audio frame index, and then extracting phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector, includes:

[0015] The initial audio feature vector is segmented to obtain multiple audio frame vectors;

[0016] Based on the reference vector of the preset codebook, the audio frame vector is indexed and queried to obtain the audio frame index corresponding to each audio frame vector. The preset codebook includes a one-to-one correspondence between the reference vector and the audio frame index, and a mapping relationship between each audio frame index and the phoneme feature.

[0017] Based on the audio frame index and the mapping relationship, extract the phoneme features corresponding to the audio frame vector;

[0018] The phoneme features of all the audio frame vectors are merged to obtain the initial phoneme feature vector.

[0019] In some embodiments, the step of performing speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector includes:

[0020] The duration of the initial phoneme feature vector is predicted based on a preset time predictor to obtain the duration sequence of the initial phoneme feature vector;

[0021] The initial phoneme feature vector is speech aligned based on the duration sequence to obtain the sample audio embedding vector.

[0022] In some embodiments, the step of predicting the duration of the initial phoneme feature vector based on a preset time predictor to obtain a duration sequence of the initial phoneme feature vector includes:

[0023] The initial phoneme feature vector is segmented to obtain multiple phoneme feature segments;

[0024] The phoneme feature segments are identified based on a preset time predictor to obtain the phoneme category of the initial phoneme feature vector and the number of phonemes in each phoneme category.

[0025] The duration sequence is obtained based on the phoneme category and the number of phonemes.

[0026] In some embodiments, the step of performing speech alignment on the initial phoneme feature vector based on the duration sequence to obtain the sample audio embedding vector includes:

[0027] The initial phoneme feature vector is embedded to obtain the audio text embedding vector;

[0028] The audio text embedding vector is segmented according to the duration sequence to obtain an intermediate vector for each phoneme category, wherein the number of intermediate vectors is equal to the number of phonemes.

[0029] The mean of the intermediate vectors is calculated to obtain the candidate vector for each phoneme category;

[0030] The candidate vectors are copied according to the number of phonemes to obtain a target vector for each phoneme category, wherein the number of target vectors is equal to the number of phonemes.

[0031] All the target vectors are concatenated to obtain the sample audio embedding vector.

[0032] In some embodiments, before decoding the sample audio embedding vector and the pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data, the model update method further includes obtaining the speech style embedding vector, specifically including:

[0033] The sample speech data is input into a preset voiceprint recognition model, wherein the voiceprint recognition model includes a segmentation layer and a hidden layer;

[0034] The sample speech data is segmented based on the segmentation layer to obtain multiple sample speech segments.

[0035] Based on the hidden layer, style recognition is performed on each of the sample speech segments to obtain multiple initial style embedding vectors;

[0036] The speaking style embedding vector is obtained by averaging the multiple initial style embedding vectors.

[0037] To achieve the above objectives, a second aspect of this application provides a speech conversion method, the method comprising:

[0038] Acquire the target speaking style information of the target speaker and the raw speech data to be processed;

[0039] The original speech data and the target speaking style information are input into the speech conversion model for speech conversion to obtain the target speech data, wherein the speech conversion model is obtained through a model update method of the first aspect.

[0040] To achieve the above objectives, a third aspect of this application provides a model update apparatus, the apparatus comprising:

[0041] The data acquisition module is used to acquire sample speech data of the sample speaking object;

[0042] An input module is used to input the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0043] The encoding module is used to encode the sample speech data through the encoding network to obtain an initial audio feature vector;

[0044] The query module is used to perform an index query on the initial audio feature vector based on a preset codebook to obtain an audio frame index, and to extract phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector.

[0045] The speech alignment module is used to perform speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector.

[0046] The decoding module is used to decode the sample audio embedding vector and the pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaking object;

[0047] The model update module is used to update the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech conversion model.

[0048] To achieve the above objectives, a fourth aspect of the present application provides an electronic device, the electronic device including a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method described in the first aspect or the method described in the second aspect.

[0049] To achieve the above objectives, a fifth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect or the method described in the second aspect.

[0050] The model update method, speech conversion method, model update device, electronic device, and storage medium proposed in this application acquire sample speech data of a sample speaker; input the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; encode the sample speech data through the encoding network to obtain an initial audio feature vector; perform an index lookup on the initial audio feature vector based on a preset codebook to obtain an audio frame index, and extract phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; perform speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector; and decode the sample audio embedding vector and a pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data, which enables... Synthetic speech data contains speech content and speaking style characteristics that are quite similar to sample audio data. Finally, updating the parameters of the neural network model based on the synthetic speech data and sample speech data allows the model to focus more on learning the similarity between the synthetic speech data and sample speech data in terms of speech content and speaking style. This effectively improves the model's update performance and enhances the accuracy of speech conversion. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue partner. By adopting dialogue methods and styles that are more interesting to the dialogue partner, the quality and effectiveness of the dialogue are improved. This enables intelligent voice dialogue services, improves customer service quality and customer satisfaction, and ultimately increases the business conversion rate. Attached Figure Description

[0051] Figure 1 This is a flowchart of the model update method provided in the embodiments of this application;

[0052] Figure 2 yes Figure 1 The flowchart of step S104 in the process;

[0053] Figure 3 yes Figure 1 The flowchart of step S105 in the process;

[0054] Figure 4 yes Figure 3 The flowchart of step S301 in the process;

[0055] Figure 5 yes Figure 3 The flowchart of step S302 in the text;

[0056] Figure 6 This is another flowchart of the model update method provided in the embodiments of this application;

[0057] Figure 7 This is a flowchart of the speech conversion method provided in the embodiments of this application;

[0058] Figure 8 This is a schematic diagram of the structure of the model update device provided in the embodiments of this application;

[0059] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0063] First, let's analyze some of the terms used in this application:

[0064] Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0065] Natural Language Processing (NLP): NLP uses computers to process, understand, and utilize human language (such as Chinese and English). NLP is a branch of artificial intelligence and an interdisciplinary field of computer science and linguistics, often referred to as computational linguistics. NLP includes syntactic analysis, semantic analysis, and discourse understanding. It is commonly used in machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, intent recognition, information extraction and filtering, text classification and clustering, sentiment analysis, and opinion mining. It involves data mining, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computation.

[0066] Phoneme: The smallest unit of speech based on the natural properties of speech. It is analyzed based on the articulation of a syllable, and one articulation constitutes one phoneme.

[0067] Fourier transform: Represents a function that satisfies certain conditions as a linear combination of trigonometric functions (sine and / or cosine functions) or their integrals. In different research fields, the Fourier transform has various variants, such as the continuous Fourier transform and the discrete Fourier transform.

[0068] Mel-Frequency Cipstal Coefficients (MFCCs) are a set of key coefficients used to construct a Mel-Frequency Cipstal spectrum. From a segment of a music signal, a set of cepstrum values ​​can be obtained that is sufficient to represent the music signal. The Mel-Frequency Cipstal Coefficients are the cepstrum values ​​derived from this cepstrum (i.e., the spectrum of the spectrum). Unlike a regular cepstrum, the most distinctive feature of the Mel-Frequency Cipstrum is that its frequency bands are uniformly distributed across the Mel scale. In other words, compared to the linear cepstrum representations commonly seen, this frequency band is closer to the non-linear human auditory system. For example, Mel-Frequency Cipstals are frequently used in audio compression techniques.

[0069] Encoder: Transforms an input sequence into a fixed-length vector.

[0070] Decoding: This involves transforming a previously generated fixed vector into an output sequence; the input sequence can be text, speech, image, or video; the output sequence can be text or image.

[0071] With the rapid development of artificial intelligence technology, intelligent voice interaction is being widely used in finance, logistics, customer service and other fields, improving the service level of enterprise customer service through functions such as intelligent marketing, intelligent debt collection and content navigation.

[0072] Currently, chatbots are frequently used in financial service scenarios such as intelligent customer service and shopping guidance to provide corresponding service support to various users. The dialogue used by these chatbots is often generated through speech-to-text conversion.

[0073] Taking insurance service robots as an example, it is often necessary to merge the descriptive text of insurance products with the speaking style of a fixed target audience to generate a voice description of the insurance product by that fixed target audience. When the insurance service robot converses with some interested individuals, it will automatically use this voice description to introduce the insurance product to those individuals. However, when the insurance service robot needs to converse with some new potential targets, the speaking style of the existing voice description can be replaced. Without changing the content of the voice description, the speaking style in the voice description is changed from the speaking style of target A to the speaking style of target B, so that the voice-converted voice description matches the conversational preferences of the new potential targets.

[0074] Speech conversion typically refers to changing the speaking style of a chatbot from one speaker to another without altering the speech content. However, current speech conversion methods using neural network models often fail to adequately learn the actual speech content and the speaker's stylistic features, resulting in poor accuracy. Therefore, improving the accuracy of speech conversion models has become a pressing technical problem.

[0075] Based on this, embodiments of this application provide a model update method, a speech conversion method, a model update device, an electronic device, and a storage medium, aiming to improve the accuracy of the model in speech conversion.

[0076] The model update method, speech conversion method, model update device, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the model update method in the embodiments of this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The model update method and speech conversion method provided in this application relate to the field of artificial intelligence technology. The model update method and speech conversion method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the model update method and speech conversion method, but is not limited to the above forms.

[0080] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user voice data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to a confirmation page. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0081] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0082] Figure 1 This is an optional flowchart of the model update method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0083] Step S101: Obtain sample speech data of the sample speaker;

[0084] Step S102: Input the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0085] Step S103: Encode the sample speech data using an encoding network to obtain the initial audio feature vector;

[0086] Step S104: Based on the preset codebook, the initial audio feature vector is indexed and queried to obtain the audio frame index, and based on the audio frame index, the initial audio feature vector is extracted to obtain the initial phoneme feature vector.

[0087] Step S105: Perform speech alignment on the initial phoneme feature vector to obtain the sample audio embedding vector;

[0088] Step S106: The sample audio embedding vector and the pre-acquired speech style embedding vector are decoded by a decoding network to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaker.

[0089] Step S107: Update the parameters of the neural network model based on the synthesized speech data and sample speech data to obtain the speech conversion model.

[0090] Steps S101 to S107 of this embodiment involve: acquiring sample speech data of the speaker; inputting the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; encoding the sample speech data through the encoding network to obtain an initial audio feature vector; indexing the initial audio feature vector based on a preset codebook to obtain an audio frame index, and extracting phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; performing speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector; decoding the sample audio embedding vector and a pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data, which enables the synthesized speech data to contain speech content and speech style characteristics that are relatively close to the sample audio data; finally, updating the parameters of the neural network model based on the synthesized speech data and the sample speech data, which enables the model to pay more attention to learning the similarity between the synthesized speech data and the sample speech data in speech content and speech style, effectively improving the model update effect and increasing the accuracy of the speech conversion model for speech conversion.

[0091] In step S101 of some embodiments, a web crawler can be written, and after setting the data source, data can be crawled in a targeted manner to obtain sample speech data of the sample speaking object. The data source can be various types of network platforms, social media, or certain specific audio databases, etc. The sample speech data can be the music material, speech report, chat dialogue, etc. of the sample speaking object. The sample speech data includes sample audio content and sample acoustic features. The sample acoustic features include the timbre information, pitch information, and other voice style characteristics of the sample speaking object.

[0092] For example, in the field of financial transactions, sample voice data is audio data containing commonly used dialogues in the financial field; in the insurance sales scenario, sample voice data is audio data containing descriptions of a certain insurance product, such as its type, cost, and target audience.

[0093] In a specific example, the sample audio content of the sample speech data is "consulting about credit card issues", "preferring credit cards with high credit limits", "handling deposit business", etc., and the sample acoustic feature is "normal speech rate".

[0094] In step S102 of some embodiments, sample speech data is input into a preset neural network model. The neural network model includes an encoding network and a decoding network. The encoding network is mainly used to reconstruct and align the input speech data, extract phoneme features from the input speech data, and adjust the length of the phoneme features according to the phoneme features and the obtained phoneme duration so that the phoneme features and the speech data can be aligned. The decoding network is mainly used to decouple the aligned phoneme features from the preset reference speaking style features, convert the speech style of the input speech data into the reference speaking style, so that the original speaking object is converted into the reference speaking object without changing the speech content of the input speech data. That is, the phoneme features of the input speech data are fused with the reference speaking style features of the reference speaking object to form new speech data. The trained neural network model can achieve a better speech conversion effect.

[0095] In step S103 of some embodiments, the sample speech data is encoded by an encoding network to extract audio content features from the sample speech data and obtain an initial audio feature vector. The initial audio feature vector is a continuous vector. This method can more conveniently extract audio content information from the sample speech data, eliminate interference caused by other redundant information to the model update, and improve the accuracy of the model update.

[0096] Please see Figure 2 In some embodiments, step S104 may include, but is not limited to, steps S201 to S204:

[0097] Step S201: The initial audio feature vector is segmented to obtain multiple audio frame vectors;

[0098] Step S202: Based on the reference vector of the preset codebook, the audio frame vector is indexed and queried to obtain the audio frame index corresponding to each audio frame vector. The preset codebook includes a one-to-one correspondence between the reference vector and the audio frame index, and the mapping relationship between each audio frame index and the phoneme feature.

[0099] Step S203: Extract the phoneme features corresponding to the audio frame vector based on the audio frame index and mapping relationship;

[0100] Step S204: Merge the phoneme features of all audio frame vectors to obtain the initial phoneme feature vector.

[0101] In step S201 of some embodiments, vector quantization technology can be used to segment the initial audio feature vector, converting the continuous initial audio feature vector into multiple discrete vectors to obtain multiple audio frame vectors, wherein the audio frame vector contains the audio content features of the sample speech data in each time frame.

[0102] In step S202 of some embodiments, the preset codebook is an initialized vector set. The preset codebook contains multiple reference vectors and audio frame indices, with a one-to-one correspondence between the reference vectors and audio frame indices, and a one-to-one mapping relationship between each audio frame index and a phoneme feature. For example, if the preset codebook contains 128 reference vectors, then each reference vector corresponds to an audio frame index, meaning the index range of the audio frame indices is 0 to 127, and each audio frame index corresponds to a phoneme feature. Therefore, an index query can be performed on the audio frame vectors based on the reference vectors of the preset codebook. That is, the cosine similarity between each audio frame vector and each reference vector in the preset codebook is calculated, and the audio frame index corresponding to the reference vector with the smallest cosine similarity is taken as the audio frame index corresponding to that audio frame vector.

[0103] In step S203 of some embodiments, since there is a one-to-one mapping relationship between each audio frame index and phoneme features, after determining the audio frame index corresponding to each audio frame vector, the phoneme features corresponding to the audio frame index are extracted according to the mapping relationship, and the extracted phoneme features are used as the phoneme features of the audio frame vector.

[0104] In step S204 of some embodiments, the queried phoneme features are sequentially concatenated according to the temporal order of the audio frame vectors to obtain an initial phoneme feature vector.

[0105] For example, after encoding the sample speech data M through an encoding network, an initial audio feature vector Z is obtained. Vector quantization is then used to segment this initial audio feature vector into multiple audio frame vectors P. Each frame in a pre-defined codebook is then queried for a reference vector corresponding to each audio frame vector P. This involves comparing the cosine similarity between each audio frame vector and each reference vector in the pre-defined codebook. The audio frame index corresponding to the reference vector with the lowest cosine similarity is taken as the audio frame index of that audio frame vector. The phoneme features corresponding to this audio frame index are then queried, and these queried phoneme features are used as the phoneme features of the audio frame vector P. This method clearly reflects the phoneme features corresponding to each audio frame, facilitating the determination of differences in phoneme features between audio frame vectors. Finally, according to the temporal order of the audio frame vectors, the queried phoneme features are concatenated sequentially to obtain the initial phoneme feature vector Q.

[0106] Through the above steps S201 to S204, the phoneme features corresponding to the sample speech data can be easily queried according to the preset codebook, and the speech text content information represented by the sample speech data can be obtained. This enables speech conversion based on the obtained speech text content information (i.e., the initial phoneme feature vector), which helps to train the model's ability to learn the text content information of the sample speech data and improve the model's speech conversion performance.

[0107] Please see Figure 3 In some embodiments, step S105 may include, but is not limited to, steps S301 to S302:

[0108] Step S301: Based on a preset time predictor, the duration of the initial phoneme feature vector is predicted to obtain the duration sequence of the initial phoneme feature vector.

[0109] Step S302: Perform speech alignment on the initial phoneme feature vector according to the duration sequence to obtain the sample audio embedding vector.

[0110] In step S301 of some embodiments, the preset time predictor may include a length adjuster, three multi-head attention layers and a fully connected layer. The length adjuster is mainly used to simulate and adjust the feature length of the initial phoneme feature vector so that the adjusted initial phoneme feature vector can achieve frame and speech alignment. The multi-head attention layer and the fully connected layer are mainly used to extract the time feature information in the initial phoneme feature vector, and to comprehensively analyze and predict the phoneme category of the initial phoneme feature vector and the number of phonemes corresponding to each phoneme category according to the importance of different time feature information. Finally, the duration sequence of the initial phoneme feature vector is constructed according to the phoneme category and the number of phonemes in each phoneme category.

[0111] In step S302 of some embodiments, before performing speech alignment on the initial phoneme feature vector, it is necessary to embed the initial phoneme feature vector to map it to a fixed vector space, obtaining an audio text embedding vector containing speech content information. The audio text embedding vector is then divided into multiple embedding vector segments, each corresponding to a frame of phoneme features in the time dimension of the sample audio data. Therefore, the number of embedding vector segments is equal to the number of frames in the sample speech data. These embedding vector segments are then classified based on phoneme categories to obtain an intermediate vector for each phoneme category. Further, the mean vector of the intermediate vector for each phoneme category is calculated, and the mean vector is copied according to the number of phonemes in that phoneme category to obtain the target vector for that phoneme category. Finally, the target vectors of all phoneme categories are concatenated to achieve speech alignment of the initial phoneme feature vector, resulting in the sample audio embedding vector.

[0112] Through the above steps S301 to S302, the number of elements and element values ​​of the duration sequence can be determined based on the phoneme information of the sample speech data. Based on the element information of the duration sequence, the speech content information of the initial phoneme feature vector is aligned with the audio length of the sample speech data to obtain a sample audio embedding vector that can represent the text content features of the sample speech data and whose audio length is consistent with the sample speech data. This is beneficial for adjusting the speech length of the generated synthetic speech data, thereby improving the accuracy of speech conversion.

[0113] Please see Figure 4 In some embodiments, step S301 may include, but is not limited to, steps S401 to S403:

[0114] Step S401: The initial phoneme feature vector is segmented to obtain multiple phoneme feature segments;

[0115] Step S402: Based on the preset time predictor, the phoneme feature segments are identified to obtain the phoneme category of the initial phoneme feature vector and the number of phonemes in each phoneme category.

[0116] Step S403: Obtain the duration sequence based on the phoneme category and the number of phonemes.

[0117] In step S401 of some embodiments, the initial phoneme feature vector is first segmented using a duration predictor, which splits the initial phoneme feature vector into frame-by-frame feature vectors to obtain multiple phoneme feature segments, where each phoneme feature segment corresponds to a Mel cepstral frame.

[0118] In step S402 of some embodiments, the feature length of the initial phoneme feature vector is first simulated and adjusted using the length adjuster in the duration predictor. Then, the temporal feature information in the initial phoneme feature vector is extracted using a multi-head attention layer and a fully connected layer. Based on the importance of different temporal feature information, the phoneme category of the initial phoneme feature vector and the number of phonemes corresponding to each phoneme category are predicted. The initial phoneme feature vector contains the phoneme category, and the number of times each phoneme appears is the number of phonemes in the phoneme category.

[0119] In step S403 of some embodiments, when constructing the duration sequence, the phoneme category can be used as the number of element types in the duration sequence, and the number of phonemes can be used as the element value of each element. For example, if the number of frames of a sample speech data is 7, then the initial phoneme feature vector contains 7 phoneme features. The initial phoneme feature vector contains three phoneme categories, namely phoneme category A, phoneme category B, and phoneme category C. The number of phonemes in phoneme category A is 2, the number of phonemes in phoneme category B is 1, and the number of phonemes in phoneme category C is 4. Then the duration sequence of the sample speech data is [2, 1, 4].

[0120] Through the above steps S401 to S403, the number of elements and element values ​​of the duration sequence can be determined based on the phoneme information of the initial phoneme feature vector, thereby converting the phoneme duration of the initial phoneme feature vector into a sequence representation. This allows the speech length to be adjusted based on the duration sequence in the subsequent speech conversion process, improving the speech coherence of the synthesized speech data obtained by conversion and improving the speech conversion effect.

[0121] Please see Figure 5 In some embodiments, step S302 may include, but is not limited to, steps S501 to S504:

[0122] Step S501: Embed the initial phoneme feature vector to obtain the audio text embedding vector;

[0123] Step S502: The audio text embedding vector is segmented according to the duration sequence to obtain the intermediate vector for each phoneme category, wherein the number of intermediate vectors is equal to the number of phonemes.

[0124] Step S503: Calculate the mean of the intermediate vectors to obtain the candidate vectors for each phoneme category;

[0125] Step S504: The candidate vectors are copied according to the number of phonemes to obtain the target vector for each phoneme category, wherein the number of target vectors is equal to the number of phonemes.

[0126] Step S505: Concatenate all target vectors to obtain the sample audio embedding vector.

[0127] In step S501 of some embodiments, the initial phoneme feature vector is embedded by an encoding network to map the initial phoneme feature vector to a fixed vector space to obtain an audio text embedding vector, wherein the audio text embedding vector contains the audio text content information of the sample speech data.

[0128] In step S502 of some embodiments, the audio text embedding vector is segmented based on the sum of the element values ​​of the duration sequence to obtain an intermediate vector for each phoneme category. Specifically, if a duration sequence is [2, 1, 4], the sum of the element values ​​is 2 + 1 + 4 = 7, so the audio text embedding vector is segmented into 7 intermediate vectors, each corresponding to a frame segment of the sample speech data. Since one frame segment corresponds to one phoneme feature, one intermediate vector corresponds to one phoneme feature, and the number of intermediate vectors is equal to the number of phonemes. For example, if the number of phonemes in phoneme category A is 2, then there are also two intermediate vectors for phoneme category A.

[0129] In step S503 of some embodiments, the average of all intermediate vectors belonging to the same phoneme category is calculated to obtain the candidate vector corresponding to each phoneme category. Specifically, the vector summation of all intermediate vectors of a certain phoneme category is first performed, and the number of phonemes in that phoneme category is determined at the same time. The quotient of the vector summation result and the number of phonemes is calculated to obtain the average vector of that phoneme category, and the average vector is used as the candidate vector of that phoneme category.

[0130] In step S504 of some embodiments, the candidate vectors of each phoneme category are copied according to the number of phonemes in that category to obtain the target vector for each phoneme category. For example, if a phoneme category contains 'a' phonemes, then the number of phonemes in that category is 'a'. The candidate vectors of that phoneme category are copied 'a' times to obtain the target vector for that phoneme category. Therefore, the number of target vectors for a phoneme category is equal to the number of phonemes.

[0131] In step S505 of some embodiments, all target vectors belonging to the same sample speech data are concatenated to achieve speech alignment of the initial phoneme feature vector and obtain the sample audio embedding vector.

[0132] Through the above steps S501 to S505, the speech content information and audio length of the sample speech data can be aligned according to the element type and element value of the duration sequence. This enables the model to better learn the speech content information of the sample speech data, improves the feature constraints on model training, enhances the model's learning and generalization ability, and enables the model to better achieve speech conversion.

[0133] Please see Figure 6 In some embodiments, prior to step S106, the model update method may include, but is not limited to, steps S601 to S604:

[0134] Step S601: Input the sample speech data into the preset voiceprint recognition model, wherein the voiceprint recognition model includes a segmentation layer and a hidden layer;

[0135] Step S602: The sample speech data is segmented based on the segmentation layer to obtain multiple sample speech segments;

[0136] Step S603: Perform style recognition on each sample speech segment based on the hidden layer to obtain multiple initial style embedding vectors;

[0137] Step S604: Calculate the mean of multiple initial style embedding vectors to obtain the speech style embedding vector.

[0138] In step S601 of some embodiments, sample speech data can be directly input into a preset voiceprint recognition model, wherein the voiceprint recognition model can be constructed based on a deep convolutional neural network, and the voiceprint recognition model includes a segmentation layer and a hidden layer.

[0139] In step S602 of some embodiments, the sample speech data is segmented based on the segmentation process, and the sample speech data is split into frame-by-frame audio segments to obtain multiple sample speech segments.

[0140] In step S603 of some embodiments, style features are extracted sequentially for each sample speech segment based on the hidden layer and the temporal order of the sample speech data to obtain an initial style embedding vector corresponding to each sample speech segment. The initial style embedding vector contains the speaking style characteristics of the sample speech segment. For example, the initial style embedding vector contains speech feature information of the sample speech segment in terms of pitch, frequency, tone, etc.

[0141] In step S604 of some embodiments, the average of multiple initial style embedding vectors is calculated to obtain the average style embedding vector of the sample speech data. The average style embedding vector is used as the speaking style embedding vector of the sample speaker. The speaking style embedding vector includes the speaking style characteristics of the sample speaker. For example, the speaking style embedding vector can characterize whether the sample speaker speaks quickly or slowly, whether the voice is high-pitched or low-pitched, etc.

[0142] Through the above steps S601 to S604, the speaking style information of the sample speech data can be extracted relatively easily, and the extracted speaking style information can be used for model training and updating, thereby improving the model's ability to learn speech style characteristics and improving the model's performance in speech conversion.

[0143] In step S106 of some embodiments, the sample audio embedding vector and the speech style embedding vector are first concatenated to obtain a synthesized audio vector. Then, the concatenated vector is decoupled through a decoding network to obtain synthesized Mel-Cepstral data. The synthesized Mel-Cepstral data is then converted into waveform form based on the vocoder in the decoding network to obtain synthesized speech data. The speech style embedding vector includes the speech style information of the sample speaker. In this way, the synthesized speech data can contain speech content and speech style information of the sample speaker that are relatively close to the sample speech data, so that the obtained synthesized speech data has better audio quality.

[0144] In step S107 of some embodiments, the process of calculating the loss of synthesized speech data and sample speech data using a preset loss function can be represented as shown in formula (1):

[0145] L recon =||xx′||1 Formula (1)

[0146] Among them, L recon Let L be the model loss value, x be the sample speech data, and x′ be the synthesized speech data. recon The magnitude of the value can clearly reflect the similarity between the sample speech data and the synthesized speech data. Simultaneously, the model loss value L... recon The size of the value can also clearly reflect the training level of the model.

[0147] Furthermore, since the sample speech data originates from the sample speakers, and the speech style embedding vectors used to train the model also originate from the sample speakers, the synthesized speech data obtained through the neural network model should be as close as possible to the sample speech data; that is, the model loss value needs to be as small as possible. Therefore, it is necessary to update the parameters of the neural network model based on the model loss value. By updating the model parameters of the neural network model, the synthesized speech data obtained through the neural network model is made closer to the sample speech data. When, after multiple parameter updates, the model loss value is less than or equal to the preset loss threshold, it indicates that the similarity between the synthesized speech data and the sample speech data is good, and the speech conversion effect of the neural network model can meet the current requirements. At this point, updating the neural network model stops, and the speech conversion model is obtained.

[0148] The model update method of this application embodiment obtains sample speech data of a sample speaker; inputs the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; encodes the sample speech data through the encoding network to obtain an initial audio feature vector; performs an index lookup on the initial audio feature vector based on a preset codebook to obtain an audio frame index, and extracts phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; performs speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector that can characterize the text content features of the sample speech data and has the same audio length as the sample speech data, which is beneficial for adjusting the speech length of the generated synthetic speech data. The decoding network decodes the sample audio embedding vector and the pre-obtained speech style embedding vector to obtain synthetic speech data, which enables the synthetic speech data to contain speech content and speech style characteristics that are relatively close to the sample audio data; finally, the neural network model is updated based on the synthetic speech data and the sample speech data, which enables the model to pay more attention to learning the similarity between the synthetic speech data and the sample speech data in speech content and speech style, effectively improving the model update effect and increasing the accuracy of the speech conversion model for speech conversion.

[0149] Please see Figure 7 This application also provides a speech conversion method, which may include, but is not limited to, steps S701 to S702:

[0150] Step S701: Obtain the target speaking style information of the target speaker and the raw speech data to be processed;

[0151] Step S702: Input the original speech data and the target speaking style information into the speech conversion model to perform speech conversion and obtain the target speech data. The speech conversion model is obtained according to the model update method in the first aspect.

[0152] In step S701 of some embodiments, a web crawler can be written, and after setting up a data source, data can be crawled in a targeted manner to obtain the raw audio data to be processed and the target speaking style information of the target speaker. The data source can be various types of online platforms, social media, or specific audio databases, etc. The raw audio data can be music material, a speech, a chat conversation, etc., of a speaker. Other methods can also be used to obtain the raw audio data and the target speaking style information of the target speaker, and are not limited to these. Furthermore, the target speaking style information can originate from the acoustic feature extraction of the target speaker's audio. For example, audio data of the target speaker can be obtained from online platforms, social media, or audio databases, and speaking style information such as pitch and timbre features of the target speaker can be obtained through a voiceprint recognition model or other d-vector techniques.

[0153] In step S702 of some embodiments, the original audio data and the speaking style information of the target speaker are input into the speech conversion model for speech conversion. The speech content of the original audio data is obtained through the speech conversion model. Then, the speaking style information of the target speaker is fused with the speech content of the original audio data, thereby realizing the conversion of the speaking style characteristics of the original audio data and obtaining the target audio data.

[0154] The speech conversion method of this application embodiment encodes and aligns the original audio data through the coding network of the speech conversion model to obtain a target audio embedding vector. The target audio embedding vector, after speech alignment, is then jointly decoupled from the target speaking style information of the target speaker to form new audio data, namely the target audio data. This ensures that the speech content of the target audio data is identical to that of the original audio data. Simultaneously, the target audio data contains the timbre and pitch characteristics of the target speaker, thus transforming the original speaker corresponding to the original audio data into the target speaker without altering the speech content information of the original audio data. This method effectively represents the speech content information and the speaking characteristics of the target speaker, significantly improving the speech conversion effect. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue object. By adopting dialogue methods and styles that are more interesting to the dialogue object, the quality and effectiveness of the dialogue are improved, enabling intelligent voice dialogue services, enhancing customer service quality and satisfaction, and ultimately increasing the business conversion rate.

[0155] Please see Figure 8 This application also provides a model update apparatus that can implement the above-described model update method. The apparatus includes:

[0156] Data acquisition module 801 is used to acquire sample speech data of the sample speaking object;

[0157] The input module 802 is used to input sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network;

[0158] The encoding module 803 is used to encode the sample speech data through an encoding network to obtain an initial audio feature vector;

[0159] The query module 804 is used to perform an index query on the initial audio feature vector based on a preset codebook to obtain the audio frame index, and to extract phoneme features from the initial audio feature vector based on the audio frame index to obtain the initial phoneme feature vector.

[0160] The speech alignment module 805 is used to perform speech alignment on the initial phoneme feature vector to obtain the sample audio embedding vector;

[0161] The decoding module 806 is used to decode the sample audio embedding vector and the pre-acquired speech style embedding vector through a decoding network to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaker.

[0162] The model update module 807 is used to update the parameters of the neural network model based on the synthesized speech data and sample speech data to obtain the speech conversion model.

[0163] The specific implementation of this model update device is basically the same as the specific implementation of the model update method described above, and will not be repeated here.

[0164] Furthermore, this application embodiment also provides a speech conversion device that can implement the above-mentioned speech conversion method. The device includes:

[0165] The acquisition module is used to acquire the target speaking style information of the target speaker and the raw speech data to be processed;

[0166] The speech conversion module is used to input the original speech data and the target speaking style information into the speech conversion model to perform speech conversion and obtain the target speech data. The speech conversion model is obtained according to the aforementioned model update device.

[0167] The specific implementation of this voice conversion device is basically the same as the specific embodiment of the above-described voice conversion method, and will not be repeated here.

[0168] This application also provides an electronic device, which includes: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for communication between the processor and the memory. When the program is executed by the processor, it implements the aforementioned model update method or speech conversion method. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0169] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0170] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0171] The memory 902 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 902 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the model update method or speech conversion method of the embodiments of this application.

[0172] The input / output interface 903 is used to implement information input and output;

[0173] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0174] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0175] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0176] This application also provides a computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the above-described model update method or speech conversion method.

[0177] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0178] The model update method, speech conversion method, model update device, electronic device, and computer-readable storage medium provided in this application embodiment acquire sample speech data of a sample speaking object; input the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; encode the sample speech data through the encoding network to obtain an initial audio feature vector; perform an index query on the initial audio feature vector based on a preset codebook to obtain an audio frame index, and extract phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector; perform speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector that can characterize the text content features of the sample speech data and has an audio length consistent with the sample speech data, which is beneficial for adjusting the speech length of the generated synthetic speech data. By decoding the sample audio embedding vector and the pre-acquired speech style embedding vector through a decoding network, synthetic speech data is obtained. This ensures that the synthetic speech data contains speech content and speech style characteristics that are closely similar to the sample audio data. Finally, the parameters of the neural network model are updated based on the synthetic speech data and the sample speech data. This allows the model to focus more on learning the similarity between the synthetic speech data and the sample speech data in terms of speech content and speech style, effectively improving the model's update performance and enhancing the accuracy of speech conversion. This speech conversion model can transform the original speaker in the original audio data into the target speaker without changing the speech content information. This method can better represent the speech content information and the speech characteristics of the target speaker, effectively improving the speech conversion effect. Consequently, in intelligent dialogues involving insurance products, financial products, etc., the synthesized speech expressed by the chatbot can better match the dialogue style preferences of the dialogue audience. By adopting dialogue methods and styles that are more interesting to the dialogue audience, the quality and effectiveness of dialogue are improved, enabling intelligent voice dialogue services, improving customer service quality and satisfaction, and ultimately increasing the business conversion rate.

[0179] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0180] It will be understood by those skilled in the art that Figure 1-7 The technical solutions shown do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0182] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0183] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0184] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0185] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0186] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0187] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0188] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0189] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A model update method, characterized in that, The method includes: Obtain sample speech data from the speaker; The sample speech data is input into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The sample speech data is encoded using the encoding network to obtain an initial audio feature vector; Based on a preset codebook, the initial audio feature vector is indexed to obtain an audio frame index, and based on the audio frame index, phoneme features are extracted from the initial audio feature vector to obtain an initial phoneme feature vector. Speech alignment is performed on the initial phoneme feature vector to obtain the sample audio embedding vector; The decoding network decodes the sample audio embedding vector and the pre-acquired speech style embedding vector to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaker. The neural network model is updated with parameters based on the synthesized speech data and the sample speech data to obtain a speech conversion model. The step of performing speech alignment on the initial phoneme feature vector to obtain the sample audio embedding vector includes: The initial phoneme feature vector is segmented based on a preset time predictor to obtain multiple phoneme feature segments. The feature length of the initial phoneme feature vector is simulated and adjusted using the length adjuster in the time predictor. The time feature information in the initial phoneme feature vector is extracted using the multi-head attention layer and fully connected layer in the time predictor. Based on the importance of different time feature information, the phoneme category of the initial phoneme feature vector and the number of phonemes corresponding to each phoneme category are predicted. The phoneme category is used as the number of element types in the duration sequence, and the number of phonemes is used as the element value of each element to obtain the duration sequence of the initial phoneme feature vector. The initial phoneme feature vector is embedded to obtain an audio text embedding vector; the audio text embedding vector is segmented according to the duration sequence to obtain an intermediate vector for each phoneme category, wherein the number of intermediate vectors is equal to the number of phonemes; the mean of the intermediate vectors is calculated to obtain a candidate vector for each phoneme category; the candidate vectors are copied according to the number of phonemes to obtain a target vector for each phoneme category, wherein the number of target vectors is equal to the number of phonemes; all the target vectors are concatenated to obtain the sample audio embedding vector.

2. The model update method according to claim 1, characterized in that, The step of indexing the initial audio feature vector based on a preset codebook to obtain an audio frame index, and then extracting phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector, includes: The initial audio feature vector is segmented to obtain multiple audio frame vectors; Based on the reference vector of the preset codebook, the audio frame vector is indexed and queried to obtain the audio frame index corresponding to each audio frame vector. The preset codebook includes a one-to-one correspondence between the reference vector and the audio frame index, and a mapping relationship between each audio frame index and the phoneme feature. Based on the audio frame index and the mapping relationship, extract the phoneme features corresponding to the audio frame vector; The phoneme features of all the audio frame vectors are merged to obtain the initial phoneme feature vector.

3. The model update method according to any one of claims 1 to 2, characterized in that, Before decoding the sample audio embedding vector and the pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data, the model update method further includes obtaining the speech style embedding vector, specifically including: The sample speech data is input into a preset voiceprint recognition model, wherein the voiceprint recognition model includes a segmentation layer and a hidden layer; The sample speech data is segmented based on the segmentation layer to obtain multiple sample speech segments. Based on the hidden layer, style recognition is performed on each of the sample speech segments to obtain multiple initial style embedding vectors; The speaking style embedding vector is obtained by averaging the multiple initial style embedding vectors.

4. A speech conversion method, characterized in that, The method includes: Acquire the target speaking style information of the target speaker and the raw speech data to be processed; The original speech data and the target speaking style information are input into the speech conversion model for speech conversion to obtain the target speech data, wherein the speech conversion model is obtained by performing the model update method according to any one of claims 1 to 3.

5. A model update device, characterized in that, The model update device includes: The data acquisition module is used to acquire sample speech data of the sample speaking object; An input module is used to input the sample speech data into a preset neural network model, wherein the neural network model includes an encoding network and a decoding network; The encoding module is used to encode the sample speech data through the encoding network to obtain an initial audio feature vector; The query module is used to perform an index query on the initial audio feature vector based on a preset codebook to obtain an audio frame index, and to extract phoneme features from the initial audio feature vector based on the audio frame index to obtain an initial phoneme feature vector. The speech alignment module is used to perform speech alignment on the initial phoneme feature vector to obtain a sample audio embedding vector. The decoding module is used to decode the sample audio embedding vector and the pre-acquired speech style embedding vector through the decoding network to obtain synthesized speech data; wherein, the speech style embedding vector includes the speech style information of the sample speaking object; The model update module is used to update the parameters of the neural network model based on the synthesized speech data and the sample speech data to obtain a speech conversion model. The step of performing speech alignment on the initial phoneme feature vector to obtain the sample audio embedding vector includes: The initial phoneme feature vector is segmented based on a preset time predictor to obtain multiple phoneme feature segments. The feature length of the initial phoneme feature vector is simulated and adjusted using the length adjuster in the time predictor. The time feature information in the initial phoneme feature vector is extracted using the multi-head attention layer and fully connected layer in the time predictor. Based on the importance of different time feature information, the phoneme category of the initial phoneme feature vector and the number of phonemes corresponding to each phoneme category are predicted. The phoneme category is used as the number of element types in the duration sequence, and the number of phonemes is used as the element value of each element to obtain the duration sequence of the initial phoneme feature vector. The initial phoneme feature vector is embedded to obtain an audio text embedding vector; the audio text embedding vector is segmented according to the duration sequence to obtain an intermediate vector for each phoneme category, wherein the number of intermediate vectors is equal to the number of phonemes; the mean of the intermediate vectors is calculated to obtain a candidate vector for each phoneme category; the candidate vectors are copied according to the number of phonemes to obtain a target vector for each phoneme category, wherein the number of target vectors is equal to the number of phonemes; all the target vectors are concatenated to obtain the sample audio embedding vector.

6. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement: The model update method as described in any one of claims 1 to 3; or, The speech conversion method as described in claim 4.

7. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to achieve the following: The model update method as described in any one of claims 1 to 3; or, The speech conversion method as described in claim 4.

Images