Voice processing method and device based on artificial intelligence, computer equipment and medium

Abstract

The invention belongs to the technical field of artificial intelligence, and relates to a voice processing method and device based on artificial intelligence, computer equipment and a storage medium, and the method comprises the steps: carrying out the feature extraction of a to-be-processed voice through a content encoder, a rhythm encoder and a pitch encoder based on a language conversion model, and obtaining a content feature, a rhythm feature and a pitch feature; adjusting the content feature, the rhythm feature and the pitch feature based on a bottleneck adapter to obtain a specified content feature, a specified rhythm feature and a specified pitch feature; optimizing the specified content features based on a consistency module to obtain target content features; generating a target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector; processing the target feature vector based on a diffusion model to obtain a target spectrum feature; and converting the target spectrum features into synthetic speech based on a vocoder. According to the invention, the voice decoupling efficiency and precision are improved, and the sound quality of the synthesized voice is improved.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence development technology and financial technology, and in particular to artificial intelligence-based speech processing methods, devices, computer equipment and storage media. Background Art

[0002] In the field of speech processing technology, particularly in finance and other fields requiring a high degree of personalization and secure authentication, voice conversion technology has attracted considerable attention due to its ability to simulate the vocal characteristics of different speakers. The goal of voice conversion is to preserve the content (i.e., semantic information) of the original speech while adjusting its vocal characteristics, such as timbre, pitch, and rhythm, to suit different application scenarios or user preferences. In recent years, with the rise of unsupervised learning methods, researchers have begun exploring how to leverage these advanced technologies to achieve more refined and natural voice feature decoupling and conversion.

[0003] Existing methods for decoupling speech features include, but are not limited to, designing specific bottleneck features to force the model to learn decoupled representations, and utilizing signal processing techniques such as data augmentation and random resampling to simulate different pitch and rhythm variations. However, these methods often have limitations. Manually designed bottleneck features may not fully capture the complex relationship between pitch and rhythm, while signal processing-based methods may introduce additional noise or distortion, resulting in poor quality of the generated synthesized speech. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to propose a speech processing method, apparatus, computer equipment and storage medium based on artificial intelligence to solve the technical problem that the existing speech feature decoupling method cannot fully capture the complex relationship between pitch and rhythm, and easily leads to poor quality of the generated synthetic speech.

[0005] In order to solve the above technical problems, the present application provides an artificial intelligence-based speech processing method, which adopts the following technical solutions:

[0006] Get the voice to be processed;

[0007] Based on the content encoder, rhythm encoder and pitch encoder in the preset language conversion model, feature extraction processing is performed on the speech to be processed to obtain corresponding content features, rhythm features and pitch features;

[0008] Based on the bottleneck adapter in the language conversion model, the content feature, the rhythm feature and the pitch feature are adjusted and processed respectively to obtain corresponding specified content features, specified rhythm features and specified pitch features;

[0009] Optimizing the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features;

[0010] Get the preset speaker embedding vector;

[0011] generating a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector;

[0012] Processing the target feature vector based on the diffusion model in the language conversion model to obtain corresponding target spectrum features;

[0013] The target spectral features are converted into corresponding synthesized speech based on a preset vocoder.

[0014] Furthermore, the step of obtaining the speech to be processed specifically includes:

[0015] Get input voice;

[0016] Performing sampling rate uniform processing on the input speech to obtain a corresponding first speech;

[0017] Performing frame processing on the first speech to obtain a corresponding second speech;

[0018] performing windowing processing on the second speech to obtain a corresponding third speech;

[0019] performing information extraction processing on the third speech based on a preset extraction strategy to obtain a corresponding fourth speech;

[0020] The fourth speech is used as the speech to be processed.

[0021] Furthermore, the step of generating a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector specifically includes:

[0022] Get the preset feature splicing strategy;

[0023] Based on the feature splicing strategy, the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector are spliced ​​to obtain a corresponding spliced ​​feature vector;

[0024] The concatenated feature vector is used as the target feature vector.

[0025] Furthermore, before the step of performing feature extraction processing on the speech to be processed by the content encoder, rhythm encoder and pitch encoder in the preset language conversion model to obtain corresponding content features, rhythm features and pitch features, the method further includes:

[0026] Obtain a pre-collected raw speech dataset;

[0027] Performing enhancement processing on the original speech data set to obtain a corresponding target speech data set;

[0028] Generating sample data based on the original speech data set and the target speech data set;

[0029] Obtain a preset initial encoder; wherein the initial encoder includes an initial content encoder, a rhythm encoder and a pitch encoder

[0030] Constructing a target loss function corresponding to the initial encoder;

[0031] The initial encoder is pre-trained based on the target loss function and a preset gradient descent algorithm to obtain a corresponding pre-trained encoder; wherein the pre-trained encoder includes the content encoder, the rhythm encoder and the pitch encoder.

[0032] Furthermore, the step of constructing a target loss function corresponding to the initial encoder specifically includes:

[0033] Obtaining a preset ranking loss function and obtaining a first weight coefficient corresponding to the ranking loss function;

[0034] Obtaining a preset contrastive learning loss function and obtaining a second weight coefficient corresponding to the contrastive learning loss function;

[0035] Call the preset loss calculation formula;

[0036] Calculate the ranking loss function, the first weight coefficient, the contrastive learning loss function, and the second weight coefficient based on the loss function calculation formula to obtain a corresponding calculation result;

[0037] The calculation result is used as the target loss function.

[0038] Furthermore, before the step of optimizing the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features, the method further includes:

[0039] Get the preset text encoder;

[0040] Get the preset speech recognition encoder;

[0041] The consistency module is constructed based on the text encoder and the speech recognition encoder.

[0042] Furthermore, after the step of converting the target spectral features into corresponding synthesized speech based on a preset vocoder, the method further includes:

[0043] performing denoising processing on the synthesized speech to obtain a corresponding first synthesized speech;

[0044] performing volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech;

[0045] performing equalization processing on the second synthesized speech to obtain a corresponding target synthesized speech;

[0046] The target synthesized speech is stored.

[0047] In order to solve the above technical problems, the present application also provides an artificial intelligence-based speech processing device, which adopts the following technical solutions:

[0048] A first acquisition module, configured to acquire the speech to be processed;

[0049] An extraction module is used to perform feature extraction processing on the speech to be processed based on the content encoder, rhythm encoder and pitch encoder in the preset language conversion model, so as to obtain corresponding content features, rhythm features and pitch features;

[0050] An adjustment module, configured to adjust the content feature, the rhythm feature, and the pitch feature based on the bottleneck adapter in the language conversion model to obtain corresponding specified content features, specified rhythm features, and specified pitch features;

[0051] an optimization module, configured to optimize the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features;

[0052] The second acquisition module is used to obtain a preset speaker embedding vector;

[0053] A generating module, configured to generate a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector;

[0054] A first processing module is used to process the target feature vector based on the diffusion model in the language conversion model to obtain a corresponding target spectrum feature;

[0055] The conversion module is used to convert the target spectrum features into corresponding synthesized speech based on a preset vocoder.

[0056] In order to solve the above technical problems, the embodiment of the present application further provides a computer device, which adopts the following technical solution:

[0057] Get the voice to be processed;

[0058] Based on the content encoder, rhythm encoder and pitch encoder in the preset language conversion model, feature extraction processing is performed on the speech to be processed to obtain corresponding content features, rhythm features and pitch features;

[0059] Based on the bottleneck adapter in the language conversion model, the content feature, the rhythm feature and the pitch feature are adjusted and processed respectively to obtain corresponding specified content features, specified rhythm features and specified pitch features;

[0060] Optimizing the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features;

[0061] Get the preset speaker embedding vector;

[0062] generating a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector;

[0063] Processing the target feature vector based on the diffusion model in the language conversion model to obtain corresponding target spectrum features;

[0064] The target spectral features are converted into corresponding synthesized speech based on a preset vocoder.

[0065] In order to solve the above technical problems, the embodiment of the present application further provides a computer-readable storage medium, which adopts the following technical solution:

[0066] Get the voice to be processed;

[0067] Based on the content encoder, rhythm encoder and pitch encoder in the preset language conversion model, feature extraction processing is performed on the speech to be processed to obtain corresponding content features, rhythm features and pitch features;

[0068] Based on the bottleneck adapter in the language conversion model, the content feature, the rhythm feature and the pitch feature are adjusted and processed respectively to obtain corresponding specified content features, specified rhythm features and specified pitch features;

[0069] Optimizing the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features;

[0070] Get the preset speaker embedding vector;

[0071] generating a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector;

[0072] Processing the target feature vector based on the diffusion model in the language conversion model to obtain corresponding target spectrum features;

[0073] The target spectral features are converted into corresponding synthesized speech based on a preset vocoder.

[0074] Compared with the prior art, the embodiments of the present application have the following beneficial effects:

[0075] The present application first obtains the speech to be processed; then, based on the content encoder, rhythm encoder and pitch encoder in the preset language conversion model, feature extraction processing is performed on the speech to be processed respectively to obtain corresponding content features, rhythm features and pitch features; then, based on the bottleneck adapter in the language conversion model, the content features, rhythm features and pitch features are adjusted and processed respectively to obtain corresponding specified content features, specified rhythm features and specified pitch features; subsequently, the specified content features are optimized based on the consistency module in the language conversion model to obtain corresponding target content features; further, a preset speaker embedding vector is obtained; and a corresponding target feature vector is generated based on the target content features, the specified rhythm features, the specified pitch features and the speaker embedding vector; finally, the target feature vector is processed based on the diffusion model in the language conversion model to obtain corresponding target spectrum features; and the target spectrum features are converted into corresponding synthesized speech based on the preset vocoder. Based on the use of the content encoder, rhythm encoder and pitch encoder in the language conversion model, the present application can automatically decompose speech into content, rhythm and pitch features. This approach avoids the shortcomings of traditional methods, which require extensive manual feature design and bottleneck adjustment, effectively improving the efficiency and accuracy of speech decoupling. Furthermore, the bottleneck adapter, consistency module, diffusion model, and pre-defined vocoder in the speech conversion model enable high-quality decoupling of speech content, pitch, and rhythm, and precise reconstruction of target feature vectors into target spectral features, effectively improving the sound quality and timbre similarity of the converted synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the solutions in this application, a brief introduction will be given below to the drawings required for use in the description of the embodiments of this application. Obviously, the drawings described below are some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0077] Figure 1 is an exemplary system architecture diagram to which the present application may be applied;

[0078] Figure 2 A flowchart of an embodiment of an artificial intelligence-based speech processing method according to the present application;

[0079] Figure 3 is a structural diagram of an embodiment of an artificial intelligence-based speech processing device according to the present application;

[0080] Figure 4 It is a structural diagram of an embodiment of a computer device according to the present application. DETAILED DESCRIPTION

[0081] Unless otherwise defined, all technical and scientific terms used herein have the same meanings as commonly understood by those skilled in the art to which this application belongs. The terms used in the specification of the application are for the purpose of describing specific embodiments only and are not intended to limit this application. The terms "including" and "having" and any variations thereof in the specification and claims of this application and the above-mentioned drawings are intended to cover non-exclusive inclusions. The terms "first", "second", etc. in the specification and claims of this application or the above-mentioned drawings are used to distinguish different objects, not to describe a specific order.

[0082] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0083] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings.

[0084] like Figure 1 As shown, system architecture 100 may include a terminal device 101, a network 102, and a server 103. Terminal device 101 may be a laptop computer 1011, a tablet computer 1012, or a mobile phone 1013. Network 102 is a medium for providing a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0085] The user can use the terminal device 101 to interact with the server 103 via the network 102 to receive or send messages, etc. Various communication client applications can be installed on the terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0086] The terminal device 101 can be various electronic devices with a display screen and supporting web browsing. In addition to the laptop computer 1011, tablet computer 1012 or mobile phone 1013, the terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a laptop computer and a desktop computer, etc.

[0087] The server 103 may be a server that provides various services, such as a background server that provides support for web pages displayed on the terminal device 101 .

[0088] It should be noted that the artificial intelligence-based speech processing method provided in the embodiments of the present application is generally executed by a server / terminal device, and accordingly, the artificial intelligence-based speech processing device is generally set in the server / terminal device.

[0089] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0090] Continue to refer Figure 2 , shows a flow chart of an embodiment of the speech processing method based on artificial intelligence according to the present application. According to different needs, the order of the steps in the flow chart can be changed, and some steps can be omitted. The speech processing method based on artificial intelligence provided by the embodiment of the present application can be applied to any scenario that requires speech processing, and the speech processing method based on artificial intelligence can be applied to products in these scenarios, for example, speech processing in the financial and insurance fields. The speech processing method based on artificial intelligence comprises the following steps:

[0091] Step S201: Acquire the speech to be processed.

[0092] In this embodiment, the electronic device (e.g. Figure 1The server / terminal device shown in the figure) can obtain the voice to be processed through a wired connection or a wireless connection. It should be noted that the above-mentioned wireless connection method may include but is not limited to 3G / 4G / 5G connection, WiFi connection, Bluetooth connection, WiMAX connection, Zigbee connection, UWB (ultra wideband) connection, and other wireless connection methods currently known or to be developed in the future. The executor of this application may specifically be a speech processing system, or simply referred to as a system. The above-mentioned voice to be processed is a voice that needs to be processed by speech synthesis. In the business scenario of speech synthesis in finance and insurance, the above-mentioned voice to be processed may be a voice related to transaction data, payment data, business data, etc. Among them, the specific implementation process of obtaining the voice to be processed will be further described in detail in the subsequent specific embodiments of this application, and will not be elaborated on here.

[0093] Step S202 : Based on the content encoder, rhythm encoder and pitch encoder in the preset speech conversion model, feature extraction processing is performed on the speech to be processed to obtain corresponding content features, rhythm features and pitch features.

[0094] In this embodiment, the construction of the above-mentioned speech conversion model is divided into two stages: in stage 1, the content encoder, rhythm encoder and pitch encoder are first pre-trained based on self-supervised learning and sorting algorithms, and positive and negative sample pairs are constructed through data enhancement. The contrastive learning idea is applied to automatically decompose the speech into content features, rhythm features and pitch features; in stage 2, the frozen pre-trained encoder is used to extract pitch features, content features and rhythm features from the speech, and then the bottleneck adapter and diffusion model are trained under the guidance of the consistency module and mutual information learning, and finally the sampling generates high-quality and high-similarity audio.

[0095] Specifically, based on the mutual information and text-guided enhanced automatic speech representation decoupling technology, a two-stage VC model (i.e., language conversion model) was designed to achieve automatic speech representation decoupling based on self-supervised learning. This model achieves high-quality speech conversion with more fine-grained characteristics such as the target speaker's timbre, intonation, and rhythm, without the need for a large number of manually designed features and bottleneck adjustments: 1. A Bottleneck Adapter (BNA) trained with MI-based vCLUB was designed to separate speaker-irrelevant information and achieve further decoupling. 2. A consistency module was proposed, which, under the guidance of transcribed text and ASR-Bottleneck (ASR-BNF), alleviates the timbre leakage problem in content extraction and enhances content consistency before and after speech conversion. 3. A diffusion model was used to reconstruct the spectrum, and the speaker embedding vector was fused with the decoupled speech component through Conditionlayer Normalization (CLN) to achieve reconstruction of the fused information into the Mel spectrum, improving sound quality and timbre similarity.

[0096] Bottleneck adapters and mutual information-enhanced decoupling are implemented in the language conversion model: Because full fine-tuning can distort pre-trained features and lead to poor performance in the presence of large distribution shifts, the pre-trained encoders are frozen in the first stage to preserve the model's decoupling capabilities. A trainable BNA layer (bottleneck adapter) is specifically designed and added to each frozen encoder, which is then trained to achieve further decoupling. The BNA fine-tuned with MI eliminates information overlap between pitch, rhythm, and content, purifying speaker-irrelevant information. During training, backpropagated gradients only update the BNA layer, while retaining the pre-trained parameters in the frozen decoupled encoder. The loss function for fine-tuning the estimate adopts the mutual information loss function. The overlap of the decoupled information is removed by using the variational upper bound of MI (mutual information). By minimizing the variational upper bound, the correlation between different speech representations can be reduced, enhancing the automatic decoupling effect.

[0097] The content encoder is responsible for extracting speaker-independent content from speech. This can be achieved using structures such as CNNs or RNNs, possibly incorporating an attention mechanism to capture key content. The rhythm encoder is responsible for extracting rhythm information from speech. The pitch encoder is responsible for extracting pitch information from speech. This can be achieved using structures such as convolutional neural networks (CNNs) or recurrent neural networks (RNNs).

[0098] Step S203 : Based on the bottleneck adapter in the language conversion model, the content feature, the rhythm feature and the pitch feature are adjusted to obtain corresponding designated content features, designated rhythm features and designated pitch features.

[0099] In this embodiment, a bottleneck adapter is added to each of the content encoder, rhythm encoder, and pitch encoder in the speech conversion model. The bottleneck adapter adjusts the features of the encoder output to better match the feature space of the target speaker. By designing the bottleneck adapter based on mutual information (MI), it further decouples speaker-independent information.

[0100] Step S204 : Optimizing the designated content features based on the consistency module in the language conversion model to obtain corresponding target content features.

[0101] In this embodiment, the consistency module is a module that implements text-guided consistency learning. This module is used to address the timbre leakage problem in the content encoder. It includes two submodules: a text encoder with a length regulator and a shared ASR encoder trained using CTC loss to extract bottleneck features.

[0102] Step S205: Obtain a preset speaker embedding vector.

[0103] In this embodiment, the speaker embedding vector refers to the embedding vector of the target speaker. The embedding vector can be retrieved from an existing speaker database or generated in real time using a speaker encoder. A speaker encoder is typically a neural network that captures speaker identity information. The speaker embedding vector is a high-dimensional numerical representation used to capture and characterize the speech characteristics of a specific speaker. It embeds the identity information of different speakers into a continuous vector space, enabling the model to distinguish and simulate the speech styles of different speakers.

[0104] Step S206 : Generate a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector.

[0105] In this embodiment, the specific implementation process of generating the corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector will be further described in detail in subsequent specific embodiments of this application and will not be elaborated on here.

[0106] Step S207 : Process the target feature vector based on the diffusion model in the language conversion model to obtain corresponding target spectrum features.

[0107] In this embodiment, the diffusion model is pre-trained to accept content features, pitch features, rhythm features, and a target speaker embedding vector as input and generate a corresponding mel-spectrogram. By inputting the prepared target feature vector into the diffusion model, the model generates a mel-spectrogram through a stepwise denoising process. This process typically involves multiple time steps, with each time step reducing input noise and increasing spectral detail. At the end of the diffusion process, the diffusion model outputs a complete mel-spectrogram that incorporates the timbre characteristics of the target speaker as well as the content, pitch, and rhythm information of the input speech.

[0108] Step S208: Convert the target spectrum features into corresponding synthesized speech based on a preset vocoder.

[0109] In this embodiment, a suitable vocoder can be selected based on actual needs, such as the Griffin-Lim algorithm, WaveGlow, or HiFi-GAN. These vocoders can convert mel-spectrograms into high-quality audio waveforms. By inputting the generated target spectral features into the selected vocoder, the vocoder will output the corresponding audio waveform, thus obtaining the synthesized speech described above.

[0110] This application is based on the use of content encoders, rhythm encoders and pitch encoders in the language conversion model, which can automatically decompose speech into content, rhythm and pitch features. This approach avoids the shortcomings of traditional methods that require a large number of manually designed features and bottleneck adjustments, and effectively improves the efficiency and accuracy of speech decoupling. In addition, based on the use of bottleneck adapters, consistency modules, diffusion models and preset vocoders in the language conversion model, high-quality decoupling of speech content, pitch and rhythm can be achieved, and accurate reconstruction of target feature vectors to target spectral features can be achieved, effectively improving the sound quality and timbre similarity of the synthesized speech generated by conversion.

[0111] In some optional implementations, step S201 includes the following steps:

[0112] Get the input voice.

[0113] In this embodiment, the input speech is the initial speech that needs to be input and subjected to speech synthesis processing.

[0114] The input speech is subjected to sampling rate uniform processing to obtain a corresponding first speech.

[0115] In this embodiment, since voice data from different sources may have different sampling rates, all voice data is resampled to a unified sampling rate to maintain processing consistency. Specifically, the audio processing library can be used to read the input voice sampling rate and data. Then, the audio resampling function is used to resample the input voice to a unified target sampling rate.

[0116] The first speech is subjected to frame processing to obtain a corresponding second speech.

[0117] In this embodiment, since speech signals are time-varying, to analyze their characteristics over a short period of time, the continuous speech signal can be segmented into a series of shorter frames. First, the frame length and frame shift of the first speech signal are determined. Then, a sliding window technique is used to segment the speech signal into multiple frames. Specifically, a fixed-length window can be slid across the speech signal, each shifted by a certain step size (i.e., frame shift).

[0118] Perform windowing processing on the second speech to obtain a corresponding third speech.

[0119] In this embodiment, in order to reduce the discontinuity of the frame edge, a window function (such as a Hamming window, a Hanning window, etc.) can be further applied to each frame after framing. Specifically, the windowing process can be completed by performing element-wise multiplication of the window function and the frame data.

[0120] performing information extraction processing on the third speech based on a preset extraction strategy to obtain a corresponding fourth speech;

[0121] In this embodiment, the extraction strategy includes extracting features from the windowed frames according to the input format used during encoder pre-training. The information extraction refers to feature extraction, including MFCC (Mel Frequency Cepstral Coefficients) and Mel spectrum.

[0122] The fourth speech is used as the speech to be processed.

[0123] In this embodiment, in order to improve the training effect of the model, the extracted features may also be normalized (such as mean normalization, standardization, etc.).

[0124] This application performs unified sampling rate processing, frame processing, windowing processing and information extraction processing on the acquired input speech, so as to quickly and accurately complete the preprocessing of the input speech and generate the speech to be processed, thereby ensuring the data accuracy of the speech to be processed, which is conducive to ensuring the consistency and accuracy of subsequent speech processing steps.

[0125] In some optional implementations of this embodiment, step S206 includes the following steps:

[0126] Get the preset feature stitching strategy.

[0127] In this embodiment, the feature splicing strategy may include concatenation, or may also include more complex fusion methods, such as weighted sum, attention mechanism and other strategies.

[0128] Based on the feature splicing strategy, the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector are spliced ​​to obtain a corresponding spliced ​​feature vector.

[0129] In this embodiment, the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector may be spliced ​​according to the strategy content of the above-mentioned feature splicing strategy to obtain a corresponding spliced ​​feature vector.

[0130] The concatenated feature vector is used as the target feature vector.

[0131] This application obtains a preset feature splicing strategy; then, based on the feature splicing strategy, splices the target content features, the specified rhythm features, the specified pitch features, and the speaker embedding vector to obtain a corresponding spliced ​​feature vector; and subsequently uses the spliced ​​feature vector as the target feature vector. By using the feature splicing strategy, this application can quickly and accurately complete the splicing of the target content features, the specified rhythm features, the specified pitch features, and the speaker embedding vector to obtain the corresponding target feature vector, effectively improving the efficiency and intelligence of generating the target feature vector.

[0132] In some optional implementations, before step S202, the electronic device may further perform the following steps:

[0133] Get a pre-collected raw speech dataset.

[0134] In this embodiment, a raw speech dataset containing a variety of speakers and speech content is pre-collected. This raw speech dataset covers a wide range of speech features, enabling the model to learn comprehensive speech representations. Furthermore, the raw speech dataset is pre-processed into a format suitable for model input, such as fixed-length audio clips or feature sequences divided by frames.

[0135] The original speech data set is enhanced to obtain a corresponding target speech data set.

[0136] In this embodiment, the original speech dataset can be enhanced by utilizing data enhancement techniques, such as randomly changing pitch, time stretching (speeding up or slowing down speech), adding noise, echo, etc., to simulate various changes that may be encountered in actual speech conversion, thereby helping the model learn more robust representations and obtain the corresponding target speech dataset.

[0137] Sample data is generated based on the original speech data set and the target speech data set.

[0138] In this embodiment, integrated data may be obtained by integrating the original speech data set and the target speech data set, and used as the sample data.

[0139] Get the preset initial encoder.

[0140] In this embodiment, the initial encoder includes an initial content encoder, a rhythm encoder, and a pitch encoder.

[0141] Construct a target loss function corresponding to the initial encoder.

[0142] In this embodiment, the specific implementation process of constructing the target loss function corresponding to the initial encoder will be further described in detail in subsequent specific embodiments of the present application and will not be elaborated on here.

[0143] The initial encoder is pre-trained based on the target loss function and a preset gradient descent algorithm to obtain a corresponding pre-trained encoder; wherein the pre-trained encoder includes the content encoder, the rhythm encoder and the pitch encoder.

[0144] In this embodiment, the construction of the speech conversion model is divided into two stages: in stage 1, the content encoder, rhythm encoder and pitch encoder are first pre-trained based on self-supervised learning and sorting algorithms, and positive and negative sample pairs are constructed through data enhancement. The contrastive learning idea is applied to automatically decompose the speech into content features, rhythm features and pitch features; in stage 2, the frozen pre-trained encoder is used to extract pitch features, content features and rhythm features from the speech, and then the bottleneck adapter and diffusion model are trained under the guidance of the consistency module and mutual information learning, and finally the sampling generates high-quality and high-similarity audio.

[0145] Specifically, during the pre-training of the initial encoder, the parameters of the encoder are optimized by using a gradient descent algorithm (such as SGD, Adam, etc.) to minimize the target loss function, thereby achieving pre-training of the encoder to obtain the corresponding pre-trained encoder, thereby laying a solid foundation for subsequent speech conversion tasks.

[0146] The present application obtains a target speech dataset by performing enhancement processing on a pre-collected original speech dataset, generates sample data based on the original speech dataset and the target speech dataset, and then constructs a target loss function corresponding to the initial encoder. The initial encoder is then pre-trained based on the target loss function and a preset gradient descent algorithm, thereby achieving a fast and intelligent construction of the required pre-trained encoder, improving the construction efficiency of the encoder and ensuring the processing effect of the obtained encoder.

[0147] In some optional implementations, constructing a target loss function corresponding to the initial encoder includes the following steps:

[0148] Obtain a preset ranking loss function, and obtain a first weight coefficient corresponding to the ranking loss function.

[0149] In this embodiment, in order to extract content features, rhythm features and pitch features (representation) from speech, three encoders (pitch encoder, content encoder, rhythm encoder) are referenced in the speech conversion model of SpeechFlow to extract the pitch features, content features and rhythm features of speech respectively. Pitch shifting and time stretching are applied to modify the pitch and rhythm of speech. By converting speech data y and its enhanced version speech data y aug is fed into the content encoder and rhythm encoder, while the pitch contour is fed into the pitch encoder. Through these three encoders, the representation is obtained from y, the content feature Z c , high-pitched characteristic Z p and rhythmic characteristics Z r , from y aug Characterized in and Then through the Rank layer, Z p and Z r Mapped to two independent scores P and R. At the same time, the content feature Z c The data is averaged into a vector C for further operation. The hyperparameter γ∈(0,1) is subsequently used to indicate the strength of data enhancement. γ<0.5 indicates negative enhancement, such as lowering pitch or rhythm, while γ>0.5 indicates positive enhancement, such as increasing pitch or rhythm. γ=0.5 means no enhancement is applied. In order for the encoder to produce a decoupled representation by identifying this enhancement strength, a sigmoid function is first applied to the pitch and rhythm scores: Then for S r and S p Calculate the ranking loss of the two. The ranking loss function includes a first ranking loss function corresponding to the rhythm feature and a second ranking loss function corresponding to the pitch feature. The first ranking loss function L r With the second ranking loss function Lp They include: Among them, the above-mentioned first weight coefficient includes a first specified weight coefficient and a second execution weight coefficient corresponding to the first ranking loss function and the second ranking loss function respectively, and the value of the first weight coefficient can be set according to actual processing requirements.

[0150] Obtain a preset contrastive learning loss function, and obtain a second weight coefficient corresponding to the contrastive learning loss function.

[0151] In this embodiment, to ensure that the content encoder only generates linguistic representations, C and C aug (Average vector of enhanced version of speech data) Apply contrastive learning loss function. Contrastive learning loss function L NCE include: sim() represents the calculation of cosine similarity. The value of the second weight coefficient can be set according to actual processing requirements.

[0152] Call the preset loss calculation formula.

[0153] In this embodiment, the above loss calculation formula may specifically adopt a weighted sum formula.

[0154] The ranking loss function, the first weight coefficient, the contrastive learning loss function and the second weight coefficient are calculated based on the loss function calculation formula to obtain corresponding calculation results.

[0155] In this embodiment, the ranking loss function, the first weight coefficient, the contrastive learning loss function and the second weight coefficient can be substituted into the above-mentioned loss function calculation formula for calculation processing to obtain the corresponding calculation result.

[0156] The calculation result is used as the target loss function.

[0157] After obtaining a preset ranking loss function, a first weight coefficient corresponding to the ranking loss function, and a preset contrastive learning loss function and a second weight coefficient corresponding to the contrastive learning loss function, the present application calculates and processes the ranking loss function, the first weight coefficient, the contrastive learning loss function, and the second weight coefficient based on the use of a loss calculation formula, thereby quickly and accurately generating a target loss function and ensuring the accuracy of the obtained target loss function. This is conducive to automatically and accurately completing the encoder pre-training process based on the obtained target loss function.

[0158] In some optional implementations of this embodiment, before step S204, the electronic device may further perform the following steps:

[0159] Get the preset text encoder.

[0160] In this embodiment, the consistency module is a module that implements text-guided consistency learning. This consistency module is used to address timbre leakage in the content encoder and comprises two submodules: (I) a text encoder with a length regulator and (II) a shared ASR encoder trained using a CTC loss for bottleneck feature extraction. Guided by the transcript and ASR-Bottleneck (ASR-BNF), the consistency module mitigates timbre leakage in content extraction and enhances content consistency before and after speech conversion.

[0161] The above-mentioned text encoder is a text encoder with a length regulator, which uses the text encoder to generate text embeddings from phonemes to guide the content encoder to generate speaker-independent content embedding vectors. The text encoder and length regulator have the same structure as those used in FastSpeech2

[20] . The alignment duration information is extracted by the MFA tool, and then the content consistency loss is calculated. Under the guidance of the transcribed text, the content consistency loss is used to encourage the content encoder to obtain speaker-independent content embedding vectors.

[0162] Get the preset speech recognition encoder.

[0163] In this embodiment, the speech recognition encoder specifically refers to a shared ASR encoder trained using CTC loss.

[0164] The consistency module is constructed based on the text encoder and the speech recognition encoder.

[0165] In this embodiment, the text encoder and the speech recognition encoder are combined to construct a corresponding consistency module.

[0166] The present application obtains a preset text encoder and a preset speech recognition encoder, and subsequently constructs the consistency module based on the text encoder and the speech recognition encoder. Based on the obtained text encoder and speech recognition encoder, the present application can quickly and intelligently construct the required consistency module, so that the subsequent use of the consistency module can effectively alleviate the timbre leakage problem in the content extraction of the content encoder and enhance the content consistency before and after speech conversion.

[0167] In some optional implementations of this embodiment, after step S208, the electronic device may further perform the following steps:

[0168] The synthesized speech is subjected to denoising processing to obtain a corresponding first synthesized speech.

[0169] In this embodiment, the denoising process is a process of removing unnecessary background noise or interfering sound from speech. Specifically, the spectral characteristics of the noise in the synthesized speech can be analyzed and the estimated noise spectrum can be subtracted from the spectrum of the noisy speech.

[0170] Volume adjustment processing is performed on the first synthesized speech to obtain a corresponding second synthesized speech.

[0171] In this embodiment, volume adjustment refers to the process of changing the overall loudness of a speech signal. Specifically, the amplitude of the speech signal can be scaled to increase or decrease its volume. This can be achieved by multiplying the audio signal by a constant (gain factor). Alternatively, automatic gain control (AGC) can be used to dynamically adjust the gain of the audio signal to maintain a consistent volume level.

[0172] The second synthesized speech is equalized to obtain a corresponding target synthesized speech.

[0173] In this embodiment, equalization is the process of adjusting the relative strengths of different frequency components in a speech signal to improve its auditory effect. Specifically, a series of bandpass filters with adjustable gain are used, each corresponding to a frequency band in the audio spectrum. By adjusting the gain of each filter, the loudness of the audio signal at different frequencies can be varied. Furthermore, precise control over specific frequencies and their surrounding frequencies is provided, allowing users to create more complex frequency response curves.

[0174] The target synthesized speech is stored.

[0175] In this embodiment, there is no specific limitation on the storage method of the target synthesized speech. For example, blockchain storage, local database storage, cloud server storage, etc. may be used. In addition, before storing the target synthesized speech, dynamic range compression processing may be further performed on the target synthesized speech to effectively reduce the difference between the highest and lowest levels in the target synthesized speech, making it sound smoother and more consistent, thereby improving the auditory quality of the target synthesized speech.

[0176] This application performs denoising on the synthesized speech to obtain a corresponding first synthesized speech; then performs volume adjustment on the first synthesized speech to obtain a corresponding second synthesized speech; then performs equalization on the second synthesized speech to obtain a corresponding target synthesized speech; and then stores the target synthesized speech. After converting the target spectral features into the corresponding synthesized speech based on a vocoder, this application further intelligently performs denoising, volume adjustment, and equalization on the synthesized speech, thereby effectively improving the speech quality of the synthesized speech. The generated target synthesized speech is then stored to ensure the data security of the target synthesized speech.

[0177] In addition, any software tools or components not provided by our company that appear in the embodiments of this application are merely examples and do not represent actual use.

[0178] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0179] It should be emphasized that in order to further ensure the privacy and security of the above-mentioned synthesized speech, the above-mentioned synthesized speech can also be stored in a node of a blockchain.

[0180] The blockchain referred to in this application is a new application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Blockchain is essentially a decentralized database, a series of data blocks generated using cryptographic methods. Each data block contains information about a batch of network transactions, which is used to verify the validity of this information (to prevent counterfeiting) and generate the next block. Blockchain can include the underlying blockchain platform, the platform product service layer, and the application service layer.

[0181] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use knowledge to achieve optimal results.

[0182] Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0183] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by instructing related hardware via computer-readable instructions. The computer-readable instructions can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes in the above-described method embodiments. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0184] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0185] Further references Figure 3 , as a response to the above Figure 2 The present application provides an embodiment of a speech processing device based on artificial intelligence, which is similar to Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0186] like Figure 3 As shown, the artificial intelligence-based speech processing device 300 of this embodiment includes: a first acquisition module 301, an extraction module 302, an adjustment module 303, an optimization module 304, a second acquisition module 305, a generation module 306, a first processing module 307, and a conversion module 308. Among them:

[0187] A first acquisition module 301 is used to acquire the speech to be processed;

[0188] Extraction module 302, configured to perform feature extraction processing on the speech to be processed based on the content encoder, rhythm encoder and pitch encoder in the preset speech conversion model, respectively, to obtain corresponding content features, rhythm features and pitch features;

[0189] An adjustment module 303 is configured to adjust the content feature, the rhythm feature, and the pitch feature based on the bottleneck adapter in the language conversion model to obtain corresponding specified content features, specified rhythm features, and specified pitch features;

[0190] An optimization module 304 is configured to optimize the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features;

[0191] A second acquisition module 305 is used to obtain a preset speaker embedding vector;

[0192] A generating module 306 is configured to generate a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector;

[0193] A first processing module 307 is configured to process the target feature vector based on the diffusion model in the language conversion model to obtain a corresponding target spectrum feature;

[0194] The conversion module 308 is configured to convert the target spectral features into corresponding synthesized speech based on a preset vocoder.

[0195] In some optional implementations of this embodiment, the first obtaining module 301 includes:

[0196] A first acquisition submodule is used to acquire input speech;

[0197] A first processing submodule, configured to perform sampling rate uniform processing on the input speech to obtain a corresponding first speech;

[0198] A second processing submodule, configured to perform frame processing on the first speech to obtain a corresponding second speech;

[0199] A third processing submodule, configured to perform windowing processing on the second speech to obtain a corresponding third speech;

[0200] a fourth processing submodule, configured to perform information extraction processing on the third speech based on a preset extraction strategy to obtain a corresponding fourth speech;

[0201] The first determining submodule is configured to use the fourth speech as the speech to be processed.

[0202] In some optional implementations of this embodiment, the generation module 306 includes:

[0203] The second acquisition submodule is used to obtain a preset feature splicing strategy;

[0204] a splicing submodule, configured to perform splicing processing on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector based on the feature splicing strategy to obtain a corresponding spliced ​​feature vector;

[0205] The second determining submodule is configured to use the concatenated feature vector as the target feature vector.

[0206] In some optional implementations of this embodiment, the artificial intelligence-based speech processing device further includes:

[0207] The third acquisition module is used to obtain the pre-collected original speech data set;

[0208] An enhancement module, configured to perform enhancement processing on the original speech data set to obtain a corresponding target speech data set;

[0209] A first building module is configured to generate sample data based on the original speech data set and the target speech data set;

[0210] A fourth acquisition module is configured to acquire a preset initial encoder, wherein the initial encoder includes an initial content encoder, a rhythm encoder, and a pitch encoder;

[0211] A second building module is used to build a target loss function corresponding to the initial encoder;

[0212] A pre-training module is used to pre-train the initial encoder based on the target loss function and a preset gradient descent algorithm to obtain a corresponding pre-trained encoder; wherein the pre-trained encoder includes the content encoder, the rhythm encoder and the pitch encoder.

[0213] In some optional implementations of this embodiment, the second building block includes:

[0214] A third acquisition submodule is used to obtain a preset ranking loss function and a first weight coefficient corresponding to the ranking loss function;

[0215] A fourth acquisition submodule, configured to acquire a preset contrastive learning loss function and a second weight coefficient corresponding to the contrastive learning loss function;

[0216] Call submodule, used to call the preset loss calculation formula;

[0217] a calculation submodule, configured to calculate the ranking loss function, the first weight coefficient, the contrastive learning loss function, and the second weight coefficient based on the loss function calculation formula to obtain a corresponding calculation result;

[0218] The third determination submodule is used to use the calculation result as the target loss function.

[0219] In some optional implementations of this embodiment, the artificial intelligence-based speech processing device further includes:

[0220] A fifth acquisition module, configured to acquire a preset text encoder;

[0221] A sixth acquisition module, configured to acquire a preset speech recognition encoder;

[0222] A third construction module is used to construct the consistency module based on the text encoder and the speech recognition encoder.

[0223] In some optional implementations of this embodiment, the artificial intelligence-based speech processing device further includes:

[0224] A second processing module is used to perform denoising on the synthesized speech to obtain a corresponding first synthesized speech;

[0225] a third processing module, configured to perform volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech;

[0226] a fourth processing module, configured to perform equalization processing on the second synthesized speech to obtain a corresponding target synthesized speech;

[0227] The storage module is used to store the target synthesized speech.

[0228] To solve the above technical problems, the present application also provides a computer device. Figure 4 , Figure 4 This is a basic structural block diagram of the computer device in this embodiment.

[0229] The computer device 4 includes a memory 41, a processor 42, and a network interface 43 that are interconnected through a system bus. It should be noted that the figure only shows a computer device 4 with components 41-43, but it should be understood that it is not required to implement all the components shown, and more or fewer components can be implemented instead. Among them, those skilled in the art can understand that the computer device here is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to microprocessors, application-specific integrated circuits (ASICs), programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0230] The computer device may be a desktop computer, notebook computer, PDA, cloud server, etc. The computer device may interact with the user via a keyboard, mouse, remote control, touchpad, or voice control device.

[0231] The memory 41 includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 41 can be an internal storage unit of the computer device 4, such as the hard disk or memory of the computer device 4. In other embodiments, the memory 41 can also be an external storage device of the computer device 4, such as a plug-in hard disk equipped on the computer device 4, a smart memory card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory 41 can also include both the internal storage unit of the computer device 4 and its external storage device. In this embodiment, the memory 41 is generally used to store the operating system and various application software installed on the computer device 4, such as computer-readable instructions for an artificial intelligence-based speech processing method. In addition, the memory 41 can also be used to temporarily store various types of data that have been output or are to be output.

[0232] In some embodiments, the processor 42 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor 42 is generally used to control the overall operation of the computer device 4. In this embodiment, the processor 42 is used to execute computer-readable instructions or process data stored in the memory 41, such as computer-readable instructions for executing the artificial intelligence-based speech processing method.

[0233] The network interface 43 may include a wireless network interface or a wired network interface. The network interface 43 is generally used to establish a communication connection between the computer device 4 and other electronic devices.

[0234] The present application also provides another embodiment, namely, providing a computer-readable storage medium, which stores computer-readable instructions, and the computer-readable instructions can be executed by at least one processor to enable the at least one processor to perform the steps of the above-mentioned artificial intelligence-based speech processing method.

[0235] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present application.

[0236] Obviously, the embodiments described above are only some of the embodiments of the present application, rather than all of the embodiments. The preferred embodiments of the present application are given in the accompanying drawings, but they do not limit the patent scope of the present application. The present application can be implemented in many different forms. On the contrary, the purpose of providing these embodiments is to make the understanding of the disclosure of the present application more thorough and comprehensive. Although the present application has been described in detail with reference to the aforementioned embodiments, for those skilled in the art, it is still possible to modify the technical solutions described in the aforementioned specific embodiments, or to make equivalent replacements for some of the technical features therein. Any equivalent structure made using the contents of the present application specification and the accompanying drawings, directly or indirectly used in other related technical fields, is also within the scope of patent protection of the present application.

Claims

1. A speech processing method based on artificial intelligence, characterized in that: The steps include: Get the voice to be processed; Based on the content encoder, the rhythm encoder and the pitch encoder in the preset language conversion model, feature extraction processing is performed on the speech to be processed respectively to obtain corresponding content features, rhythm features and pitch features; Based on the bottleneck adapter in the language conversion model, the content feature, the rhythm feature and the pitch feature are adjusted and processed respectively to obtain corresponding specified content features, specified rhythm features and specified pitch features; Optimizing the specified content features based on the consistency module in the language conversion model to obtain corresponding target content features; Get the preset speaker embedding vector; Generate a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature, and the speaker embedding vector; Processing the target feature vector based on the diffusion model in the language conversion model to obtain corresponding target spectrum features; The target spectral features are converted into corresponding synthesized speech based on a preset vocoder.

2. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of obtaining the speech to be processed specifically includes: Get input voice; Performing uniform sampling rate processing on the input speech to obtain a corresponding first speech; Performing frame processing on the first speech to obtain a corresponding second speech; Performing windowing processing on the second speech to obtain a corresponding third speech; Performing information extraction processing on the third speech based on a preset extraction strategy to obtain a corresponding fourth speech; The fourth speech is used as the speech to be processed.

3. The speech processing method based on artificial intelligence according to claim 1, characterized in that: The step of generating a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector specifically includes: Get the preset feature splicing strategy; Based on the feature splicing strategy, the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector are spliced ​​to obtain a corresponding spliced ​​feature vector; The concatenated feature vector is used as the target feature vector.

4. The speech processing method based on artificial intelligence according to claim 1, characterized in that: Before the step of respectively performing feature extraction processing on the speech to be processed by the content encoder, the rhythm encoder and the pitch encoder in the preset language conversion model to obtain corresponding content features, rhythm features and pitch features, the method further includes: Obtain a pre-collected raw speech data set; Performing enhancement processing on the original speech data set to obtain a corresponding target speech data set; Generate sample data based on the original speech data set and the target speech data set; Obtaining a preset initial encoder; wherein the initial encoder includes an initial content encoder, a rhythm encoder and a pitch encoder; Constructing a target loss function corresponding to the initial encoder; The initial encoder is pre-trained based on the target loss function and a preset gradient descent algorithm to obtain a corresponding pre-trained encoder; wherein the pre-trained encoder includes the content encoder, the rhythm encoder and the pitch encoder.

5. The artificial intelligence-based speech processing method according to claim 4, characterized in that: The step of constructing a target loss function corresponding to the initial encoder specifically includes: Obtaining a preset ranking loss function, and obtaining a first weight coefficient corresponding to the ranking loss function; Obtaining a preset contrastive learning loss function, and obtaining a second weight coefficient corresponding to the contrastive learning loss function; Call the preset loss calculation formula; Based on the loss function calculation formula, the sorting loss function, the first weight coefficient, the contrastive learning loss function and the second weight coefficient are calculated to obtain a corresponding calculation result; The calculation result is used as the target loss function.

6. The speech processing method based on artificial intelligence according to claim 1, characterized in that: Before the step of optimizing the specified content feature based on the consistency module in the language conversion model to obtain the corresponding target content feature, the method further includes: Get the preset text encoder; Get the preset speech recognition encoder; The consistency module is constructed based on the text encoder and the speech recognition encoder.

7. The speech processing method based on artificial intelligence according to claim 1, characterized in that: After the step of converting the target spectral features into corresponding synthesized speech based on a preset vocoder, the method further includes: Performing denoising processing on the synthesized speech to obtain a corresponding first synthesized speech; Performing volume adjustment processing on the first synthesized speech to obtain a corresponding second synthesized speech; performing equalization processing on the second synthesized speech to obtain a corresponding target synthesized speech; The target synthesized speech is stored.

8. A speech processing device based on artificial intelligence, characterized in that: include: A first acquisition module, used to acquire the speech to be processed; An extraction module is used to perform feature extraction processing on the speech to be processed based on a content encoder, a rhythm encoder and a pitch encoder in a preset language conversion model, respectively, to obtain corresponding content features, rhythm features and pitch features; An adjustment module, configured to adjust the content feature, the rhythm feature and the pitch feature respectively based on the bottleneck adapter in the language conversion model to obtain corresponding specified content feature, specified rhythm feature and specified pitch feature; An optimization module, configured to optimize the specified content feature based on the consistency module in the language conversion model to obtain a corresponding target content feature; A second acquisition module is used to obtain a preset speaker embedding vector; A generating module, configured to generate a corresponding target feature vector based on the target content feature, the specified rhythm feature, the specified pitch feature and the speaker embedding vector; A first processing module, configured to process the target feature vector based on the diffusion model in the language conversion model to obtain a corresponding target frequency spectrum feature; The conversion module is used to convert the target spectrum feature into corresponding synthesized speech based on a preset vocoder.

9. A computer device, characterized in that: It comprises a memory and a processor, wherein the memory stores computer-readable instructions, and when the processor executes the computer-readable instructions, the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by the processor, the steps of the artificial intelligence-based speech processing method as described in any one of claims 1 to 7 are implemented.

Images