Speech reconstruction method and device based on real-time training, computer device and medium

Abstract

The application relates to the technical field of artificial intelligence, in particular to a speech reconstruction method and device based on real-time training, computer equipment and a medium. The method divides training speech into a first speech segment and a second speech segment, respectively inputs the first speech segment and the second speech segment into a feature encoder to obtain first speech features and second speech features, inputs the training speech into a filter encoder to obtain filter features, inputs the mean of the first speech features and the second speech features and the filter features into a decoder to obtain reconstructed speech, trains a speech reconstruction model according to the first speech features, the second speech features, the reconstructed speech and the training speech, reconstructs to-be-processed speech based on the trained speech reconstruction model, improves the accuracy of feature decoding, and further improves the accuracy of speech reconstruction, can improve the simulation of machine customer service speech under a financial service platform, and further improves the user experience of users in the financial service platform.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a speech reconstruction method, apparatus, computer equipment, and medium based on real-time training. Background Technology

[0002] Currently, with the rapid development of artificial intelligence technology, voice reconstruction technology has been widely applied in financial service platforms. These platforms can be insurance systems, banking systems, transaction systems, order systems, etc. They can support functions such as shopping, social networking, interactive games, and resource transfer, and can also have functions such as applying for loans, credit cards, or purchasing insurance and wealth management products.

[0003] Voice reconstruction technology can empower advertising and customer service services of financial service platforms, such as realizing various application functions like advertising video dubbing and customer service voice synthesis. By obtaining reconstructed speech that matches the target speaker's timbre through voice reconstruction technology, the human resources consumed by the target speaker for repeated recording can be greatly reduced.

[0004] However, existing speech reconstruction models usually extract features directly from the speech to be reconstructed. The extracted features contain both semantic information and speaker timbre information. These two types of features interfere with each other during feature decoding, resulting in low accuracy of speech reconstruction. Therefore, improving the accuracy of speech reconstruction has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a speech reconstruction method, apparatus, computer device and medium based on real-time training to solve the problem of low accuracy in speech reconstruction.

[0006] In a first aspect, embodiments of the present invention provide a speech reconstruction method based on real-time training, the speech reconstruction method comprising:

[0007] The acquired training speech that meets the conditions to be reconstructed is segmented to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into the feature encoder for feature extraction to obtain the first speech feature and the second speech feature.

[0008] The trained speech input filter encoder is subjected to feature extraction to obtain filter features. The mean of the first speech feature and the second speech feature is calculated. The calculation result and the filter features are input into the decoder to obtain the reconstructed speech.

[0009] The metric loss is calculated based on the first speech feature, the second speech feature, and a preset metric loss function; the reconstruction loss is calculated based on the reconstructed speech, the training speech, and a preset reconstruction loss function.

[0010] The speech reconstruction model composed of the feature encoder, the filter encoder, and the decoder is trained based on the metric loss and the reconstruction loss to obtain a trained speech reconstruction model. The speech to be processed is then reconstructed based on the trained speech reconstruction model.

[0011] Secondly, embodiments of the present invention provide a speech reconstruction device based on real-time training, the speech reconstruction device comprising:

[0012] The speech segmentation module is used to segment the acquired training speech that meets the conditions to be reconstructed to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into the feature encoder for feature extraction to obtain the first speech feature and the second speech feature.

[0013] The speech reconstruction module is used to extract features from the trained speech input filter encoder to obtain filtered features, calculate the mean of the first speech feature and the second speech feature, and input the calculation result and the filtered features into the decoder to obtain the reconstructed speech;

[0014] The loss calculation module is used to calculate the metric loss based on the first speech feature, the second speech feature and a preset metric loss function, and to calculate the reconstruction loss based on the reconstructed speech, the training speech and a preset reconstruction loss function.

[0015] The speech reconstruction module is used to train the speech reconstruction model composed of the feature encoder, the filter encoder and the decoder according to the metric loss and the reconstruction loss, to obtain the trained speech reconstruction model, and to reconstruct the speech to be processed based on the trained speech reconstruction model.

[0016] Thirdly, embodiments of the present invention provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech reconstruction method as described in the first aspect.

[0017] Fourthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the speech reconstruction method as described in the first aspect.

[0018] The beneficial effects of the embodiments of the present invention compared with the prior art are as follows:

[0019] The acquired training speech that meets the conditions for reconstruction is segmented into a first speech segment and a second speech segment. The first and second speech segments are then input into a feature encoder for feature extraction, yielding first and second speech features. The training speech is then input into a filter encoder for feature extraction, yielding filtered features. The mean of the first and second speech features is calculated. The calculated results and the filtered features are input into a decoder to obtain the reconstructed speech. Based on the first and second speech features and a preset metric loss function, a metric loss is calculated. Based on the reconstructed speech, the training speech, and a preset reconstruction loss function, a reconstruction loss is calculated. Based on the metric loss and the reconstruction loss, the features are then processed... A speech reconstruction model consisting of a feature encoder, a filter encoder, and a decoder is trained to obtain a trained speech reconstruction model. Based on the trained speech reconstruction model, the speech to be processed is reconstructed. The speech segmentation results are trained through metric learning, enabling the feature encoder to accurately extract speech features representing speaker timbre information. Combined with the semantic features extracted by the filter encoder, speech reconstruction is achieved. In the speech reconstruction process, speech features and filter features are completely decoupled, thereby improving the accuracy of the feature decoding process and thus improving the accuracy of speech reconstruction. This can improve the simulation of machine customer service voice in financial service platforms, thereby improving the user experience in financial service platforms. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram of an application environment for a speech reconstruction method based on real-time training provided in Embodiment 1 of the present invention;

[0022] Figure 2 This is a flowchart illustrating a speech reconstruction method based on real-time training provided in Embodiment 1 of the present invention.

[0023] Figure 3 This is a flowchart illustrating a speech reconstruction method based on real-time training provided in Embodiment 2 of the present invention.

[0024] Figure 4 This is a schematic diagram of the structure of a speech reconstruction device based on real-time training provided in Embodiment 3 of the present invention;

[0025] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Detailed Implementation

[0026] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.

[0027] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0028] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0029] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0030] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0031] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0032] The embodiments of this invention can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0033] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0034] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0035] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0036] The speech reconstruction method based on real-time training provided in Embodiment 1 of this invention can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. Clients include, but are not limited to, handheld computers, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server can be a standalone server or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0037] The client and server can be deployed in a financial service platform, which can be an insurance system, a banking system, a transaction system, an order system, etc. The financial service platform can support functions such as shopping, social networking, interactive games, and resource transfer. It can also have functions such as applying for loans, credit cards, or purchasing insurance and wealth management products. The voice reconstruction task can provide synthesized voice for applications such as advertising video dubbing and customer service voice synthesis in the financial service platform, thereby improving the user experience on the financial service platform.

[0038] See Figure 2 This is a flowchart illustrating a speech reconstruction method based on real-time training provided in Embodiment 1 of the present invention. The above-described speech reconstruction method can be applied to... Figure 1 The client-side computer connects to the server to obtain training speech that meets the reconstruction criteria. The client-side computer deploys a speech reconstruction model, which may include a feature encoder, a filter encoder, and a decoder. The training speech that meets the reconstruction criteria can be used as training samples for real-time training of the speech reconstruction model. The feature encoder extracts speech features representing the speaker's timbre, the filter encoder extracts filtered features representing semantic information, and the decoder reconstructs the reconstructed speech from these features. Figure 2 As shown, the speech reconstruction method may include the following steps:

[0039] Step S201: The acquired training speech that meets the conditions for reconstruction is segmented to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into the feature encoder for feature extraction to obtain the first speech feature and the second speech feature.

[0040] Among them, the condition to be reconstructed can refer to the training speech being valid speech. Valid speech can refer to speech containing at least one complete sentence and the text information of the complete sentence is known. Training speech can refer to speech information used as training samples. In the context of financial services, training speech can refer to the speech of human customer service.

[0041] Segmentation can refer to dividing a complete training speech into multiple sub-speech segments. In this embodiment, the training speech is divided into two sub-speech segments, namely the first speech segment and the second speech segment. Each sub-speech segment contains some speech information. The feature encoder can be used to extract the speaker timbre features of the speech segments. The speaker timbre features can refer to the timbre information of the speaker corresponding to the training speech. In this embodiment, the speaker timbre features are the first speech feature and the second speech feature.

[0042] Specifically, in this embodiment, the training speech has been preprocessed. Preprocessing can refer to performing a Fourier transform on the speech signal corresponding to the training speech to obtain a spectrum vector as the training input vector of the training speech. The training input vector is also the input of the feature encoder.

[0043] The feature encoder may include convolutional layers to convolve the training input vectors and obtain the convolution result as speech features. The speech features corresponding to the first speech segment are called the first speech features, and the speech features corresponding to the second speech segment are called the second speech features.

[0044] Optionally, the acquired training speech that meets the conditions for reconstruction is segmented to obtain a first speech segment and a second speech segment, including:

[0045] Obtain the speech length of the training speech, multiply the speech length by a preset segmentation coefficient, and determine the segmentation position by the result of the multiplication;

[0046] The training speech is segmented according to the segmentation position to obtain the first speech segment and the second speech segment.

[0047] Among them, speech length can refer to the length of the phoneme sequence corresponding to the training speech. The phoneme sequence can contain several phonemes. A phoneme can refer to the basic unit of speech. The segmentation coefficient can be used to determine the position of the segmentation point, that is, the segmentation position.

[0048] Specifically, let the length of the phoneme sequence be len, and correspondingly, the number of phonemes contained in the phoneme sequence is also len. In this embodiment, the preset segmentation coefficient can be set to 0.5, so the above multiplication result is 0.5len. It should be noted that, in order to avoid the multiplication result being non-integer, [0.5len] is used as the multiplication result here, that is, 0.5len is rounded down. After obtaining the segmentation position, the subsequence from the starting phoneme to the phoneme at the segmentation position is taken as the first speech segment, and the subsequence from the phoneme one position after the segmentation position to the ending phoneme is taken as the second speech segment.

[0049] It should be noted that the preprocessing step can be performed after the segmentation operation is completed, and the size of the sub-segment vectors obtained after preprocessing is normalized. Size normalization can be achieved by zero-padding, so that the size of each sub-segment vector after normalization is the preset size. The purpose of size normalization is to facilitate the subsequent feature encoder to process the sub-segment vectors uniformly.

[0050] This embodiment determines the segmentation position by setting preset coefficients and speech length, thereby dynamically determining sub-speech segments, improving the flexibility of speech segmentation, and effectively ensuring the speech continuity of the segmentation results. This enhances the ability of the trained feature encoder to extract speaker timbre features, and thus improves the accuracy of speech reconstruction.

[0051] Optionally, the speech length is multiplied by a preset segmentation coefficient to determine the segmentation position, including:

[0052] The speech length is multiplied by N preset segmentation coefficients, and the N segmentation positions are determined based on the multiplication results.

[0053] Accordingly, the training speech is segmented according to the segmentation position to obtain the first speech segment and the second speech segment, including:

[0054] Traverse N segmentation positions and segment the training speech according to the segmentation positions to obtain N sets of segments;

[0055] Accordingly, based on the first speech feature, the second speech feature, and the preset metric loss function, the metric loss is calculated as follows:

[0056] Based on the N sets of fragments and the metric loss function, the N metric sub-losses corresponding to the N sets of fragments are calculated, and the sum of the N metric sub-losses is determined as the metric loss.

[0057] In this embodiment, the preset segmentation coefficients can be set to N, where N is an integer greater than zero. For example, N can be set to 3, and the three segmentation coefficients are 0.25, 0.5, and 0.75, respectively. This will result in three segmentation positions. By segmenting at the three positions, three sets of segmentation results can be obtained, which are also known as segment sets. Each segment set contains a first speech segment and a second speech segment, and there is a corresponding relationship between the first speech segment and the second speech segment in a segment set.

[0058] This embodiment, by setting multiple segmentation coefficients, can obtain multiple sets of input pairs for training the feature encoder for the same training speech. This effectively increases the number of training samples when the number of training speech is small, thereby improving the training fitting accuracy of the feature encoder and thus improving the accuracy of speech reconstruction.

[0059] The above steps involve segmenting the acquired training speech that meets the reconstruction conditions to obtain a first speech segment and a second speech segment. The first and second speech segments are then input into a feature encoder for feature extraction to obtain first speech features and second speech features. By segmenting the training speech into a first speech segment and a second speech segment, and then inputting them into the same feature encoder for feature extraction, the first and second speech features are obtained. This facilitates the subsequent training of the feature encoder based on the metric loss constructed from the first and second speech features. This improves the ease of training and also increases the accuracy of the feature encoder's training fit, thereby improving the accuracy of speech reconstruction.

[0060] Step S202: The trained speech input filter encoder is subjected to feature extraction to obtain filter features. The mean of the first speech feature and the second speech feature is calculated. The calculation result and the filter features are input into the decoder to obtain the reconstructed speech.

[0061] Among them, the filter encoder can be used to extract the semantic information of the training speech, the filter features can be used to represent the semantic information of the training speech, the decoder can be used to reconstruct the features, and the reconstructed speech can refer to the feature reconstruction result output by the decoder.

[0062] Specifically, the filter encoder may include a speech recognition part and a semantic extraction part. The speech recognition part may employ an acoustic model, such as a hybrid acoustic model or an end-to-end acoustic model, and may be used to convert the input training vector into a text information vector. The semantic extraction part may employ a language model, such as BERT or the Transformer model, and may be used to extract semantic information from the text information vector.

[0063] The above steps involve training the speech input filter encoder to extract features, obtaining filter features, calculating the mean of the first and second speech features, and inputting the calculation results and filter features into the decoder to obtain the reconstructed speech. The first speech features, the second speech features, and the filter features are jointly input into the decoder for reconstruction. This enables joint training of the feature encoder and the filter encoder, improving training efficiency while allowing the training processes of the feature encoder and the filter encoder to provide information to each other, thereby improving the fitting accuracy of the trained feature encoder and the trained filter encoder, and ultimately improving the accuracy of speech reconstruction.

[0064] Step S203: Calculate the metric loss based on the first speech feature, the second speech feature, and the preset metric loss function; calculate the reconstruction loss based on the reconstructed speech, the training speech, and the preset reconstruction loss function.

[0065] The preset loss function can be calculated using distance metrics such as Euclidean distance and cosine similarity. The loss can be used to characterize the constraint on the similarity between the first speech feature and the second speech feature. The preset reconstruction loss function can also be calculated using distance metrics such as Euclidean distance and cosine similarity. The reconstruction loss can be used to characterize the constraint on the similarity between the reconstructed speech and the training speech.

[0066] Specifically, since the first speech feature and the second speech feature correspond to different sub-speech segments, that is, the speech content of the first speech segment and the second speech segment are different, the first speech feature and the second speech feature obtained by conventional training should also be different. However, since the first speech segment and the second speech segment are similar in that they are both spoken by the same speaker, the metric loss in this embodiment can be used to measure the similarity between the first speech feature and the second speech feature. When the similarity is large, the metric loss is small, and vice versa. This constrains the feature encoder to output similar first speech features and second speech features for the first speech segment and the second speech segment of the same training speech. At this time, the feature encoder ignores the semantic information of the speech segment and only focuses on its speaker information, that is, the speaker's timbre information. Therefore, the speech features can be used to characterize the speaker's timbre features.

[0067] The steps described above, which calculate the metric loss based on the first speech feature, the second speech feature, and the preset metric loss function, and calculate the reconstruction loss based on the reconstructed speech, the training speech, and the preset reconstruction loss function, can effectively supervise the training of the feature encoder and the filter encoder, enabling them to fully decouple the training speech features into semantic features and speaker timbre features, thereby facilitating the convenience of subsequent speech reconstruction.

[0068] Step S204: Based on the metric loss and reconstruction loss, train the speech reconstruction model composed of the feature encoder, filter encoder and decoder to obtain the trained speech reconstruction model, and reconstruct the speech to be processed based on the trained speech reconstruction model.

[0069] Among them, the trained speech reconstruction model can be used to reconstruct speech, and the speech to be processed can refer to speech that meets the conditions for reconstruction and needs to be reconstructed.

[0070] Specifically, the sum of the metric loss and the reconstruction loss is used as the overall loss. The model parameters of the speech reconstruction model composed of the feature encoder, filter encoder and decoder are trained by the backpropagation algorithm and iterated until the overall loss converges.

[0071] Optionally, training the speech reconstruction model consisting of the feature encoder, filter encoder, and decoder based on the metric loss and reconstruction loss includes:

[0072] Multiply the measurement loss by the preset first weight to obtain the first multiplication result, and multiply the reconstruction loss by the preset second weight to obtain the second multiplication result;

[0073] The first and second multiplication results are added together, and the sum is determined as the final loss. The speech reconstruction model is then trained based on the final loss.

[0074] The preset first weight can be used to characterize the degree of influence of the measurement loss on the final loss, and the preset second weight can be used to characterize the degree of influence of the reconstruction loss on the final loss. For example, in this embodiment, both the first weight and the second weight can be set to 0.5.

[0075] This embodiment combines the metric loss and reconstruction loss in a weighted manner, thereby providing preferences for the training process of the speech reconstruction model, facilitating rapid convergence of the speech reconstruction model during training, and improving the training efficiency of the speech reconstruction model.

[0076] Optionally, the measurement loss is multiplied by a preset first weight to obtain a first multiplication result, and the reconstruction loss is multiplied by a preset second weight to obtain a second multiplication result, including:

[0077] The normalized exponential function is used to normalize the measurement loss and reconstruction loss. The normalized result of the corresponding measurement loss is determined as the first weight, and the normalized result of the corresponding reconstruction loss is determined as the second weight.

[0078] Multiply the metric loss by the first weight to obtain the first multiplication result, and multiply the reconstruction loss by the second weight to obtain the second multiplication result.

[0079] In this embodiment, let the metric loss be L1 and the reconstruction loss be L2. Then, the normalized result of the corresponding metric loss can be expressed as: The normalized result of the corresponding reconstruction loss can be expressed as:

[0080]

[0081] Since the final loss during joint training is affected by both the metric loss and the reconstruction loss, dynamically adjusting the weights corresponding to the loss based on its specific value can effectively improve the efficiency of the training process. For example, if the metric loss is larger than the reconstruction loss, the final loss should primarily consider the metric loss, with the main goal of reducing the metric loss during training. This ensures the training effect of the feature encoder. When the training effect of the feature encoder is good enough, that is, when the metric loss converges, the speech features it provides to the encoder are more accurate, thus facilitating the effective convergence of the filter encoder and decoder.

[0082] In this embodiment, the measurement loss and reconstruction loss are weighted by adaptive dynamic weights to obtain the final loss, which can effectively improve the training efficiency and training fitting effect of the speech reconstruction model, thereby improving the accuracy of the trained speech reconstruction model and thus improving the accuracy of speech reconstruction.

[0083] Optionally, reconstructing the speech to be processed based on the trained speech reconstruction model includes:

[0084] Obtain the speech to be processed and its corresponding text to be reconstructed that meet the conditions for reconstruction.

[0085] The trained speech reconstruction model is used to reconstruct the speech input to be processed, and the reconstructed speech is obtained. The text information of the reconstructed speech is consistent with the text to be reconstructed.

[0086] Here, the speech to be processed can refer to the actual speech of the target speaker, the text to be reconstructed can refer to the text information corresponding to the reconstructed speech obtained by the speech reconstruction task, and the reconstructed speech can refer to the reconstructed speech information read by the target speaker from the text to be reconstructed.

[0087] Specifically, the speech to be processed is input into a trained feature encoder to obtain target speech features, the text to be reconstructed is input into a trained filter encoder to obtain target semantic features, and the target speech features and target semantic features are input into a trained decoder to obtain reconstructed speech.

[0088] Since the speaker's timbre features and semantic features are extracted separately and decoupled in the speech reconstruction model of this embodiment, the reconstructed speech of the same target speaker reading different texts can be obtained by simply changing the text to be reconstructed. This can be applied to scenarios such as video dubbing and audio synthesis. For example, navigation speech synthesis only requires one sentence of the target speaker's speech to be processed, which can reconstruct the entire navigation speech package that matches the timbre of the target speaker. At the same time, the reconstructed speech of different target speakers reading the same text can be obtained by simply changing the speech to be processed. This can be applied to scenarios such as video dubbing selection, to select dubbing with better effect for the video.

[0089] This embodiment uses the trained speech reconstruction model to perform speech reconstruction, which can flexibly adjust the speech reconstruction task, thereby better fulfilling the personalized speech synthesis task and achieving better speech reconstruction results in specific application scenarios, that is, the accuracy of speech reconstruction is high.

[0090] The above steps involve training a speech reconstruction model composed of a feature encoder, a filter encoder, and a decoder based on metric loss and reconstruction loss to obtain a trained speech reconstruction model. The speech reconstruction model is then used to reconstruct the speech to be processed. By jointly training the speech reconstruction model with metric loss and reconstruction loss, semantic features and speaker timbre features can be completely decoupled, thereby improving the accuracy of semantic feature extraction and speaker timbre feature extraction, and thus improving the accuracy of speech reconstruction.

[0091] This embodiment trains the speech segmentation results through metric learning, enabling the feature encoder to accurately extract speech features that represent the speaker's timbre information. Combined with the semantic features extracted by the filter encoder, speech reconstruction is achieved. In the speech reconstruction process, speech features and filter features are completely decoupled, thereby improving the accuracy of the feature decoding process and thus improving the accuracy of speech reconstruction.

[0092] See Figure 3 This is a flowchart illustrating a speech reconstruction method based on real-time training provided in Embodiment 2 of the present invention. In this speech reconstruction method, the filter encoder can adopt the encoder structure of a language model to extract the semantic features of the text, or it can adopt a bottleneck encoder structure to directly extract the semantic features of the speech without the need for speech-to-text conversion, thereby improving training efficiency.

[0093] When the filter encoder adopts the encoder structure of the language model, the calculation process of the filter encoder is described in Example 1, and will not be repeated here.

[0094] When the filtered encoder adopts an encoder structure with a bottleneck, the filtered encoder includes a dimension reduction layer, a convolutional layer, and a dimension increase layer.

[0095] The calculation process of this filter encoder includes the following steps:

[0096] Step S301: Input the training speech into the dimensionality reduction layer for dimensionality reduction processing to obtain dimensionality reduction features, and input the dimensionality reduction features into the convolutional layer for feature extraction to obtain intermediate features;

[0097] Step S302: Input the intermediate features into the dimension-upgrading layer for dimension-upgrading processing to obtain the filtered features.

[0098] Among them, the filter encoder can be used to extract the semantic information of the training speech, the dimensionality reduction layer can be used to reduce the dimension of the training input vector corresponding to the training speech, the dimensionality reduction feature can be used to represent the training speech information after dimensionality reduction, the convolutional layer can be used to extract features from the dimensionality reduction feature, the intermediate feature can be used to represent the semantic information obtained after feature extraction from the dimensionality reduction feature, the dimensionality increase layer can be used to increase the dimension of the intermediate feature, and the filter feature can have the same dimension as the training input vector and can be used to represent semantic information.

[0099] Specifically, the dimensionality reduction layer can be implemented using 1*1 convolution. That is, a preset number of 1*1 convolution kernels are set, and the training input vector corresponding to the training speech is convolved using the preset number of 1*1 convolution kernels to obtain dimensionality-reduced features with unchanged size and reduced dimensionality. For example, in this embodiment, the size of the training input vector is set to 32*32*192, where 32*32 corresponds to the size parameter of the training input vector, and 192 corresponds to the dimensionality of the training input vector. Accordingly, the size of the 1*1 convolution kernel is specifically 1*1*192, and the preset number is set to 16. Then the size of the obtained dimensionality-reduced feature is 32*32*16. It can be seen that before and after the dimensionality reduction process, the size of the vector does not change, but the dimensionality is reduced.

[0100] The purpose of dimensionality reduction layer processing can be to reduce the computational load of the filter encoder. Obviously, the parameters required for subsequent processing of dimensionality-reduced features are fewer, thus effectively improving the computational efficiency of the filter encoder. In this embodiment, for the speech reconstruction task, the purpose of using dimensionality reduction layer processing also includes forcibly discarding some information of the input training speech. Since the function of the feature encoder is to extract the speaker timbre features of the training speech, it can be considered that the trained feature encoder only extracts the speaker timbre features of the training speech. At this time, when restricting the filter encoder to forcibly discard some information of the input training speech, if the filter encoder forcibly discards semantic information, it is difficult to accurately complete the reconstruction, resulting in a high reconstruction loss. Therefore, the filter encoder can accurately reconstruct semantic information only when it discards only the speaker timbre features, thus meeting the constraints of subsequent reconstruction loss.

[0101] It should be noted that the above-mentioned preset number can be used to control how much information the filter encoder discards. In order to achieve complete separation of speaker timbre features and semantic features, and to minimize the computational load of the filter encoder, the preset number is reduced by a preset step size during the training process. In this embodiment, the preset step size is set to 1 until the reconstruction loss is difficult to converge, that is, the semantic features are discarded and thus cannot be accurately reconstructed.

[0102] Similarly, the dimension-upgrading layer can also be achieved through 1*1 convolution, that is, by using 1*1 convolution kernels of the same dimension as the training input vector to perform convolution processing on the intermediate features, so as to obtain filtered features with the same dimension as the training input vector.

[0103] In this embodiment, by performing dimensionality reduction and dimensionality increase processing on the training input vector, the speaker's timbre features and semantic features are separated and decoupled. This allows speech reconstruction to be achieved simply by replacing the speaker's timbre features, improving the convenience and accuracy of speech reconstruction. This increases the accuracy of speech reconstruction and enhances the simulation of machine customer service voice on financial service platforms, thereby improving the user experience on financial service platforms.

[0104] Corresponding to the real-time training-based speech reconstruction method in the above embodiments, Figure 4A structural block diagram of a speech reconstruction device based on real-time training provided in Embodiment 3 of the present invention is shown. This speech reconstruction device is applied to a client, and the computer device corresponding to the client is connected to a server to obtain training speech that meets the conditions for reconstruction from the server. The computer device corresponding to the client is equipped with a speech reconstruction model, which may include a feature encoder, a filter encoder, and a decoder. The training speech that meets the conditions for reconstruction can be used as training samples for real-time training of the speech reconstruction model. The feature encoder can be used to extract speech features representing the speaker's timbre, the filter encoder can be used to extract filter features representing semantic information, and the decoder can be used to reconstruct the features into reconstructed speech. For ease of explanation, only the parts relevant to the embodiments of the present invention are shown.

[0105] See Figure 4 The speech reconstruction device includes:

[0106] The speech segmentation module 41 is used to segment the acquired training speech that meets the conditions to be reconstructed to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into the feature encoder for feature extraction to obtain the first speech feature and the second speech feature.

[0107] The speech reconstruction module 42 is used to extract features from the trained speech input filter encoder to obtain filtered features, calculate the mean of the first speech feature and the second speech feature, and input the calculation result and the filtered features into the decoder to obtain the reconstructed speech.

[0108] The loss calculation module 43 is used to calculate the metric loss based on the first speech feature, the second speech feature and the preset metric loss function, and to calculate the reconstruction loss based on the reconstructed speech, the training speech and the preset reconstruction loss function.

[0109] The speech reconstruction module 44 is used to train the speech reconstruction model composed of the feature encoder, filter encoder and decoder according to the metric loss and reconstruction loss, so as to obtain the trained speech reconstruction model, and to reconstruct the speech to be processed based on the trained speech reconstruction model.

[0110] Optionally, the aforementioned speech segmentation module 41 includes:

[0111] The location determination unit is used to obtain the speech length of the training speech, multiply the speech length by a preset segmentation coefficient, and determine the segmentation position by multiplying the result.

[0112] The location segmentation unit is used to segment the training speech according to the segmentation location to obtain the first speech segment and the second speech segment.

[0113] Optionally, the aforementioned position determination unit includes:

[0114] The coefficient multiplication subunit is used to multiply the speech length by N preset segmentation coefficients respectively, and determine N segmentation positions based on the multiplication results, where N is an integer greater than zero;

[0115] Accordingly, the aforementioned speech segmentation module 41 includes:

[0116] The set segmentation unit is used to traverse N segmentation positions and segment the training speech according to the segmentation positions to obtain N sets of segments, each set of segments including the first speech segment and the second speech segment.

[0117] Accordingly, the loss calculation module 43 includes:

[0118] The sub-loss calculation unit is used to calculate N metric sub-losses corresponding to the N sets of fragments and the metric loss function, and determine the sum of the N metric sub-losses as the metric loss.

[0119] Optionally, the filter encoder includes a dimensionality reduction layer, a convolutional layer, and a dimensionality increase layer;

[0120] The aforementioned speech reconstruction module 42 includes:

[0121] The feature reduction unit is used to input the training speech into the dimensionality reduction layer for dimensionality reduction processing to obtain dimensionality reduction features. The dimensionality reduction features are then input into the convolutional layer for feature extraction to obtain intermediate features.

[0122] The feature upscaling unit is used to input intermediate features into the upscaling layer for upscaling processing to obtain filtered features.

[0123] Optionally, the aforementioned speech reconstruction module 44 includes:

[0124] The weight multiplication unit is used to multiply the measurement loss by a preset first weight to obtain a first multiplication result, and to multiply the reconstruction loss by a preset second weight to obtain a second multiplication result;

[0125] The final loss determination unit is used to add the first multiplication result and the second multiplication result to determine the sum as the final loss, and to train the speech reconstruction model based on the final loss.

[0126] Optionally, the above weight multiplication unit includes:

[0127] The normalization subunit is used to normalize the measurement loss and reconstruction loss using a normalization exponential function, and to determine the normalization result of the corresponding measurement loss as the first weight and the normalization result of the corresponding reconstruction loss as the second weight.

[0128] The weight calculation subunit is used to multiply the metric loss and the first weight to obtain the first multiplication result, and to multiply the reconstruction loss and the second weight to obtain the second multiplication result.

[0129] Optionally, the aforementioned speech reconstruction module 44 includes:

[0130] The speech acquisition unit is used to acquire the speech to be processed and its corresponding text to be reconstructed that meet the conditions for reconstruction.

[0131] The speech reconstruction unit is used to reconstruct the speech from the pre-trained speech reconstruction model, resulting in reconstructed speech. The text information of the reconstructed speech is consistent with the text to be reconstructed.

[0132] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of the present invention. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0133] Figure 5 This is a schematic diagram of the structure of a computer device provided in Embodiment 4 of the present invention. Figure 5 As shown, the computer device of this embodiment includes: at least one processor ( Figure 5 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor, which, when executed by the processor, implements the steps in any of the above-described speech reconstruction method embodiments.

[0134] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 5 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0135] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0136] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of a computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0137] Those skilled in the art will understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the functions described above can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this invention. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention can implement all or part of the processes in the methods of the above embodiments by instructing related hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0138] The present invention can implement all or part of the processes in the methods of the above embodiments, or it can be accomplished by a computer program product. When the computer program product is run on a computer device, the computer device executes the steps in the above method embodiments.

[0139] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0140] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0141] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0142] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0143] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A speech reconstruction method based on real-time training, characterized in that, The speech reconstruction method includes: The acquired training speech that meets the conditions to be reconstructed is segmented to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into a feature encoder for feature extraction to obtain a first speech feature and a second speech feature. The feature encoder is used to extract the timbre information of the speaker corresponding to the training speech. The training speech is input to the filter encoder for feature extraction to obtain filter features. The mean of the first speech feature and the second speech feature is calculated. The calculation result and the filter features are input into the decoder to obtain the reconstructed speech. The filter encoder is used to extract the semantic information of the training speech. The metric loss is calculated based on the first speech feature, the second speech feature, and a preset metric loss function; the reconstruction loss is calculated based on the reconstructed speech, the training speech, and a preset reconstruction loss function. Based on the metric loss and the reconstruction loss, the speech reconstruction model composed of the feature encoder, the filter encoder and the decoder is trained to obtain a trained speech reconstruction model, and the speech to be processed is reconstructed based on the trained speech reconstruction model. The process of reconstructing the speech to be processed based on the trained speech reconstruction model includes: Obtain the speech to be processed and its corresponding text to be reconstructed that meet the conditions for reconstruction. The speech to be processed is input into the trained speech reconstruction model to reconstruct the speech, and the converted speech is obtained. The text information of the converted speech is consistent with the text to be reconstructed.

2. The speech reconstruction method according to claim 1, characterized in that, The step of segmenting the acquired training speech that meets the conditions for reconstruction to obtain a first speech segment and a second speech segment includes: Obtain the speech length of the training speech, multiply the speech length by a preset segmentation coefficient, and determine the multiplication result as the segmentation position; The training speech is segmented according to the segmentation position to obtain the first speech segment and the second speech segment.

3. The speech reconstruction method according to claim 2, characterized in that, The step of multiplying the speech length by a preset segmentation coefficient and determining the segmentation position by the multiplication result includes: The speech length is multiplied by N preset segmentation coefficients, and N segmentation positions are determined based on the multiplication results, where N is an integer greater than zero. Accordingly, segmenting the training speech according to the segmentation position to obtain the first speech segment and the second speech segment includes: Traverse the N segmentation positions and segment the training speech according to the segmentation positions to obtain N sets of segments, the sets of segments including the first speech segment and the second speech segment; Accordingly, the step of calculating the metric loss based on the first speech feature, the second speech feature, and a preset metric loss function includes: Based on the N sets of fragments and the metric loss function, N metric sub-losses corresponding to the N sets of fragments are calculated, and the sum of the N metric sub-losses is determined as the metric loss.

4. The speech reconstruction method according to claim 1, characterized in that, The filter encoder includes a dimension reduction layer, a convolutional layer, and a dimension increase layer; The step of extracting features from the trained speech input filter encoder to obtain filter features includes: The training audio is input into the dimensionality reduction layer for dimensionality reduction processing to obtain dimensionality reduction features. The dimensionality reduction features are then input into the convolutional layer for feature extraction to obtain intermediate features. The intermediate features are input into the dimension-upgrading layer for dimension-upgrading processing to obtain the filtered features.

5. The speech reconstruction method according to claim 1, characterized in that, Training the speech reconstruction model composed of the feature encoder, the filter encoder, and the decoder based on the metric loss and the reconstruction loss includes: Multiply the metric loss by a preset first weight to obtain a first multiplication result, and multiply the reconstruction loss by a preset second weight to obtain a second multiplication result; The first multiplication result and the second multiplication result are added together to determine the final loss. The speech reconstruction model is then trained based on the final loss.

6. The speech reconstruction method according to claim 5, characterized in that, The step of multiplying the measurement loss by a preset first weight to obtain a first multiplication result, and multiplying the reconstruction loss by a preset second weight to obtain a second multiplication result, includes: The measurement loss and the reconstruction loss are normalized using a normalized exponential function, and the normalized result of the corresponding measurement loss is determined as the first weight, and the normalized result of the corresponding reconstruction loss is determined as the second weight. Multiply the metric loss by the first weight to obtain the first multiplication result, and multiply the reconstruction loss by the second weight to obtain the second multiplication result.

7. A speech reconstruction device based on real-time training, characterized in that, The speech reconstruction device includes: The speech segmentation module is used to segment the acquired training speech that meets the conditions to be reconstructed to obtain a first speech segment and a second speech segment. The first speech segment and the second speech segment are respectively input into the feature encoder for feature extraction to obtain a first speech feature and a second speech feature. The feature encoder is used to extract the timbre information of the speaker corresponding to the training speech. The speech reconstruction module is used to extract features from the training speech input filter encoder to obtain filtered features, calculate the mean of the first speech feature and the second speech feature, and input the calculation result and the filtered features into the decoder to obtain the reconstructed speech. The filter encoder is used to extract the semantic information of the training speech. The loss calculation module is used to calculate the metric loss based on the first speech feature, the second speech feature and a preset metric loss function, and to calculate the reconstruction loss based on the reconstructed speech, the training speech and a preset reconstruction loss function. The speech reconstruction module is used to train the speech reconstruction model composed of the feature encoder, the filter encoder and the decoder according to the metric loss and the reconstruction loss, to obtain the trained speech reconstruction model, and to reconstruct the speech to be processed based on the trained speech reconstruction model; The speech reconstruction module includes: The speech acquisition unit is used to acquire the speech to be processed and its corresponding text to be reconstructed that meet the conditions to be reconstructed. The speech reconstruction unit is used to input the speech to be processed into the trained speech reconstruction model to reconstruct the speech and obtain the converted speech. The text information of the converted speech is consistent with the text to be reconstructed.

8. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the speech reconstruction method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the speech reconstruction method as described in any one of claims 1 to 6.

Images