A method, apparatus, device and medium for dialect speech conversion

Abstract

Embodiments of the present application provide a dialect speech conversion method, device, equipment and medium, the method comprising: obtaining a speech to be converted, wherein the speech to be converted is a Mandarin speech to be output in a conversation with a user; inputting the speech to be converted into a target translator, obtaining a dialect discrete variable by performing dimension reduction operation on the speech to be converted by the target translator; inputting the dialect discrete variable into a target restorer, obtaining a dialect spectrum corresponding to the speech to be converted by performing restoration operation on the dialect discrete variable by the target restorer; synthesizing the dialect spectrum into a dialect speech, and outputting the dialect speech. Some embodiments of the present application can output a dialect speech conforming to the expression habit of a user, thereby improving the server efficiency of an outbound server and reducing the cost of a seat.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the field of voice outbound, in particular to a dialect voice conversion method, device, equipment and medium. BACKGROUND

[0002] With the development of the Internet, outbound servers (for example, intelligent robots) are widely used in the field of voice outbound. In order to be able to communicate with users using dialects, the related art uses voice cloning, that is, cloning the voice of the target speaker. In the cloning process, part of the dialect of the speaker can be imitated, but only the tone of the user is cloned, and the dialect voice expressed by the user cannot be accurately cloned.

[0003] Therefore, how to output more accurate dialect voice becomes a problem to be solved. SUMMARY

[0004] Embodiments of the present application provide a dialect voice conversion method, device, equipment and medium. Through some embodiments of the present application, at least accurate dialect voice conforming to the expression habit of the user can be output, so as to improve the server efficiency of the outbound server and reduce the cost of the agent.

[0005] In a first aspect, the present application provides a dialect voice conversion method, comprising: obtaining a to-be-converted voice, wherein the to-be-converted voice is a Mandarin voice to be output in a conversation with a user; inputting the to-be-converted voice into a target translator, and obtaining a dialect discrete variable by performing dimension reduction operation on the to-be-converted voice through the target translator; inputting the dialect discrete variable into a target restorer, and obtaining a dialect spectrum corresponding to the to-be-converted voice by performing restoration operation on the dialect discrete variable through the target restorer; synthesizing the dialect spectrum into a dialect voice, and outputting the dialect voice.

[0006] Therefore, unlike the method of cloning the voice of the target user in the related art, the embodiments of the present application can directly map the Mandarin voice (i.e. the to-be-converted voice) to the dialect voice by discretizing and restoring the to-be-converted voice, output accurate dialect voice conforming to the expression habit of the user, so as to improve the server efficiency of the outbound server and reduce the cost of the agent.

[0007] In combination with the first aspect, in an embodiment of the present application, the obtaining of the dialect discrete variable by performing dimension reduction operation on the to-be-converted voice through the target translator comprises: encoding the to-be-converted voice through the target translator to obtain a voice code; and performing attention mechanism operation on the voice code and then decoding to obtain the dialect discrete variable.

[0008] Therefore, by encoding, attention mechanism operation and decoding on the to-be-converted speech, the embodiments of the present application can accurately convert the features of the Mandarin speech into the features of the dialect speech.

[0009] In combination with the first aspect, in an embodiment of the present application, the reducing operation on the dialect discrete variable by the target restorer comprises: performing up-sampling operation on the dialect discrete variable by the target restorer to obtain up-sampled features; performing self-attention mechanism operation on the up-sampled features to obtain attention features; and performing calculation on the attention features by a fully connected layer to obtain the dialect spectrum.

[0010] Therefore, by up-sampling and self-attention mechanism calculation on the discrete variable, the embodiments of the present application can restore the speech features into the spectrum, thereby realizing direct conversion from the Mandarin speech into the dialect speech without converting the Mandarin speech into characters and then converting the characters into the dialect speech, and further reducing the complexity of the model and the calculation time of the model.

[0011] In combination with the first aspect, in an embodiment of the present application, before the input of the to-be-converted speech into the target translator, the method further comprises: converting a dialect speech sample into a dialect sample discrete variable by a converter, wherein the plurality of dialect speeches included in the dialect speech sample are the dialects corresponding to the same region; converting the Mandarin speech sample into a discrete variable space by a to-be-trained translator, wherein the dialect sample discrete variable belongs to the discrete variable space; inputting the dialect sample discrete variable and the discrete variable space into a first loss function for calculation to obtain a first loss value; and repeating the above steps until the first loss value meets a preset condition to obtain the target translator.

[0012] In combination with the first aspect, in an embodiment of the present application, before the input of the discrete variable into the target restorer, the method further comprises: extracting a speech sample spectrum feature of a dialect speech sample; inputting the speech sample spectrum feature into a converter to obtain a speech sample hidden layer feature by the converter, and converting the hidden layer feature into a dialect sample discrete variable; performing a reducing operation on the dialect sample discrete variable by a to-be-trained restorer to obtain a dialect spectrum; inputting the dialect spectrum into a second loss function for calculation to obtain a second loss value; and repeating the above steps until the second loss value meets a preset condition to obtain the target restorer.

[0013] Therefore, by training the to-be-trained translator and the to-be-trained restorer, the embodiments of the present application can enable the translator to learn the features in the sample dialect speech, thereby obtaining accurate dialect speech in the application process.

[0014] In an implementation of the first aspect, the obtaining of the hidden layer feature of the speech sample by the converter comprises: performing down-sampling operation on the spectrum feature of the speech sample by the converter to obtain a down-sampled feature; and inputting the down-sampled feature into a fully connected layer after self-attention mechanism calculation to generate the hidden layer feature of the speech sample.

[0015] In an implementation of the first aspect, the converting of the hidden layer feature into the dialect sample discrete variable comprises: performing dot product operation on the hidden layer feature to obtain the dialect sample discrete variable.

[0016] In an implementation of the first aspect, the converting of the hidden layer feature into the dialect sample discrete variable comprises: performing dot product operation on the hidden layer feature to obtain the dialect sample discrete variable.

[0017] In an implementation of the second aspect, the feature discrete module is further configured to: encode the speech sample by the target translator to obtain a speech code; and decode the speech code after attention mechanism operation to obtain the dialect discrete variable.

[0018] In an implementation of the second aspect, the frequency spectrum restoration module is further configured to: perform up-sampling operation on the dialect discrete variable by the target restorer to obtain an up-sampled feature; perform self-attention mechanism operation on the up-sampled feature to obtain an attention feature; and calculate the attention feature by a fully connected layer to obtain the dialect frequency spectrum.

[0019] In combination with the second aspect, in an embodiment of the present application, the spectrum restoration module is further configured to: convert the dialect speech samples into dialect sample discrete variables through a converter, wherein the plurality of dialect speeches included in the dialect speech samples are of a dialect corresponding to a same region; convert the Mandarin speech samples into a discrete variable space through a translator to be trained, wherein the dialect sample discrete variables belong to the discrete variable space; input the dialect sample discrete variables, the Mandarin speech samples, and the discrete variable space into a first loss function for calculation to obtain a first loss value; repeat the above steps until the first loss value meets a preset condition, and obtain the target translator.

[0020] In combination with the second aspect, in an embodiment of the present application, the feature discrete module is further configured to: extract speech sample spectrum features of dialect speech samples; input the speech sample spectrum features into a converter to obtain speech sample hidden layer features through the converter, and convert the hidden layer features into dialect sample discrete variables; perform a restoration operation on the dialect sample discrete variables through a restorer to be trained to obtain a dialect spectrum; input the dialect spectrum into a second loss function for calculation to obtain a second loss value; repeat the above steps until the second loss value meets a preset condition, and obtain the target restorer.

[0021] In combination with the second aspect, in an embodiment of the present application, the feature discrete module is further configured to: perform a down-sampling operation on the speech sample spectrum features through the converter to obtain down-sampling features; after performing a self-attention mechanism calculation on the down-sampling features, input the down-sampling features into a full connection layer to generate the speech sample hidden layer features.

[0022] In combination with the second aspect, in an embodiment of the present application, the feature discrete module is further configured to: perform a dot product operation on the hidden layer features to obtain the dialect sample discrete variables.

[0023] In combination with the second aspect, in an embodiment of the present application, the feature discrete module is further configured to: perform a dot product operation on the hidden layer features to obtain the dialect sample discrete variables.

[0024] In combination with the second aspect, in an embodiment of the present application, the feature discrete module is further configured to: perform a dot product operation on the hidden layer features to obtain the dialect sample discrete variables. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 A scene schematic diagram of dialect speech conversion is shown for an embodiment of the present application.

[0026] Figure 2 One of the dialect voice conversion method flowcharts shown in the embodiments of the present application;

[0027] Figure 3 One of the dialect voice conversion method flowcharts shown in the embodiments of the present application;

[0028] Figure 4 One of the dialect voice conversion method flowcharts shown in the embodiments of the present application;

[0029] Figure 5 One of the dialect voice conversion method flowcharts shown in the embodiments of the present application;

[0030] Figure 6 The device composition schematic diagram of the dialect voice conversion shown in the embodiments of the present application;

[0031] Figure 7 The electronic device composition schematic diagram shown in the embodiments of the present application. DETAILED DESCRIPTION

[0032] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0033] The embodiments of the present application can be applied to the scene of converting Mandarin voice into dialect voice. In order to improve the problems in the background art, in some embodiments of the present application, by performing dimension reduction operation on the voice to be converted, discrete variables are obtained, and then the discrete variables are restored to obtain dialect spectrum. For example, in some embodiments of the present application, the outbound server is at least configured to: first, obtain the Mandarin voice to be converted, then input the Mandarin voice to be converted into a target translator to obtain discrete variables by dimension reduction operation, then input the discrete variables into a target restorer to perform restoration operation, obtain dialect spectrum, finally synthesize the dialect spectrum into dialect voice, and output the dialect voice.

[0034] The method steps in the embodiments of the present application will be described in detail below with reference to the drawings.

[0035] Figure 1 A scene diagram of dialect speech conversion in some embodiments of the present application is provided, which includes a user 110 and an outbound server 120. Specifically, in the process of communication between the outbound server 120 and the user 110, the user 110 outputs user speech to the outbound server, and the outbound server 120 generates answer speech (i.e. speech to be converted) corresponding to the user speech after receiving the user speech, and converts the answer speech into dialect speech, and then outputs the dialect speech to the user 110.

[0036] In related technologies, with the development of the Internet, big data, cloud computing and Internet of Things and other related technologies have been popularized and applied. Under this background, intelligentization is an inevitable development trend. The intelligent customer service robot of the insurance industry is one of the earliest customer service robots to be commercially used. However, most existing intelligent robots cannot communicate with customers in dialects, but only communicate with customers in Mandarin, such as fastspeech2, tacotron2, etc. The inability to synthesize dialects will make the intelligent robot lack affinity in the process of communication with customers, and in scenarios such as sales and renewal, it cannot gain the trust of customers, thereby reducing the sales rate and renewal rate of products.

[0037] To solve the above problems, in related technologies, voice cloning is used, that is, the voice of the target speaker is cloned (for example, MockingBird voice cloning software), and in the cloning process, part of the speaking dialect can be imitated, but only the user's timbre cloning effect is good, and the user's expressed dialect speech cannot be accurately cloned. The present application can reduce the dimension of voice from continuous space to limited discrete space through voice translation technology, while trying to only retain the information related to the content in the voice, and then convert it to mel-frequency spectrum features through upsampling, attention and full connection operations, and finally synthesize dialect speech through a vocoder.

[0038] The implementation process of the method of dialect speech conversion performed by the outbound server in the embodiments of the present application will be described below.

[0039] At least to solve the problems in the background art, as shown in Figure 2 Some embodiments of the present application provide a method of dialect speech conversion, which comprises:

[0040] S210, obtaining speech to be converted.

[0041] It should be noted that the to-be-converted speech can be a Mandarin speech to be output in the process of communication with the user, and the Mandarin speech to be output is generated by the outbound server according to the content expressed by the user. The to-be-converted speech can also be a Mandarin speech expressed by the agent in the process of communication with the user, and the Mandarin speech of the agent is first output to the outbound server, and then the outbound server converts the Mandarin speech into the dialect speech.

[0042] In S220, the to-be-converted speech is input into the target translator, and the dialect discrete variable is obtained by performing dimension reduction operation on the to-be-converted speech by the target translator.

[0043] In an embodiment of the present application, the to-be-trained translator and the to-be-trained restorer need to be trained before the target translator and the target restorer are used. The specific steps are as follows:

[0044] Firstly, the dialect speech samples are collected and processed.

[0045] Specifically, the dialect audio is collected through various channels, and the duration of the audio is calculated through a python script, and the audio with a short time is filtered out. After processing the text, it is divided into a training set, a validation set and a test set in a ratio of 0.7:0.2:0.1. Among them, the training set is used for model training, the validation set is used for model effect verification in the training process, and the test set is used for verifying the final effect of the trained model.

[0046] Secondly, the to-be-trained restorer is trained to obtain the target restorer.

[0047] Firstly, the speech sample spectrum features of the dialect speech samples are extracted.

[0048] It can be understood that the method of extracting the speech sample spectrum features can use the method known in the related art, and the present application does not limit the specific method of extracting the speech sample spectrum features.

[0049] Secondly, the speech sample spectrum features are input into the converter to obtain the speech sample hidden layer features through the converter, and the hidden layer features are converted into the dialect sample discrete variable.

[0050] Specifically, the speech sample spectrum features are down-sampled by the converter to obtain down-sampled features, and after the down-sampled features are calculated by the self-attention mechanism, they are input into the full connection layer to generate the speech sample hidden layer features, and then the dot product operation is performed on the hidden layer features to obtain the dialect sample discrete variable.

[0051] That is, the speech sample spectrum features are input into the converter, and then down-sampled by convolution, and then the hidden layer features are generated by Self-Attention and full connection layer, as shown in expression (1):

[0052]

[0053] wherein, represents the hidden layer feature, y represents the speech sample spectrum feature, θ conv represents the converter.

[0054] Then, the hidden layer feature is converted into the dialect sample discrete variable z by a nearest neighbor search based on a dot product, and the formula of the dot product operation is shown in expression (2):

[0055]

[0056] wherein, z represents the dialect sample discrete variable, q(z=k|y) represents the distribution of the dialect sample discrete variable z classification, ω represents the number of discrete symbols, wherein, represents the embedding space of the discrete symbol, D represents the size of each embedding vector e i , e i represents any one discrete symbol.

[0057] Then, the dialect sample discrete variable is restored by the to-be-trained restorer to obtain the dialect spectrum.

[0058] Specifically, the obtained dialect sample discrete variable is input into the to-be-trained restorer, then up-sampling operation is performed by inverse convolution, and then the mel spectrum (i.e., the dialect spectrum) is generated by Self-Attention and a fully connected layer.

[0059] Next, the dialect spectrum is input into the second loss function for calculation to obtain a second loss value.

[0060] Specifically, the to-be-trained restorer is trained by the second loss function, and the second loss function is shown in expression (3):

[0061]

[0062] wherein, L inv represents the second loss value, y represents the speech sample spectrum feature, θ inv represents the restorer, the discrete variable z is used as input, z is converted into e z in the embedding space by using the converter, γ represents the features of all dialect speech samples, f(e z ; e inv ) represents the reconstructed speech sequence.

[0063] Finally, the above steps are repeated until the second loss value meets a preset condition, and a target restorer is obtained.

[0064] That is, the preset condition can set a second threshold value for the second loss value, the second threshold value needs to be set according to the actual application, in the case of the second loss value less than or equal to the second threshold value, confirming that the second loss value meets the preset condition, then the training is ended to obtain the target restorer, in the case of the second loss value greater than the second threshold value, repeating the above training steps until the second loss value meets the preset condition, obtaining the target restorer.

[0065] As a specific embodiment of the present application, as shown in the figure, first, dialect sample collection and arrangement are carried out, and the speech sample spectrum features of the dialect sample are extracted, the speech sample spectrum features are input into the converter, the speech sample hidden layer features are extracted, and then the dialect sample discrete variables are generated based on the speech sample hidden layer features, the dialect sample discrete variables are input into the trained restorer, the dialect spectrum is obtained, the dialect spectrum is input into the second loss function, the second loss value is output, in the case of the second loss value meeting the preset requirement, the target restorer is obtained, in the case of the second loss value not meeting the preset requirement, the above steps are continued to be executed until the second loss value meets the preset requirement, and the target restorer is obtained. Figure 3 Thirdly, the trained translator is trained to obtain the target translator.

[0066] Firstly, the dialect speech sample is converted into a dialect sample discrete variable through the converter, wherein the plurality of dialect speeches included in the dialect speech sample are dialects corresponding to the same region.

[0067] It should be noted that the dialect speech sample is the dialect of the same region, that is, one dialect corresponds to one trained model, for example, Tianjin dialect corresponds to Tianjin dialect conversion model, and Hebei dialect corresponds to Hebei dialect conversion model. In the application process, the dialect conversion model (it can be understood that the dialect conversion model includes the target translator and the target restorer) can be selected according to the region where the user is located, the identity information of the user can also be obtained before the call, for example, the household registration place of the user is confirmed, so that the corresponding dialect conversion model is selected, and the dialect conversion model can also be used to convert the speech to be converted into dialect speech after the speech of the user is obtained.

[0068] Secondly, the Mandarin speech sample is converted into a discrete variable space through the trained translator, wherein the dialect sample discrete variable belongs to the discrete variable space.

[0069]

[0070] ​Specifically, the training target is represented as {(x, y) e (X, Y)}, where x represents the input audio (Mandarin), and y represents the dialect sample audio. The dialect sample audio is converted into a discrete variable by the converter, and the input audio is converted into the discrete variable space Z (it can be understood that the discrete variable space is the discrete variable corresponding to the input audio translated by the translator to be trained) by the translator to be trained, where z e Z.

[0071] Then, the dialect sample discrete variable and the discrete variable space are input into the first loss function for calculation to obtain a first loss value.

[0072] That is, the translator to be trained is trained by the first loss function, and the discrete variable corresponding to the dialect sample audio is supervised to make the discrete variable output by the translator to be trained as close as possible to the discrete variable corresponding to the dialect sample audio. Finally, the trained target translator model is saved. The first loss function is shown in expression (4):

[0073]

[0074] wherein L inv2 represents the first loss value, x represents the original audio (i.e., the Mandarin speech sample), z represents the dialect sample discrete variable, Z represents the discrete variable space, and θ trams represents the translator.

[0075] Finally, the above steps are repeated until the first loss value meets the preset condition to obtain the target translator.

[0076] That is, the preset condition can set a first threshold value for the first loss value, and the first threshold value needs to be set according to the actual application. In the case where the first loss value is less than or equal to the first threshold value, it is confirmed that the first loss value meets the preset condition, and the training is ended to obtain the target translator. In the case where the first loss value is greater than the first threshold value, the above training steps are repeated until the first loss value meets the preset condition to obtain the target translator.

[0077] As a specific embodiment of the present application, as shown in Figure 4 first, the dialect speech sample is obtained, the dialect speech sample is input into the converter to obtain the dialect sample discrete variable, at the same time, the Mandarin speech sample is obtained, the Mandarin speech sample is input into the translator to be trained to obtain the discrete variable space, then the dialect sample discrete variable and the discrete variable space are input into the first loss function to obtain the first loss value. In the case where the first loss value meets the preset requirement, the target translator is obtained, and in the case where the first loss value does not meet the preset requirement, the above steps are continuously executed until the first loss value meets the preset requirement to obtain the target translator.

[0078] In an embodiment of the present application, the specific process of dimension reduction operation on the speech to be converted to obtain the dialect discrete variable includes:

[0079] The speech to be converted is encoded by the target translator to obtain a speech code, and the speech code is decoded after attention mechanism operation to obtain the dialect discrete variable.

[0080] That is, the target translator first encodes the speech to be converted into the encoder, and then decodes the encoded result into the decoder after attention operation, and finally decodes into the dialect discrete variable.

[0081] S230, input the dialect discrete variable into the target restorer, and restore the dialect discrete variable by the target restorer to obtain the dialect spectrum corresponding to the speech to be converted.

[0082] Specifically, first, the target restorer performs upsampling operation on the dialect discrete variable to obtain upsampling features, then performs self-attention mechanism operation on the upsampling features to obtain attention features, and finally calculates the attention features by a full connection layer to obtain the dialect spectrum.

[0083] That is, the target restorer upsamples the dialect discrete variable by deconvolution, then generates a mel spectrum by self-attention and a full connection layer, and then synthesizes the dialect speech by a vocoder.

[0084] S240, synthesize the dialect spectrum into the dialect speech, and output the dialect speech.

[0085] That is, the outbound call server communicates with the user by the dialect after synthesizing the dialect speech.

[0086] As a specific embodiment of the present application, first, S510 is executed to input the speech to be converted, that is, the speech to be converted is input into the target translator 520 to obtain the dialect discrete variable, the dialect discrete variable is input into the target restorer 530 to obtain the dialect spectrum, and finally the dialect speech is generated based on the dialect spectrum, and S540 is executed to output the dialect speech.

[0087] Therefore, the present application proposes a dialect speech conversion technology for speech translation, and has the following technical points and beneficial effects:

[0088] The application maps the mel spectrum of the speech to be converted into discrete variables through an encoder (it can be understood that the encoder is a neural network structure in a restorer, which is used to convert the speech to be converted into dialect discrete variables, and the restorer includes the encoder, transpose convolution, self-attention mechanism module and full connection layer), then the corresponding discrete variables are up-sampled through transpose convolution, and then the mel spectrum of the target dialect is obtained through the self-attention mechanism and the full connection layer, and finally the mel spectrum is synthesized into the corresponding dialect speech through the vocoder. Therefore, the application directly maps the Mandarin speech to the dialect speech, without the need to map the Mandarin speech to the text and then synthesize the text into the dialect speech, thereby reducing the complexity of the model and reducing the inference time of the model.

[0089] From the perspective of the practicality of the customer service scene, the problem that the artificial customer service cannot communicate with the customer using different dialects is solved, which can be applied to intelligent robots of many companies such as customer service robots, sales robots and training robots, greatly improves the probability of customers purchasing products and the success rate of renewals, increases the service efficiency of the robot, and reduces the labor cost of the company. In most scenarios, the training robot trains the sales personnel through Mandarin, but the sales personnel in different regions have different dialects, and training the sales personnel through dialects can improve the learning efficiency. When the intelligent customer service robot and the sales robot communicate with the customer, communicating through dialects can improve the customer's sense of identity, thereby improving the customer's signing and renewal rate and increasing the company's benefits.

[0090] The above describes a dialect speech conversion method provided by the application, and the following will describe a dialect speech conversion device of the application.

[0091] As shown in Figure 6 Some embodiments of the application provide a dialect speech conversion device 600, which comprises a speech acquisition module 610, a feature discrete module 620, a spectrum restoration module 630 and a speech output module 640.

[0092] The speech acquisition module 610 is configured to acquire speech to be converted, wherein the speech to be converted is Mandarin speech to be output during a conversation with a user; the feature discrete module 620 is configured to input the speech to be converted into a target translator, and obtain dialect discrete variables by performing dimensionality reduction operation on the speech to be converted through the target translator; the spectrum restoration module 630 is configured to input the dialect discrete variables into a target restorer, and obtain dialect spectrum corresponding to the speech to be converted by performing restoration operation on the dialect discrete variables through the target restorer; and the speech output module 640 is configured to synthesize the dialect spectrum into dialect speech, and output the dialect speech.

[0093] In an embodiment of the present application, the feature discretization module 620 is further configured to: encode the speech to be converted by the target translator to obtain speech encoding; and decode the speech encoding after attention mechanism operation to obtain the dialect discrete variable.

[0094] In an embodiment of the present application, the spectrum restoration module 630 is further configured to: up-sample the dialect discrete variable by the target restorer to obtain up-sampled features; perform self-attention mechanism operation on the up-sampled features to obtain attention features; and calculate the attention features by a fully connected layer to obtain the dialect spectrum.

[0095] In an embodiment of the present application, the spectrum restoration module 630 is further configured to: convert a dialect speech sample into a dialect sample discrete variable by a converter, wherein a plurality of dialect speeches included in the dialect speech sample are dialects corresponding to a same region; convert the Mandarin speech sample into a discrete variable space by a translator to be trained, wherein the dialect sample discrete variable belongs to the discrete variable space; input the dialect sample discrete variable and the discrete variable space into a first loss function to calculate a first loss value; and repeat the above steps until the first loss value meets a preset condition to obtain the target translator.

[0096] In an embodiment of the present application, the feature discretization module 620 is further configured to: extract a speech sample spectrum feature of a dialect speech sample; input the speech sample spectrum feature into a converter to obtain a speech sample hidden layer feature by the converter, and convert the hidden layer feature into a dialect sample discrete variable; perform restoration operation on the dialect sample discrete variable by a restorer to be trained to obtain a dialect spectrum; and input the dialect spectrum into a second loss function to calculate a second loss value; and repeat the above steps until the second loss value meets a preset condition to obtain the target restorer.

[0097] In an embodiment of the present application, the feature discretization module 620 is further configured to: down-sample the speech sample spectrum feature by the converter to obtain down-sampled features; and input the down-sampled features into a fully connected layer to generate the speech sample hidden layer feature after self-attention mechanism calculation.

[0098] In an embodiment of the present application, the feature discretization module 620 is further configured to: perform dot product operation on the hidden layer feature to obtain the dialect sample discrete variable.

[0099] In an embodiment of the present application, Figure 6 the modules shown in the figure can be implemented Figures 1 to 5The various processes in the method embodiments. Figure 6 The operations and / or functions of the various modules in the method embodiments, respectively, to achieve Figures 1 to 5 The corresponding flows in the method embodiments. For details, refer to the descriptions in the method embodiments, and the detailed descriptions are appropriately omitted here to avoid repetition.

[0100] As shown in Figure 7 The embodiments of the present application provide an electronic device 700, comprising: a processor 710, a memory 720 and a bus 730, the processor is connected with the memory through the bus, the memory stores computer readable instructions, when the computer readable instructions are executed by the processor, the method is used to realize any one of the above embodiments, for details, refer to the descriptions in the method embodiments, and the detailed descriptions are appropriately omitted here to avoid repetition.

[0101] Wherein, the bus is used to realize the direct connection communication of these components. Wherein, the processor in the embodiments of the present application can be an integrated circuit chip, which has the processing capacity of signal. The processor can be a general processor, including central processing unit (CPU), network processor (NP) and the like; it can also be a digital signal processor (DSP), application specific integrated circuit (ASIC), ready programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor or the like.

[0102] The memory can be, but is not limited to, random access memory (RAM), read only memory (ROM), programmable read only memory (PROM), erasable programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM) and the like. The memory stores computer readable instructions, which can execute the method described in the above embodiments when executed by the processor.

[0103] It can be understood that Figure 7 The structure shown in the figure is only schematic, and can also include more than Figure 7The more or fewer components shown, or having the same Figure 7 The different configurations shown. Figure 7 The components shown can be implemented using hardware, software, or a combination thereof.

[0104] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a server, it implements any of the methods described in all the above embodiments. For details, please refer to the descriptions in the above method embodiments. To avoid repetition, detailed descriptions are appropriately omitted here.

[0105] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application. It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0106] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for dialect speech conversion, characterized in that, The method includes: Acquire the speech to be converted, wherein the speech to be converted is the Mandarin speech to be output during a call with the user; The speech to be converted is input into the target translator, and the target translator performs a dimensionality reduction operation on the speech to be converted to obtain dialect discrete variables. The dialect discrete variables are input into a target restorer, which performs a restoration operation on the dialect discrete variables to obtain the dialect spectrum corresponding to the speech to be converted. The restoration operation on the dialect discrete variables by the target restorer to obtain the dialect spectrum corresponding to the speech to be converted includes: performing an upsampling operation on the dialect discrete variables by the target restorer to obtain upsampling features; performing a self-attention mechanism operation on the upsampling features to obtain attention features; and calculating the attention features through a fully connected layer to obtain the dialect spectrum. The dialect spectrum is synthesized into dialect speech, and the dialect speech is output.

2. The method according to claim 1, characterized in that, The step of obtaining dialect discrete variables by performing dimensionality reduction on the speech to be converted through the target translator includes: The target translator encodes the speech to be converted to obtain speech encoding; The speech code is then decoded after undergoing an attention mechanism operation to obtain the dialect discrete variables.

3. The method according to any one of claims 1-2, characterized in that, Before inputting the speech to be converted into the target translator, the method further includes: The dialect speech samples are converted into discrete variables of dialect samples by a converter, wherein the multiple dialect speech samples included in the dialect speech samples are dialects corresponding to the same region; The translator to be trained converts Mandarin speech samples into a discrete variable space, wherein the discrete variables of the dialect samples belong to the discrete variable space. The discrete variables of the dialect samples and the space of discrete variables are input into the first loss function for calculation to obtain the first loss value; Repeat the above steps until the first loss value meets the preset condition to obtain the target translator.

4. The method according to any one of claims 1-2, characterized in that, Before inputting the discrete variable into the target reducer, the method further includes: Extracting spectral features from dialect speech samples; The spectral features of the speech samples are input into a converter to obtain the hidden features of the speech samples, and the hidden features are transformed into discrete variables of dialect samples. The dialect sample discrete variables are restored using a restorer to be trained, and the dialect spectrum is obtained. The dialect spectrum is input into a second loss function for calculation to obtain a second loss value; Repeat the above steps until the second loss value meets the preset condition to obtain the target restorer.

5. The method according to claim 4, characterized in that, The process of obtaining latent features of speech samples through the converter includes: The converter performs a downsampling operation on the spectral features of the speech sample to obtain downsampled features. After the downsampled features are calculated using a self-attention mechanism, they are input into a fully connected layer to generate the hidden layer features of the speech samples.

6. The method according to claim 4, characterized in that, The process of converting the hidden layer features into discrete variables of dialect samples includes: Perform a dot product operation on the hidden layer features to obtain the discrete variables of the dialect sample.

7. A device for dialect speech conversion, characterized in that, The device includes: The voice acquisition module is configured to acquire the voice to be converted, wherein the voice to be converted is the Mandarin voice to be output during a call with a user; The feature discrete module is configured to input the speech to be converted into the target translator, and then use the target translator to perform a dimensionality reduction operation on the speech to be converted to obtain dialect discrete variables. The spectrum restoration module is configured to input the dialect discrete variables into a target restorer, and perform a restoration operation on the dialect discrete variables through the target restorer to obtain the dialect spectrum corresponding to the speech to be converted. The step of performing the restoration operation on the dialect discrete variables through the target restorer to obtain the dialect spectrum corresponding to the speech to be converted includes: performing an upsampling operation on the dialect discrete variables through the target restorer to obtain upsampling features; performing a self-attention mechanism operation on the upsampling features to obtain attention features; and calculating the attention features through a fully connected layer to obtain the dialect spectrum. The speech output module is configured to synthesize the dialect spectrum into dialect speech and output the dialect speech.

8. An electronic device, characterized in that, include: Processor, memory, and bus; The processor is connected to the memory via the bus, and the memory stores a computer program that, when executed by the processor, can implement the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method as described in any one of claims 1-6.

Images