Cross-language voice conversion method and system, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a cross-language voice conversion method and system, electronic equipment and a storage medium, and belongs to the technical field of voice processing. The method comprises the following steps: acquiring source audio data and target audio data; inputting the source audio data and the target audio data into a pre-trained cross-language voice conversion model; performing spectrum feature analysis on the source audio data and the target audio data through an audio analyzer to respectively obtain source spectrum feature information and target spectrum feature information; performing human voice feature elimination processing on the source spectrum feature information through a content encoder to obtain target content feature information; performing conversion feature extraction processing on the target spectrum feature information through a speaker encoder to obtain target conversion feature information; and performing cross-language voice generation based on the target content feature information and the target conversion feature information through a voice generator to obtain target conversion voice. According to the embodiment of the invention, the conversion voice with a better effect can be obtained under the cross-language condition.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, is applicable to the field of financial technology, and in particular, to a cross-language speech conversion method, system, electronic device and storage medium. Background Art

[0002] Voice conversion (VC) is a technology that converts the original audio into another style of voice through audio processing. It can convert the original speaker's voice into the target speaker's voice while keeping the voice content unchanged. In the financial field, financial institutions can use voice conversion (VC) to improve customer service experience. For example, by converting the customer service staff's voice into a more natural and friendly voice or into a language familiar to the customer, the affinity and efficiency of the service can be improved.

[0003] In the related art, the converted speech effect obtained by cross-language speech conversion is not ideal. Therefore, how to obtain a good converted speech effect in a cross-language situation has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a cross-language speech conversion method, system, electronic device and storage medium, aiming to obtain better converted speech in cross-language situations.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a cross-language speech conversion method, the method comprising:

[0006] Acquire source audio data and target audio data; wherein the target audio data is speech audio in a language different from that of the source audio data;

[0007] Inputting the source audio data and the target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model includes an audio analyzer, a content encoder, a speaker encoder, and a speech generator;

[0008] Performing spectrum feature analysis on the source audio data by the audio analyzer to obtain source spectrum feature information;

[0009] Performing frequency spectrum feature analysis on the target audio data by the audio analyzer to obtain target frequency spectrum feature information;

[0010] Performing a human voice feature elimination process on the source frequency spectrum feature information by the content encoder to obtain target content feature information;

[0011] Performing conversion feature extraction processing on the target spectrum feature information by the speaker encoder to obtain target conversion feature information;

[0012] The speech generator performs cross-language speech generation based on the target content feature information and the target conversion feature information to obtain the target conversion speech.

[0013] In some embodiments, performing vocal feature removal processing on the source spectrum feature information by the content encoder to obtain target content feature information includes:

[0014] Performing instance normalization processing according to the source spectrum feature information to obtain normalized statistical data;

[0015] Performing constant information query processing based on the normalized statistical data to obtain human voice constant feature information;

[0016] The source frequency spectrum feature information is subjected to a human voice feature elimination process based on the human voice constant feature information to obtain the target content feature information.

[0017] In some embodiments, the step of performing cross-language speech generation by the speech generator based on the target content feature information and the target conversion feature information to obtain the target conversion speech includes:

[0018] Perform language conversion processing based on the target content feature information to obtain target conversion content information;

[0019] Generate spectrum information based on the target conversion content information and the target conversion feature information to obtain a target conversion spectrum;

[0020] Based on the target conversion spectrum, the target conversion speech is generated.

[0021] In some embodiments, obtaining source audio data and target audio data includes:

[0022] Acquire the source audio data;

[0023] Performing voice information analysis based on the source audio data to obtain language category information and voice identity information;

[0024] The target audio data is determined based on the language category information and the voice identity information.

[0025] In some embodiments, performing spectrum feature analysis on the source audio data by the audio analyzer to obtain source spectrum feature information includes:

[0026] Performing fundamental frequency information extraction processing on the source audio data to obtain original fundamental frequency information;

[0027] Performing pitch smoothing processing according to the original fundamental frequency information to obtain smoothed fundamental frequency information;

[0028] The source frequency spectrum feature information is synthesized according to the smoothed fundamental frequency information.

[0029] In some embodiments, before inputting the source audio data and the target audio data into a pre-trained cross-language speech conversion model, the method further includes pre-training the cross-language speech conversion model, specifically including:

[0030] Acquire a training data set and the original cross-language speech conversion model; wherein the training data set includes first audio training data and second audio training data, and the first audio training data and the second audio training data are in different languages;

[0031] Inputting the first audio training data and the second audio training data into the cross-language speech conversion model;

[0032] Performing spectrum feature analysis on the first audio training data by the audio analyzer to obtain first training spectrum feature information;

[0033] Performing spectrum feature analysis on the second audio training data by the audio analyzer to obtain second training spectrum feature information;

[0034] Performing a human voice feature elimination process on the first training spectrum feature information by the content encoder to obtain training content feature information;

[0035] Performing conversion feature extraction processing on the second training spectrum feature information by the speaker encoder to obtain training conversion feature information;

[0036] Performing cross-language speech generation by the speech generator based on the training content feature information and the training conversion feature information to obtain training conversion speech;

[0037] Comparing the first training spectrum feature information, the training conversion feature information and the training conversion speech to obtain loss deviation data;

[0038] The model parameters of the cross-language speech conversion model are updated according to the loss deviation data, and the first audio training data and the second audio training data are returned to be input into the cross-language speech conversion model until the cross-language speech conversion model meets the preset training termination condition, thereby obtaining the pre-trained cross-language speech conversion model.

[0039] In some embodiments, the loss deviation data includes first deviation data and second deviation data, and the loss deviation data is obtained by comparing the first training spectrum feature information, the training conversion feature information and the training conversion speech, including:

[0040] Performing spectrum feature analysis on the training conversion speech by the audio analyzer to obtain test spectrum feature information;

[0041] Performing a content feature comparison between the test spectrum feature information and the first training spectrum feature information to obtain the first deviation data;

[0042] Performing conversion feature extraction processing on the test spectrum feature information by the speaker encoder to obtain test conversion feature information;

[0043] A voice feature comparison is performed based on the verification conversion feature information and the training conversion feature information to obtain the second deviation data.

[0044] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application proposes a cross-language speech conversion system, the system comprising:

[0045] An audio data acquisition module, used to acquire source audio data and target audio data; wherein the target audio data is speech audio in a language different from that of the source audio data;

[0046] An input module, configured to input the source audio data and the target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model comprises an audio analyzer, a content encoder, a speaker encoder and a speech generator;

[0047] A spectrum feature extraction module, used to perform spectrum feature analysis on the source audio data through the audio analyzer to obtain source spectrum feature information; perform spectrum feature analysis on the target audio data through the audio analyzer to obtain target spectrum feature information;

[0048] A content feature extraction module, configured to perform a human voice feature elimination process on the source spectrum feature information through the content encoder to obtain target content feature information;

[0049] A human voice feature extraction module, used for performing conversion feature extraction processing on the target spectrum feature information through the speaker encoder to obtain target conversion feature information;

[0050] The speech generation module is used to perform cross-language speech generation based on the target content feature information and the target conversion feature information through the speech generator to obtain the target conversion speech.

[0051] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, the memory stores a computer program, and the processor implements the cross-language speech conversion method described in the first aspect when executing the computer program.

[0052] To achieve the above objectives, the fourth aspect of the embodiments of the present application proposes a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the cross-language speech conversion method described in the first aspect.

[0053] The cross-language speech conversion method, system, electronic device and storage medium proposed in the present application obtain source audio data and target audio data, and input them into a pre-trained cross-language speech conversion model for speech conversion processing, and respectively obtain source spectrum feature information and target spectrum feature information by performing spectrum feature analysis on the source audio data and the target audio data, and then perform human voice feature elimination processing on the source spectrum feature information to obtain target content feature information, and perform conversion feature extraction processing on the target spectrum feature information to obtain target conversion feature information, and finally perform cross-language speech generation through the target content feature information and the target conversion feature information to obtain target conversion speech. Therefore, the present application eliminates the human voice information in the source spectrum feature information by performing human voice feature elimination processing on the source spectrum feature information, and retains the feature information in terms of content, thereby avoiding the influence of the difference between different languages ​​on the extraction of the target content feature information, and then obtains the feature information in terms of human voice by performing conversion feature extraction processing on the target spectrum feature information, and finally generates a better conversion speech in the case of cross-language. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of a cross-language speech conversion method provided by an embodiment of the present application;

[0055] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0056] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.

[0057] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0058] Figure 5 yes Figure 1 Flow chart of step S107 in FIG.

[0059] Figure 6 yes Figure 1 The flowchart before step S102 in FIG.

[0060] Figure 7 yes Figure 6 Flow chart of step S608 in FIG.

[0061] Figure 8 It is a schematic diagram of the structure of a cross-language speech conversion system provided in an embodiment of the present application;

[0062] Fig. 9 yes Figure 8 Schematic diagram of the structure of the content feature extraction module;

[0063] Fig.10 yes Figure 8 A schematic diagram of the structure of the decoder of the speech generation module;

[0064] Fig.11 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0066] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] First, some nouns involved in this application are analyzed:

[0069] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0070] Natural language processing (NLP): NLP uses computers to process, understand and apply human languages ​​(such as Chinese, English, etc.). NLP is a branch of artificial intelligence and an interdisciplinary subject between computer science and linguistics. It is often referred to as computational linguistics. Natural language processing includes grammatical analysis, semantic analysis, and text understanding. Natural language processing is often used in technical fields such as machine translation, handwritten and printed character recognition, speech recognition and text-to-speech conversion, information intent recognition, information extraction and filtering, text classification and clustering, public opinion analysis and opinion mining. It involves data mining related to language processing, machine learning, knowledge acquisition, knowledge engineering, artificial intelligence research, and linguistic research related to language computing.

[0071] Voice Conversion (VC) is a technology that converts original audio into another style of sound through audio processing. It can convert the original speaker's voice into the target speaker's voice while keeping the voice content unchanged.

[0072] Automatic Speech Recognition (ASR) is a technology that converts human speech into computer-understandable information. It can convert human speech into corresponding text information.

[0073] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0074] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0075] The cross-language speech conversion method provided in the embodiment of the present application relates to the field of speech processing technology. The cross-language speech conversion method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, or can be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements a cross-language speech conversion method, etc., but is not limited to the above forms.

[0076] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0077] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0078] The embodiments of the present application provide a cross-language speech conversion method, system, electronic device, and storage medium, aiming to obtain better converted speech in a cross-language situation.

[0079] The cross-language speech conversion method, system, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the cross-language speech conversion method in the embodiments of the present application is described.

[0080] Figure 1 is an optional flow chart of the cross-language speech conversion method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0081] Step S101, obtaining source audio data and target audio data, wherein the target audio data is speech audio in a language different from that of the source audio data;

[0082] Step S102, inputting the source audio data and the target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model includes an audio analyzer, a content encoder, a speaker encoder and a speech generator;

[0083] Step S103, performing spectrum feature analysis on the source audio data by an audio analyzer to obtain source spectrum feature information;

[0084] Step S104, performing frequency spectrum feature analysis on the target audio data by an audio analyzer to obtain target frequency spectrum feature information;

[0085] Step S105, performing vocal feature elimination processing on the source spectrum feature information through a content encoder to obtain target content feature information;

[0086] Step S106, performing conversion feature extraction processing on the target spectrum feature information through a speaker encoder to obtain target conversion feature information;

[0087] Step S107: A speech generator performs cross-language speech generation based on the target content feature information and the target conversion feature information to obtain a target conversion speech.

[0088] Steps S101 to S107 shown in the embodiment of the present application are obtained by obtaining source audio data and target audio data, and inputting them into a pre-trained cross-language speech conversion model for speech conversion processing, and performing spectrum feature analysis on the source audio data and the target audio data to obtain source spectrum feature information and target spectrum feature information respectively, and then performing vocal feature elimination processing on the source spectrum feature information to obtain target content feature information, and performing conversion feature extraction processing on the target spectrum feature information to obtain target conversion feature information, and finally performing cross-language speech generation through the target content feature information and the target conversion feature information to obtain target converted speech. Therefore, the present application eliminates the vocal information in the source spectrum feature information by performing vocal feature elimination processing on the source spectrum feature information, retains the feature information in terms of content, thereby avoiding the influence of the differences in different languages ​​on the extraction of the target content feature information, and then obtains the feature information in terms of vocals by performing conversion feature extraction processing on the target spectrum feature information, and finally generates a better conversion speech in a cross-language case.

[0089] Financial institutions encounter customers from different regions. Language barriers can cause great trouble to customer service staff and customers. However, through cross-language speech conversion technology, the speech content can be switched to a more natural and friendly voice and a common language, which can improve the affinity of the service and increase the efficiency of business processing.

[0090] In step S101 of some embodiments, source audio data and target audio data are obtained. The target audio data is in a different language from the source audio data. The source audio data provides a basis for speech conversion and will be used to characterize the content features of the target converted speech, such as semantics, rhythm, and emotion, etc. The target audio data provides a reference for human voice features and will be used as acoustic attributes of the target converted speech, such as pitch, timbre, and intonation, etc.

[0091] See also Figure 2 In some embodiments, step S101 may include but is not limited to steps S201 to S203:

[0092] Step S201, obtaining source audio data;

[0093] Step S202, performing voice information analysis based on the source audio data to obtain language category information and voice identity information;

[0094] Step S203, determining target audio data based on the language category information and the voice identity information.

[0095] In step S201 of some embodiments, the source audio data is audio data input by a user object, which can be obtained through real-time communication technology or collected by a sound receiving device.

[0096] In step S202 of some embodiments, the source audio data is analyzed for speech information to obtain language category information and voice identity information, wherein the language category information indicates the language category of the speech, and the voice identity information indicates the identity information of the user object, especially the gender category, because when converting a male voice to a female voice, or vice versa, a good target converted voice cannot be obtained, and male and female voices differ in pitch, timbre, emotion, etc. In some cases, the voice identity information also includes the approximate age range of the user object, such as children, young adults, and the elderly.

[0097] In step S203 of some embodiments, target audio data is determined based on language category information and voice identity information.

[0098] Through steps S201 to S203, the source audio data is analyzed to obtain language category information and voice identity information, and the target audio data is further determined, so as to determine the adapted target audio data for user objects with different languages ​​and different identities, thereby creating conditions for generating natural and fluent converted speech.

[0099] In step S102 of some embodiments, the source audio data and the target audio data are input into a pre-trained cross-language speech conversion model.

[0100] In steps S103 to S104 of some embodiments, an audio analyzer is used to perform spectrum feature analysis on source audio data to obtain source spectrum feature information, and spectrum feature analysis is performed on target audio data to obtain target spectrum feature information.

[0101] See also Figure 3 In some embodiments, step S103 may include but is not limited to steps S301 to S303:

[0102] Step S301, extracting the fundamental frequency information of the source audio data to obtain the original fundamental frequency information;

[0103] Step S302, performing pitch smoothing processing according to the original fundamental frequency information to obtain smoothed fundamental frequency information;

[0104] Step S303: synthesize source spectrum characteristic information according to the smoothed fundamental frequency information.

[0105] In some embodiments, in step S301 to step S303, the source audio data is subjected to a fundamental frequency information extraction process to obtain original fundamental frequency information, which is related to the prosody of the speech. The original fundamental frequency information can be extracted by using a WORLD vocoder through a DIO algorithm.

[0106] Then, the original fundamental frequency information is subjected to pitch smoothing. In some embodiments, the WORLD vocoder can use the original fundamental frequency information to determine the voiceless and voiced sounds of the speech, and perform mean smoothing on the voiced sounds, thereby reducing the impact of noise and making the fundamental frequency smoother and more natural. The smoothness and consistency of the fundamental frequency are very important for generating natural speech, which can avoid the synthesized speech sounding unnatural due to the drastic change of the fundamental frequency and improve the continuity of the tone. Finally, the WORLD vocoder synthesizes the source spectrum feature information through the processed smoothed fundamental frequency information.

[0107] Through steps S301 to S303, by performing pitch smoothing processing on the fundamental frequency information of the source audio data, the influence of noise can be reduced, the fundamental frequency can be made smoother and more natural, and the continuity of the pitch can be improved, so that the finally generated speech sounds more natural.

[0108] Similarly, in step S104, the target audio data may be processed in the same manner using the methods of steps S301 to S303 to obtain target frequency spectrum feature information.

[0109] In step S105 of some embodiments, a content encoder performs vocal feature removal processing on the source frequency spectrum feature information to obtain target content feature information.

[0110] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S403:

[0111] Step S401, performing instance normalization processing according to source spectrum feature information to obtain normalized statistical data;

[0112] Step S402, performing constant information query processing based on normalized statistical data to obtain human voice constant feature information;

[0113] Step S403: performing vocal feature elimination processing on the source spectrum feature information based on the vocal constant feature information to obtain target content feature information.

[0114] In step S401 of some embodiments, instance normalization is a common technique in deep learning, which normalizes the features of each sample to have zero mean and unit variance, thereby reducing the differences between different samples. The spectrum feature information at different time points in the source spectrum feature information is used as samples and instance normalization is performed to obtain normalized statistical data, that is, statistical information such as the mean and variance of the spectrum feature information.

[0115] In step S402 of some embodiments, the normalized statistical data is subjected to constant information query processing, where constant information refers to feature information that does not change much at different time points, i.e., constant feature information of human voice. Since the source spectrum feature information is obtained from audio data input by a user object, these constant information are usually associated with the identity of the user object, and thus the human voice constant feature information related to the user object, such as the timbre and speaking habits of the user object, is generally unchanged and is a constant.

[0116] In step S403 of some embodiments, after obtaining the human voice constant feature information, human voice feature elimination processing is performed on the source spectrum feature information to obtain target content feature information related to the speech content in the audio data.

[0117] When performing cross-language speech conversion in the related art, the content information in the source audio data is generally extracted directly, and this method is greatly limited by the language category of the source audio data. For example, the speech content is extracted based on a pre-trained speech recognition model, but this method often relies on a pre-trained speech recognition model. These speech recognition models can only target audio data of a specific speech type and cannot cope well with cross-language speech conversion tasks. Unlike the related art, the content encoder of the present application does not consider the specific text or phoneme sequence of the speech content, but directly eliminates the human voice feature information related to the speaker in the source audio data through the method of instance normalization. In this way, the differences between different languages ​​will not affect the performance of the content encoder, and the feature information of the speech content, that is, the target content feature information, can be extracted.

[0118] Through steps S401 to S403, the source spectrum feature information is instance-normalized and the human voice constant feature information in the source spectrum feature information is eliminated, thereby eliminating the human voice feature information and obtaining the target content feature information related to the speech content. In addition, the content encoder does not need to be pre-trained to cope with cross-language speech conversion tasks in different languages.

[0119] In step S106 of some embodiments, a speaker encoder performs conversion feature extraction processing on the target spectrum feature information to obtain target conversion feature information.

[0120] This step aims to extract the vocal feature information in the target audio data, such as rhythm, timbre, etc. The vocal feature information will be used in the cross-language speech conversion process to generate a target converted speech with the same sound as the target audio data.

[0121] In step S107 of some embodiments, a speech generator performs cross-language speech generation based on the target content feature information and the target conversion feature information to obtain the target conversion speech.

[0122] See also Figure 5 In some embodiments, step S107 may include but is not limited to steps S501 to S503:

[0123] Step S501, performing language conversion processing based on target content feature information to obtain target conversion content information;

[0124] Step S502, generating spectrum information based on the target conversion content information and the target conversion feature information to obtain a target conversion spectrum;

[0125] Step S503: generating a target conversion speech based on the target conversion spectrum.

[0126] In step S501 of some embodiments, language conversion processing is performed based on target content feature information. Speech recognition technology can be used to identify the target content feature information and further translate it. Translation can be performed through machine translation or language model to obtain target converted content information in the language corresponding to the target audio data.

[0127] In step S502 of some embodiments, the target conversion content information is converted into feature vector information, and spectrum information is generated with the target conversion feature information to obtain the target conversion spectrum. The target conversion feature information includes the timbre features and rhythm features of the target audio data, and the spectrum information generation combines these features with the content information of the target conversion content information. Since the target audio data includes the timbre features and rhythm features of the corresponding language category, the naturalness of the generated target conversion speech will be very good. The generated target conversion spectrum represents the acoustic representation of the target conversion speech.

[0128] In step S503 of some embodiments, a final target converted speech is generated based on the target converted spectrum. The speech generator may include an efficient vocoder that can convert the spectrum information back into speech audio.

[0129] See also Figure 6 In some embodiments, before step S102, the cross-language speech conversion model is also pre-trained, which may include but is not limited to steps S601 to S609:

[0130] Step S601, obtaining a training data set and an original cross-language speech conversion model, wherein the training data set includes first audio training data and second audio training data, and the first audio training data and the second audio training data are in different languages;

[0131] Step S602, inputting the first audio training data and the second audio training data into a cross-language speech conversion model;

[0132] Step S603, performing spectrum feature analysis on the first audio training data by an audio analyzer to obtain first training spectrum feature information;

[0133] Step S604, performing spectrum feature analysis on the second audio training data by an audio analyzer to obtain second training spectrum feature information;

[0134] Step S605, performing vocal feature elimination processing on the first training spectrum feature information through a content encoder to obtain training content feature information;

[0135] Step S606, performing conversion feature extraction processing on the second training spectrum feature information through a speaker encoder to obtain training conversion feature information;

[0136] Step S607, performing cross-language speech generation based on the training content feature information and the training conversion feature information by a speech generator to obtain training conversion speech;

[0137] Step S608, comparing the first training spectrum feature information, the training conversion feature information and the training conversion speech to obtain loss deviation data;

[0138] Step S609, update the model parameters of the cross-language speech conversion model according to the loss deviation data, return to execute the input of the first audio training data and the second audio training data into the cross-language speech conversion model until the cross-language speech conversion model meets the preset training termination condition, and obtain the pre-trained cross-language speech conversion model.

[0139] In steps S601 to S607 of some embodiments, the original cross-language speech conversion model is pre-trained using a training data set, wherein the training data set includes first audio training data and second audio training data, the first audio training data and the second audio training data are in different languages, wherein the first audio training data corresponds to the source audio data, and the second audio training data corresponds to the target audio data. During the training process, the speech content of the first audio training data and the second audio training data has the same semantics. The processing of steps S602 and S607 is the same as the processing of steps S102 to S107, except that the processing here is used for sample training, and in order to save space, it will not be repeated here.

[0140] In steps S608 to S609 of some embodiments, a comparison is performed based on the first training spectrum feature information, the training conversion feature information and the training conversion speech to obtain loss deviation data, which is the deviation data obtained by the loss function and is used to correct the model parameters of the cross-language speech conversion model. The model parameters of the language speech conversion model are updated according to the loss deviation data, and it is iterated based on steps S601 to S608 until the training conversion speech generated by the cross-language speech conversion model is semantically consistent with the first audio training data, close to the second audio training data in terms of vocal feature information, and has a certain degree of speech naturalness. Finally, a pre-trained cross-language speech conversion model is obtained.

[0141] See also Figure 7 In some embodiments, step S608 may include but is not limited to steps S701 to S704:

[0142] Step S701, performing spectrum feature analysis on the training conversion speech by an audio analyzer to obtain test spectrum feature information;

[0143] Step S702, performing content feature comparison based on the test spectrum feature information and the first training spectrum feature information to obtain first deviation data;

[0144] Step S703, performing conversion feature extraction processing on the test spectrum feature information through a speaker encoder to obtain test conversion feature information;

[0145] Step S704: perform voice feature comparison based on the verification conversion feature information and the training conversion feature information to obtain second deviation data.

[0146] In step S701 of some embodiments, the processing process is the same as that of step S103, except that the data processing object here is the training converted speech, and what is obtained is the test spectrum feature information, which will not be repeated here to save space.

[0147] In step S702 of some embodiments, content features are compared between the test spectrum feature information and the first training spectrum feature information to obtain first deviation data, thereby evaluating the performance of the cross-language speech conversion model in maintaining consistency in speech content. The first deviation data reflects the degree to which the cross-language speech conversion model retains speech content during the speech conversion process, and is an important indicator for evaluating model performance.

[0148] In step S703 of some embodiments, the processing process is the same as that of step S106, except that the data processing object here is to check the spectrum characteristic information, and what is obtained is the check conversion characteristic information, which will not be described here for the sake of saving space.

[0149] In step S704 of some embodiments, the second deviation data is obtained by comparing the human voice features of the test conversion feature information with the training conversion feature information, thereby evaluating the performance of the cross-language speech conversion model in simulating the sound features of the target audio data. The second deviation data reflects the ability of the model in simulating specific sound features, which is crucial for evaluating and optimizing the timbre and melody adaptability of the cross-language speech conversion model.

[0150] Through steps S701 to S704, the performance of the cross-language speech conversion model is evaluated from two dimensions: content and vocal features. The first deviation data and the second deviation data obtained can be used to quantify the accuracy of the model in maintaining the speech content and simulating the sound of the target audio data, providing valuable feedback for the training and optimization of the model. The first deviation data and the second deviation data will be used to update the model parameters of the cross-language speech conversion model to reduce the difference between the target converted speech and the speech of the target audio data, improve the consistency of the speech content of the target converted speech and the source audio data, and improve the overall performance of the model.

[0151] The embodiment of the present application obtains source audio data and target audio data, and inputs them into a pre-trained cross-language speech conversion model for speech conversion processing, performs spectrum feature analysis on the source audio data and the target audio data, obtains source spectrum feature information and target spectrum feature information respectively, then performs vocal feature elimination processing on the source spectrum feature information to obtain target content feature information, performs conversion feature extraction processing on the target spectrum feature information to obtain target conversion feature information, and finally performs cross-language speech generation through the target content feature information and the target conversion feature information to obtain target converted speech. Therefore, the present application eliminates the vocal information in the source spectrum feature information by performing vocal feature elimination processing on the source spectrum feature information, retains the feature information in terms of content, thereby avoiding the influence of the difference between different languages ​​on the extraction of the target content feature information, and then obtains the feature information in terms of vocals by performing conversion feature extraction processing on the target spectrum feature information, and finally generates a better conversion speech in a cross-language case.

[0152] See also Figure 8 The embodiment of the present application also provides a cross-language speech conversion system, which can implement the above cross-language speech conversion method, and the system includes:

[0153] An audio data acquisition module, used to acquire source audio data and target audio data; wherein the target audio data is speech audio in a language different from that of the source audio data;

[0154] An input module, used to input source audio data and target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model includes an audio analyzer, a content encoder, a speaker encoder and a speech generator;

[0155] The spectrum feature extraction module is used to perform spectrum feature analysis on the source audio data through an audio analyzer to obtain source spectrum feature information; perform spectrum feature analysis on the target audio data through an audio analyzer to obtain target spectrum feature information;

[0156] A content feature extraction module is used to perform a human voice feature elimination process on the source spectrum feature information through a content encoder to obtain target content feature information;

[0157] A human voice feature extraction module is used to perform conversion feature extraction processing on target spectrum feature information through a speaker encoder to obtain target conversion feature information;

[0158] The speech generation module is used to generate cross-language speech based on target content feature information and target conversion feature information through a speech generator to obtain target conversion speech.

[0159] The specific implementation of the cross-language speech conversion system is substantially the same as the specific implementation of the cross-language speech conversion method described above, and will not be described in detail herein.

[0160] In some embodiments, the content feature extraction module includes at least two instance normalization layers, and its structure refers to Fig. 9 The content feature extraction module inputs the source spectrum feature information and eliminates the human voice constant feature information through the instance normalization layer to obtain the target content feature information.

[0161] In some embodiments, the speech generation module includes a decoder and a vocoder, wherein the decoder includes at least two adaptive normalization layers, and its structure is shown in FIG. Fig.10 The decoder inputs the target content feature information and inputs the target conversion feature information through the adaptive normalization layer so that the target conversion feature information is fused with the feature information of the target content feature information and finally generates the target conversion spectrum.

[0162] Both of the above structures adopt the structure of the Residual Network (ResNet) deep convolutional neural network.

[0163] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned cross-language speech conversion method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0164] See also Fig.11 , Fig.11 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0165] The processor 1101 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0166] The memory 1102 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1102 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 1102, and the processor 1101 calls and executes the cross-language speech conversion method of the embodiment of the present application;

[0167] Input / output interface 1103, used to implement information input and output;

[0168] The communication interface 1104 is used to realize the communication interaction between the device and other devices. The communication can be realized through a wired manner (such as USB, network cable, etc.) or a wireless manner (such as mobile network, WIFI, Bluetooth, etc.);

[0169] A bus 1105 that transmits information between various components of the device (e.g., the processor 1101, the memory 1102, the input / output interface 1103, and the communication interface 1104);

[0170] The processor 1101 , the memory 1102 , the input / output interface 1103 and the communication interface 1104 are connected to each other in communication within the device via the bus 1105 .

[0171] The embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the cross-language speech conversion method is implemented.

[0172] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The cross-language speech conversion method, system, electronic device and storage medium provided by the embodiment of the present application obtain source audio data and target audio data, and input them into a pre-trained cross-language speech conversion model for speech conversion processing, and respectively obtain source spectrum feature information and target spectrum feature information by performing spectrum feature analysis on the source audio data and the target audio data, and then perform vocal feature elimination processing on the source spectrum feature information to obtain target content feature information, and perform conversion feature extraction processing on the target spectrum feature information to obtain target conversion feature information, and finally perform cross-language speech generation through the target content feature information and the target conversion feature information to obtain target converted speech. Therefore, the present application eliminates the vocal information in the source spectrum feature information by performing vocal feature elimination processing on the source spectrum feature information, retains the feature information in terms of content, thereby avoiding the influence of the difference between different languages ​​on the extraction of the target content feature information, and then obtains the feature information in terms of vocals by performing conversion feature extraction processing on the target spectrum feature information, and finally generates a better conversion speech in the case of cross-language.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The system embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A cross-language speech conversion method, characterized in that: The method comprises: Acquire source audio data and target audio data; wherein the target audio data is speech audio in a language different from that of the source audio data; Inputting the source audio data and the target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model includes an audio analyzer, a content encoder, a speaker encoder, and a speech generator; Performing spectrum feature analysis on the source audio data by the audio analyzer to obtain source spectrum feature information; Performing frequency spectrum feature analysis on the target audio data by the audio analyzer to obtain target frequency spectrum feature information; Performing a human voice feature elimination process on the source frequency spectrum feature information by the content encoder to obtain target content feature information; Performing conversion feature extraction processing on the target spectrum feature information by the speaker encoder to obtain target conversion feature information; The speech generator performs cross-language speech generation based on the target content feature information and the target conversion feature information to obtain the target conversion speech.

2. The method according to claim 1, characterized in that The step of performing voice feature elimination processing on the source spectrum feature information by the content encoder to obtain target content feature information includes: Performing instance normalization processing according to the source spectrum feature information to obtain normalized statistical data; Performing constant information query processing based on the normalized statistical data to obtain human voice constant feature information; The source frequency spectrum feature information is subjected to a human voice feature elimination process based on the human voice constant feature information to obtain the target content feature information.

3. The method according to claim 1, characterized in that The step of performing cross-language speech generation by the speech generator based on the target content feature information and the target conversion feature information to obtain the target conversion speech includes: Perform language conversion processing based on the target content feature information to obtain target conversion content information; Generate spectrum information based on the target conversion content information and the target conversion feature information to obtain a target conversion spectrum; Based on the target conversion spectrum, the target conversion speech is generated.

4. The method according to claim 1, characterized in that: The obtaining of source audio data and target audio data comprises: Acquire the source audio data; Performing voice information analysis based on the source audio data to obtain language category information and voice identity information; The target audio data is determined based on the language category information and the voice identity information.

5. The method according to claim 1, characterized in that The step of performing spectrum feature analysis on the source audio data by the audio analyzer to obtain source spectrum feature information includes: Performing fundamental frequency information extraction processing on the source audio data to obtain original fundamental frequency information; Performing pitch smoothing processing according to the original fundamental frequency information to obtain smoothed fundamental frequency information; The source frequency spectrum feature information is synthesized according to the smoothed fundamental frequency information.

6. The method according to any one of claims 1 to 5, characterized in that: Before inputting the source audio data and the target audio data into the pre-trained cross-language speech conversion model, the method further includes pre-training the cross-language speech conversion model, specifically including: Acquire a training data set and the original cross-language speech conversion model; wherein the training data set includes first audio training data and second audio training data, and the first audio training data and the second audio training data are in different languages; Inputting the first audio training data and the second audio training data into the cross-language speech conversion model; Performing spectrum feature analysis on the first audio training data by the audio analyzer to obtain first training spectrum feature information; Performing spectrum feature analysis on the second audio training data by the audio analyzer to obtain second training spectrum feature information; Performing a human voice feature elimination process on the first training spectrum feature information by the content encoder to obtain training content feature information; Performing conversion feature extraction processing on the second training spectrum feature information by the speaker encoder to obtain training conversion feature information; Performing cross-language speech generation by the speech generator based on the training content feature information and the training conversion feature information to obtain training conversion speech; Comparing the first training spectrum feature information, the training conversion feature information and the training conversion speech to obtain loss deviation data; The model parameters of the cross-language speech conversion model are updated according to the loss deviation data, and the first audio training data and the second audio training data are returned to be input into the cross-language speech conversion model until the cross-language speech conversion model meets the preset training termination condition, thereby obtaining the pre-trained cross-language speech conversion model.

7. The method according to claim 6, characterized in that The loss deviation data includes first deviation data and second deviation data, and the loss deviation data is obtained by comparing the first training spectrum feature information, the training conversion feature information and the training conversion speech, including: Performing spectrum feature analysis on the training conversion speech by the audio analyzer to obtain test spectrum feature information; Performing a content feature comparison between the test spectrum feature information and the first training spectrum feature information to obtain the first deviation data; Performing conversion feature extraction processing on the test spectrum feature information by the speaker encoder to obtain test conversion feature information; A voice feature comparison is performed based on the verification conversion feature information and the training conversion feature information to obtain the second deviation data.

8. A cross-language speech conversion system, characterized in that: The system comprises: An audio data acquisition module, used to acquire source audio data and target audio data; wherein the target audio data is speech audio in a language different from that of the source audio data; An input module, configured to input the source audio data and the target audio data into a pre-trained cross-language speech conversion model; wherein the cross-language speech conversion model comprises an audio analyzer, a content encoder, a speaker encoder and a speech generator; A spectrum feature extraction module, used to perform spectrum feature analysis on the source audio data through the audio analyzer to obtain source spectrum feature information; perform spectrum feature analysis on the target audio data through the audio analyzer to obtain target spectrum feature information; A content feature extraction module, configured to perform a human voice feature elimination process on the source spectrum feature information through the content encoder to obtain target content feature information; A human voice feature extraction module, used for performing conversion feature extraction processing on the target spectrum feature information through the speaker encoder to obtain target conversion feature information; The speech generation module is used to perform cross-language speech generation based on the target content feature information and the target conversion feature information through the speech generator to obtain the target conversion speech.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the cross-language speech conversion method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the cross-language speech conversion method according to any one of claims 1 to 7 is implemented.

Images