Voice data processing method, device and equipment and computer readable storage medium

Abstract

The invention provides a voice data processing method, device and equipment and a computer readable storage medium. The method comprises the following steps: determining target audio data corresponding to a target timbre from candidate audio data corresponding to N candidate timbres; obtaining first audio data and a first text corresponding to the first audio data, and encoding the first audio data and the first text to obtain an initial mark sequence; when the first text comprises at least two languages, adding a language conversion mark to the initial mark sequence to obtain a first target mark sequence; based on the target audio data, performing feature conversion on the first target mark sequence to obtain a first audio feature of the first target mark sequence; and performing voice conversion on the first audio feature to obtain second audio data corresponding to the first text, the timbre of the second audio data being the target timbre. According to the application, the tone diversity of the voice data can be improved, and personalized tone unification is carried out on the voice data of different languages.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a voice data processing method, device, equipment and computer-readable storage medium. Background Art

[0002] In the field of speech synthesis, in order to generate natural-sounding speech, speech samples of multiple speakers are usually required. The technology of timbre unification of mixed speech can help synthesize more natural and consistent speech, and timbre unification is a key step to achieve this goal. The technology of timbre unification can be used to create personalized voice services, such as customized voice assistants, game character dubbing, etc. In the medical field, the technology of timbre unification can be used to assist diagnosis by analyzing the timbre changes of the patient's voice to diagnose certain diseases. In cultural and entertainment industries such as movies, TV series, and animation, the technology of timbre unification can be used to create more realistic and consistent voice effects.

[0003] In the speech synthesis process, the data that meets the timbre preference is insufficient, resulting in a lack of timbre richness. In addition, the differences between the training data of different languages ​​will lead to differences in the timbre of the synthesized speech and unnatural transitions, making it impossible to unify the timbre of the speech data of different languages ​​in a personalized way. Summary of the invention

[0004] The embodiments of the present application provide a voice data processing method, apparatus, device and computer-readable storage medium, which can improve the timbre diversity of voice data and perform personalized timbre unification on voice data in different languages.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides a method for processing speech data, the method comprising:

[0007] Determining target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbre;

[0008] Acquire first audio data and a first text corresponding to the first audio data, and encode the first audio data and the first text to obtain an initial tag sequence;

[0009] When the first text includes at least two languages, adding a language conversion marker to the initial marker sequence to obtain a first target marker sequence;

[0010] Based on the target audio data, feature conversion is performed on the first target tag sequence to obtain a first audio feature of the first target tag sequence;

[0011] The first audio feature is converted into speech to obtain second audio data corresponding to the first text, wherein the timbre of the second audio data is a target timbre.

[0012] The present application provides a voice data processing device, including:

[0013] A timbre determination module, used to determine target audio data corresponding to a target timbre from candidate audio data corresponding to each of N candidate timbres;

[0014] A first marking module, used to obtain first audio data and a first text corresponding to the first audio data, and encode the first audio data and the first text to obtain an initial marking sequence;

[0015] A second marking module, configured to add a language conversion mark to the initial mark sequence to obtain a first target mark sequence when the first text includes at least two languages;

[0016] a feature conversion module, configured to perform feature conversion on the first target tag sequence based on the target audio data to obtain a first audio feature of the first target tag sequence;

[0017] The speech conversion module is used to perform speech conversion on the first audio feature to obtain second audio data corresponding to the first text, wherein the timbre of the second audio data is a target timbre.

[0018] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0019] A memory for storing computer executable instructions or computer programs;

[0020] The processor is used to implement the voice data processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions or a computer program for implementing the voice data processing method provided in the embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the voice data processing method provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] By using the embodiment of the present application, the target audio data corresponding to the target timbre is determined from the candidate audio data corresponding to each of the N candidate timbres, so as to determine the audio data of the personalized timbre from the audio data of multiple timbres, and then the first audio data and the first text corresponding to the first audio data are obtained, and the first audio data and the first text are encoded to obtain an initial tag sequence. When the first text includes at least two languages, the language conversion tag is added to the initial tag sequence to obtain a first target tag sequence, which can convert different languages ​​in the text, and then based on the target audio data, the first target tag sequence is feature converted to obtain the first audio feature of the first target tag sequence, and then the first audio feature is voice converted to obtain the second audio data corresponding to the first text, and the timbre of the second audio data is the target timbre. In this way, based on the audio data of the personalized timbre, other audio data are voice-converted, so as to achieve personalized timbre unification of voice data in different languages. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is a schematic diagram of an application mode of the voice data processing method provided in an embodiment of the present application;

[0026] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0027] Figure 3A It is a first flow chart of the voice data processing method provided in an embodiment of the present application;

[0028] Figure 3B is a second flow chart of the voice data processing method provided in an embodiment of the present application;

[0029] Figure 3C 3 is a schematic diagram of a third flow chart of the voice data processing method provided in an embodiment of the present application;

[0030] Figure 3D 4 is a schematic diagram of a fourth flow chart of a method for processing voice data provided in an embodiment of the present application;

[0031] Figure 4 It is a schematic diagram of the voice data processing flow provided by an embodiment of the present application;

[0032] Figure 5 is a schematic diagram of the mel-spectrogram diffusion process provided in an embodiment of the present application;

[0033] Figure 6 is a schematic diagram of a speech tagging process provided by an embodiment of the present application;

[0034] Figure 7 is a schematic diagram of the structure of an input sequence of the same language provided in an embodiment of the present application;

[0035] Figure 8 It is a schematic diagram of the structure of input sequences in different languages ​​provided in the embodiments of the present application.

[0036] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0038] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0039] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0040] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0041] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0042] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0043] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0044] 1) Position encoding: It is a technique that introduces the position information of sequence elements into the sequence model.

[0045] 2) Mel-spectrogram: It is a visualization tool for representing the spectral characteristics of speech signals. It converts the time dimension and frequency dimension of speech signals into a two-dimensional image.

[0046] 3) Tone: refers to the quality or characteristics of sound, which is determined by many factors such as the sound's spectrum composition, waveform shape, and harmonic structure.

[0047] The embodiments of the present application provide a voice data processing method, apparatus, device and computer-readable storage medium, which can improve the timbre diversity of voice data and perform personalized timbre unification on voice data in different languages.

[0048] The following describes an exemplary application of the electronic device provided by the embodiment of the present application. The electronic device provided by the embodiment of the present application can be implemented as various types of terminals such as a laptop computer, a tablet computer, a desktop computer, a set-top box, a smart phone, a smart speaker, a smart watch, a smart TV, and a vehicle-mounted terminal, and can also be implemented as a server. The following describes an exemplary application when the device is implemented as a server.

[0049] See also Figure 1 , Figure 1 This is a schematic diagram of an application mode of the voice data processing method provided in an embodiment of the present application, for example, Figure 1 The server 200, the network 300 and the terminal 400 are involved. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0050] In the embodiment of the present application, the terminal 400 is taken as an intelligent question-answering robot for example. When the terminal 400 performs voice interaction, the terminal 400 determines the target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbres in response to the received audio selection operation. The terminal 400 collects the voice data of the dialogue object and performs voice recognition on the voice data to obtain the question text. The terminal 400 sends the question text to the server 200. The server 200 can determine the reply text based on the question text and send the reply text to the terminal 400. The terminal 400 receives the reply text and determines the reply text as the first text, obtains the first audio data corresponding to the first text, and encodes the first audio data and the first text to obtain an initial tag sequence; when the first text includes at least two languages, the language conversion tag is added to the initial tag sequence to obtain a first target tag sequence; based on the target audio data, the first target tag sequence is feature converted to obtain the first audio feature of the first target tag sequence; the first audio feature is voice converted to obtain the second audio data corresponding to the first text, the timbre of the second audio data is the target timbre, and the second audio data is the speech synthesis data, and then the speech synthesis data is output.

[0051] In some embodiments, when the voice data processing method provided in the embodiments of the present application is implemented by the server 200, after determining the target audio data, the terminal 400 can send the target audio data or the data identifier corresponding to the target audio data to the server 200, so that the server 200 can perform subsequent unified timbre processing based on the target audio data. After collecting the voice data of the conversation partner, the terminal 400 can send the voice data to the server 200. The server 200 performs voice recognition on the voice data to obtain a question text, and then determines a reply text based on the question text, and determines the reply text as the first text, obtains the first audio data corresponding to the first text, and encodes the first audio data and the first text to obtain an initial tag sequence; when the first text includes at least two languages, the language conversion tag is added to the initial tag sequence to obtain a first target tag sequence; based on the target audio data, the first target tag sequence is feature converted to obtain a first audio feature of the first target tag sequence; the first audio feature is voice converted to obtain second audio data corresponding to the first text, the timbre of the second audio data is the target timbre, and the second audio data is speech synthesis data. The server 200 sends the speech synthesis data to the terminal 400, and the terminal 400 outputs the speech synthesis data.

[0052] In some embodiments, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. Terminal 400 may be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, vehicle-mounted terminal, aircraft, intelligent robot, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0053] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, the electronic device may be a terminal or a server, Figure 2 The electronic device 500 shown includes: at least one processor 410, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0054] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0055] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0056] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0057] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0058] The operating system 451 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0059] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Authentication (WiFi), and Universal Serial Bus (USB).

[0060] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 A speech data processing device 455 stored in a memory 450 is shown, which may be software in the form of a program or plug-in, including the following software modules: a timbre determination module 4551, a first marking module 4552, a second marking module 4553, a feature conversion module 4554, and a speech conversion module 4555. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0061] The voice data processing method provided in the embodiment of the present application will be explained in combination with the exemplary application and implementation of the server device provided in the embodiment of the present application.

[0062] The following describes the voice data processing method provided in the embodiment of the present application. For ease of understanding of the voice data processing method provided in the embodiment of the present application, the following describes the application scenario of the voice data processing method provided in the embodiment of the present application. When a robot performs voice interaction, it is necessary to convert text information into natural and fluent voice output so that the robot can communicate with users in a manner familiar to humans. The voice data processing method provided in the embodiment of the present application can improve the timbre diversity of voice data and perform personalized timbre unification for voice data in different languages.

[0063] As mentioned above, the electronic device implementing the voice data processing method of the embodiment of the present application may be a terminal, a server, or a combination of the two. Next, the voice data processing method provided by the embodiment of the present application is described by taking the electronic device as a server as an example. Figure 3A , Figure 3A is a first flow chart of the voice data processing method provided in the embodiment of the present application, which will be combined with Figure 3A The steps shown are explained.

[0064] In step 301 , target audio data corresponding to a target timbre is determined from candidate audio data corresponding to each of N candidate timbre candidates.

[0065] Here, each candidate timbre corresponds to a candidate audio data, and the target timbre refers to a timbre specified from among the N candidate timbre.

[0066] In some embodiments, see Figure 3B , Figure 3B 2 is a schematic diagram of a second flow chart of the voice data processing method provided in the embodiment of the present application. Figure 3A Before step 301 shown, you can also perform Figure 3B Steps 3001 to 3003 are described in detail below.

[0067] In step 3001, an initial text is obtained, and control symbols are inserted and rewritten into the initial text to obtain a candidate text.

[0068] Here, the initial text is the text used to determine the candidate audio data, and the initial text can be obtained from the text data set of the robot's historical voice interaction. You can choose to insert control symbols and rewrite the initial text to obtain the candidate text. The control symbol is a label symbol used to adjust the initial text. For example, the control symbol is [laugh], [uv_break], etc., where [laugh] represents laughter and [uv_break] represents a pause. When you choose to insert a control symbol into the initial text, for example, the initial text is "Hello, nice to see you", and the text after the control symbol is inserted is "Hello [uv_break], nice to see you [laugh]". In the speech synthesis result of the text after the control symbol is inserted, [laugh] will be replaced by laughter, and a pause will be added at [uv_break].

[0069] Control symbols may be inserted into the initial text, or the initial text may be rewritten, based on the number of characters in the initial text. When the number of characters in the initial text is greater than or equal to a preset threshold, control symbols may be inserted into the initial text. When the number of characters in the initial text is less than a preset threshold, the initial text may be rewritten. When the initial text is rewritten, the content of the initial text is redefined, and the number of characters in the initial text is increased or decreased. For example, the initial text is "I want to sleep", and the rewritten text is "I just want to sleep".

[0070] In step 3002, N candidate timbre parameters are determined, and the N candidate timbre parameters are respectively inserted into the candidate texts to obtain N target texts.

[0071] Here, the candidate timbre parameters include the spectrum of the candidate timbre, the spatial coordinates of the candidate timbre, the timbre number of the candidate timbre, etc. The candidate timbre parameters can also be understood as the timbre seeds of the candidate timbre. Different candidate timbres correspond to different candidate timbre parameters. Cluster analysis is performed on different candidate timbres to obtain the category of each candidate timbre, and a candidate timbre parameter is assigned to each category of the candidate timbre. After determining N candidate timbre parameters, each candidate timbre parameter is inserted into the candidate text to obtain N target texts, each of which includes a control symbol and a candidate timbre parameter.

[0072] In step 3003, speech conversion is performed on the N target texts to obtain candidate audio data corresponding to the N candidate timbres.

[0073] Here, speech conversion is performed on each target text to obtain N candidate audio data, and the N candidate audio data correspond to N candidate timbres.

[0074] In some embodiments, performing speech conversion on N target texts to obtain candidate audio data corresponding to each of the N candidate timbres can be achieved by executing the following steps: for each target text, performing feature conversion on the target text to obtain a first audio tag sequence; performing feature extraction on the first audio tag sequence to obtain a first audio feature; performing speech conversion on the first audio feature to obtain candidate audio data.

[0075] Here, for each target text, according to the control symbol in the target text, the target text is feature-converted to obtain a first audio tag sequence. For example, the target text includes a control symbol [empty_spk], and the control symbol [empty_spk] represents a pause, indicating that the target text includes a pause feature, and the target text is subjected to a pause feature conversion to generate a discrete sequence audio_token, i.e., the first audio tag sequence.

[0076] The first audio tag sequence is feature extracted by an acoustic model to obtain a Mel-spectrogram feature, i.e., a first audio feature. The acoustic model may be a Diffusion Variational Autoencoder (DVAE) model, which combines a variational autoencoder and a generation model of a diffusion process, and can extract different Mel-spectrogram features for each first audio tag sequence of a target text. The vocoder model Vocos is used to convert different Mel-spectrogram features into different waveforms, thereby outputting candidate audio data corresponding to each of the N candidate timbres.

[0077] In an embodiment of the present application, N target texts are obtained by inserting N candidate timbre parameters into candidate texts respectively, and the N target texts are converted into speech to obtain candidate audio data corresponding to the N candidate timbres, thereby improving the timbre diversity of the speech data and providing diversified choices of personalized timbre.

[0078] Continue to refer Figure 3A In step 302, first audio data and a first text corresponding to the first audio data are obtained, and the first audio data and the first text are encoded to obtain an initial tag sequence.

[0079] Here, the first audio data is the audio data to be converted into timbre, the first text is the text corresponding to the first audio data, and the first text may be a reply text obtained by performing intelligent question-answering processing on the question text. The first audio data and the first text may be obtained from a dataset of historical voice interactions of the robot. The initial tag sequence includes an audio tag sequence corresponding to the first audio data and a text tag sequence corresponding to the first text.

[0080] In some embodiments, see Figure 3C , Figure 3C This is a third flow chart of the voice data processing method provided in an embodiment of the present application. Figure 3A In step 302 shown in the figure, "encoding the first audio data and the first text to obtain an initial tag sequence" can be performed by Figure 3C Steps 3021 to 3024 are implemented as described in detail below.

[0081] In step 3021, feature extraction is performed on the first audio data to obtain a first audio feature, and position encoding is performed on the first audio feature to obtain an audio representation vector.

[0082] Here, a window function such as a Hamming window or a rectangular window is used to divide the first audio data into a series of overlapping frames to obtain multiple frames of audio data. A fast Fourier transform is performed on each frame of audio data to convert the time domain signal into a frequency domain signal to obtain a spectrum of each frame of audio data. The spectrum of each frame of audio data is input into a visualization tool and plotted into a two-dimensional image to obtain a Mel-spectrogram, i.e., the first audio feature.

[0083] The first audio feature is position-encoded by the first encoder, and a fixed-size vector is assigned to each time frame of the first audio feature. The vector represents the position of each time frame in the first audio feature, and the vectors of each time frame are combined into an audio representation vector. For example, the audio representation vector can be determined by the following formula (1):

[0084] H1=Encoder1(posEnc(X)) (1)

[0085] Among them, H1 represents the audio representation vector, Encoder1 represents the first encoder, posEnc represents the position encoding, and X represents the first audio feature (Mel-spectrogram).

[0086] In step 3022, the audio representation vector is quantized to obtain a second audio tag sequence.

[0087] Here, the audio representation vector is quantized by a vector quantizer to obtain the speech mark of each time frame in the audio representation vector, and the speech mark of each time frame is combined into a second audio mark sequence. For example, the speech mark of the lth time frame can be determined by the following formula (2):

[0088]

[0089] Where u represents the speech mark of the lth time frame, c n represents the speech vector of the lth time frame in the audio representation vector, C represents the speech vector set, VQ represents the vector quantizer, h l represents the hidden layer representation of the lth time frame.

[0090] In some embodiments, the speech vector of the lth time frame may be updated by an exponential moving average method. For example, the updated speech vector of the lth time frame may be determined by the following formula (3):

[0091] c ul =ac ul +(1-a)h l (3)

[0092] Among them, c ul represents the updated speech vector of the lth time frame, a represents the attenuation coefficient, hl represents the hidden layer representation of the lth time frame.

[0093] The updated speech vector of each time frame is used as a quantized hidden layer representation and combined with the position coding as the input of the second encoder. For example, the output of the second encoder can be determined by the following formula (4):

[0094] H3=Encoder2(PosEnc(H2)) (4)

[0095] Among them, H3 represents the output of the second encoder, Encoder2 represents the second encoder, posEnc represents the position encoding, and H2 represents the quantized hidden layer representation.

[0096] In step 3023, the first text is encoded to obtain a text tag sequence.

[0097] Here, a byte pair encoding model is constructed according to the text data set, and the byte pair encoding model is used to encode the first text, and the character sequence in the first text is converted into a tag sequence to obtain a text tag sequence.

[0098] In step 3024, the second audio tag sequence and the text tag sequence are combined to obtain an initial tag sequence.

[0099] Here, the second audio tag sequence and the text tag sequence are combined to obtain an initial tag sequence, and the initial tag sequence is used as an input sequence of the large language model.

[0100] In some embodiments, the second audio marker sequence and the text marker sequence are combined to obtain an initial marker sequence, which can be achieved by performing the following steps: concatenating the text marker sequence after the second audio marker sequence to obtain an initial marker sequence.

[0101] Here, the initial tag sequence includes a sequence start tag, a text tag sequence, a second audio tag sequence, an additional information tag (including a control symbol, etc.), and a sequence end tag. The sequence start tag is located at the head of the initial tag sequence, the second audio tag sequence is spliced ​​after the sequence start tag, the text tag sequence is spliced ​​after the second audio tag sequence, the additional information tag is spliced ​​after the text tag sequence, and the sequence end tag is located at the tail of the initial tag sequence.

[0102] Continue to refer Figure 3A In step 303, when the first text includes at least two languages, a language conversion marker is added to the initial marker sequence to obtain a first target marker sequence.

[0103] Here, when the first text includes at least two languages, that is, when the first text includes different languages, a language conversion marker is added to the initial marker sequence to obtain a first target marker sequence.

[0104] In some embodiments, adding a language conversion marker to an initial marker sequence to obtain a first target marker sequence can be achieved by performing the following steps: concatenating the language conversion marker to the text marker sequence to obtain a first target marker sequence.

[0105] Here, the first target tag sequence includes a sequence start tag, a text tag sequence, a second audio tag sequence, a language conversion tag, and a sequence end tag. The sequence start tag is located at the head of the first target tag sequence, the second audio tag sequence is spliced ​​after the sequence start tag, the text tag sequence is spliced ​​after the second audio tag sequence, the language conversion tag is spliced ​​after the text tag sequence, and the sequence end tag is located at the tail of the initial tag sequence.

[0106] In an embodiment of the present application, when the first text includes at least two languages, a first target tag sequence is obtained by adding a language conversion tag to an initial tag sequence, and different languages ​​in the text can be converted based on the first target tag sequence to achieve personalized timbre unification of voice data in different languages.

[0107] Continue to refer Figure 3A In step 304, based on the target audio data, feature conversion is performed on the first target marker sequence to obtain a first audio feature of the first target marker sequence.

[0108] Here, the target audio data is encoded to obtain a target audio sequence. The optimal transmission path between the target audio sequence and the first target tag sequence is calculated through the optimal transmission condition stream matching model. The first target tag sequence is converted into a mel-spectrogram based on the optimal transmission path, that is, the first audio feature.

[0109] Continue to refer Figure 3A In step 305, the first audio feature is converted into speech to obtain second audio data corresponding to the first text.

[0110] Here, the timbre of the second audio data is the target timbre.

[0111] In some embodiments, performing speech conversion on the first audio feature to obtain second audio data corresponding to the first text can be achieved by executing the following steps: obtaining a generative adversarial network model and calling the generative adversarial network model to perform waveform extraction on the first audio feature to obtain a first audio waveform; determining audio data that matches the first audio waveform from an audio database, and determining the matching audio data as the second audio data corresponding to the first text.

[0112] Here, the generative adversarial network model can be a vocoder network HiFiGAN, and the generator in the generative adversarial network model is called to generate an audio waveform of the first audio feature, that is, a first audio waveform. After the first audio waveform is processed by denoising, adjusting the volume, etc., it is matched with the audio waveform data in the audio database to obtain matching audio data, and the matching audio data is saved as a WAV file, and it is determined as the second audio data corresponding to the first text.

[0113] In some embodiments, see Figure 3D , Figure 3D 4 is a schematic diagram of a fourth flow chart of a method for processing speech data provided in an embodiment of the present application. When the first audio data and the target audio data are in the same language, Figure 3D Steps 310 to 314 are implemented as described in detail below.

[0114] In step 310, when the first audio data and the target audio data are in the same language, the target audio data is encoded to obtain a third audio tag sequence.

[0115] Here, when the first audio data and the target audio data are in the same language, the target audio data is divided into audio data of multiple time frames using a window function, and the audio data of each time frame is subjected to fast Fourier transform and Mel filtering to obtain Mel spectrum features, and the Mel spectrum features are converted into a series of audio vectors to obtain a third audio marker sequence.

[0116] In step 311, the prompt word text corresponding to the target audio data is determined, and the prompt word text is encoded to obtain a prompt word tag sequence.

[0117] Here, speech recognition can be performed on the target audio data to obtain the prompt word text corresponding to the target audio data, and then the pre-trained embedding model can be used to encode the prompt word text corresponding to the target audio data, and the prompt word text can be converted into a series of text vectors to obtain a prompt word tag sequence.

[0118] In step 312, the third audio tag sequence and the prompt word tag sequence are inserted into the initial tag sequence to obtain a second target tag sequence.

[0119] Here, the third audio tag sequence and the prompt word tag sequence are added to the initial tag sequence to obtain a second target tag sequence.

[0120] In some embodiments, the initial tag sequence includes a second audio tag sequence and a text tag sequence, and the second audio tag sequence is located before the text tag sequence. Inserting the third audio tag sequence and the prompt word tag sequence into the initial tag sequence to obtain a second target tag sequence can be achieved by performing the following steps: inserting the prompt word tag sequence after the second audio tag sequence and before the text tag sequence, and inserting the third audio tag sequence after the text tag sequence to obtain the second target tag sequence.

[0121] Here, the second target tag sequence includes a sequence start tag, a prompt word tag sequence, a text tag sequence, a second audio tag sequence, a third audio tag sequence, an additional information tag, and a sequence end tag. The sequence start tag is located at the head of the second target tag sequence, the second audio tag sequence is spliced ​​after the sequence start tag, the prompt word tag sequence is spliced ​​after the second audio tag sequence, the text tag sequence is spliced ​​after the prompt word tag sequence, the additional information tag is spliced ​​after the text tag sequence, the third audio tag sequence is spliced ​​after the additional information tag, and the sequence end tag is located at the tail of the second target tag sequence.

[0122] Continue to refer Figure 3D In step 313, based on the target audio data, feature conversion is performed on the second target label sequence to obtain a second audio feature of the second target label sequence.

[0123] Here, the target audio data is encoded to obtain a target audio sequence. The optimal transmission path between the target audio sequence and the second target tag sequence is calculated through the optimal transmission condition stream matching model. The second target tag sequence is converted into a mel-spectrogram based on the optimal transmission path, that is, the second audio feature of the second target tag sequence.

[0124] In step 314, the second audio feature is converted into speech to obtain second audio data corresponding to the first text.

[0125] Here, the generative adversarial network model is called to perform waveform extraction on the second audio feature to obtain a second audio waveform. Audio data matching the second audio waveform is determined from the audio database, and the matching audio data is determined as the second audio data corresponding to the first text.

[0126] In an embodiment of the present application, when the first audio data and the target audio data are in the same language, the third audio mark sequence and the prompt word mark sequence corresponding to the target audio data are inserted into the initial mark sequence to obtain a second target mark sequence, and based on the second target mark sequence, personalized timbre unification of the voice data in the same language can be achieved.

[0127] In some embodiments, the map data processing method provided in the embodiments of the present application can be applied in the field of robot voice interaction. From the candidate audio data corresponding to each of the N candidate timbres, the target audio data corresponding to the target timbre is determined, so as to determine the audio data of the personalized timbre from the audio data of multiple timbres, and then the first audio data and the first text corresponding to the first audio data are obtained, and the first audio data and the first text are encoded to obtain an initial tag sequence. When the first text includes at least two languages, the language conversion tag is added to the initial tag sequence to obtain the first target tag sequence, which can convert different languages ​​in the text, and then based on the target audio data, the first target tag sequence is feature converted to obtain the first audio feature of the first target tag sequence, and then the first audio feature is voice converted to obtain the second audio data corresponding to the first text, and the timbre of the second audio data is the target timbre. In this way, based on the audio data of the personalized timbre, other audio data are voice converted, so that the robot can unify the personalized timbre of voice data in different languages.

[0128] The following describes an exemplary application of the voice data processing method provided in an embodiment of the present application in a robot voice interaction scenario.

[0129] With the development of robot technology, the function of robot voice interaction has become a hot topic in research and development, which helps to improve the usability and user experience of robots. In the robot's voice interaction system, speech synthesis technology plays a vital role. Speech synthesis technology aims to convert text information into natural and fluent speech, so that robots can communicate with users in a way that is familiar to humans.

[0130] Early speech synthesis technology was mainly based on two methods: rule synthesis and parameter synthesis. Rule synthesis synthesizes speech through artificially set linguistic rules, but the synthesized speech has low naturalness and limited adaptability. Parameter synthesis synthesizes speech by modeling the parameters of speech signals. Although it improves the speech quality to a certain extent, it still has problems such as single timbre and insufficient expressiveness. With the emergence of statistical parameter speech synthesis technology, this technology can achieve more natural and fluent speech synthesis by statistically analyzing and modeling a large amount of speech data, but it still has certain limitations when processing complex text and emotional expressions. In recent years, the breakthrough of speech synthesis technology is due to the development of deep learning technology. Speech synthesis technology based on deep neural network can automatically learn the feature representation of speech, improve the naturalness and expressiveness of speech synthesis. Its essence is a cascade model composed of acoustic model plus vocoder. For example, the use of deep learning models such as recurrent neural network (RNN) and long short-term memory network (LSTM) can better process long sequences of speech data and train more coherent speech synthesis models. At the same time, combining technologies such as Generative Adversarial Networks (GAN) as a vocoder can further improve the quality and realism of synthesized speech.

[0131] The related technology has the following problems in the speech synthesis process:

[0132] 1) Different users have different needs, for example, different timbre preferences. The timbre of synthesized speech is affected by the training data, that is, the gender, timbre and speaking style of the training data, and there is also a problem of insufficient data to meet the timbre preferences;

[0133] 2) In robot voice interaction, support for multiple languages ​​is also one of the important challenges facing speech synthesis technology. The differences between training data in different languages ​​will lead to inconsistent timbre and unnatural transitions in the output of the trained speech synthesis model. The robot cannot communicate effectively with users from different regions. The unification of the timbre of mixed Chinese and English voices is also a problem that needs to be solved urgently.

[0134] In view of the limited training data in the related art, the lack of richness of timbre, resulting in less timbre of the robot's speech synthesis, and the difference in training data for mixed voices such as Chinese and English and the lack of training data, resulting in differences in the timbre of the synthesized speech and unnatural transitions, the embodiment of the present application proposes a speech data processing method, which includes the following improvements compared to the related art:

[0135] 1) By setting the timbre seed (the candidate timbre parameters mentioned above), the text-to-speech model chatTTS can generate diversified timbre data;

[0136] 2) Based on the personalized voice selected by the user, the personalized voice is rhythmically reproduced or cloned across languages ​​through the speech synthesis model CosyVoice, thereby achieving the unified voice of personalized Chinese and English.

[0137] In the embodiment of the present application, by combining voice data of different timbres and a speech synthesis model, a variety of personalized timbres are obtained for users to choose from, and then the voice of the personalized timbre is replicated or cloned across languages, and the voice data generated is used in the speech synthesis model on the robot, so that the robot can achieve personalized timbre unification of Chinese and English voices during the voice interaction process, thereby improving the robot's voice interaction capabilities.

[0138] Example, reference Figure 4 , Figure 4 The electronic device implementing the voice data processing method of the embodiment of the present application may be a server, combined with Figure 4 The various steps shown explain the voice data processing flow provided in the embodiment of the present application.

[0139] In step 401, it is determined whether a control symbol is inserted into the text.

[0140] You can choose whether to insert control symbols into the input text. Control symbols are used to adjust the content of the text. For example, the control symbols are [l augh] and [uv_break]. [laugh] represents laughter and [uv_break] represents a pause. In the speech synthesis result of the text "Hello [uv_break], nice to meet you [la ugh]", [laugh] will be replaced by laughter and a pause will be added at [uv_break].

[0141] When the input text chooses to insert a control symbol, the control symbol is inserted into the input text through the Llama model, such as: [uv_break], [laugh], etc., to obtain text with control symbols; when the input text chooses not to insert a control symbol, it goes to step 402 to determine whether to redefine the text, that is, to regenerate the text content, for example: the input text is "I want to sleep", and the redefined text is "I just want to sleep".

[0142] In step 403, the text with the control symbol and the timbre seed number is converted into a discrete sequence audio_token.

[0143] Through the Llama model, the text with control symbols and timbre seed numbers is converted into a discrete sequence audio_token. The Llama model determines whether to perform an operation based on the control symbols in the text. For example, if a control symbol [e mpty_spk] is inserted before or after the information representing speech in the text, it is equivalent to adding a pause symbol. There is a pause in this speech information. The text is feature-converted through the Llama model, that is, the text content becomes a discrete sequence. The discrete sequence generated by this process is called audio_token.

[0144] In step 404, the discrete sequence audio_token is converted into mel-spectrogram features.

[0145] Through the diffusion variational encoder model (DVAE), that is, the acoustic model, the discrete sequence audio_token is converted into the mel-spectrogram feature mel_spec. The DVAE model combines the variational autoencoder (VAE) and the generative model of the diffusion process. By learning the potential distribution of speech data, new speech data samples are generated, which includes the modeling of the diffusion process and the learning of latent variables. It can generate speech data with various timbres. For examples, refer to Figure 5 , Figure 5 Schematic diagram of the mel-spectrogram diffusion process provided in the embodiment of the present application. Mel-spectrogram X0 is obtained by diffusion sampling to obtain mel-spectrogram X t-1 , Mel-spectrogram X t-1 After diffusion sampling, the mel spectrogram X is obtained t .

[0146] Continue to refer Figure 4 , in step 405, output the speech.

[0147] The vocoder model Vocos is used to convert the converted Mel-spectrogram features into speech. Since the acoustic model will generate a variety of different Mel-spectrogram features, the vocoder model is used to convert the Mel-spectrogram features into waveforms to export speech with different timbres for users to choose from. The personalized voice is customized according to the timbre selected by the user, and the speech data of other languages ​​is replicated or cloned across languages ​​based on the personalized voice.

[0148] In step 406, speech marking is performed.

[0149] Example, reference Figure 6 , Figure 6Schematic diagram of the speech tagging process provided by the embodiment of the present application. A vector quantization layer is inserted between the first encoder 601 and the second encoder 603, and the input speech is subjected to Mel spectrogram (Mel) feature extraction to obtain a Mel spectrogram, which is used as the input of the first encoder 601 and is encoded in combination with position coding to obtain a context-aware representation (the above-mentioned audio representation vector). For example, the context-aware representation can be determined by the following formula (1):

[0150] H1=Encoder1(posEnc(X)) (1)

[0151] Among them, H1 represents context-aware representation, Encoder1 represents the first encoder, posEnc represents position encoding, and X represents Mel-spectrogram.

[0152] The discrete tags are obtained by the vector quantizer 602, and combined with a set of audio representation vectors to obtain the speech tags at the lth time frame. For example, the speech tags at the lth time frame can be determined by the following formula (2):

[0153]

[0154] Where u represents the speech mark of the lth time frame, c n represents the speech vector of the lth time frame in the audio representation vector, C represents the speech vector set, VQ represents the vector quantizer, h l represents the hidden layer representation of the lth time frame.

[0155] The speech vector will be updated during the training process and can be updated by the exponential moving average method. For example, the updated speech vector of the lth time frame can be determined by the following formula (3):

[0156] c ul =ac ul +(1-a)h l (3)

[0157] Among them, c ul represents the updated speech vector of the lth time frame, a represents the attenuation coefficient, h l represents the hidden layer representation of the lth time frame.

[0158] The updated speech vector of each time frame is used as the quantized hidden layer representation H2 = {c u1 ,c u2 ,..}, and combined with the position encoding as the input of the second encoder, for example, the output of the second encoder can be determined by the following formula (4):

[0159] H3=Encoder2(PosEnc(H2)) (4)

[0160] Among them, H3 represents the output of the second encoder, Encoder2 represents the second encoder, and posEnc represents position encoding.

[0161] The output of the second encoder is used as the input of the Automatic Speech Recognition (ASR) decoder. In the ASR decoder 604, the output is the posterior probability P(Y|X) of the input speech corresponding to the text label. The posterior probability refers to the probability that a speech signal corresponds to a text label, that is, the probability of each text appearing in the content of the speech.

[0162] Continue to refer Figure 4 , in step 407, the text is encoded.

[0163] The input text is encoded through Byte Pair Encoding (BPE) to obtain a text vector.

[0164] In step 408, the sequences of text and speech tokens are aligned.

[0165] Each speech tag is aligned with the corresponding text vector to obtain a sequence of text and speech tags.

[0166] In step 409, the entire sequence of text and speech is learned.

[0167] Construct the input sequence of the large language model. The input sequence mainly consists of the sequence start marker, the embedded vector of the extracted speech sound, the text vector, and the sequence end marker. Since the text and speech markers are at different semantic levels, a start marker is inserted between the two, and the original sequence is used as the expected output during the training phase. During training, only the cross entropy loss of the speech marker and the sequence end marker is considered to obtain a sequence consisting of the speech marker and the text vector. For example, the cross entropy loss can be determined by the following formula (5):

[0168]

[0169] Among them, L LM represents the cross entropy loss, q(u l ) represents the posterior probability (predicted by the normalized softmax layer of the large language model).

[0170] In step 411, the tags in the input sequence are adjusted according to the speech of the reference timbre.

[0171] The selected personalized voice is used as the reference voice. When the reference voice and the input voice are in the same language, the context learning zero-shot of the large language model can be used to achieve a short reference voice sample to copy the timbre of the reference voice. This process builds the input sequence of the large language model according to actual needs, and merges the reference voice token Prompt SpeechToken (the third audio token sequence mentioned above) and the prompt voice text token Prompt Text (the prompt word token sequence mentioned above) into a unified input sequence, where the prompt voice token is pre-generated and the subsequent token is predicted by the autoregressive language model. For example, refer to Figure 7 , Figure 7 701 is a schematic diagram of the structure of the input sequence of the same language provided in the embodiment of the present application. The input sequence of the same language includes input speech 701, prompt speech text 702, input text 703 and reference speech 704.

[0172] When the reference speech (the target audio data mentioned above) and the input speech (the first audio data mentioned above) are in different languages, that is, the sequence composition of cross-language cloning does not require the reference speech and prompt word text, but needs to insert the language conversion marker LID, and the rest of the sequence structure is consistent with the input sequence of the same language. Figure 8 , Figure 8 701 is a schematic diagram of the structure of the input sequence of different languages ​​provided in the embodiment of the present application. The input sequence of different languages ​​includes input speech 701, input text 703 and language conversion mark 705.

[0173] Continue to refer Figure 4 In step 412, the speech tokens are converted into mel-spectrograms using a conditional stream matching model.

[0174] The principle of the Conditional Flow Matching (OT-FCM) model is to construct a probability density path from the prior distribution p(x) to the data distribution of the Mel-spectrogram q(x) in the continuous time normalized flow. The probability density path is represented by a time-dependent vector field. For example, the probability density path can be determined by the following formula (6):

[0175]

[0176] Where t ranges from 0 to 1. By solving the initial value problem equation, the Mel-spectrogram q(x) can be approximated with the prior distribution p(x) and sampled from it. The vector field can be learned through the optimal transmission condition flow. For example, the minimization loss can be determined by the following formulas (7) to (9):

[0177]

[0178] The speaker embedding vector, speech tag, and mel-spectrogram are all input into the neural network to match the vector field with learnable parameters. For example, the vector field can be determined by the following formula (10):

[0179]

[0180] Continue to refer Figure 4 , in step 413, the voice is converted.

[0181] The mel-spectrogram is used as the input of the vocoder to convert it into personalized speech with uniform timbre. The vocoder used here is the generative adversarial network HiFiGAN, which achieves personalized speech output with the same timbre in Chinese and English.

[0182] In the above-mentioned application scenario of robot voice interaction, by applying the chatTTS model and CosyVoice model to the robot's voice synthesis model, the effect of voice synthesis is improved, multi-timbre support and timbre unification of different languages ​​​​in Chinese and English are added, so that the robot can show a more natural, real and emotional personalized voice during voice interaction, and the timbre is consistent and the transition is natural when synthesizing mixed Chinese and English sentences. It provides a solution for personalized timbre customization and mixed language timbre unification for the robot's voice interaction, which significantly improves the robot's interactive ability.

[0183] The following is a description of an exemplary structure of the voice data processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the map data processing device 455 stored in the memory 450 may include: a timbre determination module 4551, which is used to determine the target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbres; a first marking module 4552, which is used to obtain the first audio data and the first text corresponding to the first audio data, and encode the first audio data and the first text to obtain an initial marking sequence; a second marking module 4553, which is used to add a language conversion mark to the initial marking sequence when the first text includes at least two languages, and obtain a first target marking sequence; a feature conversion module 4554, which is used to perform feature conversion on the first target marking sequence based on the target audio data to obtain the first audio feature of the first target marking sequence; a speech conversion module 4555, which is used to perform speech conversion on the first audio feature to obtain the second audio data corresponding to the first text, and the timbre of the second audio data is the target timbre.

[0184] In some embodiments, the timbre determination module 4551 is also used to obtain an initial text before determining the target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbres, and to insert control symbols and rewrite the initial text to obtain a candidate text; determine N candidate timbre parameters, and insert the N candidate timbre parameters into the candidate texts respectively to obtain N target texts; and perform speech conversion on the N target texts to obtain candidate audio data corresponding to each of the N candidate timbres.

[0185] In some embodiments, the timbre determination module 4551 is also used to perform feature conversion on each target text to obtain a first audio tag sequence; perform feature extraction on the first audio tag sequence to obtain a first audio feature; and perform speech conversion on the first audio feature to obtain candidate audio data.

[0186] In some embodiments, the first labeling module 4552 is also used to extract features from the first audio data to obtain first audio features, and position encode the first audio features to obtain an audio representation vector; quantize the audio representation vector to obtain a second audio label sequence; encode the first text to obtain a text label sequence; and combine the second audio label sequence and the text label sequence to obtain an initial label sequence.

[0187] In some embodiments, the first tag module 4552 is further used to concatenate the text tag sequence after the second audio tag sequence to obtain an initial tag sequence; and concatenate the language conversion tag after the text tag sequence to obtain a first target tag sequence.

[0188] In some embodiments, the speech conversion module 4555 is also used to obtain a generative adversarial network model, and call the generative adversarial network model to perform waveform extraction on the first audio feature to obtain a first audio waveform; determine audio data that matches the first audio waveform from an audio database, and determine the matching audio data as second audio data corresponding to the first text.

[0189] In some embodiments, the speech conversion module 4555 is also used to, when the first audio data and the target audio data are in the same language, encode the target audio data to obtain a third audio mark sequence; determine the prompt word text corresponding to the target audio data, and encode the prompt word text to obtain a prompt word mark sequence; insert the third audio mark sequence and the prompt word mark sequence into the initial mark sequence to obtain a second target mark sequence; based on the target audio data, perform feature conversion on the second target mark sequence to obtain a second audio feature of the second target mark sequence; perform speech conversion on the second audio feature to obtain second audio data corresponding to the first text.

[0190] In some embodiments, the initial tag sequence includes a second audio tag sequence and a text tag sequence, and the second audio tag sequence is located before the text tag sequence. The speech conversion module 4555 is also used to insert the prompt word tag sequence after the second audio tag sequence and before the text tag sequence, and insert the third audio tag sequence after the text tag sequence to obtain a second target tag sequence.

[0191] The embodiment of the present application provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions or computer programs from the computer readable storage medium, and the processor executes the computer executable instructions or computer programs, so that the electronic device performs the above-mentioned voice data processing method of the embodiment of the present application.

[0192] The present application embodiment provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will be caused to execute the voice data processing method provided by the present application embodiment, for example, Figure 3A The voice data processing method shown.

[0193] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface storage, optical disk, or CD-ROM; or it may be various devices including one or any combination of the above memories.

[0194] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0195] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0196] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0197] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for processing speech data, characterized in that: The method comprises: Determining target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbre; Acquire first audio data and a first text corresponding to the first audio data, and encode the first audio data and the first text to obtain an initial tag sequence; When the first text includes at least two languages, adding a language conversion marker to the initial marker sequence to obtain a first target marker sequence; Based on the target audio data, feature conversion is performed on the first target tag sequence to obtain a first audio feature of the first target tag sequence; The first audio feature is converted into speech to obtain second audio data corresponding to the first text, wherein the timbre of the second audio data is a target timbre.

2. The method according to claim 1, characterized in that Before determining the target audio data corresponding to the target timbre from the candidate audio data corresponding to each of the N candidate timbres, the method further includes: Acquire an initial text, insert control symbols and rewrite the initial text to obtain a candidate text; Determine N candidate timbre parameters, and insert the N candidate timbre parameters into the candidate texts respectively to obtain N target texts; The N target texts are converted into voice, and candidate audio data corresponding to the N candidate timbres are obtained.

3. The method according to claim 2, characterized in that The performing speech conversion on the N target texts to obtain candidate audio data corresponding to the N candidate timbres respectively includes: For each of the target texts, perform feature conversion on the target text to obtain a first audio tag sequence; Performing feature extraction on the first audio tag sequence to obtain a first audio feature; Perform voice conversion on the first audio feature to obtain candidate audio data.

4. 根据权利要求1所述的方法 ， 其特征在于 ， 所述对所述第 一 音频数据和所述第 一 文本进行编码处 Processing, get the initial tag sequence, including: Performing feature extraction on the first audio data to obtain a first audio feature, and performing position encoding on the first audio feature to obtain an audio representation vector; quantizing the audio representation vector to obtain a second audio label sequence; Encoding the first text to obtain a text tag sequence; The second audio tag sequence and the text tag sequence are combined to obtain an initial tag sequence.

5. 根据权利要求4所述的方法 ， 其特征在于 ， 所述对所述第二音频标记序列和所述文本标记序列进 The rows are combined and processed to obtain the initial tag sequence, including: concatenating the text tag sequence after the second audio tag sequence to obtain the initial tag sequence; The step of adding the language conversion token to the initial token sequence to obtain a first target token sequence includes: The language conversion tag is concatenated to the text tag sequence to obtain the first target tag sequence.

6. 根据权利要求1所述的方法 ， 其特征在于 ， 所述对所述第 一 音频特征进行语音转换 ， 得到所述第 The second audio data corresponding to a text includes: Obtaining a generative adversarial network model, and calling the generative adversarial network model to perform waveform extraction on the first audio feature to obtain a first audio waveform; Audio data matching the first audio waveform is determined from an audio database, and the matching audio data is determined as second audio data corresponding to the first text.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: When the first audio data and the target audio data are in the same language, encoding the target audio data to obtain a third audio tag sequence; Determine the prompt word text corresponding to the target audio data, and encode the prompt word text to obtain a prompt word tag sequence; Inserting the third audio tag sequence and the prompt word tag sequence into the initial tag sequence to obtain a second target tag sequence; Based on the target audio data, performing feature conversion on the second target label sequence to obtain a second audio feature of the second target label sequence; The second audio feature is converted into speech to obtain second audio data corresponding to the first text.

8. 根据权利要求7所述的方法 ， 其特征在于 ， 所述初始标记序列包括所述第二音频标记序列和所述 A text tag sequence, and the second audio tag sequence is located before the text tag sequence, The step of inserting the third audio tag sequence and the prompt word tag sequence into the initial tag sequence to obtain a second target tag sequence includes: The prompt word tag sequence is inserted after the second audio tag sequence and before the text tag sequence, and the third audio tag sequence is inserted after the text tag sequence to obtain the second target tag sequence.

9. A voice data processing device, characterized in that: The device comprises: A timbre determination module, used to determine target audio data corresponding to a target timbre from candidate audio data corresponding to each of N candidate timbres; A first marking module, used to obtain first audio data and a first text corresponding to the first audio data, and encode the first audio data and the first text to obtain an initial marking sequence; A second marking module, configured to add a language conversion mark to the initial mark sequence to obtain a first target mark sequence when the first text includes at least two languages; a feature conversion module, configured to perform feature conversion on the first target tag sequence based on the target audio data to obtain a first audio feature of the first target tag sequence; The speech conversion module is used to perform speech conversion on the first audio feature to obtain second audio data corresponding to the first text, wherein the timbre of the second audio data is a target timbre.

10. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; The processor is used to implement the voice data processing method according to any one of claims 1 to 8 when executing the computer executable instructions or computer programs stored in the memory.

11. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the voice data processing method according to any one of claims 1 to 8 is implemented.

Images