Timbre conversion method and device, electronic equipment, storage medium and program product

Abstract

This application provides a timbre conversion method, apparatus, electronic device, storage medium, and program product. The method includes: acquiring first audio data; extracting a first prosodic feature, semantic feature, and non-semantic feature from the first audio data; wherein the first prosodic feature represents a preset timbre and a first prosodic pattern corresponding to the preset timbre; determining a second prosodic feature based on the first prosodic feature and pre-determined target timbre data; wherein the second prosodic feature represents the target timbre and the first prosodic pattern; fusing the semantic feature and the non-semantic feature based on the second prosodic feature to obtain a speech fusion feature; and determining second audio data based on the speech fusion feature and the second prosodic feature.

Description

Technical Field

[0001] This application relates to the field of electronic digital data processing technology, and in particular to a timbre conversion method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] Speech conversion is a technology that can alter a speaker's voice while preserving semantic information. It can be used in fields such as film and television dubbing, privacy protection, and personalized speech synthesis. With technological advancements, speech conversion technology is increasingly focused on converting highly expressive speech. The goal of highly expressive speech conversion is to achieve excellent conversion results for input audio that is emotionally charged and contains rich non-semantic content (crying, laughing, coughing, etc.). However, most existing speech conversion technologies attempt to balance speaker similarity, intelligibility, and expressiveness, but cannot achieve both simultaneously. Summary of the Invention

[0003] In view of this, the purpose of this application is to provide a timbre conversion method, apparatus, electronic device, storage medium and program product to achieve a highly expressive speech conversion effect on input audio containing non-semantic content.

[0004] To achieve the above objectives, this application provides a timbre conversion method, the method comprising:

[0005] Acquire first audio data, and extract first prosodic features, semantic features, and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic pattern corresponding to the preset timbre;

[0006] A second prosodic feature is determined based on the first prosodic feature and pre-determined target timbre data; wherein, the second prosodic feature represents the target timbre and the first prosodic feature;

[0007] The semantic features and non-semantic features are fused based on the second prosodic features to obtain speech fusion features;

[0008] The second audio data is determined based on the speech fusion features and the second prosodic features.

[0009] For the same inventive purpose, this application also provides a timbre conversion device, the device comprising:

[0010] The acquisition module is configured to acquire first audio data and extract first prosodic features, semantic features, and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic feature corresponding to the preset timbre.

[0011] The feature extraction module is configured to determine a second prosodic feature based on the first prosodic feature and pre-determined target timbre data; wherein the second prosodic feature represents the target timbre and the first prosodic feature;

[0012] The fusion module is configured to fuse the semantic features and the non-semantic features based on the second prosodic features to obtain speech fusion features;

[0013] The speech generation module is configured to determine second audio data based on the speech fusion features and the second prosodic features.

[0014] For the purposes described above, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the timbre conversion method as described in any of the above claims.

[0015] For the purposes described above, this application also provides a non-transitory computer-readable storage medium that stores computer instructions for causing the computer to execute the timbre conversion method described in any of the above-described methods.

[0016] For the purposes described above, this application also provides a computer program product, including computer program instructions that, when the computer program instructions are run on a computer, cause the computer to perform the timbre conversion method as described in the first aspect.

[0017] As can be seen from the above description, the timbre conversion method, apparatus, electronic device, storage medium, and program product provided in this application first acquire first audio data, and extract first prosodic features, semantic features, and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic corresponding to the preset timbre; further, a second prosodic feature is determined based on the first prosodic features and pre-determined target timbre data; wherein, the second prosodic feature represents the target timbre and the first prosodic; further, the semantic features and non-semantic features are fused based on the second prosodic features to obtain speech fusion features; finally, second audio data is determined based on the speech fusion features and the second prosodic features. This application can convert the preset timbre of the original speaker to the target timbre while ensuring that the semantics and prosody of the original speaker remain unchanged, and completely extracts and converts the non-semantic information in the original speaker's audio. The converted audio data can not only accurately change the timbre of the audio, but also completely preserve the non-semantic content in the original speaker's audio, achieving a low-response-latency, high-efficiency, and accurate real-time voice changing effect. It achieves the technical effect of capturing highly expressive non-semantic content from the original speaker's audio, as well as excellent conversion between the original speaker's timbre and the target timbre, while maintaining the high definition of the converted audio. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only for this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram illustrating an application scenario of a timbre conversion method provided in an embodiment of this application.

[0020] Figure 2 This is a flowchart illustrating a timbre conversion method provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of the structure of a tone conversion device provided in an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of the hardware structure of an electronic device provided in this embodiment. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the principles and spirit of this application will be described below with reference to several exemplary embodiments. It should be understood that these embodiments are provided merely to enable those skilled in the art to better understand and implement this application, and are not intended to limit the scope of this application in any way. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0024] According to embodiments of this application, a timbre conversion method, apparatus, electronic device, storage medium, and program product are proposed.

[0025] In this article, it is important to understand that any number of elements in the accompanying figures is for illustrative purposes and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0026] It should be noted that, unless otherwise defined, the technical or scientific terms used in the embodiments of this application should have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms "first," "second," and similar terms used in the embodiments of this application do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as "comprising" or "including" mean that the element or object preceding the word encompasses the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Terms such as "connected" or "linked" are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as "upper," "lower," "left," and "right" are only used to indicate relative positional relationships; when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0027] The technical solution of this application will be further described in detail below through specific embodiments.

[0028] refer to Figure 1This diagram illustrates an application scenario for a timbre conversion method provided in this application. The application scenario includes a terminal device 101, a server 102, and a data storage system 103. The terminal device 101, server 102, and data storage system 103 can all be connected via wired or wireless communication networks. The terminal device 101 includes, but is not limited to, desktop computers, mobile phones, mobile computers, tablets, game consoles, media players, smart wearable devices, personal digital assistants (PDAs), or other electronic devices capable of performing the aforementioned functions. The server 102 and data storage system 103 can both be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.

[0029] Server 102 provides voice conversion services to users of terminal device 101. Terminal device 101 has a client installed that communicates with server 102. Users can input one or more audio data segments to be converted through the client. The client sends the first audio data to be converted to server 102. Server 102 processes the first audio data, extracting the first prosodic features, semantic features, and non-semantic features of the first audio data. Based on the first prosodic features and the pre-determined target voice data, it determines the second prosodic features. Based on the second prosodic features, it re-fuses the semantic features and non-semantic features to obtain speech fusion features. Finally, based on the obtained speech fusion features and the second prosodic features, it determines the second audio data, completing the voice conversion. Server 102 sends the converted second audio data to the client, and the client plays the voice converted speech to the user.

[0030] The data storage system 103 stores a large amount of training data, including a pre-acquired speech dataset containing various timbre information. This speech dataset can have better timbre information to represent speech data. The sources of the training data include, but are not limited to, existing databases, data crawled from the Internet, or data uploaded by users when using the client.

[0031] The following is combined Figure 1 The application scenarios described above illustrate the timbre conversion method according to exemplary embodiments of this application. It should be noted that the above application scenarios are merely shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this regard. Rather, the embodiments of this application can be applied to any applicable scenario.

[0032] refer to Figure 2 This is a flowchart illustrating a timbre conversion method provided in an embodiment of this application.

[0033] Step S201: Obtain first audio data, and extract first prosodic features, semantic features, and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic pattern corresponding to the preset timbre.

[0034] In practice, the first audio data refers to the speech data that needs to be converted to different timbres. Different sound-producing bodies produce different timbres due to their different materials and structures. For example, the sounds produced by a piano, a violin, and a person are different, and the sounds produced by each person are also different. Therefore, timbre reflects the unique quality of the sound produced by each object.

[0035] The information conveyed in human speech can be broadly categorized into prosody, semantics, and non-semantics, representing rhythm, emotion, language, and speaker identity, respectively. Generally, speaker identity can be represented by a pre-defined timbre. Furthermore, non-semantic sounds, such as breathing, laughter, and crying, are also essential in verbal communication. Voice-to-text (VC) is a technique that alters relevant speaker information in a given audio recording to make it sound like another speaker while preserving other aspects of the speech.

[0036] In practice, a meaningful audio data segment can be the audio of a speaker's speech over a period of time, containing semantic and non-semantic components. In order to perform better speech conversion, it is usually necessary to decouple the various information in the audio for separate analysis.

[0037] As an alternative embodiment, the pitch of the first audio data can be determined and used as the first prosodic feature.

[0038] Pitch refers to the different levels of sound, that is, the height of a sound, and is one of the basic characteristics of sound. Volume can also be understood as the energy of sound; volume, also known as intensity or loudness, refers to the subjective perception of the strength or weakness of a sound heard, and its objective evaluation scale is the amplitude of the sound. In the process of speech conversion, since the converted speech mainly contains semantic content, it largely loses the expressive content of the original audio data. Therefore, this application additionally uses prosodic features to address this issue.

[0039] As an optional embodiment, during the audio playback process, the sound may be sometimes loud and sometimes soft. Therefore, it is necessary to perform normalization processing on the audio. The normalization processing includes two types: one is peak normalization processing, and the other is pitch normalization processing. Peak normalization processing can be understood as adjusting the place with the largest audio peak in the audio to a peak of a specific size, and then performing corresponding enlargement or reduction processing on the remaining peaks. In the embodiment of the present application, it is preferable to perform pitch normalization processing on the first audio data, calculate the average pitch of the first audio data, and then obtain the expected volume difference based on the average pitch as the adjustment basis. Its advantage is simplicity. Only by finding the maximum pitch and performing processing based on the maximum pitch can the overall volume adjustment be determined. The processed pitch is the first prosodic feature, where the first prosodic feature represents a preset timbre and the first prosody corresponding to the preset timbre.

[0040] Semantic features refer to the units of meaning (or content) in language. Semantic features are the smallest meaning units that make up words or sentences, that is, the distinctive features of words or sentences. It is a semantic unit at a lower level than words or sentences obtained by decomposing words or sentences. It belongs to the microscopic level of semantics because it cannot be directly observed in the semantic system. For example, for the word "sister", the following semantic components can be obtained: [relative], [sibling], [older], [female]. For the semantic unit "success", the following semantic components can be obtained: [competition], [defeat the opponent].

[0041] As an optional embodiment, the semantic features of the first audio data can be extracted by a pre-trained semantic extraction model (ASR). The first audio data is input into a pre-constructed automatic speech recognition model to determine the bottleneck features of the first audio data.

[0042] In the process of voice conversion, the most core task is to extract the text information and prosodic information of the source speaker's audio, which is essentially to remove the timbre information in the source speaker's audio features. Among them, the text information of the original speaker's audio can be represented by bottleneck features, also known as bnf features.

[0043] The training target of the semantic extraction model (ASR) is text, and the bottleneck features it extracts only contain semantic information, which can remove the expressive information and non-semantic information in the first audio data. Therefore, in the present application, the first audio data is input into the semantic extraction model to extract the bottleneck features of the first audio data. Further, the bottleneck features are input into the BNF encoder to further extract the context information related to the bottleneck features. By integrating the context information, the converted audio can sound more expressive and coherent at the connection points within and between sentences. For content with rich variations and high expressiveness, the speech synthesized from the context model will be more stable and natural.

[0044] As an optional embodiment, non-semantic features of the first audio data can be extracted using a pre-built perturbed-wav model. The first audio data can be input into the pre-built perturbed-wav model to determine the perturbed waveform features of the first audio data.

[0045] The speech perturbation model can remove speaker information from the audio by adjusting the fundamental frequency and formants of the first audio data, thereby obtaining the perturbation waveform features of the first audio data. Furthermore, the perturbation waveform features are input into the perturbation waveform encoder to obtain non-semantic features.

[0046] Step S202: Determine a second prosodic feature based on the first prosodic feature and the predetermined target timbre data; wherein the second prosodic feature represents the target timbre and the first prosodic feature.

[0047] In this embodiment, in order to better preserve the prosodic features of the first audio data during feature fusion and to make the timbre-converted audio have a higher similarity, a prosodic encoder can be used to embed the target timbre into the first prosodic feature to obtain a second prosodic feature that can characterize the target timbre.

[0048] As an optional embodiment, the target timbre features of the target timbre data can first be determined. Further, the target timbre features and the pitch obtained in step S201 are input into the prosody encoder to determine the second prosody features.

[0049] Specifically, pitch can be normalized to identify the point with the largest pitch amplitude. This amplitude is then increased to near 1, and the increase percentage is recorded. All other points are then stretched according to this percentage. This process removes the original speaker's timbre from the first audio data, generating a prosody unrelated to the original speaker. A conditional normalization layer is then used to embed the target timbre feature as conditional information, resulting in a second prosodic feature. This second prosodic feature can represent both the target timbre and the first prosodic feature in the first audio data that is unrelated to the original speaker.

[0050] Step S203: The semantic features and the non-semantic features are fused according to the second prosodic features to obtain speech fusion features.

[0051] In practical implementation, an attention-based fusion model can be selected to fuse semantic and non-semantic features. The attention mechanism can focus on important information with high weights and ignore irrelevant information with low weights. Furthermore, it can continuously adjust the weights so that important information can be selected under different circumstances, thus exhibiting higher scalability and robustness.

[0052] As an optional embodiment, the second prosodic feature, semantic feature, and non-semantic feature can be input into an attention-based fusion model, and the dot product of the second prosodic feature and the semantic feature can be calculated. The weight value of the semantic feature is determined based on the dot product value. Finally, the semantic feature and the non-semantic feature are fused according to the weight value to obtain the speech fusion feature.

[0053] As an optional embodiment, since a rough judgment of speech content can be made based on prosodic features, in the attention-based fusion model, the second prosodic feature is used as the query, the semantic feature is used as the key, and the non-semantic feature is used as the value to calculate the weight vector, and then the semantic feature and the non-semantic feature are weighted and summed.

[0054] In the context of attention mechanisms, we refer to autonomous cues as queries. Given any query, the attention mechanism guides selection towards sensory inputs (e.g., intermediate feature representations) through attention pooling. In attention mechanisms, these sensory inputs are called values. More simply, each value is paired with a key, which can be thought of as a non-autonomous cue for the sensory input. Attention pooling is designed so that a given query (autonomous cue) can be matched with a key (non-autonomous cue), leading to the most matching value (sensory input). The attention mechanism combines the query (autonomous cue) and the key (non-autonomous cue) through attention pooling, thus achieving a preference for selecting values ​​(sensory inputs). The fused speech features contain complete semantic information while also preserving speaker-independent prosodic and non-semantic information from the initial audio data.

[0055] As an alternative implementation, since the bottleneck feature extracted by the semantic extraction model (ASR) is already a robust semantic feature, the model tends to directly use the bottleneck feature to reconstruct the audio during training, rather than extracting information from the learnable perturbed-wav model. The weights in the attention-based fusion model will be completely biased towards the semantic feature, making it difficult to fully train the perturbed-wav model. Therefore, further improvements in convergence speed and content extraction capabilities are needed.

[0056] As an alternative implementation, non-semantic features are directly fused with second prosodic features to obtain non-semantic enhancement features. By bypassing semantic features and directly fusing non-semantic features with second prosodic features, the perturbation wave encoder can be trained more fully.

[0057] As can be seen from the above description, the timbre conversion method, apparatus, electronic device, storage medium, and program product provided in this application first acquire first audio data, and extract first prosodic features, semantic features, and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic corresponding to the preset timbre; further, a second prosodic feature is determined based on the first prosodic features and pre-determined target timbre data; wherein, the second prosodic feature represents the target timbre and the first prosodic; further, the semantic features and non-semantic features are fused based on the second prosodic features to obtain speech fusion features; finally, second audio data is determined based on the speech fusion features and the second prosodic features. This application can convert the preset timbre of the original speaker to the target timbre while ensuring that the semantics and prosody of the original speaker remain unchanged, and completely extracts and converts the non-semantic information in the original speaker's audio. The converted audio data can not only accurately change the timbre of the audio, but also completely preserve the non-semantic content in the original speaker's audio, achieving a low-response-latency, high-efficiency, and accurate real-time voice changing effect. It achieves the technical effect of capturing highly expressive non-semantic content from the original speaker's audio, as well as excellent conversion between the original speaker's timbre and the target timbre, while maintaining the high definition of the converted audio.

[0058] It should be noted that the method in this embodiment can be executed by a single device, such as a computer or server. The method can also be applied in a distributed scenario, where multiple devices cooperate to complete the task. In such a distributed scenario, one of these devices may execute only one or more steps of the method in this embodiment, and the multiple devices will interact with each other to complete the method described.

[0059] It should be noted that the above description describes some embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in a different order than that shown in the above embodiments and still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0060] Based on the same inventive concept, corresponding to any of the above embodiments, this application also provides a timbre conversion device.

[0061] refer to Figure 3 This is a schematic diagram of the structure of a tone conversion device provided in an embodiment of this application.

[0062] The acquisition module 301 is configured to acquire first audio data and extract first prosodic features, semantic features and non-semantic features from the first audio data; wherein, the first prosodic features represent a preset timbre and a first prosodic feature corresponding to the preset timbre.

[0063] The feature extraction module 302 is configured to determine a second prosodic feature based on the first prosodic feature and pre-determined target timbre data; wherein the second prosodic feature represents the target timbre and the first prosodic feature;

[0064] The fusion module 303 is configured to fuse the semantic features and the non-semantic features according to the second prosodic features to obtain speech fusion features;

[0065] The speech generation module 304 is configured to determine second audio data based on the speech fusion features and the second prosodic features.

[0066] Optionally, the acquisition module 301 is further configured to:

[0067] The pitch of the first audio data is determined, and the pitch is used as the first prosodic feature.

[0068] Optionally, the acquisition module 301 is further configured to:

[0069] The first audio data is input into a pre-built automatic speech recognition model to determine the bottleneck features of the first audio data, and the context information of the bottleneck features is extracted to obtain the semantic features; wherein, the bottleneck features represent the semantic information of the first audio data; and the context information represents the semantic coherence in the first audio data.

[0070] Optionally, the acquisition module 301 is further configured to:

[0071] The first audio data is input into a pre-built speech perturbation model, and the perturbation waveform features of the first audio data are extracted. The non-semantic features are obtained based on the perturbation waveform features; wherein, the perturbation waveform features characterize the non-semantic information of the first audio data.

[0072] Optionally, the feature extraction module 302 is further configured to:

[0073] Determine the target timbre features of the target timbre data;

[0074] The target timbre feature and the pitch are input into the prosodic encoder to determine the second prosodic feature.

[0075] Optionally, the fusion module 303 is further configured to:

[0076] The second prosodic feature, the semantic feature, and the non-semantic feature are input into an attention-based fusion model to calculate the dot product of the second prosodic feature and the semantic feature.

[0077] The weight value of the semantic feature is determined based on the dot product value;

[0078] The semantic features and non-semantic features are fused according to the weight values ​​to obtain the speech fusion features.

[0079] Optionally, the fusion module 303 is further configured to:

[0080] The non-semantic features are fused with the second prosodic features to obtain non-semantic enhancement features.

[0081] Optionally, the speech generation module 304 is further configured to:

[0082] The speech fusion feature, the second prosodic feature, and the non-semantic enhancement feature are input into the speech decoder to obtain the second speech waveform data;

[0083] The second speech data is generated based on the second speech waveform data.

[0084] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, in implementing this application, the functions of each module can be implemented in one or more software and / or hardware.

[0085] The apparatus described above is used to implement the corresponding timbre conversion method in any of the foregoing embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0086] Figure 4 This illustration shows an exemplary structural diagram of an electronic device provided in an embodiment of this application.

[0087] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the timbre conversion method described in any of the above embodiments. Figure 3 This embodiment illustrates a more specific hardware structure of an electronic device, which may include a processor 310, a memory 320, an input / output interface 330, a communication interface 340, and a bus 350. The processor 310, memory 320, input / output interface 330, and communication interface 340 are interconnected internally via the bus 350.

[0088] The processor 310 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0089] The memory 320 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 320 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 320 and is called and executed by the processor 310.

[0090] Input / output interface 330 is used to connect input / output modules to realize information input and output. Input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touch screens, microphones, various sensors, etc., and output devices may include displays, speakers, vibrators, indicator lights, etc.

[0091] The communication interface 340 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0092] Bus 350 includes a pathway for transmitting information between various components of the device (e.g., processor 310, memory 320, input / output interface 330, and communication interface 340).

[0093] It should be noted that although the above-described device only shows the processor 310, memory 320, input / output interface 330, communication interface 340, and bus 350, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0094] The electronic devices described above are used to implement the corresponding timbre conversion methods in any of the foregoing embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0095] Based on the same inventive concept, corresponding to the methods of any of the above embodiments, this application also provides a non-transitory computer-readable storage medium storing computer instructions for causing the computer to execute the timbre conversion method as described in any of the above embodiments.

[0096] The computer-readable medium of this embodiment includes permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0097] The computer instructions stored in the storage medium of the above embodiments are used to cause the computer to execute the timbre conversion method as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0098] Based on the same inventive concept, corresponding to the timbre conversion method described in any of the above embodiments, this disclosure also provides a computer program product, which includes computer program instructions. In some embodiments, the computer program instructions can be executed by one or more processors of a computer to cause the computer and / or the processor to perform the timbre conversion method. Corresponding to the execution entity for each step in each embodiment of the timbre conversion method, the processor executing the corresponding step can belong to the corresponding execution entity.

[0099] The computer program product of the above embodiments is used to cause the computer and / or the processor to execute the timbre conversion method as described in any of the above embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be repeated here.

[0100] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of this application (including the claims) is limited to these examples; within the framework of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of the embodiments of this application as described above, which are not provided in the details for the sake of brevity.

[0101] Additionally, to simplify the description and discussion, and to avoid obscuring the embodiments of this application, the well-known power / ground connections to integrated circuit (IC) chips and other components may or may not be shown in the provided drawings. Furthermore, the apparatus may be shown in block diagram form to avoid obscuring the embodiments of this application, and this also takes into account the fact that the details of the implementation of these block diagram apparatuses are highly dependent on the platform on which the embodiments of this application will be implemented (i.e., these details should be fully understood by those skilled in the art). While specific details (e.g., circuits) have been set forth to describe exemplary embodiments of this application, it will be apparent to those skilled in the art that the embodiments of this application can be implemented without these specific details or with variations thereof. Therefore, these descriptions should be considered illustrative rather than restrictive.

[0102] Although this application has been described in conjunction with specific embodiments thereof, many substitutions, modifications, and variations of these embodiments will be apparent to those skilled in the art from the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may be used with the embodiments discussed.

[0103] The embodiments of this application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the embodiments of this application should be included within the protection scope of this application.

Claims

1. A timbre conversion method characterized by, The method comprises: acquiring first audio data, extracting first prosody features, semantic features and non-semantic features of the first audio data; wherein the first prosody features represent a preset timbre and a first prosody corresponding to the preset timbre; determining second prosody features according to the first prosody features and predetermined target timbre data; wherein the second prosody features represent a target timbre and the first prosody; inputting the second prosody features, the semantic features and the non-semantic features into a fusion model based on an attention mechanism, and calculating a dot product value of the second prosody features and the semantic features; determining a weight value of the semantic features according to the dot product value; fusing the semantic features and the non-semantic features according to the weight value to obtain speech fusion features; determining second audio data according to the speech fusion features and the second prosody features.

2. The method of claim 1, wherein, The acquiring first audio data and extracting first prosody features of the first audio data comprise: determining a pitch of the first audio data, and taking the pitch as the first prosody features.

3. The method of claim 1, wherein, The acquiring first audio data and extracting semantic features of the first audio data comprise: inputting the first audio data into a pre-constructed automatic speech recognition model, determining bottleneck features of the first audio data, and extracting context information of the bottleneck features to obtain the semantic features; wherein the bottleneck features represent semantic information of the first audio data; and the context information represents coherence of the semantic information in the first audio data.

4. The method of claim 1, wherein, The extracting non-semantic features of the first audio data comprises: inputting the first audio data into a pre-constructed speech perturbation model, extracting perturbation waveform features of the first audio data, and obtaining the non-semantic features according to the perturbation waveform features; wherein the perturbation waveform features represent non-semantic information of the first audio data.

5. The method of claim 2, wherein, The determining second prosody features according to the first prosody features and predetermined target timbre data comprises: determining target timbre features of the target timbre data; inputting the target timbre features and the pitch into a prosody encoder to determine the second prosody features.

6. The method of claim 1, wherein, The method further comprises: fusing the non-semantic features and the second prosody features to obtain non-semantic enhanced features.

7. The method of claim 6, wherein, The determining second audio data according to the speech fusion features and the second prosody features comprises: inputting the speech fusion features, the second prosody features and the non-semantic enhanced features into a speech decoder to obtain second speech waveform data; generating the second audio data according to the second speech waveform data.

8. A timbre conversion apparatus characterized by comprising: The device comprises: an acquisition module configured to acquire first audio data, and extract first prosody features, semantic features and non-semantic features of the first audio data; wherein the first prosody features represent a preset timbre and a first prosody corresponding to the preset timbre; The feature extraction module is configured to determine a second prosody feature according to the first prosody feature and predetermined target timbre data; wherein the second prosody feature characterizes the target timbre and the first prosody; The fusion module is configured to input the second prosody feature, the semantic feature and the non-semantic feature into a fusion model based on an attention mechanism, calculate a dot product value of the second prosody feature and the semantic feature, determine a weight value of the semantic feature according to the dot product value, and fuse the semantic feature and the non-semantic feature according to the weight value to obtain speech fusion features; The speech generation module is configured to determine second audio data according to the speech fusion features and the second prosody feature.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1-7 when executing the program.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer instructions for causing the computer to execute the method of any one of claims 1-7.

11. A computer program product comprising computer program instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1-7.

Images