Global prosodic type transfer without text transcription
By employing machine learning models and a two-stage training strategy, the challenge of prosodic information conversion under textless transcription was solved, achieving effective prosodic type transfer in high-resource languages, and making it applicable to a variety of speech applications.
Patent Information
- Application Number
- CN202280036866.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-06-03
- Filing Date
- 2022-05-24
- Publication Date
- 2026-11-10
- Estimated Expiration
- 2042-05-24
AI Technical Summary
Existing speech type transfer algorithms struggle to effectively convert prosodic information without text transcription, especially in high-resource languages, thus limiting their application.
Unsupervised processing using machine learning models is employed. Through self-expression learning and a two-stage training strategy, prosody separation and fuzziness are achieved, generating prosodic fuzzy content code, and synthesizing speech without text transcription.
It enables efficient conversion of prosodic types without text transcription, and is applicable to a variety of speech applications, including speech analysis, speech conversion, prosodic modification and emotional speech synthesis, without relying on text annotation.
Smart Images

Figure CN117355893B_ABST
Abstract
Description
Background Technology
[0001] This invention relates generally to speech processing, and more specifically to global prosody type transfer without text transcription.
[0002] Prosody plays a crucial role in characterizing the type of speaker or emotion, but most non-parallel speech or emotion type transfer algorithms do not translate any prosodic information. The two main components of prosody are pitch and rhythm. Rhythm outlines the sequence of speech duration and expresses aspects such as phrases, speech rate, pauses, and salience. Pitch reflects intonation. Separating prosodic information (especially the rhythmic component) from speech is challenging because it involves synchronizing the input speech with the separated speech expression. Therefore, most existing prosodic type transfer algorithms will need to rely on some form of text transcription to identify content information, limiting their application to high-resource languages. High-resource languages are those with abundant data resources, enabling the development of machine learning-based systems for these languages.
[0003] Speech type transfer refers to the task of transferring source speech to a target domain while preserving its content. For example, in speech type transfer, the domain corresponds to speaker identity. In emotion type transfer, the domain corresponds to emotion category. In both tasks, prosody is considered an important part of the domain type; for example, different speakers or emotions have unique prosodic patterns. However, few existing algorithms can fully transform the prosodic aspect in both applications, and transcription is required. Summary of the Invention
[0004] According to an aspect of the invention, a computer-implemented method for separating prosody in spoken natural language using a machine learning model is provided. The method includes encoding the spoken natural language by a computing device to generate content codes. The method further includes resampling the content codes to blur the prosody by applying unsupervised techniques to the machine learning model without text transcription, thereby generating prosodic-blurred content codes. The method additionally includes decoding the prosodic-blurred content codes by the computing device to indirectly synthesize speech based on the content codes.
[0005] In this embodiment, a similarity-based random resampling technique can be used to resample content code. This technique shortens (using similarity-based downsampling) or lengthens (using similarity-based upsampling) content code segments with similarity exceeding a threshold to an equal length, thereby creating prosodic fuzzy content code. In this embodiment, the similarity exceeding the threshold can be a prosodic similarity threshold.
[0006] In this embodiment, the resampling step can be guided by self-expressive representation learning.
[0007] In one embodiment, a resampling step can blur the rhythmic component among multiple prosodic components in the content code to generate prosodic-blurred content code.
[0008] In one embodiment, a target domain label can be used to decode prosodic ambiguity content codes. In another embodiment, the target domain label can be a domain summary specifying at least one speaker identity and at least one emotion category.
[0009] In one embodiment, a two-stage training technique can be used to perform the resampling step. This technique prevents the encoder performing the encoding step from colluding with the decoder performing the decoding step to jointly encode and decode rhythmic information. In another embodiment, the two-stage training technique may include a synchronous training portion and an asynchronous training portion. The synchronous training portion uses a sample length aligner to align the sample length and recover the input prosody, while the asynchronous training portion lacks a sample length aligner to enable the decoder to infer the prosody.
[0010] According to another aspect of the invention, a computer program product for the separation of prosody in spoken natural language using a machine learning model is provided. The computer program product includes a computer-readable storage medium having program instructions. The program instructions are executable by a computer to cause the computer to encode spoken natural language by an encoder of the computer to produce content codes. The program instructions are executable by a computer to cause the computer to resample the content codes by applying unsupervised techniques to a machine learning model to blur the prosody, without text transcription, to generate prosodic-blurred content codes. The program instructions are executable by a computer to cause the computer to decode the prosodic-blurred content codes by a decoder of the computer to indirectly synthesize speech based on the content codes.
[0011] According to further aspects of the present invention, a global prosody transfer system is provided. The system includes a memory device for storing program code. The system also includes a processor device operatively coupled to the memory device for running the program code to encode spoken natural language to produce content code. The processor device further runs program code to blur prosody by applying unsupervised techniques to a machine learning model to resample the content code without text transcription, to generate prosodic-blurred content code. The processor device further runs program code to decode the prosodic-blurred content code to indirectly synthesize speech based on the content code.
[0012] These and other features and advantages will become clear from the following detailed description of illustrative embodiments, which will be read in conjunction with the accompanying drawings. Attached Figure Description
[0013] The following description will provide details of preferred embodiments with reference to the following figures, in which:
[0014] Figure 1 This is a block diagram illustrating an exemplary computing device according to an embodiment of the present invention;
[0015] Figure 2 This is a block diagram illustrating an exemplary first stage of a two-stage training system for global prosodic type transfer for textless transcription according to an embodiment of the present invention;
[0016] Figure 3 This is a block diagram illustrating an exemplary second stage of a two-stage training system for global prosodic type transfer for textless transcription according to an embodiment of the present invention;
[0017] Figure 4 This is a flowchart illustrating an exemplary method for global prosodic type transfer for textless transcription according to an embodiment of the present invention;
[0018] Figure 5 This is a block diagram illustrating an exemplary downsampling scenario according to an embodiment of the present invention;
[0019] Figure 6 This is a block diagram illustrating an exemplary upsampling situation according to an embodiment of the present invention;
[0020] Figure 7 This is a block diagram illustrating an illustrative cloud computing environment according to an embodiment of the present invention, having one or more cloud computing nodes communicating with a local computing device used by a cloud consumer; and
[0021] Figure 8 This is a block diagram illustrating a set of functional abstraction layers provided by a cloud computing environment according to an embodiment of the present invention. Detailed Implementation
[0022] Embodiments of the present invention address global prosodic type transfer without text transcription. Embodiments of the present invention enable the decoupling of global prosodic type from speech, independent of text transcription.
[0023] Embodiments of the present invention may relate to an autoencoder-based prosodic type transfer framework with a thorough rhythm removal module guided by self-expression learning. Embodiments of the present invention employ a two-level training strategy to ensure the transmission of complete content information without leaking rhythm.
[0024] Embodiments of this invention globally transfer prosodic types. In global prosodic transfer, the decoder receives only a high-level summary of the target prosodic, such as speaker identity, emotion, etc. The decoder infers the details of the prosodic based on such speaker and / or emotion. For example, the decoder determines which word is sped up, slowed down, rises in pitch, or falls in pitch, etc. In local prosodic transfer, the decoder is provided with the details of the target prosodic, and the decoder precisely follows the details of the provided prosodic. Obtaining the target prosodic is difficult without parallel datasets (i.e., speech utterances with the same content as the input but different prosodic) or text annotations. This invention overcomes these drawbacks of local prosodic transfer.
[0025] Embodiments of the present invention can also globally transfer timbre type in a manner similar to that of prosody, as further described in detail herein.
[0026] Embodiments of the present invention provide an unsupervised speech separation algorithm that (1) does not require text annotations and (2) can efficiently transform prosodic types given a domain profile (e.g., speaker identity and emotion category) that provides only high-level global information.
[0027] Various embodiments of the present invention can be used to perform prosody type transfer using cloud-based services or cloud computing systems as described herein.
[0028] Embodiments of the present invention can be used in many different types of speech applications. Some exemplary speech applications include, but are not limited to, reducing interference in speech analysis tasks, speech conversion, prosody modification, emotional speech synthesis, and low bit-rate speech coding (where prosodic information has been obscured at low bit rates).
[0029] Figure 1 This is a block diagram illustrating an exemplary computing device 100 according to an embodiment of the present invention. The computing device 100 is configured to perform global prosody type transfer without text transcription.
[0030] The computing device 100 can be implemented as any type of computing or computer device capable of performing the functions described herein, including but not limited to computers, servers, rack-based servers, blade servers, workstations, desktop computers, laptop computers, notebook computers, tablet computers, mobile computing devices, wearable computing devices, network devices, web devices, distributed computing systems, processor-based systems, and / or consumer electronics devices. Alternatively or additionally, the computing device 100 can be implemented as one or more compute sleds, memory sleds, or other racks, sleds, chassis, or other components that are physically separated from the computing device. Figure 1As shown, computing device 100 illustratively includes processor 110, input / output subsystem 120, memory 130, data storage device 140, and communication subsystem 150, and / or other components and devices typically found in servers or similar computing devices. Of course, in other embodiments, computing device 100 may include other or additional components, such as those typically found in server computers (e.g., different input / output devices). Additionally, in some embodiments, one or more of the illustrative components may be incorporated into another component or otherwise formed part of another component. For example, in some embodiments, memory 130 or a portion thereof may be incorporated into processor 110.
[0031] Processor 110 may be implemented as any type of processor capable of performing the functions described herein. Processor 110 may be implemented as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single-core or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuitry.
[0032] Memory 130 may be embodied as any type of volatile or non-volatile memory or data storage device capable of performing the functions described herein. In operation, memory 130 may store various data and software used during the operation of computing device 100, such as operating systems, applications, programs, libraries, and drivers. Memory 130 is communicatively coupled to processor 110 via I / O subsystem 120, which may be embodied as circuitry and / or components facilitating input / output operations with processor 110, memory 130, and other components of computing device 100. For example, I / O subsystem 120 may be embodied as or otherwise include a memory controller hub, input / output control hub, platform controller hub, integrated control circuitry, firmware devices, communication links (e.g., point-to-point links, bus links, wires, cables, optical fibers, printed circuit board traces, etc.), and / or other components and subsystems to facilitate input / output operations. In some embodiments, the I / O subsystem 120 may be part of a system-on-a-chip (SOC) and may be integrated on a single integrated circuit chip along with the processor 110, memory 130 and other components of the computing device 100.
[0033] Data storage device 140 can be embodied as one or more devices of any type configured for short-term or long-term data storage, such as, for example, memory devices and circuitry, memory cards, hard disk drives, solid-state drives, or other data storage devices. Data storage device 140 can store program code for global prosodic type transfer for textless transcription. The communication subsystem 150 of computing device 100 can be embodied as any network interface controller or other communication circuitry, device, or combination thereof capable of enabling communication between computing device 100 and other remote devices via a network. Communication subsystem 150 can be configured to use any one or more communication technologies (e.g., wired or wireless communication) and associated protocols (e.g., Ethernet, etc.). Bluetooth This communication can be achieved using technologies such as WiMAX.
[0034] As shown in the figure, the computing device 100 may also include one or more peripheral devices 160. Peripheral devices 160 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 160 may include a display, touchscreen, graphics circuitry, keyboard, mouse, speaker system, microphone, network interface, and / or other input / output devices, interface devices, and / or peripheral devices.
[0035] Of course, the computing device 100 may also include other components (not shown), as readily apparent to those skilled in the art, and some components may be omitted. For example, different other input and / or output devices may be included in the computing device 100, depending on the specific implementation of the computing device 100, as readily understood by those skilled in the art. For example, different types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memories, etc., in different configurations may be utilized. Further, in another embodiment, a cloud configuration may be used (e.g., see...). Figure 6-7 Given the teachings of the invention provided herein, those skilled in the art will readily conceive of these and other variations of the processing system 100.
[0036] As used herein, the terms "hardware processor subsystem" or "hardware processor" can refer to a processor, memory (including RAM, cache, etc.), software (including memory management software), or a combination thereof that cooperate to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuitry, processing circuitry, instruction execution devices, etc.). One or more data processing elements may be included in a central processing unit, a graphics processing unit, and / or a separate processor- or computing element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more on-board memories (e.g., cache, dedicated memory array, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, basic input / output system (BIOS), etc.), which may be on-board or underboard, or may be dedicated to use by the hardware processor subsystem.
[0037] In some embodiments, the hardware processor subsystem may include and execute one or more software elements. The one or more software elements may include an operating system and / or one or more applications and / or specific code for implementing a specified result.
[0038] In other embodiments, the hardware processor subsystem may include dedicated, special-purpose circuitry that performs one or more electronic processing functions to achieve a specified result. Such circuitry may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.
[0039] These and other variations of the hardware processor subsystem are also considered according to embodiments of the invention.
[0040] The following Figure 2 and Figure 3 The first stage 200 and the second stage 300 of a two-stage training system for global prosodic type transfer for textless transcription are described.
[0041] Phase 1 (200) corresponds to the synchronous training portion of the two-stage training system, and Phase 2 (300) corresponds to the asynchronous training portion. The synchronous training portion is used for realignment. And Z(m). Realignment And Z(m) makes Synchronization with Z(m) is achieved, thus offsetting the effects of random resampling. Therefore, the first stage 200 is referred to as synchronous training. Realignment is the key operation in the first stage. The asynchronous training portion is used to update only the decoder.
[0042] Figure 2This is a block diagram illustrating an exemplary first stage 200 of a two-stage training system for global prosodic type transfer for textless transcription according to an embodiment of the present invention.
[0043] The first stage 200 includes an encoder 210, a resampler 220, a calibrator 230, and a decoder 240.
[0044] Encoder 210 receives input of Mel-Frequency Cepstrum Coefficients (MFCC) C(t) at time t, and encodes C(t) to output an encoded expression Z(t). Note that after encoding, the constituent elements of the encoded expression Z(t) are vertically aligned.
[0045] Resampler 220 receives the encoded expression Z(t) and resamples the encoded expression Z(t) to output the resampled expression. It is important to note that after resampling, the expression of resampling... The constituent elements are not vertically aligned.
[0046] Aligner 230 receives resampled expression And align the length of its constituent elements to output the aligned expression.
[0047] Decoder 240 receives the aligned expression Z′(t) and decodes the aligned expression Z′(t) in response to domain ID D to output the synthesized speech.
[0048] Figure 3 This is a block diagram illustrating an exemplary second stage 300 of a two-stage training system for global prosodic type transfer for textless transcription according to an embodiment of the present invention.
[0049] The first stage 200 includes a first-stage encoder 210, a resampler 220, and a decoder 240.
[0050] Encoder 210 (whose parameters are frozen in the second stage 300) receives input of Mel-frequency cepstral coefficients (MFCC) C(t) at time t and encodes C(t) to output an encoded expression Z(t). It should be noted that the constituent elements of the encoded expression Z(t) are not longitudinally aligned after encoding. The encoded Z(t) remains aligned with the input. Only resampling changes the alignment. Figure 2 and Figure 3 The main difference between them is that, Figure 3 It does not have an aligner 230. Figure 2 In this process, the aligner 230 restores the input prosody, so that the decoder 240 no longer needs to focus on reconstructing the prosody. Figure 3In the absence of an aligner 230, the decoder 240 must learn how to infer prosody from the resampled content code and domain ID. This stage 300 is where the decoder 240 truly learns to infer prosody. The first stage 200 prepares the decoder 240 by learning how to reconstruct non-prosodic components.
[0051] Resampler 220 receives the encoded expression Z(t) and resamples the encoded expression to output the resampled expression. It is important to note that after resampling, the resampled expression changes because the resampled length is altered. The constituent elements are not vertically aligned.
[0052] Decoder 240 receives the resampled expression And in response to the domain ID D, the expression for resampling is... Decode to output synthesized speech
[0053] Figure 4 This is a flowchart illustrating an exemplary method 400 for global prosodic type transfer for textless transcription according to an embodiment of the present invention.
[0054] At box 410, the spoken natural language is encoded to produce content code.
[0055] At box 420, content code is resampled to blur prosody by applying an unsupervised technique to the machine learning model, resulting in prosodic-blurred content code. Note that the use of an unsupervised approach leads to a lack of text transcription.
[0056] In box 430, prosodic ambiguous content codes are decoded without text transcription to indirectly synthesize speech based on the content codes. In an embodiment, a target domain label can be used to decode the prosodic ambiguous content codes. In an embodiment, the target domain label can be a domain profile specifying at least one speaker identity and at least one emotion category.
[0057] The refactoring process is as follows:
[0058]
[0059]
[0060] Here, Enc represents the encoder, Res represents the resampler, and Dec represents the decoder.
[0061] Figure 5 This is a block diagram illustrating an exemplary downsampling scenario 500 according to an embodiment of the present invention. Figure 6 This is a block diagram illustrating an exemplary upsampling scenario 600 according to an embodiment of the present invention.
[0062] exist Figure 5 and 6 The image shows a resampler 220 (left) and an aligner 230 (right). Figure 5 and Figure 6 The merging arrows in the diagram represent mean-pooling. Figure 5 and Figure 6 The separating arrows in the diagram indicate that the input is copied into multiple repetitions. The shaded lines of each block describe the similarity between the representations. About Figure 5 When τ≤1, the sequence is segmented based on similarity, and each segment is merged into a single code using average pooling. (Regarding...) Figure 6 When τ > 1, each segment contains only one piece of code. Furthermore, empty segments are inserted at points of high temporal similarity, with the corresponding output positions copying the previous code.
[0063] Similarity-based downsampling
[0064] The resampling scheme of this invention takes advantage of the observation that relatively stable segments in speech tend to have more flexible durations.
[0065] This invention therefore modifies the Self-Representation Autoencoder (SEA) algorithm to a similarity-based downsampling scheme. SEA derives frame-level speech representations, denoted as A(t), which conversely promote high cosine similarity between similar frames and low cosine similarity between dissimilar frames. This invention then creates a Gram matrix G to record the cosine similarity between any pair of frames:
[0066]
[0067] like Figure 5 As shown in the left figure, the downsampling scheme for Z(t) involves two steps. First, the invention divides Z(t) into continuous segments such that the cosine similarity of A(t) is high within each segment and decreases across segment boundaries. Second, each segment is merged into a single code by averaging. Formally, t... m This is represented as the left boundary of the m-th segment. The boundaries are determined sequentially until t. m When all boundaries are determined, if t is in(t m If the minimum time in (,∞) is found, then the next boundary t m +1 is set to t, where t and t m The cosine similarity between them drops below the threshold:
[0068]
[0069] τ(t) is a predefined threshold that can vary across t. After all segments are determined, each segment is reduced to one code segment using average pooling.
[0070]
[0071] Figure 6 An example of an input sequence of length 4 is shown. The second and third codes are very similar. Then, with an appropriate choice of τ(t), downsampling divides the input sequence into three segments, and each segment is folded into a code by mean pooling. Note that the threshold τ(t) governs the algorithm's tolerance for dissimilarity. If τ(t) = 1, each code will be assigned to a separate segment, resulting in no length reduction.
[0072] The segmentation results for the two utterances show that, despite a significant difference in length, they can be divided into approximately the same number of segments, and the segments have a high degree of correspondence in content. Since the downsampled output is obtained by mean pooling each segment, it can be expected that their downsampled outputs will be very similar and temporally aligned, which means that the necessary condition for the loss of rhythmic information is roughly met.
[0073] Random thresholding
[0074] For any fixed threshold τ in Equation (3), there is a trade-off between rhythm separation and content loss. The lower the τ, the more rhythm information is removed, but the more content is also lost. Ideally, during testing, it is desirable to set the threshold to 1 to deliver full content information. And make the decoder ignore All rhythmic information within it. This can be achieved using random thresholding rules.
[0075] To understand why, pay attention to if the decoder uses... The decoder needs to know the value of τ to recover rhythmic information, because how the rhythmic information is recovered depends on how it is folded, which is determined by τ. However, large variations in speech rate, speech length, and rhythmic patterns in the training speech will mask changes in τ, making it extremely difficult to estimate its value. Therefore, the decoder will ignore... Any preserved rhythmic information. This invention employs a double-randomized thresholding scheme. This invention first randomly extracts global variables. It is shared throughout the discourse, in which, Represents the interval [u l ,u r The uniform distribution within ] Then, in order to determine whether time t should be the boundary of the next segment (i.e., t in formula (3) m +1), This invention draws local variables Then
[0076] τ(t)=L(t)-quantile[G(t m ,t m -b:t m +b)]. (5)
[0077] q-quantile[·] means taking q-quantile (q-quantile).
[0078] The motivation for setting two randomization levels is that G can obscure global speech rate information and L(t) can obscure local fine-grained rhythm patterns.
[0079] Similarity-based upsampling
[0080] To further obscure rhythmic information, this invention summarizes a resampling module to accommodate upsampling. Just as downsampling aims to primarily shorten segments with high similarity (and thus reduce scaling imbalance), upsampling aims to primarily lengthen segments with high similarity (and thus increase scaling imbalance).
[0081] In the case of downsampling, τ = 1 means there is no reduction in length at all. Therefore, we attempt to extrapolate to the case where τ > 1, where the higher τ is, the longer the sequence becomes. The upsampling algorithm according to the invention achieves this by inserting new code between adjacent codes. Specifically, it is assumed that up to t... m All boundaries. When τ(t)>1, according to formula (3), t m +1 will be explicitly set to t. Furthermore, this invention will also add another sentence boundary to t, namely, t m +2 = t, if
[0082]
[0083] In other words, the present invention inserts an empty segment for the (m+1)th segment (because t m +1=t m +2). During the average pooling phase, the empty segment will be mapped to the code at its left boundary, i.e.
[0084] If t m =t m+1 (7)
[0085] Non-empty segments will still be averaged in the same way as in Equation (4).
[0086] Figure 6The left figure illustrates the upsampling process for an example of length 4. Similar to the case where τ=1, all codes are segmented individually. The difference is the insertion of a new empty segment after the third code, where the cosine similarity is very high. During the average pooling phase, this empty segment becomes additional code that copies the previous code.
[0087] Two-stage training
[0088] Despite the presence of a resampling module, the encoder and decoder may still find alternative ways to deliver temporal resampling-robust rhythmic information. Therefore, this invention introduces a two-stage training scheme to prevent any possible collusion.
[0089] The first stage, known as synchronous training, will Realign with Z(m), as follows Figure 5 and Figure 6 As shown in the right figure. Specifically, for the downsampling case, the present invention replicates each To match the length of the original segment from its average pooling code; for the case of upsampling, the invention deletes the newly inserted... The network is then trained end-to-end to reconstruct the input using realignment modules, such as... Figure 2 As shown. Since the decoder has full access to the rhythmic information, the encoder will be trained to convey content information rather than rhythmic information. The second phase, called asynchronous training, removes the realignment module, freezes the encoder, and updates only the decoder, as... Figure 3 As shown.
[0090] It should be understood that while this disclosure includes a detailed description of cloud computing, the implementation of the teachings cited herein is not limited to cloud computing environments. Rather, embodiments of the invention can be implemented in conjunction with any other type of computing environment now known or developed hereafter.
[0091] Cloud computing is a service delivery model that enables convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services), which can be rapidly provisioned and released with minimal management effort or interaction with the service provider. This cloud model may include at least five features, at least three service models, and at least four deployment models.
[0092] The features are as follows:
[0093] On-demand self-service: Cloud consumers can unilaterally and automatically provide computing power, such as server time and network storage, as needed, without requiring human interaction with the service provider.
[0094] Extensive network access: Capabilities are available through networks and accessed via standard mechanisms that facilitate the use of heterogeneous thin client or thick client platforms (e.g., mobile phones, laptops, and PDAs).
[0095] Resource pooling: A provider's computing resources are pooled to serve multiple consumers using a multi-tenant model, where different physical and virtual resources are dynamically assigned and reassigned as needed. There is a sense of location independence because consumers typically do not have control or knowledge of the exact location of the resources provided, but may be able to specify the location at a higher level of abstraction (e.g., country, state, or data center).
[0096] Rapid flexibility: The ability to provide capacity quickly and flexibly, automatically scaling down and up rapidly in some situations to scale up rapidly. For consumers, the available supply capacity often appears unlimited and can be purchased in any quantity at any time.
[0097] Measuring services: Cloud systems automatically control and optimize resource usage by leveraging metering capabilities at a level of abstraction appropriate to the service type (e.g., storage, processing, bandwidth, and active user accounts). Resource usage can be monitored, controlled, and reported, providing transparency to both service providers and consumers.
[0098] The service model is as follows:
[0099] Software as a Service (SaaS): This provides consumers with the ability to use the provider's applications running on cloud infrastructure. Applications can be accessed from different client devices via thin client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including the network, servers, operating system, storage, or even individual application capabilities, with possible exceptions such as limited user-specific application configuration settings.
[0100] Platform as a Service (PaaS): This provides consumers with the ability to deploy applications created or acquired by the consumer using programming languages and tools supported by the provider onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they have control over the deployed applications and the configuration of any application hosting environment.
[0101] Infrastructure as a Service (IaaS): This provides consumers with the capability to deliver processing, storage, networking, and other basic computing resources that enable them to deploy and run arbitrary software, which may include operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but rather have control over the operating system, storage, deployed applications, and potentially limited control over selected networking components (e.g., host firewalls).
[0102] The deployment model is as follows:
[0103] Private cloud: A cloud infrastructure that operates solely for an organization. It can be managed by the organization or a third party and can exist on-site or off-site.
[0104] Community cloud: A cloud infrastructure shared by several organizations and supporting a specific community with shared concerns (e.g., tasks, security requirements, policies, and compliance considerations). It can be managed by an organization or a third party and can exist on-site or off-site.
[0105] Public cloud: Makes cloud infrastructure available to the public or large industry groups and is owned by an organization that sells cloud services.
[0106] Hybrid cloud: A cloud infrastructure is a combination of two or more clouds (private, community, or public) that remain a single entity but are bound together by standardized or proprietary technologies that enable data and applications to be ported (e.g., cloud bursting for load balancing between clouds).
[0107] Cloud computing environments are service-oriented, focusing on statelessness, loose coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure comprising a network of interconnected nodes.
[0108] Now for reference Figure 7 This describes an illustrative cloud computing environment 750. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 that can communicate with local computing devices used by cloud consumers, such as, for example, personal digital assistants (PDAs) or cellular phones 754A, desktop computers 754B, laptop computers 754C, and / or automotive computer systems 754N. The nodes 710 can communicate with each other. They can be physically or virtually grouped (not shown) in one or more networks, such as private clouds, community clouds, public clouds, or hybrid clouds, or combinations thereof, as described above. This allows the cloud computing environment 750 to provide infrastructure, platform, and / or software as a service, without requiring cloud consumers to maintain resources on their local computing devices. It should be understood that... Figure 7 The types of computing devices 754A-N shown are intended to be illustrative only, and the computing node 710 and cloud computing environment 750 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).
[0109] See now Figure 8 This demonstrates the 750 (cloud computing environment) Figure 7 This provides a set of functional abstractions. It should be understood beforehand. Figure 8The components, layers, and functions shown are intended to be illustrative only, and embodiments of the invention are not limited thereto. As described, the following layers and corresponding functions are provided:
[0110] The hardware and software layer 860 includes hardware and software components. Examples of hardware components include: a mainframe 861; a server 862 based on a RISC (Reduced Instruction Set Computer) architecture; a server 863; a blade server 864; a storage device 865; and a network and network components 866. In some embodiments, the software components include network application server software 867 and database software 868.
[0111] The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities can be provided: virtual server 871; virtual storage 872; virtual network 873, including virtual private network; virtual application and operating system 874; and virtual client 875.
[0112] In one example, management layer 880 can provide the functions described below: Resource Provisioning 881 provides dynamic procurement of computing resources and other resources used to perform tasks within the cloud computing environment. Metering and Pricing 882 provides cost tracking as resources are utilized within the cloud computing environment and bills or invoices for the consumption of these resources. In one example, these resources may include application software licenses. Security provides authentication for cloud consumers and tasks, as well as protection for data and other resources. User Portal 883 provides access to the cloud computing environment for consumers and system administrators. Service Level Management 884 provides cloud resource allocation and management to ensure that required service levels are met. Service Level Agreement (SLA) Planning and Fulfillment 885 provides pre-scheduling and procurement of cloud resources, anticipating future requirements for those resources according to the SLA.
[0113] Workload layer 890 provides examples of functionalities that can leverage cloud computing environments. Examples of workloads and functionalities that can be provided from this layer include: mapping and navigation 891; software development and lifecycle management 892; virtual classroom education delivery 893; data analytics and processing 894; transaction processing 895; and global prosodic type transfer without text transcription 896.
[0114] This invention can be a system, method, and / or computer program product with any possible level of technical detail integration. The computer program product may include a computer-readable storage medium having computer-readable program instructions thereon for causing a processor to execute aspects of the invention.
[0115] Computer-readable storage media can be tangible devices capable of retaining and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of computer-readable storage media includes: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital universal disk (DVD), memory sticks, floppy disks, mechanical encoding devices such as punch cards or protrusions in slots having instructions recorded thereon, and any suitable combination of the foregoing. As used herein, computer-readable storage media should not be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.
[0116] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a suitable computing / processing device via a network (e.g., the Internet, a local area network, a wide area network, and / or a wireless network), or to an external computer or external storage device. The network may include copper cables, optical fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to a computer-readable storage medium within the suitable computing / processing device.
[0117] Computer-readable program instructions used to perform the operations of this invention may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages (such as SMALLTALK, C++, etc.) and conventional procedural programming languages (such as the "C" programming language or similar programming languages). The computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as a standalone software package, partially on a user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer may be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)) or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs) may execute computer-readable program instructions by utilizing state information from the computer-readable program instructions to personalize the electronic circuitry in order to perform aspects of this invention.
[0118] The present invention will now be described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.
[0119] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner, such that the computer-readable storage medium storing the instructions includes an article of manufacture containing instructions that implement aspects of the functions / actions specified in one or more blocks of a flowchart and / or block diagram.
[0120] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce computer-implemented processing, such that the instructions executed on the computer, other programmable apparatus, or other device perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.
[0121] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. Each block in a flowchart or block diagram may represent a module, segment, or portion of instructions, including one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than indicated in the figures. For example, depending on the functions involved, two consecutively shown blocks may actually be executed substantially simultaneously, or these blocks may sometimes be executed in reverse order. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action or executes a combination of dedicated hardware and computer instructions.
[0122] References to the invention in this specification as "one embodiment" or "embodiment" and other variations thereof mean that a particular feature, structure, characteristic, etc., described in connection with that embodiment is included in at least one embodiment of the invention. Therefore, the phrases "in one embodiment" or "embodiment" appearing in various places throughout the specification, as well as any other variations, do not necessarily refer to the same implementation.
[0123] It should be understood that, for example, in the cases of “A / B,” “A and / or B,” and “at least one of A and B,” the use of any of the following “ / ,” “and / or,” and “at least one of” is intended to include selecting only the first listed option (A), or only the second listed option (B), or selecting both options (A and B). As another example, in the cases of “A, B, and / or C” and “at least one of A, B, and C,” this wording is intended to cover selecting only the first listed option (A), or only the second listed option (B), or only the third listed option (C), or only the first and second listed options (A and B), or only the first and third listed options (A and C), or only the second and third listed options (B and C), or selecting all three options (A, B, and C). It will be apparent to those skilled in the art that this can be extended for many of the listed items.
[0124] Preferred embodiments of the systems and methods have been described (these are intended to be illustrative and not restrictive), and it should be noted that modifications and variations can be made by those skilled in the art based on the foregoing teachings. Therefore, it should be understood that changes may be made to the specific embodiments disclosed within the scope of the invention as outlined in the appended claims. Various aspects of the invention, having the details and features required by patent law, have thus been described, and the claimed and desired protection by a patent certificate is set forth in the claims.
Claims
1. A computer-implemented method for separating prosody in spoken natural language using a machine learning model, the method comprising: Spoken natural language is encoded by computing devices to produce content codes; The computing device, without text transcription, applies unsupervised techniques to the machine learning model to resample the content code to blur the prosody, thereby generating prosodic-blurred content code. The content code is resampled using a similarity-based random resampling technique, which uses similarity-based downsampling to shorten or similarity-based upsampling to extend content code segments with similarity exceeding a threshold to equal length, thus forming prosodic-blurred content code. as well as The computing device decodes the prosodic fuzzy content code to indirectly synthesize speech based on the content code.
2. The computer-implemented method according to claim 1, wherein, Rhythm conveys the rhythm and pitch of spoken natural language.
3. The computer-implemented method according to claim 1, wherein, The encoding step is performed by the encoder of the computing device, the resampling step is performed by the resampler of the computing device, and the decoding step is performed by the decoder of the computing device.
4. The computer-implemented method according to claim 1, wherein, Similarity values above the threshold are prosodic similarity thresholds.
5. The computer-implemented method according to claim 1, wherein, The resampling step is guided by self-expressive learning.
6. The computer-implemented method according to claim 1, wherein, The resampling step blurs the rhythmic component among multiple prosodic components in the content code to generate prosodic-blurred content code.
7. The computer-implemented method according to claim 1, wherein, The prosodic fuzzy content code is decoded using target domain tags.
8. The computer-implemented method according to claim 7, wherein, The target domain label is a domain summary that specifies at least one speaker identity and at least one emotion category.
9. The computer-implemented method according to claim 1, wherein, The method is configured as a cloud-based service with encoding, resampling, and decoding components.
10. The computer-implemented method of claim 1, wherein the resampling step is performed by a two-stage training technique that prevents the encoder performing the encoding step from colluding with the decoder performing the decoder step to jointly encode and decode rhythm information.
11. The computer-implemented method according to claim 10, wherein, The two-stage training technique includes a synchronous training part and an asynchronous training part. The synchronous training part uses a sample length aligner to align the sample length and recover the input prosody. The asynchronous training part lacks a sample length aligner so that the decoder can infer the prosody.
12. A computer program product for separating prosody in spoken natural language using a machine learning model, the computer program product comprising program instructions executable by a computer to cause the computer to: The spoken natural language is encoded by the computer's encoder to produce content code; The computer's resampler, without text transcription, applies unsupervised techniques to the machine learning model to resample the content code to blur the prosody, generating prosodic-blurred content code, wherein... The content code is resampled using a similarity-based random resampling technique, which uses similarity-based downsampling to shorten or similarity-based upsampling to extend content code segments with similarity exceeding a threshold to equal length, thereby forming rhythmically ambiguous content code. as well as The computer's decoder decodes the prosodic fuzzy content code to indirectly synthesize speech based on the content code.
13. The computer program product according to claim 12, wherein, The resampling step is guided by self-expressive learning.
14. The computer program product according to claim 12, wherein, The prosodic fuzzy content code is decoded using target domain tags.
15. The computer program product according to claim 14, wherein, The target domain label is a domain summary that specifies at least one speaker identity and at least one emotion category.
16. The computer program product according to claim 12, wherein, The resampling step is performed using a two-stage training technique that prevents the encoder performing the encoding step from colluding with the decoder performing the decoding step to jointly encode and decode rhythm information.
17. The computer program product according to claim 16, wherein, The two-stage training technique includes a synchronous training part and an asynchronous training part. The synchronous training part uses a sample length aligner to align the sample length and recover the input prosody. The asynchronous training part lacks a sample length aligner so that the decoder can infer the prosody.
18. A global prosody transfer system for separating prosody in spoken natural language using a machine learning model, comprising: Memory devices used to store program code; as well as A processor device, operatively coupled to the memory device, is used to execute the program code to: Encode spoken natural language to produce content code; In the absence of text transcription, the content code is resampled to blur the prosody by applying unsupervised techniques to the machine learning model to generate prosodic-blurred content code. The content code is resampled using a similarity-based random resampling technique, which uses similarity-based downsampling to shorten or similarity-based upsampling to extend content code segments with similarity exceeding a threshold to equal length, thereby forming prosodic-blurred content code. as well as The prosodic fuzzy content code is decoded to indirectly synthesize speech based on the content code.
