Overall prosodic style transformation without text transcription

Abstract

A computer-implemented method is provided that uses a machine learning model for disentangling prosody in spoken natural language. The method includes encoding the spoken natural language by a computing device to generate a content code. The method further includes resampling the content code to disambiguate prosody by the computing device without text transcription by applying unsupervised techniques to the machine learning model to generate a prosodically ambiguous content code. The method further includes decoding the prosodically ambiguous content code by the computing device to indirectly synthesize speech based on the content code.

Description

[Technical Field]

[0001] This invention relates in general to speech processing, and more particularly to the transformation of overall prosodic style without text transcription.

[0002] Prosody plays a crucial role in characterizing a speaker's or emotional style, yet most non-parallel voice or emotional style transformation algorithms do not transform any prosodic information. The two main components of prosody are pitch and rhythm. Rhythm summarizes the sequence of phoneme durations and represents several aspects of phrasing, speech speed, pauses, and prominences. Pitch reflects intonation. Deconstructing prosodic information, particularly the rhythmic component, from speech is difficult because it involves disrupting the synchronization between the input speech and the deconstructed speech representation. As a result, most existing prosodic style transformation algorithms must rely on some form of text transcription to identify content information, limiting their applications to high-resource languages ​​only. High-resource languages ​​are those for which a large amount of data resources exist, making it possible to develop machine learning-based systems for these languages.

[0003] Speech style conversion refers to the task of transforming source speech into the style of a target domain without altering its content. For example, in voice style conversion, the domain corresponds to speaker identity. In sentiment style conversion, the domain corresponds to sentiment category. In both of these tasks, prosody is considered a crucial part of the domain style; for example, different speakers or sentiments have distinct prosodic patterns. However, very few of the most advanced algorithms for these two applications can transform the prosodic aspects at all and require transcription. [Overview of the Initiative]

[0004] According to aspects of the present invention, a computer implementation method is provided that uses a machine learning model for undoing prosodic entanglement in spoken natural language. The method comprises encoding spoken natural language by a computing device to generate a content code. The method further comprises resampling the content code to make it prosodic by a computing device without text transcription by applying unsupervised techniques to a machine learning model to generate a prosodic ambiguous content code. The method further comprises decoding the prosodic ambiguous content code by a computing device to indirectly synthesize speech based on the content code.

[0005] In one embodiment, content codes can be resampled using similarity-based random resampling techniques, and content code segments with similarity exceeding a threshold are shortened using similarity-based downsampling or lengthened using similarity-based upsampling to form prosodic ambiguous content codes. In one embodiment, the similarity exceeding a threshold may be a prosodic similarity threshold.

[0006] In one embodiment, the resampling step may be guided by learning self-expressive representations.

[0007] In one embodiment, the content code can be made ambiguous by resampling the rhythm component from among multiple prosodic components, thereby generating a content code with ambiguous prosodicity.

[0008] In one embodiment, ambiguous content codes can be decoded using a subject area label. In one embodiment, the subject area label may be a summary of a subject area that specifies at least one speaker identity and at least one sentiment category.

[0009] In one embodiment, the resampling step can be performed by a two-stage training technique that prevents the encoder performing the encoding step from coordinating with the decoder performing the decoding step to encode and decode the rhythm information in a single step. In one embodiment, the two-stage training technique may include a synchronous training portion that uses a sample length aligner to adjust the sample length and reconstruct the input prosody, and an asynchronous training portion that does not use a sample length aligner to allow the decoder to infer the prosody.

[0010] According to another aspect of the present invention, a computer program product is provided for using a machine learning model to untangle prosody in natural speech. The computer program product includes a computer-readable storage medium having program instructions embodied in the computer program product. The program instructions are computer-executable such that the computer encodes natural speech using the computer's encoder in order to generate content code. The program instructions are computer-executable such that the computer resamples the content code to make it prosody ambiguous by applying unsupervised techniques to a machine learning model, without text transcription, in order to generate prosody ambiguous content code. The program instructions are computer-executable such that the computer decodes the prosody ambiguous content code using the computer's decoder in order to indirectly synthesize speech based on the content code.

[0011] According to yet another aspect of the present invention, an overall prosodic conversion system is provided. The system includes a memory device for storing program code. The system further includes a processor device operably coupled to the memory device for executing the program code to encode natural language into speech and generate content code. The processor device further executes the program code to resample the content code to make it prosodic, without text transcription, by applying unsupervised techniques to a machine learning model, in order to generate prosodic ambiguous content code. The processor device also executes the program code to decode the prosodic ambiguous content code in order to indirectly synthesize speech based on the content code.

[0012] These and other features and advantages will become apparent from the following detailed description of the exemplary embodiment, which will be read in conjunction with the attached drawings. [Brief explanation of the drawing]

[0013] The following description provides details of a preferred embodiment with reference to the following figures.

[0014] [Figure 1] This is a block diagram showing an exemplary computing device according to an embodiment of the present invention.

[0015] [Figure 2] This block diagram shows an exemplary first stage of a two-stage training system for the transformation of overall prosodic style without text transcription, according to an embodiment of the present invention.

[0016] [Figure 3] This block diagram shows an exemplary second stage of a two-stage training system for the transformation of overall prosodic style without text transcription, according to an embodiment of the present invention.

[0017] [Figure 4]A flowchart showing an exemplary method for converting an overall rhythm style without text transcription according to an embodiment of the present invention.

[0018] [Figure 5] A block diagram showing an exemplary downsampling case according to an embodiment of the present invention.

[0019] [Figure 6] A block diagram showing an exemplary upsampling case according to an embodiment of the present invention.

[0020] [Figure 7] A block diagram showing an exemplary cloud computing environment having one or more cloud computing nodes with which a local computing device used by a cloud consumer communicates according to an embodiment of the present invention.

[0021] [Figure 8] A block diagram showing a set of functional abstraction layers provided by a cloud computing environment according to an embodiment of the present invention.

Mode for Carrying Out the Invention

[0022] Embodiments of the present invention are directed to the conversion of an overall rhythm style without text transcription. Embodiments of the present invention can decompose the overall rhythm style from speech without relying on text transcription.

[0023] Embodiments of the present invention can include a rhythm style conversion framework based on an autoencoder with a thorough rhythm removal module derived from expression learning of self-expression. Embodiments of the present invention adopt a two-stage training strategy to ensure that complete content information passes through without leaking rhythm.

[0024] Embodiments of the present invention perform a global stylistic transformation. In a global stylistic transformation, the decoder receives only a high-level summary of the target prosody, such as speaker identity and sentiment. Prosodic details are inferred by the decoder for such speaker and / or sentiment. For example, the decoder determines which words to accelerate, decelerate, raise or lower the tone of, etc. In a local stylistic transformation, the decoder is provided with details of the target prosody, and the decoder adheres precisely to the provided prosodic details. Without parallel datasets (i.e., speech utterances with the same content as the input but with different prosodic syllables) or text annotations, obtaining the target prosody is difficult. The present invention overcomes these shortcomings of local stylistic transformation.

[0025] Embodiments of the present invention can also transform the sound quality style as well as the metrical structure, as will be described in more detail herein.

[0026] Embodiments of the present invention provide an unsupervised speech decomposition algorithm that (1) does not require text annotations and (2) can effectively transform a given prosodic style into a region that provides only high-level overall information (e.g., speaker identity and sentiment category).

[0027] Embodiments of the present invention can be used to perform prosodic style transformation using a cloud-based service or cloud computing system, as described herein.

[0028] Embodiments of the present invention can be applied to many different types of speech applications. Some exemplary speech applications include, but are not limited to, speech analysis tasks, voice conversion, prosodic correction, sentiment speech synthesis, and interference reduction in low-bitrate speech encoding (low bitrate in the sense that prosodic information is ambiguous).

[0029] Figure 1 is a block diagram showing an exemplary computing device 100 according to an embodiment of the present invention. The computing device 100 is configured to perform an overall prosodic style transformation without text transcription.

[0030] Computing device 100 may be embodied as any type of computing device or computer device capable of performing the functions described herein, including but not limited to computers, servers, rack-based servers, blade servers, workstations, desktop computers, laptop computers, notebook computers, tablet computers, mobile computing devices, wearable computing devices, network appliances, web appliances, distributed computing systems, processor-based systems, and / or consumer electronic devices. Additionally or alternatively, computing device 100 may be embodied as one or more compute threads, memory threads, or other components of other racks, threads, computing chassis, or physically distributed computing devices. As shown in Figure 1, computing device 100 includes a processor 110, an input / output subsystem 120, memory 130, a data storage device 140, and a communication subsystem 150, and / or other components and devices commonly found in servers or similar computing devices. Of course, in other embodiments, computing device 100 may include other or additional components, such as components commonly found in server computers (e.g., various input / output devices). Furthermore, in some embodiments, one or more of the exemplary components may be incorporated into another component or otherwise form part of another component. For example, in some embodiments, the memory 130 or a part thereof may be incorporated into the processor 110.

[0031] The processor 110 can be embodied as any type of processor capable of performing the functions described herein. The processor 110 can be embodied as a single processor, multiple processors, a central processing unit (CPU), a graphics processing unit (GPU), a single or multi-core processor, a digital signal processor, a microcontroller, or other processor or processing / control circuit.

[0032] Memory 130 may be embodied as any type of volatile or non-volatile memory or data storage capable of performing the functions described herein. During operation, memory 130 may store various data and software used during the operation of computing device 100, such as operating systems, applications, programs, libraries, and drivers. Memory 130 is communicably coupled to processor 110 via I / O subsystem 120, which may be embodied as circuits and / or components that facilitate input / output operations with processor 110, memory 130, and other components of computing device 100. For example, I / O subsystem 120 may be embodied as a memory controller hub, input / output control hub, platform controller hub, integrated control circuit, firmware device, communication link (e.g., point-to-point link), bus link, wire, cable, optical conductor, printed circuit board trace, etc.) and / or other components and subsystems that facilitate input / output operations, or otherwise may include them. In some embodiments, the I / O subsystem 120 may form part of a system-on-a-chip (SOC) and be integrated onto a single integrated circuit chip together with the processor 110, memory 130, and other components of the computing device 100.

[0033] The data storage device 140 may be embodied as one or more devices of any type configured for short-term or long-term storage of data, such as memory devices and circuits, memory cards, hard disk drives, solid-state drives, or other data storage devices. The data storage device 140 can store program code for translating the overall prosodic style without text transcription. The communication subsystem 150 of the computing device 100 may be embodied as any network interface controller, or other communication circuits, devices, or ensembles thereof, which can enable communication between the computing device 100 and other remote devices over a network. The communication subsystem 150 may be configured to use any one or more communication technologies (e.g., wired or wireless) and associated protocols (e.g., Ethernet®, InfiniBand®, Bluetooth®, Wi-Fi®, WiMAX®, etc.) to perform such communication.

[0034] As shown in the figure, the computing device 100 may also include one or more peripheral devices 160. Peripheral devices 160 may include any number of additional input / output devices, interface devices, and / or other peripheral devices. For example, in some embodiments, peripheral devices 160 may include a display, a touchscreen, graphics circuitry, a keyboard, a mouse, a speaker system, a microphone, a network interface, and / or other input / output devices, interface devices, and / or peripheral devices.

[0035] Of course, the computing device 100 may include other elements (not shown) and may omit certain elements, as can be easily conceivable by those skilled in the art. For example, as can be easily understood by those skilled in the art, various other input and / or output devices may be included in the computing device 100 depending on the specific implementation of the computing device 100. For example, various types of wireless and / or wired input and / or output devices may be used. Furthermore, additional processors, controllers, memory, etc., may be used in various configurations. Furthermore, in another embodiment, a cloud configuration may be used (see, for example, Figures 6-7). These and other variations of the processing system 100 will be easily conceivable by those skilled in the art, considering the teachings of the invention provided herein.

[0036] As used herein, the terms “hardware processor subsystem” or “hardware processor” may refer to a processor, memory (including RAM, cache, etc.), software (including memory management software), or a combination thereof, that work together to perform one or more specific tasks. In useful embodiments, a hardware processor subsystem may include one or more data processing elements (e.g., logic circuits, processing circuits, instruction execution devices, etc.). One or more data processing elements may include a central processing unit, a graphics processing unit, and / or a separate processor-based or compute-element-based controller (e.g., logic gates, etc.). A hardware processor subsystem may include one or more onboard memories (e.g., caches, dedicated memory arrays, read-only memory, etc.). In some embodiments, a hardware processor subsystem may include one or more memories (e.g., ROM, RAM, Basic Input / Output System (BIOS), etc.) that may be onboard or offboard, or that may be dedicated to the use of the hardware processor subsystem.

[0037] In some embodiments, the hardware processor subsystem may include and be executable one or more software elements. One or more software elements may include an operating system and / or one or more applications and / or specific code for achieving a specified result.

[0038] In other embodiments, the hardware processor subsystem may include specialized circuits dedicated to performing one or more electronic processing functions to achieve a specified result. Such circuits may include one or more application-specific integrated circuits (ASICs), FPGAs, and / or PLAs.

[0039] These and other variations of the hardware processor subsystem are also intended in accordance with embodiments of the present invention.

[0040] Figures 2 and 3 below illustrate the first stage 200 and the second stage 300 of a two-stage training system for overall prosodic style conversion without text transcription.

[0041] The first stage 200 corresponds to the synchronous training portion of the two-stage training system, and the second stage 300 corresponds to the asynchronous training portion of the two-stage training system. The synchronous training portion is

number

number

[0042] Figure 2 is a block diagram illustrating the first stage 200 of an exemplary two-stage training system for overall prosodic style conversion without text transcription, according to an embodiment of the present invention.

[0043] The first stage 200 includes an encoder 210, a resampler 220, an aligner 230, and a decoder 240.

[0044] Encoder 210 receives an input of Mel-frequency cepstrum coefficients (MFCCs) C(t) at time t, encodes C(t), and outputs the encoded representation Z(t). Note that the constituents of the encoded representation Z(t) are aligned vertically after encoding.

[0045] The resampler 220 receives the encoded representation Z(t), resamples the encoded representation Z(t), and the resampled representation

number

number

[0046] Aliner 230 is a resampled representation.

number

number

[0047] Decoder 240 receives the aligned representation Z'(t), decodes the aligned representation Z'(t) in response to domain ID D, and synthesizes speech.

number

[0048] Figure 3 is a block diagram illustrating the second stage 300 of an exemplary two-stage training system for overall prosodic style conversion without text transcription, according to an embodiment of the present invention.

[0049] The first stage 200 includes the encoder 210, resampler 220, and decoder 240 of the first stage.

[0050] Encoder 210 receives an input of Mel-frequency cepstrum coefficients (MFCCs) C(t) at time t, with its parameters frozen in the second stage 300, encodes C(t), and outputs the encoded representation Z(t). Note that the constituents of the encoded representation Z(t) are not vertically aligned after encoding. The encoded Z(t) still matches the input. Only resampling changes the alignment. The main difference between Figure 2 and Figure 3 is that Figure 3 does not have an aligner 230. In Figure 2, the aligner 230 reconstructs the input prosody, and as a result, decoder 240 does not need to focus on reconstructing the prosody at this point. In Figure 3, without the aligner 230, decoder 240 needs to learn how to infer the prosody from the resampled content code and domain ID. This stage 300 is the stage in which decoder 240 actually learns to infer the prosody. The first stage 200 prepares decoder 240 by learning how to reconstruct the non-prosodic components.

[0051] The resampler 220 receives the encoded representation Z(t), resamples the encoded representation, and the resampled representation

number

number

[0052] Decoder 240 is a resampled representation

number

number

number

[0053] Figure 4 is a flowchart illustrating an exemplary method 400 for the transformation of overall prosodic style without text transcription, according to an embodiment of the present invention.

[0054] In block 410, natural language is encoded via speech to generate a content code.

[0055] In block 420, content codes are resampled to make them prosodic by applying unsupervised techniques to a machine learning model in order to generate prosodic ambiguous content codes. Note that using unsupervised methods results in a lack of text transcription.

[0056] In block 430, a prosodic ambiguous content code is decoded without text transcription, and speech is synthesized indirectly based on the content code. In one embodiment, the prosodic ambiguous content code can be decoded using a subject area label. In one embodiment, the subject area label may be a summary of a region that specifies at least one speaker identity and at least one sentiment category.

[0057] The reconstruction process is as follows:

number

[0058] Figure 5 is a block diagram showing an exemplary downsampling case 500 according to an embodiment of the present invention. Figure 6 is a block diagram showing an exemplary upsampling case 600 according to an embodiment of the present invention.

[0059] The resampler 220 (left) and aligner 230 (right) are shown in Figures 5 and 6. The converging arrows in Figures 5 and 6 indicate mean pooling. The splitting arrows in Figures 5 and 6 indicate copying the input into multiple copies. The shading in each block indicates the similarity between representations. Regarding Figure 5,

number

number

[0060] The resampling method of the present invention utilizes the observation that relatively stable segments within speech tend to have more flexible durations.

[0061] Therefore, the present invention modifies the Self-Expressive Autoencoder (SEA) algorithm into a similarity-based downsampling scheme. The SEA derives a frame-level speech representation denoted by A(t) that contrastsly promotes high cosine similarity between similar frames and low cosine similarity between different frames. The present invention then constructs a Gram matrix G, which records the cosine similarity between any pair of frames.

number

[0062] As shown in the left-hand section of Figure 5, the downsampling method for Z(t) involves two steps. First, the present invention divides Z(t) into continuous segments such that the cosine similarity of A(t) is high within each segment and decreases across segment boundaries. Next, each segment is merged into a single code by mean pooling. Formally, t m This is shown as the left boundary of the m-th segment. The boundaries are determined sequentially. m Once all boundaries up to are determined, t is determined to be t and t m The cosine similarity between them falls below the threshold.

number

number

[0063] τ(t) is a predetermined threshold that changes with t. After all segments have been determined, each segment is shortened to a single code by mean pooling, i.e., as follows:

number

[0064] Figure 6 shows a small example of an input sequence of length 4. The second and third codes are very similar. Then, with an appropriate selection of τ(t), downsampling divides the input sequence into three segments, and mean pooling shortens each segment to one code. Note that the threshold τ(t) determines the degree of tolerance the algorithm has for dissimilarity. If τ(t)=1, each code is assigned to a separate segment, and no length reduction occurs.

[0065] The segmentation results of the two utterances show that, despite their significantly different lengths, they can be divided into nearly the same number of segments, indicating a high degree of content-based segmental agreement. Since the downsampled outputs are obtained by averaging each segment, it can be predicted that these downsampled outputs are very similar and temporally aligned, which implies that the necessary conditions for rhythmic information loss are nearly met. Randomized thresholding

[0066] For any fixed threshold τ in equation (3), there is a trade-off between untangling rhythmic information and content loss. The lower τ, the more rhythmic information is removed, but more content is also lost. Ideally, the threshold should be set to 1 during testing to obtain complete content information.

number

number

[0067] To understand the reason, the decoder

number

number

number

number

number

number

number

[0068]

number

[0069] The motivation for setting two levels of randomization is that the overall speech rate information can be ambiguous due to G, and L(t) can obscure local fine rhythm patterns. Similarity-based upsampling

[0070] To further obscure the rhythm information, the present invention generalizes the resampling module to accommodate upsampling. Similar to downsampling, which aims mainly to shorten segments with higher similarity (thus reducing imbalance), upsampling aims mainly to lengthen segments with higher similarity (thus increasing imbalance).

[0071] In the downsampling case, τ = 1 implies that there is no shortening of the length. Thus, this case is

Number

Number

[0072] In other words, the present invention inserts an empty segment into the (m + 1)-th segment (because t m +1 = t m +2). In the average pooling stage, this empty segment is mapped to the code at its left boundary, that is,

Number

[0073] Non-empty segments are still averaged and pooled in the same way as in equation (4).

[0074] The left-hand section of Figure 6 shows the upsampling process using a small example of length 4. Similar to the case where τ=1, all codes are segmented individually. The difference is that a new empty segment is inserted after a third code with very high cosine similarity. During the mean pooling phase, this empty segment is replaced by an additional code that copies the previous code. Two-stage training

[0075] Despite the resampling module, encoders and decoders can still find alternative methods to transmit rhythmic information that is robust to temporal resampling. Therefore, the present invention introduces a two-stage training scheme to prevent any possible collusion.

[0076] In the first stage of training, called synchronous training, as shown in the right-hand section of Figures 5 and 6,

number

number

number

[0077] While this disclosure includes a detailed description of cloud computing, it should be understood that the implementation of the teachings described herein is not limited to cloud computing environments. Rather, embodiments of the present invention can be implemented in combination with any other type of computing environment that is currently known or may be developed in the future.

[0078] Cloud computing is a service delivery model that enables convenient on-demand network access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, services) that can be rapidly provisioned and deployed with minimal management effort or interaction with service providers. This cloud model may include at least five characteristics, at least three service models, and at least four deployment models.

[0079] The features are as follows: On-Demand Self-Service: Cloud consumers can unilaterally provision computing power, such as server time and network storage, automatically and automatically as needed, without requiring human interaction with a service provider. Broad Network Access: Capabilities are available over the network and accessed through standard mechanisms that facilitate use by heterogeneous thin-client platforms or thick-client platforms (e.g., mobile phones, laptops, PDAs®). Resource Pooling: Provider computing resources are pooled to serve multiple consumers using a multi-tenant model, with different physical and virtual resources dynamically allocated and reallocated according to demand. Consumers generally have a sense of location independence, in that they have no control or knowledge of the exact location of the resources provided, but can specify the location at a higher level of abstraction (e.g., country, state, or data center). Rapid Elasticity: Capabilities can be provisioned quickly and elastically, and in some cases, automatically and rapidly scale out, rapidly release, and rapidly scale in. To consumers, the capacity available for provisioning often appears unlimited and can be purchased in any quantity at any time. Measured Services: Cloud systems automatically control and optimize resource usage by leveraging metric capabilities at a level of abstraction appropriate to the type of service (e.g., storage, processing, bandwidth, active user accounts). They can monitor, control, and report on resource usage, providing transparency to both providers and consumers of the services they use.

[0080] The service model is as follows: Software as a Service (SaaS): The ability provided to consumers is to use the provider's applications running on cloud infrastructure. These applications are accessible from various client devices via thin-client interfaces such as web browsers (e.g., web-based email). Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, storage, or individual application functionalities, except for limited user-specific application configuration settings. Platform as a Service (PaaS): The ability provided to consumers is to deploy applications they have created or acquired, written using programming languages ​​and tools supported by the provider, onto cloud infrastructure. Consumers do not manage or control the underlying cloud infrastructure, including networks, servers, operating systems, or storage, but they do control the deployed applications and, in some cases, the configuration of the application hosting environment. Infrastructure as a Service (IaaS): The ability provided to consumers is to provision processing, storage, networking, and other basic computing resources that allow consumers to deploy and run any software, including operating systems and applications. Consumers do not manage or control the underlying cloud infrastructure, but they control the operating systems, storage, deployed applications, and, in some cases, limit their control over selected networking components (e.g., host firewalls).

[0081] The deployment model is as follows: Private Cloud: The cloud infrastructure is operated solely for an organization. It may be managed by the organization or a third party and may reside on-premises or off-premises. Community Cloud: The cloud infrastructure is shared by multiple organizations and supports a specific community with shared concerns (e.g., mission, security requirements, policies, compliance considerations). It may be managed by the organization or a third party and may reside on-premises or off-premises. Public Cloud: The cloud infrastructure is made available to the general public or large industry groups and is owned by an organization that sells cloud services. Hybrid Cloud: The cloud infrastructure remains a unique entity but is a configuration of two or more clouds (private, community, or public) joined by standardized or proprietary technologies (e.g., cloudburst for load balancing between clouds) that enable data and application portability.

[0082] Cloud computing environments are service-oriented, emphasizing statelessness, low coupling, modularity, and semantic interoperability. At the heart of cloud computing is the infrastructure, including a network of interconnected nodes.

[0083] Referring here to Figure 7, an exemplary cloud computing environment 750 is shown. As shown, the cloud computing environment 750 includes one or more cloud computing nodes 710 that can communicate with local computing devices used by cloud consumers, such as a personal digital assistant (PDA®) or mobile phone 754A, a desktop computer 754B, a laptop computer 754C, and / or an automotive computer system 754N. The nodes 710 can communicate with each other. They may be physically or virtually grouped (not shown) within one or more networks, such as a private cloud, community cloud, public cloud, or hybrid cloud, or a combination thereof, as described above. This allows the cloud computing environment 750 to provide infrastructure, platform and / or software as a service, eliminating the need for cloud consumers to maintain resources on their local computing devices. The types of computing devices 754A-N shown in Figure 7 are for illustrative purposes only, and it should be understood that the computing node 710 and the cloud computing environment 750 can communicate with any type of computerized device via any type of network and / or network-addressable connection (e.g., using a web browser).

[0084] Referring now to Figure 8, a set of functional abstraction layers provided by the cloud computing environment 750 (Figure 7) is shown. It should be understood in advance that the components, layers, and functions shown in Figure 8 are for illustrative purposes only, and embodiments of the present invention are not limited thereto. As shown, the following layers and corresponding functions are provided:

[0085] The hardware and software layer 860 includes hardware and software components. Examples of hardware components include a mainframe 861, a RISC (Reduced Instruction Set Computer) architecture-based server 862, a server 863, a blade server 864, a storage device 865, and network and networking components 866. In some embodiments, the software components include network application server software 867 and database software 868.

[0086] The virtualization layer 870 provides an abstraction layer from which the following examples of virtual entities may be provided: a virtual server 871, virtual storage 872, a virtual network 873 including a virtual private network, a virtual application and operating system 874, and a virtual client 875.

[0087] For example, the management layer 880 may provide the following functions: Resource provisioning 881 provides dynamic procurement of computing and other resources used to perform tasks within the cloud computing environment. Metering and pricing 882 provides cost tracking as resources are used within the cloud computing environment and billing or invoices for the consumption of these resources. For example, these resources may include application software licenses. Security provides identification and verification of cloud consumers and tasks, and protection for data and other resources. User portal 883 provides consumers and system administrators with access to the cloud computing environment. Service level management 884 provides allocation and management of cloud computing resources to ensure that the required service levels are met. Service level agreement (SLA) planning and execution 885 provides proactive preparation and procurement of cloud computing resources for which future requirements are anticipated in accordance with the SLA.

[0088] The workload layer 890 provides examples of functions that may be available in a cloud computing environment. Examples of workloads and functions that may be provided from this layer include mapping and navigation 891, software development and lifecycle management 892, virtual classroom education delivery 893, data analysis processing 894, transaction processing 895, and overall prosodic style conversion without text transcription 896.

[0089] The present invention may be a system, method, and / or computer program product in any possible level of technical detail. The computer program product may include a computer-readable storage medium (or more mediums) having computer-readable program instructions thereon for causing a processor to perform aspects of the present invention.

[0090] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium may, for example, be an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. A non-exhaustive list of further specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices such as punch cards or grooved structures on which instructions are recorded, and any suitable combination thereof. The computer-readable storage media used herein should not be interpreted as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through fiber optic cables), or electrical signals transmitted through wires.

[0091] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and translates them for storage on a computer-readable storage medium within each computing / processing device.

[0092] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as SMALLTALK®, C++, and conventional procedural programming languages ​​such as the "C" programming language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or on a server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or wide area network (WAN), or it may be connected to an external computer (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by utilizing state information of computer-readable program instructions for personalizing the electronic circuit in order to perform an aspect of the present invention.

[0093] Aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block in the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0094] These computer-readable program instructions are provided to the processors of general-purpose computers, dedicated computers, or other programmable data processing devices, which can generate machines, and as a result, instructions executed via the processors of the computer or other programmable data processing devices create means for performing functions / operations specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions can also be stored in computer-readable storage media that can instruct computers, programmable data processing devices, and / or other devices to function in a particular way, and as a result, computer-readable storage media having instructions stored therein comprises products containing instructions that perform modes of functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0095] Computer-readable program instructions can also be loaded into a computer, other programmable data processing device, or other device, causing a series of operational steps to be executed on the computer, other programmable device, or other device, thereby generating a computer implementation process, the resulting instructions executed on the computer, other programmable device, or other device, implement the functions / operations specified in one or more blocks of a flowchart and / or block diagram.

[0096] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagram may represent a module, segment, or part of an instruction having one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or the blocks may be executed in reverse order depending on the related functions. It should also be noted that each block in the block diagram and / or flowchart diagram, and combinations of blocks in the block diagram and / or flowchart diagram, can be implemented by a dedicated hardware-based system that performs a specified function or action, or a combination of dedicated hardware and computer instructions.

[0097] In this specification, references to “one embodiment” or “an embodiment” of the present invention, and to other variations thereof, mean that certain features, structures, properties, etc., described in relation to the embodiments are included in at least one embodiment of the present invention. Therefore, when the phrases “in one embodiment” or “in an embodiment” and any other variations appear in various places throughout this specification, they do not necessarily all refer to the same embodiment.

[0098] Please understand that the use of any of the following " / ", "and / or", and "at least one of" is intended to include, for example, "A / B", "A and / or B", and "at least one of A and B", the selection of only the first enumerated option (A), or only the second enumerated option (B), or the selection of both options (A and B). As a further example, in the case of "A, B, and / or C", and "at least one of A, B, and C", such phrasing is intended to include the selection of only the first enumerated option (A), or only the second enumerated option (B), or only the third enumerated option (C), or only the first and second enumerated options (A and B), or only the first and third enumerated options (A and C), or only the second and third enumerated options (B and C), or the selection of all three options (A, B, and C). This can be extended to the same number of items as listed, as is readily apparent to those skilled in the art.

[0099] Preferred embodiments of the system and method (intended to illustrate, but not limiting) have been described, but it should be noted that modifications and variations can be made by those skilled in the art with respect to the above teachings. Therefore, it should be understood that changes can be made within the specific embodiments disclosed within the scope of the present invention and outlined in the appended claims. Accordingly, aspects of the present invention have been described with the details and specificity required by patent law, but what is claimed and desired to be protected by patent is set forth in the appended claims.

Claims

1. A computer implementation method that uses a machine learning model to resolve prosody entanglement in natural language spoken by voice, wherein the computer implementation method is A step in which a computing device encodes the natural language spoken in order to generate content code; The steps include: applying unsupervised techniques to the machine learning model to generate prosodic ambiguous content code, the computing device resampling the content code without text transcription to make it prosodic, the content code being resampled using similarity-based random resampling techniques, and content code segments having similarity above a threshold being shortened using similarity-based downsampling or lengthened using similarity-based upsampling to equalize their lengths, thereby forming the prosodic ambiguous content code; and The computing device decodes the ambiguous content code in order to indirectly synthesize speech based on the content code. A computer implementation method comprising the following:

2. The computer implementation method according to claim 1, wherein the prosody conveys the rhythm and pitch of natural language through the aforementioned speech.

3. The computer implementation method according to claim 1, wherein the encoding step is performed by the encoder of the computing device, the resampling step is performed by the resampler of the computing device, and the decoding step is performed by the decoder of the computing device.

4. The computer implementation method according to claim 1, wherein the similarity exceeding the threshold is the prosodic similarity threshold.

5. The computer implementation method according to claim 1, wherein the resampling step is guided by self-expression learning.

6. The computer implementation method according to claim 1, wherein the rhythm component is made ambiguous within the content code by the step of resampling a plurality of prosodic components, thereby generating a content code with ambiguous prosodicity.

7. The computer implementation method according to claim 1, wherein the aforementioned ambiguous content code is decoded using a target area label.

8. The computer implementation method according to claim 7, wherein the target region label is a summary of a region that specifies at least one speaker identity and at least one sentiment category.

9. The computer implementation method according to claim 1, wherein the computer implementation method is configured as a cloud-based service having an encoding component, a resampling component, and a decoding component.

10. A computer implementation method for using a machine learning model to resolve prosody entanglement in natural language spoken by voice, wherein the computer implementation method is A step in which a computing device encodes the natural language spoken in order to generate content code; The steps include: applying unsupervised techniques to the machine learning model to generate prosodic ambiguous content code, thereby causing the computing device to resample the content code without text transcription in order to make the prosodic ambiguous; and The computing device decodes the ambiguous content code in order to indirectly synthesize speech based on the content code. Equipped with, The resampling step is performed by a two-stage training technique that prevents the encoder performing the encoding step from coordinating with the decoder performing the decoding step to encode and decode the rhythm information all at once. The two-stage training technique comprises a synchronous training portion that uses a sample length aligner to adjust the sample length and reconstruct the input prosody, and an asynchronous training portion without the sample length aligner to allow the decoder to infer the prosody. Computer implementation method.

11. A computer program for using a machine learning model to untangle prosody in natural speech, wherein the computer program comprises program instructions, and the program instructions are used by a computer In order to generate content code, the computer's encoder encodes the natural language spoken; To generate prosodic ambiguous content codes, an unsupervised technique is applied to the machine learning model, causing the computer's resampler to resample the content codes without text transcription in order to make them prosodic, where the content codes are resampled using similarity-based random resampling techniques, and content code segments with similarity exceeding a threshold are shortened using similarity-based downsampling or lengthened using similarity-based upsampling to equalize their lengths, thereby forming prosodic ambiguous content codes; and In order to indirectly synthesize speech based on the aforementioned content code, the computer's decoder is used to decode the content code with ambiguous prosody. A computer program that is executable by the computer.

12. The computer program according to claim 11, wherein the resampling step is guided by self-expression learning.

13. The computer program according to claim 11, wherein the aforementioned ambiguous content code is decoded using a target region label.

14. The computer program according to claim 13, wherein the target region label is a summary of a region that specifies at least one speaker identity and at least one sentiment category.

15. A computer program for using a machine learning model to untangle prosody in natural speech, wherein the computer program comprises program instructions, and the program instructions are transmitted to a computer In order to generate content code, the computer's encoder encodes the natural language spoken; To generate prosodic ambiguous content codes, an unsupervised technique is applied to the machine learning model, causing the computer's resampler to resample the content codes without text transcription in order to make them prosodic; and In order to indirectly synthesize speech based on the aforementioned content code, the computer's decoder is used to decode the content code with ambiguous prosody. Thus, it is executable by the aforementioned computer, The resampling step is performed by a two-stage training technique that prevents the encoder performing the encoding step from coordinating with the decoder performing the decoding step to encode and decode the rhythm information all at once. The two-stage training technique comprises a synchronous training portion that uses a sample length aligner to adjust the sample length and reconstruct the input prosody, and an asynchronous training portion without the sample length aligner to allow the decoder to infer the prosody. Computer program.

16. A memory device for storing program code; and To generate content code, natural language is encoded via speech; To generate prosodic ambiguous content codes, unsupervised techniques are applied to a machine learning model to resample the content codes without text transcription to make them prosodic, where the content codes are resampled using similarity-based random resampling techniques, and content code segments with similarity exceeding a threshold are shortened using similarity-based downsampling or lengthened using similarity-based upsampling to equalize their lengths, thereby forming prosodic ambiguous content codes; and In order to indirectly synthesize speech based on the aforementioned content code, the content code with ambiguous prosody is decoded. A processor device operably coupled to the memory device in order to execute the aforementioned program code. A comprehensive prosodic conversion system equipped with these features.

17. A memory device for storing program code; and To generate content code, natural language is encoded via speech; To generate prosodic ambiguous content codes, unsupervised techniques are applied to machine learning models to resample the content codes without text transcription in order to make them prosodic; and In order to indirectly synthesize speech based on the aforementioned content code, the content code with ambiguous prosody is decoded. A processor device operably coupled to the memory device in order to execute the aforementioned program code. Equipped with, The resampling step is performed by a two-stage training technique that prevents the encoder performing the encoding step from coordinating with the decoder performing the decoding step to encode and decode the rhythm information all at once. The two-stage training technique comprises a synchronous training portion that uses a sample length aligner to adjust the sample length and reconstruct the input prosody, and an asynchronous training portion without the sample length aligner to allow the decoder to infer the prosody. A comprehensive prosodic conversion system.

Images