Source speech modification based on input speech characteristics

Abstract

An apparatus includes one or more processors configured to process an input audio spectrum of an input speech to detect a first characteristic associated with the input speech. The one or more processors are further configured to select one or more reference inserts from a plurality of reference inserts based at least in part on the first characteristic. The one or more processors are further configured to process the representation of the source speech using the one or more reference embedding to generate an output audio spectrum of the output speech.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of priority to commonly owned U.S. non-provisional patent application No. 17 / 931,755, filed on September 13, 2022, the contents of which are expressly incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to modifying source speech based on characteristics of input speech to generate output speech. Background Art

[0004] Technological advances have led to smaller and more powerful computing devices. For example, there are currently a variety of portable personal computing devices, including small, lightweight and user-friendly wireless phones (such as mobile and smart phones, tablet computers and laptop computers). These devices can communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functions, such as digital still cameras, digital video cameras, digital recorders, and audio file players. In addition, such devices can process executable instructions, including software applications, such as web browser applications, which can be used to access the Internet. As such, these devices can include significant computing power.

[0005] Such computing devices often incorporate the function of receiving audio signals from one or more microphones. For example, the audio signal may represent the user's voice captured by the microphone, the external sound captured by the microphone, or a combination thereof. Such devices may include personal assistant applications, language translation applications, or other applications that generate audio signals representing voice for playback by one or more speakers. In some examples, the device incorporates the function of performing audio modification to have fixed predetermined characteristics. For example, configuration settings may be updated to adjust the bass in the source audio file. Voice modification based on the characteristics detected in the input voice representation is unavailable, which may result in limited enhancement possibilities. Summary of the invention

[0006] According to a specific implementation of the present disclosure, a device includes one or more processors configured to process an input audio spectrum of an input speech to detect a first characteristic associated with the input speech. The one or more processors are also configured to select one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic. The one or more processors are further configured to process a representation of a source speech using the one or more reference embeddings to generate an output audio spectrum of an output speech.

[0007] According to another specific implementation of the present disclosure, a method includes processing an input audio spectrum of an input speech at a device to detect a first characteristic associated with the input speech. The method also includes selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic. The method further includes processing a representation of a source speech using the one or more reference embeddings to generate an output audio spectrum of an output speech.

[0008] According to another specific implementation of the present disclosure, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to process an input audio spectrum of an input speech to detect a first characteristic associated with the input speech. When the instructions are executed by the one or more processors, the one or more processors are further caused to select one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic. When the instructions are executed by the one or more processors, the one or more processors are further caused to use the one or more reference embeddings to process a representation of a source speech to generate an output audio spectrum of an output speech.

[0009] According to another specific implementation of the present disclosure, a device includes a component for processing an input audio spectrum of an input speech to detect a first characteristic associated with the input speech. The device also includes a component for selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic. The device also includes a component for processing a representation of a source speech using the one or more reference embeddings to generate an output audio spectrum of an output speech.

[0010] Other aspects, advantages, and features of the present disclosure will become apparent upon review of the entire application, including the Brief Description of the Drawings, Detailed Description, and Claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of certain illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0012] Figure 2 According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of a characteristic detector of a system.

[0013] Figure 3A According to some examples of the present disclosure Figure 2 Illustration of illustrative aspects of the operation of an emotion detector of a characteristic detector.

[0014] Figure 3B According to some examples of the present disclosure Figure 2 Illustration of illustrative aspects of the operation of an emotion detector of a characteristic detector.

[0015] Figure 4 According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of the operation of an embedded selector of a system.

[0016] Figure 5A According to some examples of the present disclosure Figure 4 A diagram of illustrative aspects of the operation of an emotion adjuster embedded in a selector.

[0017] Figure 5B According to some examples of the present disclosure Figure 4 A diagram of illustrative aspects of the operation of an emotion adjuster embedded in a selector.

[0018] Figure 5C According to some examples of the present disclosure Figure 4 A diagram of illustrative aspects of the operation of an emotion adjuster embedded in a selector.

[0019] Figure 5D According to some examples of the present disclosure Figure 4 A diagram of illustrative aspects of the operation of an emotion adjuster embedded in a selector.

[0020] Figure 6 According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of the operation of an embedded selector of a system.

[0021] Fig. 7A According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of the operation of an embedded selector of a system.

[0022] Figure 7B According to some examples of the present disclosure Figure 1 Illustration of illustrative aspects of the operation of an embedded selector of a system.

[0023] Fig. 8A According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of a transformation embedding generator for a system.

[0024] Figure 8B According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of a transformation embedding generator for a system.

[0025] Figure 8C According to some examples of the present disclosure Figure 1 A diagram of illustrative aspects of the operation of a transformation embedding generator for a system.

[0026] Fig. 9is a block diagram of illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0027] Fig.10 is a block diagram of illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0028] Fig.11 is a block diagram of illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0029] Fig.12 According to some examples of the present disclosure Figures 9 to 11 A diagram of illustrative aspects of the operation of a representation generator for any system in the system.

[0030] Fig.13A is a block diagram of illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0031] Fig. 13B is a block diagram of illustrative aspects of a system operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0032] Fig.14 According to some examples of the present disclosure, it is possible to train Figure 1 Block diagram of illustrative aspects of a system of an audio analyzer of a system.

[0033] Fig.15 An example of an integrated circuit operable to perform source speech modification based on input speech characteristics according to some examples of the present disclosure is illustrated.

[0034] Fig.16 is a diagram of a mobile device operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0035] Fig.17 is a diagram of a headset operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0036] Fig.18 is a diagram of an earbud headset operable to perform source speech modification based on input speech characteristics according to some examples of the present disclosure.

[0037] Fig.19 is a diagram of a wearable electronic device operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0038] Fig. 20is a diagram of a voice-controlled speaker system operable to perform source speech modification based on input speech characteristics according to some examples of the present disclosure.

[0039] Fig.21 is a diagram of a camera operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0040] Fig. 22 is a diagram of a first example of a vehicle operable to perform source speech modification based on input speech characteristics according to some examples of the present disclosure.

[0041] Fig.23 is a diagram of a headset, such as an extended reality headset, operable to perform source voice modification based on input voice characteristics, according to some examples of the present disclosure.

[0042] Fig.24 is a diagram of glasses (such as extended reality glasses) that are operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure.

[0043] Fig.25 is a diagram of a second example of a vehicle operable to perform source speech modification based on input speech characteristics according to some examples of the present disclosure.

[0044] Fig.26 According to some examples of the present disclosure, Figure 1 A diagram of a specific implementation of a method for performing source voice modification based on input voice characteristics performed by a device.

[0045] Fig. 27 is a block diagram of a specific illustrative example of a device operable to perform source voice modification based on input voice characteristics according to some examples of the present disclosure. DETAILED DESCRIPTION

[0046] In some examples, the device incorporates functionality to perform audio modification to have fixed predetermined characteristics. For example, a configuration setting may be updated to adjust the bass in a source audio file. Voice modification based on characteristics detected in the input audio may result in various enhancement possibilities. In an example, a source voice generated, for example, by a personal assistant application may be updated to match voice characteristics detected in a user voice received from a microphone. For example, the user voice may have a higher intensity during the day and a lower intensity at night, and the source voice of the personal assistant may be adjusted to have a corresponding intensity. In some examples, the source voice may be adjusted to have a lower absolute intensity relative to the user voice. For example, when the user voice sounds tired, the source voice may be adjusted to sound calm, and when the user voice sounds excited, the source voice may be adjusted to sound happy.

[0047] Disclosed are systems and methods for performing source speech modification based on input speech characteristics. For example, an audio analyzer determines input characteristics of input speech audio. In some examples, the input speech audio may correspond to an input signal received from a microphone. The input characteristics may include emotion, speaker identity, speech style (e.g., volume, pitch, speed, etc.), or a combination thereof. The audio analyzer determines target characteristics based on the input characteristics and updates the source speech audio to have the target characteristics to generate output speech audio. In some examples, the source speech audio is generated by an application.

[0048] In some aspects, the target characteristic is the same as the input characteristic, so that the output voice audio sounds similar to the input voice audio (e.g., has the same characteristic as the input voice audio). For example, the output voice audio has the same intensity as the input voice audio. In some aspects, the target characteristic, although based on the input characteristic, is different from the input characteristic, so that the output voice audio changes based on the input voice audio, but sounds different from the input voice audio. For example, the output voice audio has a positive intensity relative to the input voice audio. For example, the mental health application is designed to generate a response (e.g., output voice audio) with a positive intensity relative to the received user voice (e.g., input voice audio).

[0049] Optionally, in some aspects, the source voice audio is the same as the input voice audio. For example, the audio analyzer updates the input voice audio received from the microphone based on the characteristics of the input voice audio to generate the output voice audio. For example, the output voice audio has a positive intensity relative to the input voice audio. For example, a user of a live game channel wants their voice to have a higher energy to attract the attention of the audience.

[0050] Certain aspects of the present disclosure are described below with reference to the accompanying drawings. In this specification, common features are designated by common reference numerals. As used herein, various terms are used only for the purpose of describing a particular implementation and are not intended to limit the implementation. For example, the singular forms "a", "an", and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. In addition, some features described herein are singular in some implementations and plural in other implementations. For illustration, Figure 1 Depicts a system including one or more processors ( Figure 1 190 ), indicating that in some implementations, the device 102 includes a single processor 190 , while in other implementations, the device 102 includes multiple processors 190 .

[0051] In some of the drawings, multiple instances of a particular type of feature are used. Although the features are physically and / or logically different, the same reference numeral is used for each feature, and the different instances are distinguished by adding a letter to the reference numeral. When features as a group or type are referred to herein (e.g., when no reference is made to a particular one of the features), the reference numeral is used without a distinguishing letter. However, when a particular feature of a plurality of features of the same type is referred to herein, the reference numeral is used with a distinguishing letter. For example, reference numerals are used to identify the features of the same type. Figure 4 , a plurality of operating modes are illustrated and are associated with reference numerals 105A and 105B. When referring to a particular one of the operating modes, such as operating mode 105A, a distinguishing letter "A" is used. However, when referring to any one of the operating modes or to the operating modes as a group, reference numeral 105 is used without a distinguishing letter.

[0052] As used herein, the term "comprise" may be used interchangeably with "include". Additionally, the term "wherein" may be used interchangeably with "wherein". As used herein, "exemplary" indicates an example, a specific implementation, and / or an aspect, and should not be interpreted as restrictive or indicating a preferred or preferred specific implementation. As used herein, ordinal terms (e.g., "first", "second", "third", etc.) used to modify elements (such as structures, components, operations, etc.) do not themselves indicate any priority or order of the element relative to another element, but only distinguish the element from another element with the same name (but using ordinal terms). As used herein, the term "set" refers to one or more specific elements in a specific element, and the term "plurality" refers to multiple (e.g., two or more) specific elements.

[0053] As used herein, "coupling" may include "communicatively coupled", "electrically coupled" or "physically coupled", and may also (or alternatively) include any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), etc. As illustrative, non-limiting examples, two electrically coupled devices (or components) may be included in the same device, or may be included in different devices, and may be connected via electronic devices, one or more connectors, or inductive coupling. In some specific implementations, two devices (or components) that are communicatively coupled (such as electrically connected) may directly or indirectly transmit and receive signals (e.g., digital signals or analog signals) via one or more wires, buses, networks, etc. As used herein, "direct coupling" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled or physically coupled) without intermediate components.

[0054] In the present disclosure, terms such as "determine", "calculate", "estimate", "shift", "adjust", etc. may be used to describe how to perform one or more operations. It should be noted that such terms should not be interpreted as limiting, and other technologies may be used to perform similar operations. Additionally, as mentioned herein, "generate", "calculate", "estimate", "use", "select", "access", and "determine" may be used interchangeably. For example, "generating", "calculating", "estimating" or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating or determining the parameter (or signal), or may refer to using, selecting or accessing a parameter (or signal) that has already been generated (such as by another component or device).

[0055] refer to Figure 1 , discloses certain illustrative aspects of a system configured to perform source voice modification based on input voice characteristics and is generally designated 100. The system 100 includes a device 102 that includes one or more processors 190. The one or more processors 190 include an audio analyzer 140 that is configured to perform source voice modification based on input voice characteristics. In certain aspects, the audio analyzer 140 is trained by a trainer, such as reference Fig.14 Further described.

[0056] The audio analyzer 140 includes an audio spectrum generator 150, which is coupled to a transform embedding generator 158 via a feature detector 154 and an embedding selector 156. The transform embedding generator 158 is coupled to an audio synthesizer 166 via a speech converter 164. In some aspects, the speech converter 164 corresponds to a generator and the audio synthesizer 166 corresponds to a decoder. Optionally, in some implementations, the speech converter 164 is also coupled to the transform embedding generator 158 via a baseline embedding generator 160.

[0057] The audio spectrum generator 150 is configured to generate an input audio spectrum 151 of an input speech representation 149 (e.g., a representation of the input speech). In an example, the input speech representation 149 corresponds to audio including the input speech, and the audio spectrum generator 150 is configured to apply a transform (e.g., a Fast Fourier Transform (FFT)) to the audio in the time domain to generate the input audio spectrum 151 in the frequency domain.

[0058] The characteristic detector 154 is configured to process the input audio spectrum 151 to detect input characteristics 155 associated with the input speech, as shown in FIG. Figure 2 As further described. Input characteristics 155 may include the emotion, style (e.g., volume, pitch, speed, or a combination thereof) or both of the input speech. In some aspects, characteristic detector 154 is configured to perform speaker recognition to determine that input audio spectrum 151 may correspond to the input speech of a particular user. In these aspects, input characteristics 155 may include a speaker identifier (e.g., a user identifier) ​​of the particular user.

[0059] The embedding selector 156 is configured to select one or more reference embeddings 157 from a plurality of reference embeddings based at least in part on the input characteristics 155, such as the reference Figures 4 to 7B For example, the embedding selector 156 is configured to determine a target characteristic 177 based on the input characteristic 155 and select one or more reference embeddings 157 corresponding to the target characteristic 177. For example, the reference embedding 157 may correspond to a specific emotion, a specific style, a specific speaker identifier, or a combination thereof.

[0060] In a specific implementation, a reference embedding 157 corresponding to a specific emotion (e.g., excitement) indicates a set (e.g., a vector) of speech feature values ​​(e.g., high pitch) indicating a specific emotion. In a specific implementation, a reference embedding 157 corresponding to a specific speaker identifier indicates a set (e.g., a vector) of speech feature values ​​indicating the speech of a specific speaker (e.g., a user) associated with the specific speaker identifier. In a specific implementation, a reference embedding 157 corresponding to a specific pitch indicates a set (e.g., a vector) of speech feature values ​​indicating a specific pitch. In a specific implementation, a reference embedding 157 corresponding to a specific speed indicates a set (e.g., a vector) of speech feature values ​​indicating a specific speed. In a specific implementation, a reference embedding 157 corresponding to a specific volume indicates a set (e.g., a vector) of speech feature values ​​indicating a specific volume.

[0061] Non-limiting examples of speech features include Mel-frequency cepstral coefficients (MFCC), shifted delta cepstral coefficients (SDCC), spectral centroid, spectral roll-off, spectral flatness, spectral contrast, spectral bandwidth, chrominance-based features, zero-crossing rate, root mean square energy, linear prediction cepstral coefficients (LPCC), spectral subband centroid, line spectral frequency, single-frequency cepstral coefficients, formant frequency, power normalized cepstral coefficients (PNCC), or a combination thereof.

[0062] The audio analyzer 140 is configured to process a source speech representation 163 (e.g., a representation of a source speech) using one or more reference embeddings 157 to generate an output audio spectrum 165 of an output speech. As an illustrative example, the source speech representation 163 is processed using one or more reference embeddings 157 corresponding to a single input speech representation 149. In other examples, a set of one or more reference embeddings 157 corresponding to a plurality of input speech representations 149 may be used to process the source speech representation 163, such as a reference Figure 8C Further described.

[0063] In an example, the transformation embedding generator 158 is configured to generate a transformation embedding 159 based on one or more reference embeddings 157, such as reference FIG. 8A to FIG. 8CFurther described. In certain aspects, one or more reference embeddings 157 include a single reference embedding, and the conversion embedding 159 is the same as the single reference embedding. In some aspects, one or more reference embeddings 157 include multiple reference embeddings, and the conversion embedding 159 is a combination of multiple reference embeddings. The speech converter 164 is configured to apply the conversion embedding 159 to the source speech representation 163 to generate an output audio spectrum 165 for the output speech. For example, the conversion embedding 159 corresponds to a set (e.g., a vector) of first speech feature values, and applying the conversion embedding 159 to the source speech representation 163 corresponds to adjusting the second speech feature value of the source speech representation 163 based on the first speech feature value to generate the output audio spectrum 165. In a specific implementation, a specific second speech feature value of the source speech representation 163 is replaced or modified based on the corresponding first speech feature value of the conversion embedding 159.

[0064] In certain implementations, source speech representation 163 includes encoded source speech. Speech converter 164 applies transform embedding 159 to the encoded source speech to generate transformed encoded source speech, and decodes the transformed encoded source speech to generate output audio spectrum 165.

[0065] The audio synthesizer 166 is configured to process the output audio spectrum 165 to generate the output signal 135. For example, the audio synthesizer 166 is configured to apply a transform (e.g., an inverse FFT (iFFT)) to the output audio spectrum 165 to generate the output signal 135. The output signal 135 has an output characteristic that matches a target characteristic 177. In some examples, the target characteristic 177 is the same as the input characteristic 155. In these examples, the output characteristic matches the input characteristic 155. For illustration, a first speech characteristic of the output signal 135 (representing the output speech) matches a second speech characteristic of the input speech representation 149 (representing the input speech). In particular aspects, a "speech characteristic" corresponds to a speech feature.

[0066] In a specific implementation including the baseline embedding generator 160, the speech converter 164 is further configured to provide the output audio spectrum 165 to the baseline embedding generator 160. The baseline embedding generator 160 is configured to determine a baseline embedding 161 based at least in part on the output audio spectrum 165, and to provide the baseline embedding 161 to the transformation embedding generator 158. The transformation embedding generator 158 is configured to generate a subsequent transformation embedding based at least in part on the baseline embedding 161. Using the baseline embedding generator 160, a gradual change in the characteristics of the output speech in the output signal 135 can be achieved.

[0067] In some implementations, device 102 corresponds to or is included in one of various types of devices. In the illustrative example, one or more processors 190 are integrated into a headphone device (such as a headset device). Fig.17 described) or in-ear headphones (as described in Fig.18 In other examples, the one or more processors 190 are integrated into at least one of the following: Fig.16 The mobile phone or tablet computer device described, as referenced Fig.19 The wearable electronic device described in reference Fig. 20 The voice controlled speaker system described in reference Fig.21 The camera device described in reference Fig.23 The extended reality headset described, or as described in reference Fig.24 In another illustrative example, one or more processors 190 are integrated into a vehicle, such as a vehicle shown in FIG. Fig. 22 and Fig.25 As further described.

[0068] During operation, the audio spectrum generator 150 is configured to obtain an input speech representation 149 of the input speech. In some examples, the input speech representation 149 is based on the input speech audio. For illustration, the input speech representation 149 may be based on one or more input audio signals received from one or more microphones that capture the input speech, such as reference Fig. 9 In another example, the input speech representation 149 may be based on one or more input audio signals generated by an application of the device 102 or another device.

[0069] In an example, the input speech representation 149 may be based on an input speech text (e.g., a script, a chat session, etc.). For illustration, the audio spectrum generator 150 performs text-to-speech conversion on the input speech text to generate the input speech audio. In some implementations, the input speech text is associated with one or more characteristic indicators (such as an emotion indicator, a style indicator, a speaker indicator, or a combination thereof). The emotion indicator may include punctuation (e.g., an exclamation mark to indicate surprise), a word (e.g., "I am happy"), an emoticon (e.g., a smiley face), etc. The style indicator may include words that are generally associated with a particular style (e.g., "you guys"), metadata indicating a style, or both. The speaker indicator may include one or more speaker identifiers. In some aspects, the text-to-speech conversion generates the input speech audio to include characteristics, such as an emotion indicated by an emotion indicator, a style indicated by a style indicator, speech characteristics corresponding to a speaker indicator, or a combination thereof.

[0070] In some aspects, the input speech representation 149 includes at least one of an input speech spectrum, linear predictive coding (LPC) coefficients, or MFCCs of the input speech audio. In some examples, the input speech representation 149 is based on decoded data. For example, a decoder of the device 102 receives encoded data from another device and decodes the encoded data to generate the input speech representation 149, as described in reference to Fig. 13B Further described.

[0071] The audio spectrum generator 150 generates an input audio spectrum 151 of the input speech representation 149. For example, the audio spectrum generator 150 applies a transform (e.g., a fast Fourier transform (FFT)) to the input speech audio in the time domain to generate the input audio spectrum 151 in the frequency domain. FFT is provided as an illustrative example of a transform applied to the input speech audio to generate the input audio spectrum 151. In other examples, the audio spectrum generator 150 may process the input speech representation 149 using various transforms and techniques to generate the input audio spectrum 151. The audio spectrum generator 150 provides the input audio spectrum 151 to the characteristics detector 154.

[0072] The characteristic detector 154 processes the input audio spectrum 151 of the input speech to detect the input characteristic 155 associated with the input speech, as shown in FIG. Figure 2 As further described, for example, input characteristics 155 indicate a mood, style, speaker identifier, or a combination thereof associated with the input speech.

[0073] Optionally, in some examples, characteristic detector 154 determines input characteristic 155 (e.g., emotion, style, speaker identifier, or a combination thereof) based at least in part on image data 153, user input 103 from user 101, or both, as described in reference to FIG. Figure 2 In some aspects, the image data 153 corresponds to an image (e.g., a still image, an image frame from a video, a generated image, or a combination thereof) associated with the input speech. For example, a camera captures an image concurrently with a microphone capturing the input speech, as described in reference to Fig. 9 In some examples, the encoded data received from another device includes image data 153, input speech representation 149, or both, as described in reference to Fig. 13B In some examples, user input 103 indicates a speaker identifier. Feature detector 154 provides input feature 155 to embedding selector 156 .

[0074] In some examples, target characteristic 177 is the same as input characteristic 155. Optionally, in some examples, embedding selector 156 maps input characteristic 155 to target characteristic 177 according to operating mode 105, as shown in FIG. Figures 4 to 5CIn some aspects, the operating mode 105 is based on configuration settings, default data, user input, or a combination thereof.

[0075] The embedding selector 156 selects one or more reference embeddings 157 corresponding to the target feature 177 from a plurality of reference embeddings, such as the reference Figures 6 to 7B For example, one or more reference embeddings 157 include one or more emotion reference embeddings corresponding to the emotion indicated by target characteristic 177, one or more style reference embeddings corresponding to the style indicated by target characteristic 177, one or more speaker reference embeddings corresponding to the speaker identifier indicated by target characteristic 177, or a combination thereof.

[0076] Optionally, in some aspects, one or more reference embeddings 157 include a plurality of reference embeddings, and embedding selector 156 determines weights 137 associated with the plurality of one or more reference embeddings 157. For example, one or more reference embeddings 157 include a first emotion reference embedding and a second emotion reference embedding. In this example, weights 137 include a first weight and a second weight associated with the first emotion reference embedding and the second emotion reference embedding, respectively.

[0077] Transformation embedding generator 158 generates transformation embedding 159 based at least in part on one or more reference embeddings 157. In some examples, one or more reference embeddings 157 include a single reference embedding, and transformation embedding 159 is the same as the single reference embedding. In some examples, one or more reference embeddings 157 include multiple reference embeddings, and transformation embedding generator 158 combines the multiple reference embeddings to generate transformation embedding 159, such as reference embeddings 157 and 157. FIG. 8A to FIG. 8C Optionally, in some implementations, the transformation embedding generator 158 combines one or more reference embeddings 157 and baseline embeddings 161 to generate a transformation embedding 159, such as reference Figure 8B In certain aspects, baseline embedding generator 160 generates and updates baseline embedding 161 during an audio analysis session of audio analyzer 140 such that changes in characteristics of output signal 135 are gradual. Transformation embedding generator 158 provides transformation embedding 159 to speech converter 164.

[0078] The speech converter 164 obtains a source speech representation 163 of the source speech. In some aspects, the input speech is used as the source speech. In other aspects, the input speech is different from the source speech. In certain aspects, the device 102 includes a representation generator configured to generate the source speech representation 163, as shown in FIG. Fig.12In some examples, the source speech representation 163 is based on source speech audio. For illustration, the source speech representation 163 may be based on one or more source audio signals received from one or more microphones that capture the source speech, such as reference Fig.10 In another example, the source speech representation 163 can be based on one or more source audio signals generated by an application of the device 102 or another device.

[0079] In an example, the source speech representation 163 may be based on a source speech text (e.g., a script, a chat session, etc.). For illustration, the speech converter 164 performs text-to-speech conversion on the source speech text to generate the source speech audio. In some implementations, the source speech text is associated with one or more characteristic indicators (such as an emotion indicator, a style indicator, a speaker identifier, or a combination thereof). The emotion indicator may include punctuation (e.g., an exclamation mark to indicate surprise), a word (e.g., "I am happy"), an emoticon (e.g., a smiley face), etc. The style indicator may include words (e.g., "you guys") that are typically associated with a particular style, metadata indicating a style, or both. In some aspects, the text-to-speech conversion generates the source speech audio to include characteristics, such as the emotion indicated by the emotion indicator, the style indicated by the style indicator, speech characteristics corresponding to the speaker identifier, or a combination thereof.

[0080] In some aspects, the source speech representation 163 is based on at least one of the source speech audio, the source speech spectrum of the source speech audio, the LPC coefficients of the source speech audio, or the MFCC of the source speech audio. In some examples, the source speech representation 163 is based on decoded data. For example, a decoder of the device 102 receives encoded data from another device and decodes the encoded data to generate the source speech representation 163, as described in reference to Fig. 13B Further described.

[0081] The speech converter 164 is configured to apply the conversion embedding 159 to the source speech representation 163 to generate an output audio spectrum 165 of the output speech. For example, the source speech representation 163 indicates the source speech amplitude associated with a specific frequency. Based on determining that the conversion embedding 159 indicates the adjustment amplitude for a specific frequency, the speech converter 164 determines the output speech amplitude based on the source speech amplitude, the adjustment amplitude, or both. In a specific example, the speech converter 164 determines the output speech amplitude by adjusting the source speech amplitude based on the adjustment amplitude. In another example, the output speech amplitude is the same as the adjustment amplitude. The speech converter 164 generates an output audio spectrum 165 indicating the output speech amplitude for a specific frequency. The speech converter 164 provides the output audio spectrum 165 to the audio synthesizer 166.

[0082] The audio synthesizer 166 generates an output speech representation (e.g., a representation of the output speech) based on the output audio spectrum 165. For example, the audio synthesizer 166 applies a transform (e.g., iFFT) to the output audio spectrum 165 to generate an output signal 135 (e.g., an audio signal) representing the output speech. In some examples, the audio synthesizer 166 performs a speech-to-text conversion on the output signal 135 to generate an output speech text. In certain aspects, the output speech representation includes the output signal 135, the output speech text, or both. In certain aspects, the input speech representation 149 includes the input speech text, and the output speech representation includes the output speech text.

[0083] In certain aspects, the output speech representation has the target characteristic 177. For example, the output signal 135 includes the output speech audio having the target characteristic 177. As another example, the output speech text includes a characteristic indicator (e.g., a word, an emoticon, a speaker identifier, metadata, etc.) corresponding to the target characteristic 177.

[0084] The audio analyzer 140 provides the output speech representation (e.g., the output signal 135, the output speech text, or both) to one or more devices, such as a speaker, a storage device, a network device, another device, or a combination thereof. In some examples, the audio analyzer 140 outputs the output signal 135 via one or more speakers, as shown in FIG. Fig.11 In some examples, the audio analyzer 140 encodes the output signal 135 to generate encoded data and provides the encoded data to another device, such as reference Fig.13A Further described.

[0085] In a specific example, the audio analyzer 140 receives input speech of the user 101 via one or more microphones, updates the input speech based on the input characteristics 155 of the input speech (e.g., uses the input speech as the source speech and updates the source speech representation 163) to generate output speech (e.g., output signal 135). For illustration, the user 101 streams a game channel, and the output speech has a target characteristic 177 that is amplified relative to the input characteristic 155.

[0086] In a specific example, the audio analyzer 140 receives input speech from another device, and updates the source speech (e.g., source speech representation 163) based on the input characteristics 155 of the input speech to generate output speech (e.g., output signal 135). For illustration, the audio analyzer 140 receives input speech from another device during a call with the device, receives the source speech of the user 101 via one or more microphones, and updates the source speech (e.g., source speech representation 163) based on the input characteristics 155 of the input speech to generate output speech (e.g., output signal 135) transmitted to the other device. In a specific aspect, the output speech has a positive intensity relative to the input speech.

[0087] Thus, system 100 enables dynamically updating source speech based on characteristics of input speech to generate output speech. In some aspects, the source speech is updated in real time. For example, device 102 receives data corresponding to the input speech, data corresponding to the source speech, or both, concurrently with audio analyzer 140 providing output signal 135 to a playback device (e.g., a speaker, another device, or both).

[0088] refer to Figure 2 , a diagram 200 showing illustrative aspects of the operation of the characteristic detector 154 is shown. The characteristic detector 154 includes an emotion detector 202, a speaker detector 204, a style detector 206, or a combination thereof. The style detector 206 includes a volume detector 212, a pitch detector 214, a speed detector 216, or a combination thereof.

[0089] The feature detector 154 is configured to process (e.g., using a neural network or other feature detection technique) the image data 153, the input audio spectrum 151, the user input 103, or a combination thereof to determine input features 155. The input features 155 include an emotion 267, a volume 272, a pitch 274, a speed 276, or a combination thereof detected as corresponding to the input speech associated with the input audio spectrum 151. In some examples, the input features 155 include a speaker identifier 264 of a predicted speaker (e.g., a person, a character, etc.) of the input speech associated with the input audio spectrum 151.

[0090] In certain aspects, the emotion detector 202 is configured to determine the emotion 267 based on the image data 153, the input audio spectrum 151, or both, as shown in FIG. FIG. 3A to FIG. 3B In some implementations, the emotion detector 202 includes one or more neural networks trained to process the image data 153, the input audio spectrum 151, or both to determine the emotion 267, as described in reference FIG. 3A to FIG. 3B Further described.

[0091] In some examples, the emotion detector 202 processes the input audio spectrum 151 using audio emotion detection techniques to detect a first emotion of the input voice representation 149. In some examples, the emotion detector 202 processes the image data 153 using image emotion analysis techniques to detect a second emotion. For illustration, the emotion detector 202 performs facial detection on the image data 153 to determine that a face is detected in a facial portion of the image data 153, and performs facial emotion detection on the facial portion to detect the second emotion. In a particular aspect, the emotion detector 202 performs context detection on the image data 153 to determine a context and a corresponding context emotion. For example, a particular context (e.g., a concert) is mapped to a particular context emotion (e.g., excitement). The second emotion is based on the context emotion, the facial emotion detected in the facial portion, or both.

[0092] The emotion detector 202 determines the emotion 267 based on the first emotion, the second emotion, or both. For example, the emotion 267 corresponds to an average of the first emotion and the second emotion. For illustration, the first emotion is represented by a first coordinate in the emotion map, and the second emotion is represented by a second coordinate in the emotion map, as shown in FIG. Figure 3A As further described, sentiment 267 corresponds to a midpoint (eg, an average) between the first coordinate and the second coordinate in the sentiment map.

[0093] In certain aspects, speaker detector 204 is configured to determine speaker identifier 264 based on image data 153, input audio spectrum 151, user input 103, or a combination thereof. In certain implementations, speaker detector 204 performs facial recognition (e.g., using a neural network or other facial recognition technology) on image data 153 to detect a face and predict that the face may correspond to a user (e.g., a person, a character, etc.) associated with a user identifier. Speaker detector 204 selects the user identifier as the speaker identifier for the image prediction.

[0094] In a particular implementation, speaker detector 204 performs speaker identification (e.g., using a neural network or other speaker identification technique) on input audio spectrum 151 to predict that speech characteristics indicated by input audio spectrum 151 may correspond to a user (e.g., a person, a character, etc.) associated with a user identifier, and selects the user identifier as a speaker identifier for the audio prediction.

[0095] In a particular implementation, the user input 103 indicates a user-predicted speaker identifier. As an example, the user input 103 indicates a logged-in user. As another example, the user input 103 indicates a call with a particular user and receiving input speech during the call, and the user-predicted speaker identifier corresponds to a user identifier of the particular user.

[0096] The speaker detector 204 determines the speaker identifier 264 based on the image-predicted speaker identifier, the audio-predicted speaker identifier, the user-predicted speaker identifier, or a combination thereof. For example, in a specific implementation where the speaker detector 204 generates a single predicted speaker identifier among the image-predicted speaker identifier, the audio-predicted speaker identifier, or the user-predicted speaker identifier, the speaker detector 204 selects the single predicted speaker identifier as the speaker identifier 264.

[0097] In specific implementations where the speaker detector 204 generates multiple predicted speaker identifiers among the image predicted speaker identifier, the audio predicted speaker identifier, or the user predicted speaker identifier, the speaker detector 204 selects one of the multiple predicted speaker identifiers as the speaker identifier 264. For example, the speaker detector 204 selects the speaker identifier 264 based on confidence scores associated with the multiple predicted speaker identifiers, priorities associated with the multiple predicted speaker identifiers, or a combination thereof. In particular aspects, the priorities associated with the predicted speaker identifiers are based on default data, configuration settings, user input, or a combination thereof.

[0098] In certain aspects, the style detector 206 is configured to determine a volume 272, a pitch 274, a speed 276, or a combination thereof based on the input audio spectrum 151. In some implementations, the volume detector 212 processes (e.g., using a neural network or other volume detection technique) the input audio spectrum 151 to determine the volume 272. In some implementations, the pitch detector 214 processes (e.g., using a neural network or other pitch detection technique) the input audio spectrum 151 to determine the pitch 274. In some implementations, the speed detector 216 processes (e.g., using a neural network or other speed detection technique) the input audio spectrum 151 to determine the speed 276.

[0099] refer to Figure 3A , a diagram 300 showing illustrative aspects of the operation of the emotion detector 202 is shown. The emotion detector 202 includes an audio emotion detector 354.

[0100] The audio emotion detector 354 performs audio emotion detection (e.g., using a neural network or other audio emotion detection techniques) on the input audio spectrum 151 to determine an audio emotion 355. In some implementations, the audio emotion detection includes determining an audio emotion confidence score 355 associated with the audio emotion. The emotion 267 includes the audio emotion 355.

[0101] Diagram 300 includes emotion map 347. In certain aspects, emotion 267 corresponds to a particular value on emotion map 347. In some examples, the horizontal value (e.g., x-coordinate) of the particular value indicates the valence of emotion 267, and the vertical value (e.g., y-coordinate) of the particular value indicates the intensity of emotion 267.

[0102] The distance (e.g., Cartesian distance) between a pair of emotions 267 indicates the similarity between the emotions 267. For example, the emotion map 347 indicates a first distance (e.g., first Cartesian distance) between a first coordinate corresponding to the emotion 267A (e.g., anger) and a second coordinate corresponding to the emotion 267B (e.g., relaxation) and a second distance (e.g., second Cartesian distance) between the first coordinate corresponding to the emotion 267A and a third coordinate corresponding to the emotion 267C (e.g., sadness). The second distance is less than the first distance, indicating that the emotion 267A (e.g., anger) is more similar to the emotion 267C (e.g., sadness) than to the emotion 267B (e.g., relaxation).

[0103] As a non-limiting example, the emotion map 347 is illustrated as a two-dimensional space. In other examples, the emotion map 347 can be a multi-dimensional space.

[0104] refer to Figure 3B , a diagram 350 showing illustrative aspects of the operation of the emotion detector 202 is shown. The emotion detector 202 includes an audio emotion detector 354, an image emotion detector 356, or both. In some implementations, the emotion detector 202 includes a emotion analyzer 358 coupled to the audio emotion detector 354 and the image emotion detector 356.

[0105] In some implementations, emotion detector 202 performs facial detection on image data 153 and determines emotion 267 based at least in part on an output of the facial detection. For example, facial detection indicates that a facial image portion of image data 153 corresponds to a face. In certain implementations, emotion detector 202 processes the facial image portion (e.g., using a neural network or other facial emotion detection technique) to determine a predicted facial emotion.

[0106] In some examples, the emotion detector 202 performs context detection on the image data 153 (e.g., using a neural network or other context detection technique) and determines the emotion 267 based at least in part on the output of the context detection. For example, the context detection indicates that the image data 153 corresponds to a particular context (e.g., a party, a concert, a meeting, etc.), and the emotion detector 202 determines a predicted context emotion (e.g., excitement) corresponding to the particular context (e.g., a concert). In certain aspects, the emotion detector 202 determines the image emotion 357 based on the predicted facial emotion, the predicted context emotion, or both. In some implementations, the emotion detector 202 determines an image emotion confidence score associated with the image emotion 357.

[0107] The emotion detector 202 determines the emotion 267 based on the audio emotion 355, the image emotion 357, or both. For example, the emotion analyzer 358 determines the emotion 267 based on the audio emotion 355 and the image emotion 357. In a specific implementation, the emotion analyzer 358 selects one of the audio emotion 355 or the image emotion 357 with a higher confidence score as the emotion 267. In a specific implementation, in response to determining that a single emotion of the audio emotion 355 or the image emotion 357 is associated with a confidence score greater than a threshold, the emotion analyzer 358 selects the single emotion of the audio emotion 355 or the image emotion 357 as the emotion 267.

[0108] In a particular implementation, the emotion analyzer 358 determines an average value (e.g., average x-coordinate and average y-coordinate) of the audio emotion 355 and the image emotion 357 as the emotion 267. For example, in response to determining that each of the audio emotion 355 and the image emotion 357 is associated with a corresponding confidence score greater than a threshold confidence score, the emotion analyzer 358 determines the average value of the audio emotion 355 and the image emotion 357 as the emotion 267.

[0109] refer to Figure 4 , a diagram 400 showing illustrative aspects of the operation of the embedding selector 156. In certain aspects, the embedding selector 156 initializes the target characteristic 177 to be the same as the input characteristic 155. Optionally, in some implementations, the embedding selector 156 includes a characteristic adjuster 492 configured to update the target characteristic 177 based on the input characteristic 155 and the operating mode 105.

[0110] In certain aspects, the operating mode 105 is based on default data, configuration settings, user input, or a combination thereof.The characteristic adjuster 492 includes an emotion adjuster 452, a speaker adjuster 454, a volume adjuster 456, a pitch adjuster 458, a speed adjuster 460, or a combination thereof.

[0111] Emotion adjuster 452 is configured to update emotion 267 of target characteristic 177 based on operating mode 105. In a particular implementation, emotion adjuster 452 uses emotion adjustment data 449 to map an original emotion (e.g., emotion 267 indicated by input characteristic 155) to a target emotion (e.g., emotion 267 to be included in target characteristic 177). For example, in response to determining that operating mode 105 corresponds to operating mode 105A (e.g., "positive boost"), emotion adjuster 452 updates emotion 267 based on emotion adjustment data 449A, as shown in FIG. Figure 5A Further described.

[0112] In another example, in response to determining that the operating mode 105 corresponds to the operating mode 105B (eg, "complementary"), the emotion adjuster 452 updates the emotion 267 based on the emotion adjustment data 449B, as shown in FIG. Figure 5B In yet another example, in response to determining that the operating mode 105 corresponds to the operating mode 105C (eg, “fluent”), the emotion adjuster 452 updates the emotion 267 based on the emotion adjustment data 449C, as shown in FIG. Figure 5C In certain aspects, the operating mode 105 is based on a user selection of one of a plurality of operating modes, such as the operating mode 105A, the operating mode 105B, the operating mode 105C, or a combination thereof.

[0113] In certain aspects, emotion adjustment data 449A indicates a first mapping between the emotions indicated in emotion map 347. Emotion adjustment data 449B indicates a second mapping between the emotions indicated in emotion map 347. Emotion adjustment data 449C indicates a third mapping between the emotions indicated in emotion map 347. In some aspects, the second mapping includes at least one mapping not included in the first mapping, the first mapping includes at least one mapping not included in the second mapping, or both. In some aspects, the third mapping includes at least one mapping not included in the first mapping, the first mapping includes at least one mapping not included in the third mapping, or both. In some aspects, the third mapping includes at least one mapping not included in the second mapping, the second mapping includes at least one mapping not included in the third mapping, or both.

[0114] In some aspects, the operating mode 105 indicates a specific emotion, and the emotion adjuster 452 sets the emotion 267 of the target characteristic 177 to the specific emotion, such as reference Figure 5D For example, the operating mode 105 is based on a user selection of a particular emotion. In some aspects, the emotion adjustment data 449 does not include a mapping for a particular original emotion, and the emotion adjuster 452 estimates a mapping from the particular original emotion to the particular target emotion based on one or more other mappings, such as reference Figure 7B Further described.

[0115] Speaker adaptor 454 is configured to update speaker identifier 264 of target characteristics 177 based on operating mode 105. In a particular implementation, operating mode 105 includes speaker mapping data indicating that an original speaker identifier (e.g., speaker identifier 264 indicated in input characteristics 155) is to be mapped to a particular target speaker identifier, and speaker adaptor 454 updates target characteristics 177 to indicate the particular target speaker identifier as speaker identifier 264. For example, operating mode 105 is based on a user selection indicating that the voice of a first user (e.g., Susan) associated with the original speaker identifier is to be modified to sound like the voice of a second user (e.g., Tom) associated with the particular target speaker identifier.

[0116] In a particular implementation, the operating mode 105 indicates selection of a particular target speaker identifier, and the speaker adaptor 454 updates the target characteristics 177 to indicate the particular target speaker identifier as the speaker identifier 264. For example, the operating mode 105 is based on a user selection indicating that the speech is to be modified to sound like the speech of a user (e.g., a person, a character, etc.) associated with the particular target speaker identifier.

[0117] Volume adjuster 456 is configured to update volume 272 of target characteristic 177 based on operating mode 105. In a particular implementation, operating mode 105 includes volume mapping data indicating that the original volume (e.g., volume 272 indicated in input characteristic 155) is to be mapped to a particular target volume, and volume adjuster 456 updates target characteristic 177 to indicate the particular target volume as volume 272. For example, operating mode 105 is based on a user selection indicating that the volume is to be reduced by a particular amount. Volume adjuster 456 determines the particular target volume based on the difference between volume 272 and the particular amount, and updates target characteristic 177 to indicate the particular target volume as volume 272. In a particular implementation, operating mode 105 indicates selection of the particular target volume, and speaker adjuster 454 updates target characteristic 177 to indicate the particular target volume as volume 272.

[0118] The pitch adjuster 458 is configured to update the pitch 274 of the target characteristic 177 based on the operating mode 105. In a particular implementation, the operating mode 105 includes pitch mapping data indicating that the original pitch (e.g., the pitch 274 indicated in the input characteristics 155) is to be mapped to a particular target pitch, and the pitch adjuster 458 updates the target characteristic 177 to indicate the particular target pitch as the pitch 274. For example, the operating mode 105 is based on a user selection that the pitch is to be reduced by a particular amount. The pitch adjuster 458 determines the particular target pitch based on the difference between the pitch 274 and the particular amount, and updates the target characteristic 177 to indicate the particular target pitch as the pitch 274. In a particular implementation, the operating mode 105 indicates the selection of the particular target pitch, and the speaker adjuster 454 updates the target characteristic 177 to indicate the particular target pitch as the pitch 274.

[0119] The speed adjuster 460 is configured to update the speed 276 of the target characteristic 177 based on the operating mode 105. In a particular implementation, the operating mode 105 includes speed mapping data indicating that the original speed (e.g., the speed 276 indicated in the input characteristics 155) is to be mapped to a particular target speed, and the speed adjuster 460 updates the target characteristic 177 to indicate the particular target speed as the speed 276. For example, the operating mode 105 is based on a user selection indicating that the speed is to be reduced by a particular amount. The speed adjuster 460 determines the particular target speed based on the difference between the speed 276 and the particular amount, and updates the target characteristic 177 to indicate the particular target speed as the speed 276. In a particular implementation, the operating mode 105 indicates the selection of the particular target speed, and the speaker adjuster 454 updates the target characteristic 177 to indicate the particular target speed as the speed 276.

[0120] The embedding selector 156 determines one or more reference embeddings 157 associated with the target characteristic 177 based on the characteristic mapping data 457, such as the reference Figure 6 The characteristic adjuster 492 enables dynamic selection of one or more reference embeddings 157 corresponding to the target characteristic 177 based on the input characteristic 155 .

[0121] refer to Figure 5A , showing Figure 4 Graph 500 of illustrative aspects of the operation of emotion adjuster 452. Graph 500 includes an example of emotion adjustment data 449A corresponding to operating mode 105A (eg, positive promotion).

[0122] The emotion adjustment data 449A indicates that each original emotion in the emotion map 347 is mapped to a corresponding target emotion in the emotion map 347 that has a higher (e.g., positive) intensity, a higher (e.g., positive) valence, or both relative to the original emotion. For example, the first original emotion (e.g., anger) is mapped to the first target emotion (e.g., excitement), the second original emotion (e.g., sadness) is mapped to the second target emotion (e.g., happiness), and the third original emotion (e.g., relaxation) is mapped to the third target emotion (e.g., joy). The first target emotion, the second target emotion, and the third target emotion have a higher intensity and a higher valence than the first original emotion, the second original emotion, and the third original emotion, respectively.

[0123] As an illustrative example, emotion adjustment data 449A is provided that indicates a mapping of three original emotions to three target emotions. In other examples, emotion adjustment data 449A may include fewer than three mappings or more than three mappings.

[0124] When the operating mode 105A is selected (e.g., positive boost), the emotion adjustment data 449A causes the embedding selector 156 to select a target emotion (e.g., emotion 267 of the target characteristic 177) that enables the audio analyzer 140 to generate an audio signal corresponding to a positive emotion relative to the original emotion of the input speech representation 149 (e.g., emotion 267 of the input characteristic 155). Figure 1 The output signal 135 of FIG. 101 is a signal that is generated by the user 101. In an example, the user 101 selects the operation mode 105A (e.g., aggressive boost) to increase the positivity and energy of the voice in the live video, where the input voice is used as the source voice. In another example, the user 101 selects the operation mode 105A (e.g., aggressive boost) to increase the positivity and energy of the voice in a marketing call, where the input voice corresponds to the voice of the recipient of the call and the source voice corresponds to the recorded message.

[0125] refer to Figure 5B , showing Figure 4 520 of illustrative aspects of the operation of the emotion adjuster 452. The diagram 520 includes an example of emotion adjustment data 449B corresponding to the operating mode 105B (eg, complementary).

[0126] The emotion adjustment data 449B indicates that each original emotion in the emotion map 347 is mapped to a corresponding target emotion in the emotion map 347 that has a complementary (e.g., opposite) intensity, a complementary (e.g., opposite) valence, or both relative to the original emotion. In a particular aspect, a first specific emotion is represented by a first horizontal coordinate (e.g., 10 as the x-coordinate) and a first vertical coordinate (e.g., 5 as the y-coordinate). A second specific emotion that is complementary to the first specific emotion has a second horizontal coordinate (e.g., -10 as the x-coordinate) and a second vertical coordinate (e.g., -5 as the y-coordinate). The second horizontal coordinate is the negative value of the first horizontal coordinate, and the second vertical coordinate is the negative value of the first vertical coordinate.

[0127] The emotion adjustment data 449B indicates that a first emotion (e.g., anger) maps to a second emotion (e.g., relaxation), and vice versa. As another example, a third emotion (e.g., sadness) maps to a fourth emotion (e.g., happiness), and vice versa. The first emotion (e.g., anger) has a complementary intensity and a complementary valence relative to the second emotion (e.g., relaxation). The third emotion (e.g., sadness) has a complementary intensity and a complementary valence relative to the fourth emotion (e.g., happiness).

[0128] As an illustrative example, emotion adjustment data 449B is provided that indicates two mappings. In other examples, emotion adjustment data 449B may include less than two mappings or more than two mappings.

[0129] When the operating mode 105B (e.g., complementary) is selected, the emotion adjustment data 449B causes the embedding selector 156 to select a target emotion (e.g., the emotion 267 of the target characteristic 177) that enables the audio analyzer 140 to generate an audio signal that corresponds to a complementary emotion relative to the original emotion of the input speech representation 149 (e.g., the emotion 267 of the input characteristic 155). Figure 1 The output signal 135.

[0130] refer to Figure 5C , showing Figure 4 Graph 550 shows illustrative aspects of the operation of emotion adjuster 452. Graph 550 includes an example of emotion adjustment data 449C corresponding to operating mode 105C (eg, flow).

[0131] The emotion adjustment data 449C indicates that each original emotion in the emotion map 347 is mapped to a corresponding target emotion in the emotion map 347, which has a complementary intensity, a complementary (e.g., opposite) valence, or both relative to the original emotion within the same emotion quadrant of the emotion map 347. In a particular aspect, the first emotion quadrant corresponds to a positive valence value (e.g., greater than the 0x coordinate) and a positive intensity value (e.g., greater than the 0y coordinate), the second emotion quadrant corresponds to a negative valence value (e.g., less than the 0x coordinate) and a positive intensity value (e.g., greater than the 0y coordinate), the third emotion quadrant corresponds to a negative valence value (e.g., less than the 0x coordinate) and a negative intensity value (e.g., less than the 0y coordinate), and the fourth emotion quadrant corresponds to a positive valence value (e.g., greater than the 0x coordinate) and a negative intensity value (e.g., less than the 0y coordinate).

[0132] In each of the first and third emotion quadrants, complementary emotions can be determined by changing the x-coordinate and the y-coordinate and maintaining the same sign. In the example of the first emotion quadrant, the first specific emotion is represented by a first horizontal coordinate (e.g., 10 as the x-coordinate) and a first vertical coordinate (e.g., 5 as the y-coordinate). The second specific emotion complementary to the first specific emotion in the first emotion quadrant has a second horizontal coordinate (e.g., 5 as the x-coordinate) and a second vertical coordinate (e.g., 10 as the y-coordinate). The second horizontal coordinate is the same as the first vertical coordinate, and the second vertical coordinate is the same as the first horizontal coordinate. The emotion adjustment data 449C indicates that the first specific emotion is mapped to the second specific emotion, and vice versa.

[0133] In the example of the third emotion quadrant, a first specific emotion is represented by a first horizontal coordinate (e.g., -10 as the x-coordinate) and a first vertical coordinate (e.g., -5 as the y-coordinate). A second specific emotion that is complementary to the first specific emotion in the third emotion quadrant has a second horizontal coordinate (e.g., -5 as the x-coordinate) and a second vertical coordinate (e.g., -10 as the y-coordinate). The second horizontal coordinate (e.g., -5) is the same as the first vertical coordinate (e.g., -5), and the second vertical coordinate (e.g., -10) is the same as the first horizontal coordinate (e.g., -10). The emotion adjustment data 449C indicates that the first specific emotion maps to the second specific emotion, and vice versa.

[0134] In each of the second and fourth emotion quadrants, complementary emotions can be determined by changing the x-coordinate and the y-coordinate and changing the sign. In the example of the second emotion quadrant, the first specific emotion is represented by the first horizontal coordinate (e.g., -10 as the x-coordinate) and the first vertical coordinate (e.g., 5 as the y-coordinate) in the second emotion quadrant. The second specific emotion complementary to the first specific emotion in the second emotion quadrant has a second horizontal coordinate (e.g., -5 as the x-coordinate) and a second vertical coordinate (e.g., 10 as the y-coordinate). The second horizontal coordinate (e.g., -5) is the negative value of the first vertical coordinate (e.g., 5), and the second vertical coordinate (e.g., 10) is the negative value of the first horizontal coordinate (e.g., -10). The emotion adjustment data 449C indicates that the first specific emotion is mapped to the second specific emotion, and vice versa.

[0135] In the example of the fourth emotion quadrant, the first specific emotion is represented by a first horizontal coordinate (e.g., 10 as the x-coordinate) and a first vertical coordinate (e.g., -5 as the y-coordinate) in the fourth emotion quadrant. A second specific emotion that is complementary to the first specific emotion in the fourth emotion quadrant has a second horizontal coordinate (e.g., 5 as the x-coordinate) and a second vertical coordinate (e.g., -10 as the y-coordinate). The second horizontal coordinate (e.g., 5) is the negative value of the first vertical coordinate (e.g., -5), and the second vertical coordinate (e.g., -10) is the negative value of the first horizontal coordinate (e.g., 10). The emotion adjustment data 449C indicates that the first specific emotion is mapped to the second specific emotion, and vice versa.

[0136] As an illustrative example, emotion adjustment data 449C is provided that indicates four mappings. In other examples, emotion adjustment data 449C may include fewer than four mappings or more than four mappings.

[0137] When the operating mode 105C (e.g., smooth) is selected, the emotion adjustment data 449C causes the embedding selector 156 to select a target emotion (e.g., emotion 267 of the target characteristic 177) that enables the audio analyzer 140 to generate an audio signal corresponding to a supplementary emotion in the same emotion quadrant relative to the original emotion of the input speech representation 149 (e.g., emotion 267 of the input characteristic 155). Figure 1 The output signal 135.

[0138] refer to Figure 5D , showing Figure 4 560 of illustrative aspects of the operation of emotion adjuster 452. Diagram 560 includes an example of an operational mode 105 corresponding to a user input indicating a target emotion.

[0139] In a specific example, the user input corresponds to a selection of a target emotion for the emotion map 347 via the graphical user interface (GUI) 549. In this example, the emotion adjuster 452 selects the target emotion as the emotion 267 for the target characteristic 177.

[0140] refer to Figure 6 , a diagram 600 showing illustrative aspects of the operation of the embedding selector 156 is shown. The embedding selector 156 is configured to select one or more reference embeddings 157 based on the target characteristic 177.

[0141] The embedding selector 156 includes feature mapping data 457 that maps features to reference embeddings. In certain aspects, the feature mapping data 457 includes emotion mapping data 671 that maps emotions 267 to reference embeddings. For example, the emotion mapping data 671 indicates that emotion 267A (e.g., anger) is associated with reference embedding 157A. As another example, the emotion mapping data 671 indicates that emotion 267B (e.g., relaxation) is associated with reference embedding 157B. In yet another example, the emotion mapping data 671 indicates that emotion 267C (e.g., sadness) is associated with reference embedding 157C. As an illustrative example, emotion mapping data 671 including mappings for three emotions is provided. In other examples, the emotion mapping data 671 may include mappings for less than three emotions or more than three emotions.

[0142] In some aspects, the emotion 267 of the target characteristic 177 is included in the emotion mapping data 671, and the embedding selector 156 selects the corresponding reference embedding 157 as one or more reference embeddings 681 associated with the emotion 267. In the example, the emotion 267 corresponds to the emotion 267A (e.g., anger). In this example, in response to determining that the emotion mapping data 671 indicates that the emotion 267A (e.g., anger) corresponds to the reference embedding 157A, the embedding selector 156 selects the reference embedding 157A as one or more reference embeddings 681 associated with the emotion 267.

[0143] In some aspects, the emotion 267 of the target characteristic 177 is not included in the emotion mapping data 671, and the embedding selector 156 selects the reference embedding 157 associated with multiple emotions as the reference embedding 681, such as the reference Fig. 7A Further described. In some implementations, embedding selector 156 also generates sentiment weights 691 associated with reference embeddings 681. Weights 137 include sentiment weights 691 (if any), and one or more reference embeddings 157 include one or more reference embeddings 681.

[0144] In certain aspects, feature mapping data 457 includes speaker identifier mapping data 673 that maps speaker identifiers to reference embeddings. For example, speaker identifier mapping data 673 indicates that a first speaker identifier (e.g., a first user identifier) ​​is associated with reference embedding 157A. As another example, speaker identifier mapping data 673 indicates that a second speaker identifier (e.g., a second speaker identifier) ​​is associated with reference embedding 157B. As an illustrative example, speaker identifier mapping data 673 including two mappings for two speaker identifiers is provided. In other examples, speaker identifier mapping data 673 may include mappings for less than two speaker identifiers or more than two speaker identifiers.

[0145] In some aspects, the speaker identifier 264 of the target characteristic 177 is included in the speaker identifier mapping data 673. For example, in response to determining that the speaker identifier mapping data 673 indicates that the speaker identifier 264 (e.g., the first speaker identifier) ​​corresponds to the reference embedding 157A, the embedding selector 156 selects the reference embedding 157A as the one or more reference embeddings 683 associated with the speaker identifier 264.

[0146] In some aspects, the speaker identifier 264 of the target feature 177 includes multiple speaker identifiers. For example, the source speech is updated to sound like a combination of multiple speakers in the output speech. The embedding selector 156 selects the reference embedding 157 associated with the multiple speaker identifiers as the reference embedding 683, and generates a speaker weight 693 associated with the reference embedding 683. For example, in response to determining that the speaker identifier 264 includes a first speaker identifier and a second speaker identifier indicated by the speaker identifier mapping data 673 as being mapped to the reference embedding 157A and the reference embedding 157B, respectively, the embedding selector 156 selects the reference embedding 157A and the reference embedding 157B as the reference embedding 683. In a particular aspect, the speaker weight 693 corresponds to an equal weight for each reference embedding in the reference embedding 683. In another aspect, operating mode 105 includes user input indicating a first speaker weight associated with a first speaker identifier and a second speaker weight associated with a second speaker identifier, and speaker weights 693 include a first speaker weight for reference embedding 157A and a second speaker weight for reference embedding 157B in one or more reference embeddings 683. Weights 137 include speaker weights 693, if any, and one or more reference embeddings 157 include one or more reference embeddings 683.

[0147] In certain aspects, the characteristic mapping data 457 includes volume mapping data 675 that maps a particular volume to a reference embedding. For example, the volume mapping data 675 indicates that a first volume (e.g., high) is associated with the reference embedding 157A. As another example, the volume mapping data 675 indicates that a second volume (e.g., low) is associated with the reference embedding 157B. As an illustrative example, volume mapping data 675 including two mappings for two volumes is provided. In other examples, the volume mapping data 675 may include mappings for less than two volumes or more than two volumes.

[0148] In some aspects, in response to determining that volume mapping data 675 indicates that volume 272 (e.g., first volume) of target characteristic 177 corresponds to reference embedding 157A, embedding selector 156 selects reference embedding 157A as one or more reference embeddings 685 associated with volume 272. One or more reference embeddings 157 include one or more reference embeddings 685.

[0149] In some aspects, volume 272 (e.g., medium) of target characteristic 177 is not included in volume mapping data 675, and embedding selector 156 selects reference embedding 157 associated with a plurality of volumes as reference embedding 685. For example, embedding selector 156 selects reference embedding 157A and reference embedding 157B corresponding to a first volume (e.g., high) and a second volume (e.g., low), respectively, as reference embedding 685. For illustration, embedding selector 156 selects a next volume greater than volume 272 and a next volume less than volume 272 included in volume mapping data 675.

[0150] In some implementations, embedding selector 156 also generates volume weights 695 associated with reference embedding 685. For example, volume weights 695 include a first weight for reference embedding 157A and a second weight for reference embedding 157B. The first weight is based on the difference between volume 272 (e.g., medium) and a first volume (e.g., high). The second weight is based on the difference between volume 272 (e.g., medium) and a second volume (e.g., low). Weights 137 include volume weights 695 (if any), and one or more reference embeddings 157 include one or more reference embeddings 685.

[0151] In certain aspects, the characteristic mapping data 457 includes pitch mapping data 677 that maps a particular pitch to a reference embedding. For example, the pitch mapping data 677 indicates that a first pitch (e.g., high) is associated with the reference embedding 157A. As another example, the pitch mapping data 677 indicates that a second pitch (e.g., low) is associated with the reference embedding 157B. As an illustrative example, a pitch mapping data 677 including two mappings for two pitches is provided. In other examples, the pitch mapping data 677 may include mappings for less than two pitches or more than two pitches.

[0152] In some aspects, in response to determining that pitch mapping data 677 indicates that pitch 274 (e.g., first pitch) of target characteristic 177 corresponds to reference embedding 157A, embedding selector 156 selects reference embedding 157A as one or more reference embeddings 687 associated with pitch 274. One or more reference embeddings 157 include one or more reference embeddings 687.

[0153] In some aspects, pitch 274 (e.g., medium) of target characteristic 177 is not included in pitch mapping data 677, and embedding selector 156 selects reference embeddings 157 associated with a plurality of pitches as reference embedding 687. For example, embedding selector 156 selects reference embedding 157A and reference embedding 157B corresponding to a first pitch (e.g., high) and a second pitch (e.g., low), respectively, as reference embedding 687. For illustration, embedding selector 156 selects a next pitch greater than pitch 274 and a next pitch less than pitch 274 included in pitch mapping data 677.

[0154] In some implementations, embedding selector 156 also generates pitch weights 697 associated with reference embedding 687. For example, pitch weights 697 include a first weight for reference embedding 157A and a second weight for reference embedding 157B. The first weight is based on the difference between pitch 274 (e.g., medium) and a first pitch (e.g., high). The second weight is based on the difference between pitch 274 (e.g., medium) and a second pitch (e.g., low). Weights 137 include pitch weights 697 (if any), and one or more reference embeddings 157 include one or more reference embeddings 687.

[0155] In certain aspects, the characteristic mapping data 457 includes speed mapping data 679 that maps a particular speed to a reference embedding. For example, the speed mapping data 679 indicates that a first speed (e.g., high) is associated with the reference embedding 157A. As another example, the speed mapping data 679 indicates that a second speed (e.g., low) is associated with the reference embedding 157B. As an illustrative example, speed mapping data 679 including two mappings for two speeds is provided. In other examples, the speed mapping data 679 may include mappings for less than two speeds or more than two speeds.

[0156] In some aspects, in response to determining that speed mapping data 679 indicates that speed 276 (e.g., first speed) of target characteristic 177 corresponds to reference embedding 157A, embedding selector 156 selects reference embedding 157A as one or more reference embeddings 689 associated with speed 276. One or more reference embeddings 157 include one or more reference embeddings 689.

[0157] In some aspects, the speed 276 (e.g., medium) of the target characteristic 177 is not included in the speed mapping data 679, and the embedding selector 156 selects the reference embedding 157 associated with the plurality of speeds as the reference embedding 689. For example, the embedding selector 156 selects the reference embedding 157A and the reference embedding 157B corresponding to the first speed (e.g., high) and the second speed (e.g., low), respectively, as the reference embedding 689. For illustration, the embedding selector 156 selects the next speed greater than the speed 276 and the next speed less than the speed 276 included in the speed mapping data 679.

[0158] In some implementations, the embedding selector 156 also generates a speed weight 699 associated with the reference embedding 689. For example, the speed weight 699 includes a first weight for the reference embedding 157A and a second weight for the reference embedding 157B. The first weight is based on the difference between the speed 276 (e.g., medium) and the first speed (e.g., high). The second weight is based on the difference between the speed 276 (e.g., medium) and the second speed (e.g., low). The weight 137 includes the speed weight 699 (if any), and one or more reference embeddings 157 include one or more reference embeddings 689.

[0159] refer to Fig. 7A , a diagram 700 showing illustrative aspects of the operation of the embedding selector 156 is shown. The input characteristics 155 include the emotion 267D (e.g., boredom). The emotion adjuster 452 selects the emotion adjustment data 449 based on the operating mode 105. For example, if the operating mode 105 includes the operating mode 105A (e.g., positive promotion), the emotion adjuster 452 selects the emotion adjustment data 449A associated with the operating mode 105A, as shown in reference Figure 4As another example, if the operating mode 105 includes the operating mode 105B (eg, complementary), the emotion adjuster 452 selects the emotion adjustment data 449B associated with the operating mode 105B, as shown in FIG. Figure 4 as described.

[0160] The emotion adjuster 452 determines that the emotion adjustment data 449 indicates that the emotion 267D (e.g., boredom) is mapped to the emotion 267E. The emotion adjuster 452 updates the target feature 177 to include the emotion 267E. In response to determining that the emotion mapping data 671 does not include any reference embeddings corresponding to the emotion 267E, the emotion adjuster 452 selects a plurality of mappings corresponding to emotions within a threshold distance of the emotion 267E in the emotion map 347 from the emotion mapping data 671. For example, the emotion adjuster 452 selects a first mapping for the emotion 267B (e.g., relaxation) based on determining that the emotion 267B is within the threshold distance of the emotion 267E. As another example, the emotion adjuster 452 selects a second mapping for the emotion 267F (e.g., calm) based on determining that the emotion 267F is within the threshold distance of the emotion 267E.

[0161] The emotion adjuster 452 adds the reference embedding corresponding to the selected mapping to the one or more reference embeddings 681 associated with the emotion 267E. For example, in response to determining that the first mapping indicates that the emotion 267B (e.g., relaxation) corresponds to the reference embedding 157B, the emotion adjuster 452 includes the reference embedding 157B in the one or more reference embeddings 681 associated with the emotion 267E. In a particular aspect, the emotion adjuster 452 determines the weight 137B based on the distance between the emotion 267E and the emotion 267B (e.g., relaxation), and includes the weight 137B in the emotion weight 691.

[0162] In another example, in response to determining that the second mapping indicates that the emotion 267F (e.g., calm) corresponds to the reference embedding 157F, the emotion adjuster 452 includes the reference embedding 157F in the one or more reference embeddings 681 associated with the emotion 267E. In a particular aspect, the emotion adjuster 452 determines the weight 137F based on the distance between the emotion 267E and the emotion 267F (e.g., calm), and includes the weight 137F in the emotion weight 691.

[0163] Therefore, the emotion adjuster 452 selects multiple reference embeddings 157 (e.g., reference embedding 157B and reference embedding 157F) as one or more reference embeddings 681, which can be combined to generate an estimated emotion embedding corresponding to emotion 267E, as shown in reference Fig. 8A As further described, one or more reference embeddings 681 are combined based on a sentiment weight 691 .

[0164] refer to Figure 7B , a diagram 750 showing illustrative aspects of the operation of the embedding selector 156 is shown. The emotion adjuster 452 selects the emotion adjustment data 449 based on the operating mode 105.

[0165] In an example, the emotion adjustment data 449 includes a first mapping indicating that the emotion 267C (e.g., sad) is mapped to the emotion 267B (e.g., relaxed) and a second mapping indicating that the emotion 267H (e.g., frustrated) is mapped to the emotion 267J (e.g., content). In an example, the emotion mapping data 671 indicates that the emotion 267B (e.g., relaxed) is mapped to the reference embedding 157B and the emotion 267J (e.g., content) is mapped to the reference embedding 157J. In a particular aspect, the emotion adjustment data 449 includes a mapping to the emotion whose reference embedding the emotion mapping data 671 includes.

[0166] The input characteristics 155 include emotion 267G. In response to determining that the emotion adjustment data 449 does not include any mappings corresponding to emotion 267G, the emotion adjuster 452 selects a plurality of mappings corresponding to emotions within a threshold distance of emotion 267G in the emotion map 347 from the emotion adjustment data 449. For example, the emotion adjuster 452 selects a first mapping (e.g., from emotion 267H to emotion 267J) based on determining that emotion 267H is within a threshold distance of emotion 267G. As another example, the emotion adjuster 452 selects a second mapping (e.g., from emotion 267C to emotion 267B) based on determining that emotion 267C is within a threshold distance of emotion 267G.

[0167] In a particular implementation, the emotion adjuster 452 estimates that the emotion 267G maps to the emotion 267K based on determining that the relative distance of the emotion 267K from the emotion 267J (e.g., contentment) and the emotion 267B (e.g., relaxation) is the same as the relative distance of the emotion 267G from the emotion 267H (e.g., frustration) and the emotion 267C (e.g., sadness). The target characteristic 177 includes the emotion 267K.

[0168] In response to determining that the emotion mapping data 671 does not indicate any reference embeddings corresponding to the emotion 267K, the emotion adjuster 452 selects a plurality of mappings from the emotion mapping data 671 to determine a reference embedding corresponding to the emotion 267K, such as the reference Fig. 7A For example, the emotion adjuster 452 selects a first mapping for emotion 267B (eg, relaxed) and a second mapping for emotion 267J (eg, satisfied) from the emotion mapping data 671.

[0169] The emotion adjuster 452 adds the reference embedding corresponding to the selected mapping to the one or more reference embeddings 681 associated with the emotion 267K. For example, the emotion adjuster 452 adds the reference embedding 157B and the reference embedding 157J corresponding to the emotion 267B and the emotion 267J, respectively, to the one or more reference embeddings 681.

[0170] In certain aspects, the emotion adjuster 452 determines the weight 137J based on the distance between the emotion 267J and the emotion 267K, the distance between the emotion 267H and the emotion 267G, or both. In certain aspects, the emotion adjuster 452 determines the weight 137B based on the distance between the emotion 267B and the emotion 267K, the distance between the emotion 267C and the emotion 267G, or both. The emotion weight 691 includes the weight 137B and the weight 137J.

[0171] Therefore, the emotion adjuster 452 selects multiple reference embeddings 157 (e.g., reference embedding 157B and reference embedding 157J) as one or more reference embeddings 681, which can be combined to generate an estimated emotion embedding, such as reference Fig. 8A As further described, the estimated emotion embedding corresponds to emotion 267K which is an estimated target emotion of emotion 267G. One or more reference embeddings 681 are combined based on emotion weights 691 .

[0172] refer to Fig. 8A , a diagram 800 showing illustrative aspects of the operation of an illustrative implementation of the transformation embedding generator 158 is shown. The transformation embedding generator 158 includes an embedding combiner 852 that is configured to generate an embedding 859 based at least in part on one or more reference embeddings 157.

[0173] In response to determining that one or more reference embeddings 157 include a single reference embedding, embedding combiner 852 designates the single reference embedding as embedding 859. Alternatively, in response to determining that one or more reference embeddings 157 include multiple reference embeddings, embedding combiner 852 combines the multiple reference embeddings to generate embedding 859.

[0174] In certain aspects, in response to determining that the one or more reference embeddings 157 include multiple reference embeddings, the embedding combiner 852 generates a specific reference embedding for the corresponding type of feature. In an example, the embedding combiner 852 combines the one or more reference embeddings 681 to generate an emotion embedding 871, combines the one or more reference embeddings 683 to generate a speaker embedding 873, combines the one or more reference embeddings 685 to generate a volume embedding 875, combines the one or more reference embeddings 687 to generate a pitch embedding 877, combines the one or more reference embeddings 689 to generate a velocity embedding 879, or a combination thereof.

[0175] In some aspects, the embedding combiner 852 combines multiple reference embeddings for a particular type of feature based on corresponding weights. For example, the embedding combiner 852 combines one or more reference embeddings 681 based on the emotion weight 691. For illustration, the emotion weight 691 includes a first weight for a reference embedding 157A in one or more reference embeddings 681 and a second weight for a reference embedding 157B of one or more reference embeddings 681. The embedding combiner 852 applies the first weight to the reference embedding 157A to generate a first weighted reference embedding, and applies the second weight to the reference embedding 157B to generate a second weighted reference embedding. In some examples, the reference embedding 157 corresponds to a set (e.g., a vector) of speech feature values, and applying a specific weight to the reference embedding 157 corresponds to multiplying each speech feature value in the speech feature value by a specific weight to generate a weighted reference embedding. The embedding combiner 852 generates the emotion embedding 871 based on a combination (e.g., a sum) of the first weighted reference embedding and the second weighted reference embedding.

[0176] In some aspects, the embedding combiner 852 combines multiple reference embeddings for a particular type of feature independently of corresponding weights (e.g., in the absence of corresponding weights). In an example, in response to determining that the speaker weights 693 are not available, the embedding combiner 852 combines one or more reference embeddings 683, wherein each reference embedding in the one or more reference embeddings 683 has an equal weight. For illustration, the embedding combiner 852 generates the speaker embedding 873 as a combination (e.g., an average) of the reference embedding 157A in the one or more reference embeddings 683 and the reference embedding 157B in the one or more reference embeddings 683.

[0177] The embedding combiner 852 generates the embedding 859 as a combination of specific reference embeddings for corresponding types of features. For example, the embedding combiner 852 generates the embedding 859 as a combination (e.g., a cascade) of the emotion embedding 871, the speaker embedding 873, the volume embedding 875, the pitch embedding 877, the velocity embedding 879, or a combination thereof. In a particular aspect, the embedding 859 represents the target feature 177. In a particular aspect, the embedding 859 is used as the transformation embedding 159.

[0178] refer to Figure 8B , a diagram 850 showing illustrative aspects of the operation of another illustrative implementation of the transformation embedding generator 158. The transformation embedding generator 158 includes an embedding combiner 854 coupled to the embedding combiner 852.

[0179] Embedding combiner 854 is configured to combine embedding 859 with baseline embedding 161 to generate transformed embedding 159. In certain aspects, in response to determining that no baseline embedding associated with the audio analysis session is available, embedding combiner 854 designates embedding 859 as transformed embedding 159 and stores transformed embedding 159 as baseline embedding 161.

[0180] In response to determining that a baseline embedding 161 associated with the ongoing audio analysis session is available, the embedding combiner 854 generates a transformed embedding 159 based on a combination of the embedding 859 and the baseline embedding 161. In an example, the transformed embedding 159 corresponds to a combination (e.g., a cascade) of the emotion embedding 861, the speaker embedding 863, the volume embedding 865, the pitch embedding 867, the velocity embedding 869, or a combination thereof.

[0181] The embedding combiner 854 generates a transformation embedding 159 corresponding to a combination (e.g., a cascade) of the emotion embedding 881, the speaker embedding 883, the volume embedding 885, the pitch embedding 887, the velocity embedding 889, or a combination thereof.

[0182] The embedding combiner 854 generates a feature embedding of the transformed embedding 159 based on the first corresponding feature embedding of the baseline embedding 161, the second corresponding feature embedding of the embedding 859, or both. For example, the embedding combiner 854 generates the emotion embedding 881 as a combination (e.g., an average) of the emotion embedding 861 and the emotion embedding 871. For illustration, the emotion embedding 861 includes a first set of speech feature values ​​(e.g., x1, x2, x3, ...), and the emotion embedding 871 includes a second set of speech feature values ​​(e.g., y1, y2, y3, etc.). The embedding combiner 854 generates the emotion embedding 881 including a third set of speech feature values ​​(e.g., z1, z2, z3, etc.), wherein each Nth speech feature value (zN) in the third set of speech feature values ​​is an average of the corresponding Nth speech feature value (xN) in the first set of speech feature values ​​and the corresponding Nth feature value (yN) in the second set of speech feature values.

[0183] In some examples, one of emotion embedding 861 or emotion embedding 871 is available, but not both, because baseline embedding 161 does not include emotion embedding 861 or embedding 859 does not include emotion embedding 871. In these examples, emotion embedding 881 includes whichever of emotion embedding 861 or emotion embedding 871 is available. In some examples, neither emotion embedding 861 nor emotion embedding 871 is available. In these examples, transformed embedding 159 does not include emotion embedding 881.

[0184] Similarly, the embedding combiner 854 generates a speaker embedding 883 based on the speaker embedding 863, the speaker embedding 873, or both. As another example, the embedding combiner 854 generates a volume embedding 875 based on the volume embedding 885, the volume embedding 865, or both. As yet another example, the embedding combiner 854 generates a pitch embedding 887 based on the pitch embedding 867, the pitch embedding 877, or both. Similarly, the embedding combiner 854 generates a velocity embedding 889 based on the velocity embedding 869, the velocity embedding 879, or both. In a particular aspect, the embedding combiner 854 stores the transition embedding 159 as a baseline embedding 161 for use in generating the transition embedding 159 based on one or more reference embeddings 157 of the input speech representation 149 corresponding to a subsequent portion of the input speech. Using the baseline embedding 161 to generate the transition embedding 159 can achieve a gradual change of the transition embedding 159 and the output signal 135.

[0185] refer to Figure 8C , a diagram 890 showing illustrative aspects of the operation of the transformation embedding generator 158 is shown. Diagram 890 includes an example 892 of components of the audio analyzer 140, an example 894 of an illustrative implementation of the transformation embedding generator 158 of example 894, and an example 896 of generating an embedding 859 by the embedding combiner 856 of the transformation embedding generator 158 of example 892.

[0186] In example 892, the audio spectrum generator 150 generates an input audio spectrum 151 corresponding to each input voice representation of a plurality of input voice representations 149 (such as input voice representations 149A to input voice representations 149N), wherein the input voice representation 149N corresponds to the Nth input representation, wherein N corresponds to a positive integer greater than 1. For example, the audio spectrum generator 150 processes the input voice representation 149A to generate the input audio spectrum 151A, as shown in reference Figure 1 Similarly, the audio spectrum generator 150 generates one or more additional input audio spectra 151. For example, the audio spectrum generator 150 processes the input speech representation 149N to generate the input audio spectrum 151N, as shown in FIG. Figure 1 As described.

[0187] The characteristic detector 154 determines an input characteristic 155 corresponding to each input audio spectrum in the input audio spectrum 151. For example, the characteristic detector 154 processes the input audio spectrum 151A to determine the input characteristic 155A, as shown in FIG. Figure 1 Similarly, the characteristic detector 154 determines one or more additional input characteristics 155. For example, the characteristic detector 154 processes the input audio spectrum 151N to determine the input characteristic 155N, as described in reference Figure 1 As described.

[0188] Embedding selector 156 determines a target feature 177 and one or more reference embeddings 157 corresponding to each of input features 155. For example, embedding selector 156 determines a target feature 177A corresponding to input feature 155A, and determines a reference embedding 157A, a weight 137A, or a combination thereof corresponding to target feature 177A, such as reference embedding 157A. Figure 1 Similarly, embedding selector 156 determines one or more additional target features 177 and one or more additional reference embeddings 157 corresponding to each of input features 155. For example, embedding selector 156 determines a target feature 177N corresponding to input feature 155N, and determines a reference embedding 157N, a weight 137N, or a combination thereof corresponding to target feature 177N, such as reference embedding 137N. Figure 1 As described.

[0189] Transformed embedding generator 158 generates transformed embedding 159 based on reference embedding 157, weights 137, or multiple sets of both. In example 894, embedding combiner 852 is coupled to embedding combiner 856. Optionally, in some implementations, embedding combiner 856 is coupled to embedding combiner 854.

[0190] The embedding combiner 852 generates an embedding 859 corresponding to each set of one or more reference embeddings 157, weights 137, or both. For example, the embedding combiner 852 generates an embedding 859A corresponding to one or more reference embeddings 157A, weights 137A, or a combination thereof, such as reference Fig. 8A Similarly, the embedding combiner 852 generates one or more additional embeddings 859 corresponding to each set of one or more reference embeddings 157, weights 137, or combinations thereof. For example, the embedding combiner 852 generates an embedding 859N corresponding to one or more reference embeddings 157N, weights 137N, or combinations thereof, such as reference Fig. 8A As described.

[0191] Embedding combiner 856 generates embedding 859 based on a combination (eg, an average) of embeddings 859A through 859N. In certain aspects, embedding 859 corresponds to a weighted average of embeddings 859A through 859N.

[0192] As shown in example 896, embedding 859A corresponds to a combination (e.g., a cascade) of at least two of emotion embedding 871A, speaker embedding 873A, volume embedding 875A, pitch embedding 877A, or velocity embedding 879A. Embedding 859N corresponds to a combination (e.g., a cascade) of at least two of emotion embedding 871N, speaker embedding 873N, volume embedding 875N, pitch embedding 877N, or velocity embedding 879N. Embedding combiner 856 generates embedding 859 corresponding to a combination (e.g., a cascade) of at least two of emotion embedding 871, speaker embedding 873, volume embedding 875, pitch embedding 877, or velocity embedding 879. As illustrative examples, each of embedding 859A, embedding 859N, and embedding 859 that includes at least two of emotion embedding, speaker embedding, volume embedding, pitch embedding, or velocity embedding is provided. In some examples, one or more of embedding 859A, embedding 859N, or embedding 859 may include a single embedding of an emotion embedding, a speaker embedding, a volume embedding, a pitch embedding, or a velocity embedding.

[0193] Embedding combiner 856 generates a feature embedding for embedding 859 based on the first corresponding feature embedding of embedding 859A and the additional corresponding feature embeddings of one or more additional embeddings 859. For example, embedding combiner 856 generates emotion embedding 871 as a combination (e.g., an average) of emotion embeddings 871A through 871N. In some examples, fewer than N emotion embeddings are available, and embedding combiner 856 generates emotion embedding 871 based on the available emotion embeddings in embeddings 859A through 859N. In examples where an emotion embedding is not included in embeddings 859A through 859N, embeddings 159 do not include emotion embedding 871.

[0194] Similarly, embedding combiner 856 generates speaker embedding 873 based on speaker embedding 873A to speaker embedding 873N. As another example, embedding combiner 856 generates volume embedding 875 based on volume embedding 875A to volume embedding 875N. As yet another example, embedding combiner 856 generates pitch embedding 877 based on pitch embedding 877A to pitch embedding 877N. Similarly, embedding combiner 856 generates velocity embedding 879 based on velocity embedding 879A to velocity embedding 879N. In a particular aspect, embedding 859 corresponds to transformation embedding 159. In another aspect, embedding combiner 854 processes embedding 859 and baseline embedding 161 to generate transformation embedding 159, as shown in reference Figure 8B As described.

[0195] refer to Fig. 9 , a system 900 is shown. The system 900 is operable to perform source voice modification based on input voice characteristics. In certain aspects, Figure 1System 100 includes one or more components of system 900 .

[0196] The audio analyzer 140 is coupled to the input interface 914, the input interface 924, or both. The input interface 914 is configured to be coupled to one or more cameras 910. The input interface 924 is configured to be coupled to one or more microphones 920. As a non-limiting example, the one or more cameras 910 and the one or more microphones 920 are illustrated as being external to the device 102. In other examples, at least one of the one or more cameras 910, at least one of the one or more microphones 920, or a combination thereof may be integrated into the device 102.

[0197] One or more cameras 910 are provided as illustrative, non-limiting examples of image sensors, in other examples, other types of image sensors may be used. One or more microphones 920 are provided as illustrative, non-limiting examples of audio sensors, in other examples, other types of audio sensors may be used.

[0198] In some aspects, the device 102 includes a representation generator 930 coupled to the audio analyzer 140. The representation generator 930 is configured to process the source speech data 928 to generate the source speech representation 163, as shown in FIG. Fig.12 Further described.

[0199] The audio analyzer 140 receives an audio signal 949 from the input interface 924. The audio signal 949 corresponds to the microphone output 922 (e.g., audio data) received from the one or more microphones 920. The input speech representation 149 is based on the audio signal 949. In some examples, the audio signal 949 is used as the source speech data 928. In some examples, the source speech data 928 is generated by an application or other component of the device 102. In some examples, the source speech data 928 corresponds to decoded data, such as reference speech data. Fig. 13B Further described.

[0200] In some aspects, the audio analyzer 140 receives an image signal 916 from the input interface 914. The image signal 916 corresponds to the camera output 912 from the one or more cameras 910. Optionally, in some examples, the image data 153 is based on the image signal 916.

[0201] The audio analyzer 140 generates an output signal 135 based on the input speech representation 149 and the source speech representation 163, as shown in FIG. Figure 1 Optionally, in some examples, the audio analyzer 140 also generates an output signal 135 based on the image data 153, as described in reference Figure 1As described. In an example, the input speech representation 149 corresponds to the input speech of the user 101 captured by one or more microphones 920, concurrently with the one or more cameras 910 capturing an image (e.g., a still image or a video) corresponding to the image data 153. Therefore, the source speech corresponding to the source speech data 928 can be updated in real time based on the camera output 912 and the microphone output 922 to generate an output signal 135 corresponding to the output speech. In some examples, the audio analyzer 140 outputs the output signal 135 concurrently with the device 102 receiving the microphone output 922, receiving the camera output 912, or both.

[0202] refer to Fig.10 , a system 1000 is shown. The system 1000 is operable to perform source voice modification based on input voice characteristics. In certain aspects, Figure 1 The system 100 includes one or more components of the system 1000 .

[0203] The source speech data 928 is based on an audio signal 949. In some examples, the audio signal 949 is also used as an input speech representation 149. In some examples, the input speech representation 149 is generated by an application or other component of the device 102. In some examples, the input speech representation 149 corresponds to decoded data, such as reference Fig. 13B The representation generator 930 processes the audio signal 949 into the source speech data 928 to generate the source speech representation 163, as described in reference Fig.12 Further described.

[0204] In some examples, image data 153 is based on Fig. 9 In some examples, image data 153 is generated by an application or other component of device 102. In some examples, image data 153 corresponds to decoded data, such as reference Fig. 13B Further described.

[0205] The audio analyzer 140 generates an output signal 135 based on the input speech representation 149 and the source speech representation 163, as shown in FIG. Figure 1 Optionally, in some examples, the audio analyzer 140 also generates an output signal 135 based on the image data 153, as described in reference Figure 1 In an example, the source speech data 928 corresponds to the source speech of the user 101 captured by one or more microphones 920. Therefore, the source speech corresponding to the source speech data 928 can be updated in real time based on the input speech representation 149 and the image data 153 to generate an output signal 135 corresponding to the output speech. In some examples, the audio analyzer 140 outputs the output signal 135 concurrently with the device 102 receiving the microphone output 922.

[0206] refer to Fig.11 , a system 1100 is shown. The system 1100 is operable to perform source voice modification based on input voice characteristics. In certain aspects, Figure 1 System 100 includes one or more components of system 1100 .

[0207] The audio analyzer 140 is coupled to an output interface 1124, which is configured to be coupled to one or more speakers 1110. As a non-limiting example, the one or more speakers 1110 are illustrated as being external to the device 102. In other examples, at least one of the one or more speakers 1110 may be integrated into the device 102.

[0208] The audio analyzer 140 generates an output signal 135 based on the input speech representation 149 and the source speech representation 163, as shown in FIG. Figure 1 Optionally, in some examples, the audio analyzer 140 also generates an output signal 135 based on the image data 153, as described in reference Figure 1 For example, the audio analyzer 140 provides the output signal 135 to the one or more speakers 1110 via the output interface 1124. In some examples, the audio analyzer 140 provides the output signal 135 to the one or more speakers 1110, which is consistent with the device 102 receiving the output signal 135 from the device 102. Fig. 9 The one or more microphones 920 of the device 102 receive the microphone output 922 concurrently. In some examples, the audio analyzer 140 provides an output signal 135 to one or more speakers 1110, which is consistent with the device 102 receiving the microphone output 922. Fig. 9 One or more cameras 910 receive camera output 912 concurrently.

[0209] refer to Fig.12 , a diagram 1200 representing illustrative aspects of the operation of the generator 930 is shown. The audio spectrum generator 150 is coupled to the combiner 1246 via the encoder 1242 and the base frequency (F0) extractor 1244.

[0210] The audio spectrum generator 150 generates a source audio spectrum 1240 of the source speech data 928. In a particular aspect, the source speech data 928 includes source speech audio. In an alternative aspect, the source speech data 928 includes non-audio data, and the audio spectrum generator 150 generates the source speech audio based on the source speech data 928. In an example, the source speech data 928 includes speech text (e.g., chat script, screen playback, closed caption text, etc.). The audio spectrum generator 150 generates the source speech audio based on the speech text. For example, the audio spectrum generator 150 performs text-to-speech conversion on the speech text to generate the source speech audio. In some examples, the source speech data 928 includes one or more characteristic indicators, such as one or more emotion indicators, one or more speaker indicators, one or more style indicators, or a combination thereof, and the audio spectrum generator 150 generates the source speech audio to have source characteristics corresponding to the one or more characteristic indicators.

[0211] In some implementations, the audio spectrum generator 150 applies a transform (e.g., a fast Fourier transform (FFT)) to the source speech audio in the time domain to generate a source audio spectrum 1240 (e.g., a mel-spectrogram) in the frequency domain. FFT is provided as an illustrative example of a transform applied to the source speech audio to generate the source audio spectrum 1240. In other examples, the audio spectrum generator 150 can process the source speech audio using various transforms and techniques to generate the source audio spectrum 1240. The audio spectrum generator 150 provides the source audio spectrum 1240 to the encoder 1242 and the F0 extractor 1244.

[0212] The encoder 1242 (e.g., a spectral encoder) processes the source audio spectrum 1240 using a spectral coding technique to generate a source speech embedding 1243. In certain aspects, the source speech embedding 1243 represents the potential features of the source speech audio. The F0 extractor 1244 processes the source audio spectrum 1240 using a baseband extraction technique to generate an F0 embedding 1245. In certain aspects, the F0 extractor 1244 includes a pre-trained joint detection and classification (JDC) network that includes a convolution layer followed by a bidirectional long short-term memory (BLSTM) unit, and the F0 embedding 1245 corresponds to the convolution output. The combiner 1246 generates a source speech representation 163 corresponding to a combination (e.g., a sum, product, average, or concatenation) of the source speech embedding 1243 and the F0 embedding 1245.

[0213] refer to Fig.13A , a system 1300 is shown. The system 1300 is operable to perform source voice modification based on input voice characteristics. The device 102 includes an audio encoder 1320 coupled to the audio analyzer 140. The system 1300 includes a device 1304 including an audio decoder 1330.

[0214] The device 102 is configured to couple to the device 1304. In an example, the device 102 is configured to couple to the device 1304 via a network. The network may include one or more wireless networks, one or more wired networks, or a combination thereof.

[0215] The audio analyzer 140 provides the output signal 135 to the audio encoder 1320. The audio encoder 1320 encodes the output signal 135 to generate encoded data 1322. The audio encoder 1320 provides the encoded data 1322 to the device 1304. The audio decoder 1330 decodes the encoded data 1322 to generate an output signal 1335. In certain aspects, the output signal 1335 estimates the output signal 135. For example, due to network loss, decoding errors, etc., the output signal 1335 may be different from the output signal 135. The audio decoder 1330 outputs the output signal 1335 via one or more speakers 1310. In certain aspects, the device 1304 outputs the output signal 1335 via the one or more speakers 1310 concurrently with receiving the encoded data 1322 from the device 102.

[0216] refer to Fig. 13B , a system 1350 is shown. The system 1350 is operable to perform source voice modification based on input voice characteristics. The device 102 includes an audio decoder 1370 coupled to the audio analyzer 140.

[0217] In certain aspects, device 102 is coupled to one or more speakers 1360. As a non-limiting example, one or more speakers 1360 are illustrated as being external to device 102. In other examples, at least one of one or more speakers 1360 can be integrated into device 102.

[0218] System 1350 includes device 1306 configured to be coupled to device 102. In an example, device 102 is configured to be coupled to device 1306 via a network. The network may include one or more wireless networks, one or more wired networks, or a combination thereof.

[0219] The audio decoder 1370 receives the encoded data 1362 from the device 1306. The audio decoder 1370 decodes the encoded data 1362 to generate decoded data 1372. The audio analyzer 140 generates an output signal 135 based on the decoded data 1372. In certain aspects, the decoded data 1372 includes the input speech representation 149, the image data 153, the user input 103, the operation mode 105, the source speech representation 163, or a combination thereof. In certain aspects, the audio analyzer 140 outputs the output signal 135 via one or more speakers 1360.

[0220] refer to Fig.14 , a system 1400 is shown. The system 1400 is operable to train the audio analyzer 140. The system 1400 includes a device 1402. In some aspects, the device 1402 is the same as the device 102. In other aspects, the device 1402 is external to the device 102 and the device 102 receives the trained version of the audio analyzer 140 from the device 1402.

[0221] The device 1402 includes one or more processors 1490. The one or more processors 1490 include a trainer 1466 configured to train the audio analyzer 140 using the training data 1460. The training data 1460 includes the input speech representation 149 and the source speech representation 163. The training data 1460 also indicates one or more target characteristics, such as emotion 1467, speaker identifier 1464, volume 1472, pitch 1474, speed 1476, or a combination thereof.

[0222] In some examples, the target characteristic is the same as an input characteristic of the input speech representation 149. In some examples, the input characteristic is mapped to a target characteristic of the operating mode 105, the image data 153, the user input 103, or a combination thereof.

[0223] The trainer 1466 provides the input speech representation 149 and the source speech representation 163 to the audio analyzer 140. Optionally, in some examples, the trainer 1466 also provides the user input 103, the image data 153, the operating mode 105, or a combination thereof to the audio analyzer 140. The audio analyzer 140 generates the output signal 135 based on the input speech representation 149, the source speech representation 163, the user input 103, the image data 153, the operating mode 105, or a combination thereof, as shown in FIG. Figure 1 As described.

[0224] Trainer 1466 includes emotion detector 202, speaker detector 204, style detector 206, synthesized audio detector 1440, or a combination thereof. Emotion detector 202 processes output signal 135 to determine emotion 1487 of output signal 135. Speaker detector 204 processes output signal 135 to determine that output signal 135 corresponds to speech of a speaker (e.g., a user) who may have speaker identifier 1484. Style detector 206 processes output signal 135 to determine volume 1492, pitch 1494, speed 1496, or a combination thereof of output signal 135, as described in reference to FIG. Figure 2 Synthesized audio detector 1440 processes output signal 135 to generate indicator 1441, which indicates whether output signal 135 is likely to correspond to a live human voice or to a synthesized voice.

[0225] The error analyzer 1442 determines a loss metric 1445 based on a comparison of one or more target characteristics associated with the input speech representation 149 (as indicated by the training data 1460) and corresponding detected characteristics (as determined by the emotion detector 202, the speaker detector 204, the style detector 206, the synthesized audio detector 1440, or a combination thereof). For example, the loss metric 1445 is based at least in part on a comparison of the emotion 1467 and the emotion 1487, where the emotion 1467 corresponds to the target emotion corresponding to the input speech representation 149 as indicated by the training data 1460, and the emotion 1487 is detected by the emotion detector 202. As another example, the loss metric 1445 is based at least in part on a comparison of the volume 1472 and the volume 1492, where the volume 1472 corresponds to the target volume corresponding to the input speech representation 149 as indicated by the training data 1460, and the volume 1492 is detected by the style detector 206.

[0226] In a particular example, the loss metric 1445 is based at least in part on a comparison of the pitch 1474 to the pitch 1494, where the pitch 1474 corresponds to a target pitch corresponding to the input speech representation 149 as indicated by the training data 1460, and the pitch 1494 is detected by the style detector 206. As another example, the loss metric 1445 is based at least in part on a comparison of the speed 1476 to the speed 1496, where the speed 1476 corresponds to a target speed corresponding to the input speech representation 149 as indicated by the training data 1460, and the speed 1496 is detected by the style detector 206.

[0227] In certain aspects, the loss metric 1445 is based at least in part on a comparison of a first speaker representation associated with a speaker identifier 1464 and a second speaker representation associated with a speaker identifier 1484, wherein the speaker identifier 1464 corresponds to a target speaker identifier corresponding to the input speech representation 149 as indicated by the training data 1460, and the speaker identifier 1484 is detected by the speaker detector 204. In certain aspects, the loss metric 1445 is based on the indicator 1441. For example, a first value of the indicator 1441 indicates that the output signal 135 is detected as approximately human speech, while a second value of the indicator 1441 indicates that the output signal 135 is detected as synthesized speech. In this example, the loss metric 1445 is decreased based on the indicator 1441 having the first value, or increased based on the indicator 1441 having the second value.

[0228] The error analyzer 1442 generates an update command 1443 to update the audio analyzer 140 (e.g., weights and biases of its neural network) based on the loss metric 1445. For example, the error analyzer 1442 iteratively provides the audio analyzer 140 with a training data set including the input speech representation 149, the source speech representation 163, the user input 103, the image data 153, the operating mode 105, or a combination thereof to generate the output signal 135 and update the audio analyzer 140 to reduce the loss metric 1445. In response to determining that the loss metric 1445 is within a threshold loss, the loss metric 1445 has stopped changing, at least a threshold iteration count has been performed, or a combination thereof, the error analyzer 1442 determines that the training of the audio analyzer 140 is complete. In certain aspects, in response to determining that the training is complete, the trainer 1466 provides the audio analyzer 140 to the device 102.

[0229] In certain aspects, the audio analyzer 140 and the trainer 1466 correspond to a generative adversarial network (GAN). Fig.12 F0 extractor 1244, combiner 1246 and Figure 1 The voice converter 164 corresponds to the generator of the GAN, and the emotion detector 202, the speaker detector 204 and the style detector 206 correspond to the discriminators of the GAN.

[0230] In certain aspects, updating the audio analyzer 140 includes updating the GAN. In some implementations, the audio analyzer 140 includes an automatic speech recognition (ASR) model and a F0 network, and the trainer 1466 transmits the update command 1443 to update the ASR model, the F0 network, or both. In certain aspects, Fig.12 The F0 extractor 1244 includes a F0 network. In certain aspects, Figure 1 The feature detector 154 includes an ASR model.

[0231] Fig.15 The device 102 is depicted as an implementation 1500 of an integrated circuit 1502 including one or more processors 190. The integrated circuit 1502 includes a signal input 1504 (such as one or more bus interfaces) to enable receiving input data 1549 for processing. The input data 1549 includes an input speech representation 149, a source speech representation 163, image data 153, user input 103, an operating mode 105, or a combination thereof.

[0232] The integrated circuit 1502 also includes an audio output 1506 (such as a bus interface) to enable transmission of the output signal 135. The integrated circuit 1502 enables source speech modification based on input speech characteristics to be implemented as a component in a system, such as: Fig.16 A mobile phone or tablet computer as depicted in Fig.17 The headphones depicted in Fig.18 The earbud-type headphones depicted in Fig.19 Wearable electronic devices such as Fig. 20 The voice controlled speaker system described in Fig.21 The camera depicted in Fig.23 The extended reality headset depicted in Fig.24 The extended reality glasses depicted in Fig. 22 or Fig.25 The means of transport depicted in .

[0233] As an illustrative, non-limiting example, Fig.16 A specific implementation 1600 is depicted in which the device 102 includes a mobile device 1602, such as a phone or tablet computer. The mobile device 1602 includes one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, and a display screen 1604. Components of the one or more processors 190, including the audio analyzer 140, are integrated into the mobile device 1602 and are shown using dashed lines to indicate internal components that are not generally visible to a user of the mobile device 1602.

[0234] In certain aspects, the one or more cameras 1630 include Fig. 9 One or more cameras 910. One or more cameras 1630 are provided as non-limiting examples of image sensors. In some examples, one or more other types of image sensors may be used in addition to or in place of a camera. In certain aspects, one or more microphones 1610 include Fig. 9 One or more microphones 920. One or more microphones 1610 are provided as non-limiting examples of audio sensors. In some examples, one or more other types of audio sensors may be used in addition to or in place of a microphone. In certain aspects, one or more speakers 1620 include Fig.11 One or more speakers 1110, Fig.13A One or more speakers 1310, Fig. 13B One or more speakers 1360, or a combination thereof.

[0235] In a particular example, the audio analyzer 140 operates to detect user voice activity and then processes the user voice activity to perform one or more operations at the mobile device 1602, such as launching a graphical user interface or otherwise displaying other information associated with the user's voice at the display screen 1604 (e.g., via an integrated "smart assistant" application).

[0236] In the example, Figure 1The source speech representation 163 represents source speech associated with the virtual assistant application of the mobile device 1602. The input speech representation 149 represents input speech received by the audio analyzer 140 via one or more microphones 1610. The audio analyzer 140 determines input characteristics 155 of the input speech representation 149 and updates the source speech representation 163 of the source speech based on the input characteristics 155 to generate an output signal 135 representing the output speech, as shown in FIG. Figure 1 The output speech corresponds to a social interaction response from the virtual assistant application based on the input characteristics 155. For example, the response from the virtual assistant is updated based on the input characteristics 155 of the input speech.

[0237] Fig.17 A specific implementation 1700 is depicted in which the device 102 includes a headphone device 1702. The headphone device 1702 includes one or more microphones 1610, one or more speakers 1620, or a combination thereof. Components of the one or more processors 190, including the audio analyzer 140, are integrated into the headphone device 1702. In a specific example, the audio analyzer 140 operates to detect user voice activity (which may cause the headphone device 1702 to perform one or more operations at the headphone device 1702), send audio data corresponding to the user voice activity to a second device (not shown) for further processing, or a combination thereof.

[0238] In some examples, the source speech representation 163 corresponds to a source audio signal to be played by one or more speakers 1620. In these examples, the headphone device 1702 updates the source speech representation 163 to generate the output signal 135 and outputs the output signal 135 via the one or more speakers 1620 (rather than the source audio signal).

[0239] In some examples, the source speech representation 163 corresponds to a source audio signal received from one or more microphones 1610. In these examples, the headphone device 1702 updates the source speech representation 163 to generate an output signal 135 and provides the output signal 135 to another device or component.

[0240] Fig.18 An implementation 1800 is depicted in which the device 102 includes a portable electronic device corresponding to a pair of earbuds 1806, the pair of earbuds including a first earbud 1802 and a second earbud 1804. Although earbuds are described, it should be understood that the disclosed technology may be applied to other in-ear or over-ear playback devices.

[0241] The first earbud 1802 includes: a first microphone 1820, such as a high signal-to-noise ratio microphone positioned to capture the voice of the wearer of the first earbud 1802; an array of one or more other microphones that are configured to detect ambient sounds and are spatially distributed to support beamforming, which are illustrated as microphones 1822A, 1822B, and 1822C; an "internal" microphone 1824 close to the wearer's ear canal (e.g., to assist in active noise cancellation); and a self-voice microphone 1826, such as a bone conduction microphone configured to convert sound vibrations of the wearer's ear bones or skull into audio signals.

[0242] In a particular implementation, the one or more microphones 1610 include a first microphone 1820, microphones 1822A, 1822B, and 1822C, an internal microphone 1824, a self-speech microphone 1826, or a combination thereof. In a particular aspect, the audio analyzer 140 of the first earbud 1802 receives audio signals from the first microphone 1820, microphones 1822A, 1822B, and 1822C, the internal microphone 1824, the self-speech microphone 1826, or a combination thereof.

[0243] The second earbud 1804 can be configured in a substantially similar manner to the first earbud 1802. In some implementations, the audio analyzer 140 of the first earbud 1802 is further configured to receive one or more audio signals generated by one or more microphones of the second earbud 1804, such as via wireless transmission between the earbuds 1802, 1804 or via wired transmission (in implementations where the earbuds 1802, 1804 are coupled via a transmission line). In other implementations, the second earbud 1804 also includes the audio analyzer 140, so that the techniques described herein can be performed by a user wearing a single earbud of either earbud 1802, 1804.

[0244] In some implementations, the earbuds 1802, 1804 are configured to automatically switch between various operating modes, such as a pass-through mode, in which ambient sounds are played via the speaker 1830; a playback mode, in which non-ambient sounds (e.g., streaming audio corresponding to a phone conversation, media playback, video games, etc.) are played back through the speaker 1830; and an audio zoom mode or beamforming mode, in which one or more ambient sounds are emphasized and / or other ambient sounds are suppressed for playback at the speaker 1830. In other implementations, the earbuds 1802, 1804 may support fewer modes, or may support one or more other modes instead of, or in addition to, the described modes.

[0245] In one illustrative example, the earbuds 1802, 1804 may automatically switch from playback mode to pass-through mode in response to detecting the wearer's voice, and may automatically switch back to playback mode after the wearer has stopped speaking. In some examples, the earbuds 1802, 1804 may operate in two or more modes concurrently, such as by performing audio zoom on a particular ambient sound (e.g., a dog barking), and playing the audio zoom sound superimposed on the sound being played while the wearer is listening to music (the volume may be reduced while the audio zoom sound is playing). In this example, the wearer may be alerted to the ambient sound associated with the audio event without stopping music playback.

[0246] Fig.19 A specific implementation 1900 is depicted in which the device 102 includes a wearable electronic device 1902 (illustrated as a "smart watch"). An audio analyzer 140, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof are integrated into the wearable electronic device 1902. In a specific example, the audio analyzer 140 operates to detect user voice activity and then processes the user voice activity to perform one or more operations at the wearable electronic device 1902, such as launching a graphical user interface or otherwise displaying other information associated with the user's voice at a display screen 1904 of the wearable electronic device 1902. For illustration, the wearable electronic device 1902 may include a display screen 1904 configured to display a notification based on the user's voice detected by the wearable electronic device 1902. In a specific aspect, the display screen 1904 displays a notification indicating that an input characteristic 155 was detected, a target characteristic 177 was applied to generate an output signal 135, or both.

[0247] In a specific example, the wearable electronic device 1902 includes a haptic device that provides a tactile notification (e.g., a vibration) in response to detection of user voice activity. For example, the tactile notification may cause the user to look at the wearable electronic device 1902 to see a displayed notification indicating that a keyword spoken by the user was detected. Thus, the wearable electronic device 1902 may alert a user with hearing loss or a user wearing a headset that the user's voice activity was detected.

[0248] Fig. 202000 is a specific implementation in which the device 102 includes a wireless speaker and a voice-activated device 2002. The wireless speaker and voice-activated device 2002 may have wireless network connectivity and be configured to perform assistant operations. A processor 190 including an audio analyzer 140, one or more microphones 1610, one or more cameras 1630, or a combination thereof is included in the wireless speaker and voice-activated device 2002. The wireless speaker and voice-activated device 2002 also includes one or more sensors 1620. During operation, in response to receiving a verbal command identified as a user's voice via the operation of the audio analyzer 140, the wireless speaker and voice-activated device 2002 may perform an assistant operation, such as via the execution of a voice activation system (e.g., an integrated assistant application). The assistant operation may include adjusting the temperature, playing music, turning on the lights, etc. For example, the assistant operation is performed in response to receiving a command after a keyword or key phrase (e.g., "Hello, assistant"). In an example, the audio analyzer 140 generates a source voice representation 163 using the assistant's voice as the source voice, updates the source voice representation 163 based on input voice received via one or more microphones 1610 to generate an output signal 135 , and outputs the output signal 135 via one or more speakers 1620 .

[0249] Fig.21 An implementation 2100 is depicted in which the device 102 includes a portable electronic device corresponding to a camera device 2102. The audio analyzer 140, one or more microphones 1610, one or more speakers 1620, or a combination thereof are included in the camera device 2102. In certain aspects, the one or more cameras 1630 include the camera device 2102. During operation, in response to receiving a spoken command identified as a user's voice via operation of the audio analyzer 140, the camera device 2102 may perform operations responsive to the spoken user command, such as adjusting image or video capture settings, image or video playback settings, or image or video capture instructions, as illustrative examples.

[0250] In an example, the camera device 2102 includes an assistant application, and the audio analyzer 140 generates a source voice representation 163 using the voice of the assistant application as the source voice, updates the source voice representation 163 based on the input voice received via one or more microphones 1610 to generate an output signal 135, and outputs the output signal 135 via one or more speakers 1620.

[0251] Fig. 22An implementation 2200 is depicted in which the device 102 corresponds to or is integrated within a vehicle 2202 (shown as a manned or unmanned aerial device (e.g., a package delivery drone)). The audio analyzer 140, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof are integrated into the vehicle 2202. User voice activity detection may be performed based on audio signals received from the one or more microphones 1610 of the vehicle 2202, such as for delivery instructions from an authorized user of the vehicle 2202.

[0252] In an example, the vehicle 2202 includes an assistant application, and the audio analyzer 140 generates a source voice representation 163 using the voice of the assistant application as the source voice, updates the source voice representation 163 based on the input voice received via one or more microphones 1610 to generate an output signal 135, and outputs the output signal 135 via one or more speakers 1620.

[0253] Fig.23 An implementation 2300 is depicted in which the device 102 includes a portable electronic device corresponding to an extended reality (XR) headset 2302. The headset 2302 may include an augmented reality headset, a mixed reality headset, or a virtual reality headset. The audio analyzer 140, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof are integrated into the headset 2302. User voice activity detection may be performed based on audio signals received from the one or more microphones 1610 of the headset 2302.

[0254] The visual interface device is positioned in front of the user's eyes to enable display of augmented reality, mixed reality, or virtual reality images or scenes to the user while wearing the headset 2302. In a particular example, the visual interface device is configured to display a notification indicating that the user's voice is detected in the audio signal. In a particular aspect, the visual interface device displays a notification indicating that the input characteristic 155 is detected, the target characteristic 177 is applied to generate the output signal 135, or both.

[0255] Fig.24An implementation 2400 is depicted in which the device 102 includes a portable electronic device corresponding to XR glasses 2402. The glasses 2402 may include augmented reality glasses, mixed reality glasses, or virtual reality glasses. The glasses 2402 include a projection unit 2404 configured to project visual data onto a surface of a lens 2406, or to reflect the visual data from a surface of the lens 2406 onto a retina of the wearer. An audio analyzer 140, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof are integrated into the glasses 2402. The audio analyzer 140 may be used to generate an output signal 135 based on an audio signal received from the one or more microphones 1610. For example, the audio signal received from the one or more microphones 1610 may correspond to an input speech representation 149, a source speech representation 163, or both.

[0256] In a specific example, the projection unit 2404 is configured to display a notification indicating a user's voice detected in the audio signal. In a specific example, the projection unit 2404 is configured to display a notification indicating a detected audio event. For example, the notification may be superimposed on the user's field of view at a specific location that coincides with the location of the source of the sound associated with the audio event. For example, the user may perceive the sound as being emitted from the direction of the notification. In an illustrative specific implementation, the projection unit 2404 is configured to display a notification indicating that an input characteristic 155 is detected, an application target characteristic 177 is applied to generate an output signal 135, or both.

[0257] Fig.25Another specific implementation 2500 is depicted in which the device 102 corresponds to a vehicle 2502 (shown as a car) or is integrated within the vehicle. The vehicle 2502 includes one or more processors 190, which include an audio analyzer 140. The vehicle 2502 also includes one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof. In some aspects, at least one of the one or more microphones 1610 is positioned to capture the speech of the operator of the vehicle 2502. User voice activity detection can be performed based on audio signals received from the one or more microphones 1610 of the vehicle 2502. In some specific implementations, user voice activity detection can be performed based on audio signals received from an internal microphone (e.g., at least one of the one or more microphones 1610), such as for voice commands from authorized passengers. For example, user voice activity detection may be used to detect voice commands from an operator of the vehicle 2502 (e.g., a voice command from a parent to set the volume to 5 or set a destination for the autonomous vehicle), and to ignore voice commands from another passenger (e.g., a voice command from a child to set the volume to 10 or another passenger discussing another location). In some implementations, user voice activity detection may be performed based on an audio signal received from an external microphone (e.g., at least one of the one or more microphones 1610), such as an authorized user of the vehicle. In a specific implementation, in response to receiving a verbal command identified as a user's voice via operation of the audio analyzer 140, the voice activation system initiates one or more operations of the vehicle 2502 based on one or more keywords detected in the output signal 135 (e.g., "unlock", "start the engine", "play music", "show weather forecast", or another voice command), such as by providing feedback or information via the display 2520 or one or more speakers (e.g., speakers 1620).

[0258] In some aspects, the audio signals received from the one or more microphones 1610 are used as the source speech representation 163, the input speech representation 149, or both. In an example, the audio signals received from the one or more microphones 1610 are used as the input speech representation 149, and the audio signals played by the one or more speakers 1620 are used as the source speech representation 163. The audio analyzer 140 updates the source speech representation 163 to generate the output signal 135, as shown in FIG. Figure 1 For illustration, speech to be played by the one or more speakers 1620 is updated based on characteristics of the input speech of the passenger of the vehicle 2502 before being played back by the one or more speakers 1620 .

[0259] In another example, an audio signal received from one or more microphones 1610 is used as the source speech representation 163, and an audio signal received by the vehicle 2502 during a call from another device is used as the input speech representation 149. The audio analyzer 140 updates the source speech representation 163 to generate the output signal 135, as shown in FIG. Figure 1 For illustration, before the outgoing speech of the passenger of the vehicle 2502 is transmitted to another device, the outgoing speech is updated based on the incoming speech received from the other device.

[0260] refer to Fig.26 , shows a specific implementation of a method 2600 for performing source voice modification based on input voice characteristics. In certain aspects, one or more operations of the method 2600 are performed by at least one of the following: Figure 1 feature detector 154, embedding selector 156, conversion embedding generator 158, speech converter 164, audio analyzer 140, one or more processors 190, device 102, system 100; Figure 2 emotion detector 202, speaker detector 204, style detector 206, volume detector 212, pitch detector 214, speed detector 216; Figure 3A Audio emotion detector 354; Figure 3B Image emotion detector 356, emotion analyzer 358; Figure 4 The characteristic adjuster 492, the emotion adjuster 452, the speaker adjuster 454, the volume adjuster 456, the pitch adjuster 458, and the speed adjuster 460; Figure 8B Embedding combiner 852 and embedding combiner 854; Figure 8C The embedded combiner 856; Fig.14 One or more processors 1490, device 1402, system 1400; or a combination thereof.

[0261] Method 2600 includes, at 2602, processing an input audio spectrum of input speech to detect a first characteristic associated with the input speech. For example, Figure 1 The characteristic detector 154 processes the input audio spectrum 151 of the input speech (represented by the input speech representation 149) to detect the input characteristic 155 associated with the input speech, as shown in FIG. Figure 1 as described.

[0262] Method 2600 also includes selecting one or more reference embeddings from the plurality of reference embeddings based at least in part on the first characteristic at 2604. For example, embedding selector 156 selects one or more reference embeddings 157 from the plurality of reference embeddings based at least in part on input characteristic 155, such as reference Figure 1 as described.

[0263] The method 2600 further includes processing a representation of the source speech using one or more reference embeddings to generate an output audio spectrum of the output speech at 2606. For example, the speech converter 164 processes the source speech representation 163 using one or more reference embeddings 157 to generate an output audio spectrum 165 of the output speech (represented by the output signal 135). In certain aspects, using the one or more reference embeddings 157 includes using a conversion embedding 159 based on the one or more reference embeddings 157.

[0264] Thus, method 2600 enables dynamically updating source speech based on characteristics of input speech to generate output speech. In some aspects, the source speech is updated in real time. For example, data corresponding to the input speech, data corresponding to the source speech, or both are received by device 102 concurrently with audio analyzer 140 providing output signal 135 to a playback device (e.g., a speaker, another device, or both).

[0265] Fig.26 The method 2600 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit (such as a central processing unit (CPU)), a digital signal processor (DSP), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.26 The method 2600 may be performed by a processor executing instructions, such as reference Fig. 27 described.

[0266] refer to Fig. 27 , depicts a block diagram of a particular exemplary implementation of a device and generally designated as 2700. In various implementations, the device 2700 may have more Fig. 27 More or fewer components may be illustrated. In an exemplary implementation, device 2700 may correspond to device 102. In an exemplary implementation, device 2700 may execute the reference Figures 1 to 26 One or more operations described.

[0267] In certain implementations, the device 2700 includes a processor 2706 (e.g., a central processing unit (CPU)). The device 2700 may include one or more additional processors 2710 (e.g., one or more DSPs). In certain aspects, Figure 1 The one or more processors 190 correspond to the processor 2706, the processor 2710, or a combination thereof. The processor 2710 may include a speech and music coder-decoder (codec) 2708, which includes a speech decoder ("vocoder") encoder 2736, a vocoder decoder 2738, an audio analyzer 140, or a combination thereof.

[0268] The device 2700 may include a memory 2786 and a codec 2734. The memory 2786 may include instructions 2756 that may be executed by one or more additional processors 2710 (or processor 2706) to implement the functionality described with reference to the audio analyzer 140. The device 2700 may include a modem 2770 coupled to an antenna 2752 via a transceiver 2750. In a particular aspect, the modem 2770 transmits the encoded data 1322 of FIG. 13 to the device 1304 via the transceiver 2750. In a particular aspect, the modem 2770 receives the encoded data 1362 of FIG. 13 from the device 1306 via the transceiver 2750.

[0269] Device 2700 may include a display 2728 coupled to display controller 2726. In certain aspects, display 2728 includes Fig.16 Display screen 1604, Fig.19 Display screen 1904, Fig.23 A visual interface device of a headset 2302, Fig.24 Lens 2406, Fig.25 display 2520, or a combination thereof.

[0270] One or more microphones 1610, one or more speakers 1620, one or more cameras 1630, or a combination thereof may be coupled to the codec 2734. The codec 2734 may include a digital-to-analog converter (DAC) 2702, an analog-to-digital converter (ADC) 2704, or both. In a specific implementation, the codec 2734 may receive an analog signal from the one or more microphones 1610, convert the analog signal to a digital signal using the analog-to-digital converter 2704, and provide the digital signal to the voice and music codec 2708. The voice and music codec 2708 may process the digital signal, and the digital signal may be further processed by the audio analyzer 140. In a specific implementation, the audio analyzer 140 may generate a digital signal. The voice and music codec 2708 may provide the digital signal to the codec 2734. The codec 2734 may convert the digital signal to an analog signal using the digital-to-analog converter 2702 and may provide the analog signal to the one or more speakers 1620.

[0271] In a particular implementation, the device 2700 may be included in a system-in-package or system-on-chip device 2722. In a particular implementation, the memory 2786, the processor 2706, the processor 2710, the display controller 2726, the decoder 2734, and the modem 2770 are included in the system-in-package or system-on-chip device 2722. In a particular implementation, the input device 2730 and the power supply 2744 are coupled to the system-in-package or system-on-chip device 2722. In addition, in a particular implementation, as shown in FIG. Fig. 27 As illustrated, the display 2728, input device 2730, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, antenna 2752, and power supply 2744 are external to the system-in-package or system-on-chip device 2722. In a particular implementation, each of the display 2728, input device 2730, speaker 2792, one or more microphones 1610, one or more speakers 1620, one or more cameras 1630, antenna 2752, and power supply 2744 may be coupled to a component of the system-in-package or system-on-chip device 2722, such as an interface or a controller.

[0272] Device 2700 may include a smart speaker, a speaker pole, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and voice-activated device, a portable electronic device, a gaming device, an automobile, a computing device, a communication device, an Internet of Things (IoT) device, an XR device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0273] In conjunction with the described implementation, an apparatus includes means for processing an input audio spectrum of an input speech to detect a first characteristic associated with the input speech. For example, the means for processing the input audio spectrum may correspond to: Figure 1 characteristic detector 154, audio analyzer 140, one or more processors 190, device 102, system 100; Figure 2 emotion detector 202, speaker detector 204, style detector 206, volume detector 212, pitch detector 214, speed detector 216; Figure 3A Audio emotion detector 354; Fig.14One or more processors 1490, device 1402, system 1400; voice and music codec 2708, processor 2706, processor 2710, device 2700; one or more other circuits or components configured to process an input audio spectrum of input speech to detect a first characteristic associated with the input speech; or any combination thereof.

[0274] The apparatus also includes means for selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic. For example, the means for selecting may correspond to: Figure 1 The embedding selector 156, the audio analyzer 140, the one or more processors 190, the device 102, the system 100; Figure 4 The characteristic adjuster 492, the emotion adjuster 452, the speaker adjuster 454, the volume adjuster 456, the pitch adjuster 458, and the speed adjuster 460; Fig.14 One or more processors 1490, device 1402, system 1400; voice and music codec 2708, processor 2706, processor 2710, device 2700; one or more other circuits or components configured to select one or more reference embeddings from a plurality of reference embeddings based at least in part on a first characteristic; or any combination thereof.

[0275] The apparatus further includes a component for processing a representation of a source speech using one or more reference embeddings to generate an output audio spectrum of an output speech. For example, the component for processing may correspond to: Figure 1 The speech converter 164, the audio analyzer 140, the one or more processors 190, the device 102, and the system 100; Fig.14 One or more processors 1490, device 1402, system 1400; speech and music codec 2708, processor 2706, processor 2710, device 2700; one or more other circuits or components configured to process a representation of source speech using one or more reference embeddings to generate an output audio spectrum for output speech; or any combination thereof.

[0276] In some specific implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as memory 2786) includes instructions (e.g., instructions 2756) that, when executed by one or more processors (e.g., one or more processors 2710 or processor 2706), cause the one or more processors to process an input audio spectrum of input speech (e.g., input audio spectrum 151) to detect a first characteristic associated with the input speech (e.g., input characteristic 155). The instructions, when executed by the one or more processors, also cause the one or more processors to select one or more reference embeddings (e.g., one or more reference embeddings 157) from a plurality of reference embeddings based at least in part on the first characteristic. The instructions, when executed by the one or more processors, further cause the one or more processors to process a representation of a source speech (e.g., source speech representation 163) using the one or more reference embeddings to generate an output audio spectrum of an output speech (e.g., output audio spectrum 165).

[0277] Specific aspects of the present disclosure are described below in various sets of related embodiments:

[0278] According to embodiment 1, a device comprises: one or more processors, wherein the one or more processors are configured to: process an input audio spectrum of an input speech to detect a first characteristic associated with the input speech; select one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; and use the one or more reference embeddings to process a representation of a source speech to generate an output audio spectrum of an output speech.

[0279] Embodiment 2 includes the apparatus of embodiment 1, wherein the first characteristic comprises an emotion of the input speech.

[0280] Embodiment 3 includes the apparatus of embodiment 1 or embodiment 2, wherein the first characteristic comprises a volume of the input speech.

[0281] Embodiment 4 includes the apparatus of any one of Embodiments 1 to 3, wherein the first characteristic comprises a pitch of the input speech.

[0282] Embodiment 5 includes the apparatus of any one of Embodiments 1 to 4, wherein the first characteristic comprises a speed of the input speech.

[0283] Embodiment 6 includes a device according to any one of Embodiments 1 to 5, wherein the one or more processors are further configured to: process a source audio spectrum of the source speech using an encoder to generate a source speech embedding; and process the source audio spectrum using a fundamental frequency (F0) extractor to generate an F0 embedding, wherein the representation of the source speech is based on the source speech embedding and the F0 embedding.

[0284] Embodiment 7 includes the apparatus of any one of Embodiments 1 to 6, wherein the input speech is used as the source speech.

[0285] Embodiment 8 includes a device according to any one of Embodiments 1 to 6, wherein the one or more processors are further configured to receive the input speech via one or more microphones, wherein the source speech is associated with a virtual assistant, and wherein the output speech corresponds to a social interaction response from the virtual assistant based on the first characteristic.

[0286] Embodiment 9 includes the apparatus of any one of Embodiments 1 to 8, wherein a second characteristic associated with the output speech matches the first characteristic.

[0287] Embodiment 10 includes the apparatus of any one of Embodiments 1 to 9, wherein a first speech characteristic of the output speech matches a second speech characteristic of the input speech.

[0288] Embodiment 11 includes a device according to any one of Embodiments 1 to 10, wherein the representation of the source speech includes encoded source speech, and wherein the one or more processors are further configured to: generate a transformation embedding based on the one or more reference embeddings; apply the transformation embedding to the encoded source speech to generate a transformed encoded source speech; and decode the transformed encoded source speech to generate the output audio spectrum.

[0289] Embodiment 12 includes the apparatus of embodiment 11, wherein the one or more processors are configured to combine the one or more reference embeddings and a baseline embedding to generate the transformed embedding.

[0290] Embodiment 13 includes a device according to embodiment 11 or embodiment 12, wherein the one or more processors are configured to: select multiple reference embeddings from the multiple reference embeddings based at least in part on the first characteristic; and combine the multiple reference embeddings to generate the transformed embedding.

[0291] Embodiment 14 includes an apparatus according to any one of Embodiments 1 to 13, wherein the representation of the source speech is based on at least one of source speech audio, source speech text, source speech spectrum, linear predictive coding (LPC) coefficients, or Mel-frequency cepstral coefficients (MFCC).

[0292] Embodiment 15 includes a device according to any one of Embodiments 1 to 14, wherein the one or more processors are configured to: map the first characteristic to a target characteristic according to an operating mode; and select the one or more reference embeddings corresponding to the target characteristic from the multiple reference embeddings.

[0293] Embodiment 16 includes the device of embodiment 15, wherein the operating mode is based on user input, configuration settings, default data, or a combination thereof.

[0294] Embodiment 17 includes a device according to any one of Embodiments 1 to 16, wherein the one or more processors are further configured to: process the input audio spectrum to detect a first emotion; process the image data to detect a second emotion; and select the one or more reference embeddings from the multiple reference embeddings based on the first emotion and the second emotion.

[0295] Embodiment 18 includes the apparatus of Embodiment 17, wherein the one or more processors are further configured to perform facial detection on the image data, and wherein the second emotion is detected based at least in part on an output of the facial detection.

[0296] Embodiment 19 includes a device according to embodiment 17 or embodiment 18, wherein the one or more processors are further configured to receive audio data from one or more microphones concurrently with receiving the image data from one or more image sensors, and wherein the audio data represents the input voice, the source voice, or both.

[0297] Embodiment 20 includes the device of Embodiment 19, further comprising the one or more microphones and the one or more image sensors.

[0298] Embodiment 21 comprises a device according to any one of Embodiments 1 to 20, wherein the one or more processors are configured to: obtain a representation of the input speech; process the representation of the input speech to generate the input audio spectrum; and generate a representation of the output speech based on the output audio spectrum.

[0299] Embodiment 22 includes the apparatus of embodiment 21, wherein the representation of the input speech comprises a first text, and wherein the representation of the output speech comprises a second text.

[0300] Embodiment 23 includes a device according to any one of Embodiments 1 to 22, wherein the one or more processors are integrated into at least one of a vehicle, a communication device, a gaming device, an extended reality (XR) device, or a computing device.

[0301] According to embodiment 24, a method includes: processing an input audio spectrum of input speech at a device to detect a first characteristic associated with the input speech; selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; and using the one or more reference embeddings to process a representation of a source speech to generate an output audio spectrum of an output speech.

[0302] Embodiment 25 includes the method of embodiment 24, wherein the first characteristic comprises an emotion of the input speech.

[0303] Embodiment 26 includes the method of Embodiment 24 or Embodiment 25, wherein the first characteristic includes a volume of the input speech.

[0304] Embodiment 27 includes the method of any one of Embodiments 24 to 26, wherein the first characteristic includes a pitch of the input speech.

[0305] Embodiment 28 includes the method of any one of Embodiments 24 to 27, wherein the first characteristic includes a speed of the input speech.

[0306] Embodiment 29 includes a method according to any one of Embodiments 24 to 28, the method further comprising: processing a source audio spectrum of the source speech using an encoder to generate a source speech embedding; and processing the source audio spectrum using a fundamental frequency (F0) extractor to generate an F0 embedding, wherein the representation of the source speech is based on the source speech embedding and the F0 embedding.

[0307] Embodiment 30 includes the method according to any one of Embodiments 24 to 29, wherein the input speech is used as the source speech.

[0308] Embodiment 31 includes a method according to any one of Embodiments 24 to 29, the method further including receiving the input speech via one or more microphones at the device, wherein the source speech is associated with a virtual assistant, and wherein the output speech corresponds to a social interaction response from the virtual assistant based on the first characteristic.

[0309] Embodiment 32 includes the method of any one of Embodiments 24 to 31, wherein a second characteristic associated with the output speech matches the first characteristic.

[0310] Embodiment 33 includes the method of any one of Embodiments 24 to 32, wherein a first speech characteristic of the output speech matches a second speech characteristic of the input speech.

[0311] Embodiment 34 includes a method according to any one of Embodiments 24 to 33, the method further comprising: generating a conversion embedding at the device based on the one or more reference embeddings; applying the conversion embedding to the encoded source speech at the device to generate a converted encoded source speech, wherein the representation of the source speech includes the encoded source speech; and decoding the converted encoded source speech at the device to generate the output audio spectrum.

[0312] Embodiment 35 includes the method of embodiment 34, further comprising combining the one or more reference embeddings and a baseline embedding at the device to generate the transformed embedding.

[0313] Embodiment 36 includes a method according to embodiment 34 or embodiment 35, the method further comprising: selecting multiple reference embeddings from the multiple reference embeddings based at least in part on the first characteristic; and combining the multiple reference embeddings at the device to generate the converted embedding.

[0314] Embodiment 37 includes a method according to any one of embodiments 24 to embodiment 36, wherein the representation of the source speech is based on at least one of source speech audio, source speech text, source speech spectrum, linear predictive decoding (LPC) coefficients or Mel-frequency cepstral coefficients (MFCC).

[0315] Embodiment 38 includes a method according to any one of Embodiments 24 to 37, the method further comprising: mapping the first characteristic to a target characteristic according to an operating mode at the device; and selecting the one or more reference embeddings corresponding to the target characteristic from the multiple reference embeddings.

[0316] Embodiment 39 includes a method according to embodiment 38, wherein the operating mode is based on user input, configuration settings, default data, or a combination thereof.

[0317] Embodiment 40 includes a method according to any one of Embodiments 24 to 39, the method further comprising: processing the input audio spectrum at the device to detect a first emotion; processing image data at the device to detect a second emotion; and selecting the one or more reference embeddings from the multiple reference embeddings based on the first emotion and the second emotion.

[0318] Embodiment 41 includes the method according to Embodiment 40, the method further comprising performing facial detection on the image data, wherein the second emotion is detected based at least in part on an output of the facial detection.

[0319] Embodiment 42 includes a method according to embodiment 40 or embodiment 41, the method further comprising receiving audio data from one or more microphones at the device concurrently with receiving the image data from one or more image sensors at the device, wherein the audio data represents the input voice, the source voice, or both.

[0320] Embodiment 43 includes a method according to any one of Embodiments 24 to 42, the method further comprising: obtaining a representation of the input speech at the device; processing the representation of the input speech at the device to generate the input audio spectrum; and generating a representation of the output speech at the device based on the output audio spectrum.

[0321] Embodiment 44 includes the method of embodiment 43, wherein the representation of the input speech comprises a first text, and wherein the representation of the output speech comprises a second text.

[0322] According to embodiment 44, a device includes: a memory configured to store instructions; and a processor configured to execute the instructions to perform the method according to any one of embodiments 24 to 44.

[0323] According to embodiment 45, a non-transitory computer-readable medium stores instructions, which, when executed by a processor, cause the processor to perform a method according to any one of embodiments 24 to 44.

[0324] According to embodiment 46, an apparatus includes components for performing the method according to any one of embodiments 24 to embodiment 44.

[0325] According to embodiment 47, a non-transitory computer-readable medium stores instructions which, when executed by one or more processors, cause the one or more processors to: process an input audio spectrum of an input speech to detect a first characteristic associated with the input speech; select one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; and use the one or more reference embeddings to process a representation of a source speech to generate an output audio spectrum of an output speech.

[0326] According to embodiment 30, a device comprises: a component for processing an input audio spectrum of an input speech to detect a first characteristic associated with the input speech; a component for selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; and a component for processing a representation of a source speech using the one or more reference embeddings to generate an output audio spectrum of an output speech.

[0327] It will also be appreciated by the skilled person that the various illustrative logic blocks, configurations, modules, circuits, and algorithmic steps described in conjunction with the specific implementations disclosed herein can be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above in terms of their functions. Whether such functions are implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the overall system. The skilled person can implement the described functions in different ways for each specific application, and such specific implementation decisions will not be interpreted as causing a departure from the scope of the present disclosure.

[0328] The steps of the method or algorithm described in conjunction with the specific implementation disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integral with the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components.

[0329] The previous description of the disclosed aspects is provided to enable those skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but should be granted the broadest scope that may be consistent with the principles and novel features as defined by the following claims.

Claims

1. A device, comprising: One or more processors configured to: processing an input audio spectrum of input speech to detect a first characteristic associated with the input speech; selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; as well as A representation of the source speech is processed using the one or more reference embeddings to generate an output audio spectrum for the output speech. 2 . The apparatus of claim 1 , wherein the first characteristic comprises an emotion of the input speech.

3. The apparatus of claim 1, wherein the first characteristic comprises a volume of the input speech. The apparatus of claim 1 , wherein the first characteristic comprises a pitch of the input speech.

5. The apparatus of claim 1, wherein the first characteristic comprises a speed of the input speech.

6. The device of claim 1, wherein the one or more processors are further configured to: processing a source audio spectrum of the source speech using an encoder to generate a source speech embedding; and The source audio spectrum is processed using a fundamental frequency (F0) extractor to generate an F0 embedding, wherein the representation of the source speech is based on the source speech embedding and the F0 embedding.

7. The apparatus according to claim 1, wherein the input speech is used as the source speech.

8. The device of claim 1, wherein the one or more processors are further configured to receive the input speech via one or more microphones, wherein the source speech is associated with a virtual assistant, and wherein the output speech corresponds to a social interaction response from the virtual assistant based on the first characteristic.

9. The apparatus of claim 1, wherein a second characteristic associated with the output speech matches the first characteristic.

10. The apparatus of claim 1, wherein a first voice characteristic of the output speech matches a second voice characteristic of the input speech.

11. The apparatus of claim 1 , wherein the representation of the source speech comprises encoded source speech, and wherein the one or more processors are further configured to: generating a transformed embedding based on the one or more reference embeddings; applying the transform embedding to the encoded source speech to generate transformed encoded source speech; as well as The converted encoded source speech is decoded to generate the output audio spectrum.

12. The apparatus of claim 11, wherein the one or more processors are configured to combine the one or more reference embeddings and a baseline embedding to generate the transformed embedding.

13. The device of claim 11, wherein the one or more processors are configured to: selecting a plurality of reference embeddings from the plurality of reference embeddings based at least in part on the first characteristic; and The plurality of reference embeddings are combined to generate the transformed embedding.

14. The apparatus of claim 1, wherein the representation of the source speech is based on at least one of source speech audio, source speech text, source speech spectrum, linear predictive coding (LPC) coefficients, or Mel-frequency cepstral coefficients (MFCCs).

15. The device of claim 1, wherein the one or more processors are configured to: mapping the first characteristic to a target characteristic according to an operating mode; and The one or more reference embeddings corresponding to the target characteristic are selected from the plurality of reference embeddings.

16. The device of claim 15, wherein the operating mode is based on user input, configuration settings, default data, or a combination thereof.

17. The device of claim 1, wherein the one or more processors are further configured to: processing the input audio spectrum to detect a first emotion; processing the image data to detect a second emotion; and The one or more reference embeddings are selected from the plurality of reference embeddings based on the first emotion and the second emotion.

18. The device of claim 17, wherein the one or more processors are further configured to perform facial detection on the image data, and wherein the second emotion is detected based at least in part on an output of the facial detection.

19. The device of claim 17, wherein the one or more processors are further configured to receive audio data from one or more microphones concurrently with receiving the image data from one or more image sensors, and wherein the audio data represents the input speech, the source speech, or both.

20. The device of claim 19, further comprising the one or more microphones and the one or more image sensors.

21. The device of claim 1, wherein the one or more processors are configured to: obtaining a representation of the input speech; processing the representation of the input speech to generate the input audio spectrum; and A representation of the output speech is generated based on the output audio spectrum.

22. The apparatus of claim 21, wherein the representation of the input speech comprises a first text, and wherein the representation of the output speech comprises a second text.

23. The device of claim 1, wherein the one or more processors are integrated into at least one of a vehicle, a communication device, a gaming device, an extended reality (XR) device, or a computing device.

24. A method comprising: processing, at a device, an input audio spectrum of input speech to detect a first characteristic associated with the input speech; selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; as well as A representation of the source speech is processed using the one or more reference embeddings to generate an output audio spectrum for the output speech.

25. The method according to claim 24, further comprising: Processing a source audio spectrum of the source speech using an encoder to generate a source speech embedding; as well as The source audio spectrum is processed using a fundamental frequency (F0) extractor to generate an F0 embedding, wherein the representation of the source speech is based on the source speech embedding and the F0 embedding.

26. The method of claim 24, further comprising receiving the input speech via one or more microphones at the device, wherein the source speech is associated with a virtual assistant, and wherein the output speech corresponds to a social interaction response from the virtual assistant based on the first characteristic.

27. The method of claim 24, further comprising: generating, at the device, a transformed embedding based on the one or more reference embeddings; applying the transform embedding to the encoded source speech to generate transformed encoded source speech, wherein the representation of the source speech comprises the encoded source speech; as well as The converted encoded source speech is decoded at the device to generate the output audio spectrum.

28. The method of claim 27, further comprising combining, at the device, the one or more reference embeddings and a baseline embedding to generate the transformed embedding.

29. A non-transitory computer readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to: processing an input audio spectrum of input speech to detect a first characteristic associated with the input speech; selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; as well as A representation of the source speech is processed using the one or more reference embeddings to generate an output audio spectrum for the output speech.

30. An apparatus, comprising: means for processing an input audio spectrum of input speech to detect a first characteristic associated with said input speech; means for selecting one or more reference embeddings from a plurality of reference embeddings based at least in part on the first characteristic; and Means for processing a representation of the source speech using the one or more reference embeddings to generate an output audio spectrum for the output speech.

Images