Speech generation for virtual characters

Abstract

This disclosure describes techniques for generating speech for a virtual character. A plurality of source sounds may be received. The plurality of source sounds may correspond to a plurality of frames of the video. The video may include a virtual character. The plurality of source sounds may be converted into a plurality of representations in a hidden space using a first model. Each of the plurality of representations may include a plurality of parameters. The plurality of parameters may correspond to a plurality of sound features. A plurality of sounds may be generated in real-time for the virtual character in the video based at least in part on modifying at least one of the plurality of parameters of each representation.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the U.S. application serial number 17 / 751,324, filed on May 23, 2022, entitled “VOICE GENERATION FOR VIRTUALCHARACTERS,” the disclosure of which is incorporated herein by reference in its entirety. Background Art

[0003] People are increasingly using internet-based tools to communicate. Internet-based tools can be any software or platform. Existing social media platforms enable users to communicate with each other by sharing images, videos, and other information via static applications or web pages. As communication devices such as mobile phones become more sophisticated, people continue to look forward to new ways of entertainment, social networking, and communication. BRIEF DESCRIPTION OF THE DRAWINGS

[0004] The following detailed description will be better understood when read in conjunction with the accompanying drawings.For purposes of illustration, exemplary embodiments of various aspects of the disclosure are shown in the drawings; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0005] Figure 1 An exemplary system for virtual character voice generation according to the present disclosure is shown.

[0006] Figure 2 An exemplary framework for virtual character voice generation according to the present disclosure is shown.

[0007] Figure 3 An exemplary user interface for virtual character voice generation according to the present disclosure is shown.

[0008] Figure 4 An example process for virtual character voice generation according to the present disclosure is shown.

[0009] Figure 5 Another example process for virtual character voice generation according to the present disclosure is shown.

[0010] Figure 6 Another example process for virtual character voice generation according to the present disclosure is shown.

[0011] Figure 7 Another example process for virtual character voice generation according to the present disclosure is shown.

[0012] Figure 8 Another example process for virtual character voice generation according to the present disclosure is shown.

[0013] Fig. 9 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0014] A virtual character (e.g., a digital human, an image, a simulated character, etc.) is a simulation of a human being on a computer. Virtual characters are becoming increasingly popular in today's digital ecosystem. For example, many movies, television programs, simulated combat, games, virtual worlds, and industries utilize virtual characters in some way. An important aspect of a virtual character is its voice. However, it may be difficult to generate a desired voice for a virtual character. For example, it may be difficult to generate a voice with desired characteristics, such as desired volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. Therefore, it is desirable to use improved techniques for virtual character voice generation.

[0015] Figure 1 An exemplary system 100 for virtual character voice generation is shown. The system 100 may include a server 102 and a plurality of client devices 104. The server 102 and the plurality of client devices 104a to 104n may communicate with each other via one or more networks 132.

[0016] The servers 102 may be located at a data center, such as a single location, or distributed at different geographic locations (e.g., at several locations). The servers 102 may provide services via one or more networks 120. The networks 132 include various network devices, such as routers, switches, multiplexers, hubs, modems, bridges, repeaters, firewalls, proxy devices, etc. The networks 132 may include physical links, such as coaxial cable links, twisted pair cable links, optical fiber links, combinations thereof, etc. The networks 132 may include wireless links, such as cellular links, satellite links, Wi-Fi links, etc.

[0017] The service 102 may include multiple computing nodes that host various services. In an embodiment, the node hosts a virtual character service 112. The virtual character service 112 may include a content streaming service, such as an Internet Protocol video streaming service. The virtual character service 112 may be configured to generate a virtual character. The virtual character service 112 may be configured to provide the generated virtual character to the client devices 104a to 104n via the network 132. For example, the virtual character service 112 may include a video sharing service, a video hosting platform, a content distribution platform, a collaborative gaming platform, etc.

[0018] The plurality of client devices 104 may include any type of computing device, such as a mobile device, a tablet device, a laptop computer, a desktop computer, a smart TV or other smart device (e.g., a smart watch, a smart speaker, smart glasses, a smart helmet), a gaming device, a set-top box, a digital streaming device, a robot, etc. The plurality of client devices 104 may be associated with one or more users. A single user may use one or more of the plurality of client devices 104 to access the server 102. The plurality of client devices 104 may travel to various locations and use different networks to access the server 102.

[0019] In an embodiment, the client device 104 may include an application 106. The application 106 may be associated with the generation of a virtual character. For example, the application 106 may be configured to generate a virtual character locally on the client device 104. Alternatively, the application 106 may be configured to facilitate the virtual character service 112 to generate a virtual character. The application 106 outputs (e.g., displays, renders, presents) content associated with the generation of a virtual character to a user associated with the client device 104 via an interface 108 of the client device 104. The content may include video, audio, graphics, text data, etc. Each client device in the plurality of client devices 104 may include one or more microphones 107a to 107n. The microphones 107a to 107n may be configured to capture audio. For example, the audio may include audio spoken by one or more users.

[0020] In an embodiment, the virtual character service 112 and / or the application 106 are configured to generate a voice for the virtual character. The virtual character for which the voice is generated may, for example, be included in a video comprising a plurality of frames. In an embodiment, the virtual character for which the voice is generated may be presented in a movie, a television program, a simulated combat, a game, or a virtual world.

[0021] To generate a voice for a virtual character, the virtual character service 112 and / or the application 106 may receive a source sound, such as a source sound from a client device 104. For example, the source sound may be captured by one or more microphones 107a to 107n. The virtual character service 112 and / or the application 106 may generate a voice for the virtual character based on the source sound. For example, if the source sound includes audio of a user saying "Hello, my name is Kat", the virtual character service 112 and / or the application 106 may generate a voice for the virtual character that also says "Hello, my name is Kat". However, despite the same words as the source audio, the voice of the virtual character may sound different from the user's voice. In some embodiments, the voice of the virtual character may be generated in real time or near real time as the source sound is received or captured.

[0022] In an embodiment, the virtual character service 112 and / or the application 106 includes a first model 113. The first model 113 can be configured to convert the source sound into multiple representations in a latent space. Each representation in the multiple representations may include multiple parameters, such as multidimensional parameters. Multiple parameters may each correspond to a sound feature in multiple sound features. As an example and not limitation, multiple sound features may include volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. For example, the first model 113 can be configured to process sounds at a specific frequency (e.g., 30FPS). Each frame can be represented by one of the multiple representations. Multiple representations can be stored in databases 122a to 122n and / or local memory 115a as representation data 124.

[0023] In an embodiment, the first model 113 includes an encoder. If the first model 113 includes the encoder 113, the first model 113 may encode the source sound into a plurality of representations (each of the plurality of representations includes a plurality of parameters) through a trained artificial intelligence (AI) model.

[0024] In an embodiment, multiple parameters can be adjusted or modified by the user. For example, the user who generates the source sound may want to adjust or modify the parameters in order to generate the desired voice for the virtual character. For example, the desired voice can have one or more desired characteristics, such as the desired volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. Information indicating multiple parameters can be presented to the user, such as via interface 108. The user can modify these multiple parameters via interface 108. For example, if the desired voice has a very high pitch, the user can adjust or modify the parameters corresponding to the pitch accordingly. If one or more of the multiple parameters are modified, the modified multiple representations can be stored in databases 122a to 122n and / or local memory 115a as modified representation data 125.

[0025] In an embodiment, the databases 122a to 122n and / or the local memory 115a include one or more templates 126. The template 126 may indicate predefined parameters for the user. For example, a "female" template may indicate parameters that may change a male voice to a female voice. Other examples of templates include "child", "male", "alien", "demon", "cartoon character", etc. A user may select one or more templates to apply to a representation. For example, a user may select a "female" template to apply to a representation. If a user selects one or more templates to apply to a source sound, the predefined parameters associated with the selected one or more templates may be automatically applied to the representation. In some embodiments, the user may be able to further modify the parameters after applying one or more templates to a representation.

[0026] In some embodiments, the user can customize the parameter modification by providing two different voices or speech. For example, the user can provide a female voice and a male voice by reading the same sentence. The two inputs can be analyzed and the best parameters can be recommended for the voice change. The user can be able to further modify the parameters.

[0027] In an embodiment, the virtual character service 112 and / or the application 106 is configured to generate multiple sounds for the virtual character in real time based at least in part on the modified representation. For example, the first model 113 may include a decoder. The decoder may be configured to decode the modified representation into a sound signal (i.e., the virtual character voice). The sound signal may be output, such as via a speaker of the client device 104. The sound signal may be output at the same frame as the corresponding source sound.

[0028] In some embodiments, the virtual character service 112 and / or the application 106 includes a second model 114. The second model 114 may be configured to receive user camera data 127 as input. The camera data 127 may include one or more frames (i.e., images) captured by the camera of the client device 104. The camera data 127 may be stored in the databases 122a to 122n and / or the local memory 115a to 115n. The second model 114 may utilize the user camera input to analyze the skeleton of the user's body and his or her facial landmarks, etc., via one or more AI model-driven algorithms. The input of the second model 114 may be an image, and the output may be body and / or facial landmark coordinates 128. These body and / or facial landmark coordinates 128 may be used to drive the body and facial movements of the virtual character. The body and / or facial landmark coordinates 128 may be stored in the databases 122a to 122n and / or the local memory 115a.

[0029] In addition to receiving user camera input, the second model 114 can also receive the source sound discussed above as input. The source sound can be used to help improve the stability of the facial animation of the virtual character. For example, this can be particularly useful when the user's face is not facing the camera, is blocked by the hand, etc. In order to use the source sound as an additional dimension input, an encoder can be used to convert the sound input of the current time window into an image. Additional image layers can be added to the current camera image frame as additional input for training the second model 114. Training can be performed according to a traditional convolutional neural network (CNN) model training process.

[0030] Figure 2 An exemplary framework 200 for generating virtual character voices is shown. The framework 200 includes an encoder 204, a control user interface 206, and a decoder 208. A sound signal 202 may be input into the sound encoder 204. For example, the sound signal may be a source sound received from a client device. For example, the source sound may be captured by one or more microphones of the client device.

[0031] The sound encoder 204 can be configured to convert the input sound signal 202 into multiple representations in the latent space. Each representation in the multiple representations may include multiple parameters 207a to 207n, such as multidimensional parameters. Each parameter in the multiple parameters 207a to 207n may correspond to a sound feature in multiple sound features. As an example and not limitation, multiple sound features may include volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. For example, the encoder 204 may encode the input sound signal 202 into multiple representations (each representation in the multiple representations includes multiple parameters 207a to 207n) through a trained artificial intelligence (AI) model.

[0032] The plurality of parameters 207a to 207n may be adjusted or modified by a user via the control user interface 206. For example, a user generating a source sound may want to adjust or modify the parameters 207a to 207n in order to generate a desired voice for a virtual character. For example, the desired voice may have one or more desired characteristics, such as a desired volume, speech rate, pitch, velocity, rhythm, fluency, clarity, pronunciation, articulation, tone, etc. Information indicating the plurality of parameters 207a to 207n may be presented to the user, for example, via the interface 206.

[0033] Figure 3An exemplary user interface (UI) 300 for generating a virtual character voice is shown. In particular, UI 300 can be used to modify multiple parameters in order to generate a desired voice for a virtual character. For example, a user can modify a first parameter 302a by adjusting a first slider 304a to a desired position on UI 300. The first parameter 302a can be any voice parameter, including but not limited to volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain or dry gain. Similarly, a user can modify a second parameter 302b by adjusting a second slider 304b to a desired position on UI 300. The second parameter 302b can be any voice parameter, including but not limited to volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain or dry gain. In addition, a user can modify a third parameter 302c by adjusting a third slider 304c to a desired position on UI 300. The third parameter 302c may be any speech parameter, including but not limited to volume, speech rate, pitch, pitch ratio, velocity, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, or dry gain.

[0034] In an embodiment, a user may select one or more templates to apply to a source sound. A template may indicate predefined parameters, and such predefined parameters may be automatically applied to a representation. For example, sliders 304a to 304c may be automatically adjusted to positions corresponding to the selected template. In some embodiments, a user may be able to further modify sliders 304a to 304c after applying a template.

[0035] Return to reference Figure 2 , multiple sounds can be generated in real time for the virtual character based at least in part on the modified representation. For example, the decoder 208 can be configured to decode the modified representation into a sound signal 210 (i.e., the virtual character voice). The sound signal 210 can be output, for example, via a speaker of the client device 104. The sound signal 210 can be output at the same frame as the corresponding source sound.

[0036] Figure 4 An example process 400 is shown that is performed by one or more components shown in FIG. 100. For example, the process 400 may be performed at least in part by the server 102 and / or the client device 104. The process 400 may be performed to generate a voice for a virtual character. Figure 4 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, remove, reorder, or modify the operations depicted.

[0037] At 402, a plurality of source sounds may be received. The plurality of source sounds may correspond to a plurality of frames of a video. The video may include a virtual character. For example, the source sounds may be captured by one or more microphones of a client device 104. The source sounds may include audio of a user saying a word or phrase that the user wants the virtual character to say. For example, if the user wants the virtual character to say "Hello, my name is Kat," the user may speak into the microphone of his or her client device 104 and say "Hello, my name is Kat."

[0038] Speech can be generated for a virtual character based on source sounds. At 404, a first model can be used to convert multiple source sounds into multiple representations in a latent space. Each of the multiple representations can include multiple parameters. These multiple parameters can correspond to multiple sound features. As an example and not limitation, multiple sound features can include volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc.

[0039] The user may adjust or modify the multiple parameters. For example, a user generating a source sound may want to adjust or modify the parameters in order to generate a desired voice for a virtual character. For example, the desired voice may have one or more desired characteristics, such as a desired volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. Information indicating the multiple parameters may be presented to the user, for example, via an interface. The user may modify these multiple parameters via the interface.

[0040] At 406, multiple sounds can be generated in real time for the virtual character in the video based at least in part on modifying at least one of the multiple parameters of each representation. For example, if the parameter is modified so that the pitch of the source sound increases, the multiple sounds generated in real time for the virtual character can have a higher pitch than the source sound. These multiple sounds can be used as the voice of the virtual character. Any number of parameters may have been modified. Therefore, the multiple sounds generated in real time for the virtual character may sound very different from the source sound. For example, the source sound may show a man talking, and the multiple sounds generated in real time for the virtual character may sound like a child, a woman, a cartoon character, etc. talking.

[0041] Figure 5 An example process 500 is shown that is performed by one or more components shown in FIG. 100. For example, the process 500 may be performed at least in part by the server 102 and / or the client device 104. The process 500 may be performed to generate a voice for a virtual character. Figure 5 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, remove, reorder, or modify the operations depicted.

[0042] At 502, a plurality of source sounds may be received. The plurality of source sounds may correspond to a plurality of frames of a video. The video may include a virtual character. For example, the source sounds may be captured by one or more microphones of a client device 104. The source sounds may include audio of a user saying a word or phrase that the user wants the virtual character to say. For example, if the user wants the virtual character to say "Hello, my name is Kat," the user may speak into the microphone of his or her client device 104 and say "Hello, my name is Kat."

[0043] Voices can be generated for virtual characters based on source sounds. At 504, multiple source sounds can be encoded into multiple representations. Each of the multiple representations may include multiple parameters. These multiple parameters may correspond to multiple sound features. As an example and not limitation, multiple sound features may include volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. For example, the source sound can be encoded into multiple representations (each of the multiple representations includes multiple parameters) through a trained AI model.

[0044] The user may adjust or modify the plurality of parameters. For example, a user generating a source sound may want to adjust or modify the parameters in order to generate a desired voice for a virtual character. At 506, information indicating the plurality of parameters may be presented to the user. For example, a UI such as UI 300 may be presented to the user. The user may utilize the UI to modify one or more of the plurality of parameters in order to generate a desired voice for the virtual character.

[0045] For example, a user can modify a first parameter by adjusting a first slider to a desired position on the UI. The first parameter can be any speech parameter, including but not limited to volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, or dry gain. Similarly, a user can modify a second parameter by adjusting a second slider to a desired position on the UI. The second parameter can be any speech parameter, including but not limited to volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, or dry gain. In addition, a user can modify a third parameter by adjusting a third slider to a desired position on the UI. The third parameter can be any speech parameter, including but not limited to volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, or dry gain.

[0046] At 508, multiple modified representations can be generated by modifying at least one parameter in multiple parameters of each representation based on user input. For example, multiple modified representations can reflect any modification (if any) made by the user to multiple parameters. At 510, multiple sounds can be generated in real time for the virtual character in the video by decoding multiple modified representations into sound signals. The decoder can be configured to decode the modified representations into sound signals (i.e., virtual character voices). The sound signal can be output such as via a speaker of a client device. The sound signal can be output at the same frame as the corresponding source sound. At 512, multiple sounds can be output for the virtual character in a way that the virtual character emits multiple sounds in multiple frames. For example, it may look like the virtual character is talking. Multiple sounds can include words that look like the virtual character is saying.

[0047] Figure 6 An example process 600 is shown that is performed by one or more components shown in FIG. 100. For example, process 600 may be performed at least in part by server 102 and / or client device 104. Process 600 may be performed to generate speech for a virtual character. Figure 6 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, remove, reorder, or modify the operations depicted.

[0048] At 602, a plurality of source sounds may be received. The plurality of source sounds may correspond to a plurality of frames of a video. The video may include a virtual character. For example, the source sounds may be captured by one or more microphones of a client device 104. The source sounds may include audio of a user speaking a word or phrase that the user wants the virtual character to speak. For example, if the user wants the virtual character to say "Hello, my name is Kat", the user may speak into the microphone of his or her client device 104 and say "Hello, my name is Kat". The method 600 may proceed to step 604 or step 606.

[0049] Multiple source sounds can be used to generate voice for a virtual character. The virtual character for which voice is generated can be included in a video including multiple frames, for example. At 604, multiple source sounds can be processed at a predetermined rate to generate multiple sounds for the virtual character in the video in real time.

[0050] For example, processing multiple source sounds at a predetermined rate may include converting the source sounds into multiple representations in a latent space. Each representation in the multiple representations may include multiple parameters, such as multidimensional parameters. Multiple parameters may each correspond to a sound feature in multiple sound features. As an example and not limitation, multiple sound features may include volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc. Generating voice for a virtual character is a real-time running process. Multiple source sounds corresponding to multiple video frames are processed at a specific frequency. For example, the source sound can be processed at 30 frames per second (FPS).

[0051] A user may adjust or modify a number of parameters. For example, a user generating a source sound may want to adjust or modify parameters in order to generate a desired voice for a virtual character. For example, the desired voice may have one or more desired characteristics, such as a desired volume, speech rate, pitch, pitch ratio, rate, rhythm, fluency, clarity, pronunciation, articulation, tone, pass-through, wet gain, dry gain, etc.

[0052] A plurality of sounds may be generated in real time for the virtual character based at least in part on the modified representation. The plurality of sounds may be output, for example, via a speaker of a client device. The sound signal may be output at the same frame as the corresponding source sound.

[0053] The second model can be configured to receive user camera data as input. The camera data may include one or more frames (i.e., images) captured by the camera of the client device. The camera data may be stored in a database and / or local memory. The second model may use the user camera input to analyze the skeleton of the user's body and his or her facial landmarks, etc. via one or more AI model-driven algorithms. The input of the second model may be an image, and the output may be body and / or facial landmark coordinates. These body and / or facial landmark coordinates may be used to drive the body and facial movements of the virtual character.

[0054] In addition to receiving user camera input, the second model can also receive the source sound discussed above as input. At 606, the animation of the virtual character in the video can be improved by using multiple source sounds as additional inputs to the second model, and the second model is configured to control the animation of the virtual character in the video. The source sound can be used to help improve the stability of the facial animation of the virtual character. For example, this can be particularly useful when the user's face is not facing the camera, blocked by the hand, etc. In order to utilize the source sound as an additional dimensional input, the sound input of the current time window can be converted into an image using an encoder. Additional image layers can be added to the current camera image frame as additional inputs for training the second model. Training can be performed according to a traditional convolutional neural network (CNN) model training process.

[0055] Figure 7 An example process 700 is shown that is performed by one or more components shown in FIG. 100. For example, process 700 may be performed at least in part by server 102 and / or client device 104. Process 700 may be performed to generate speech for a virtual character. Figure 7 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, remove, reorder, or modify the operations depicted.

[0056] The template may indicate predefined voice parameters for the user. At 702, a plurality of templates may be generated, each of the plurality of templates including a set of predefined parameters, wherein each of the plurality of templates corresponds to a particular type of sound or voice. For example, a "female" template may indicate parameters that may change a male voice to a female voice. Other examples of templates include "child," "male," "alien," "demon," "cartoon character," and the like. The user may select one or more templates to apply to the representation.

[0057] At 704, information indicating a plurality of templates may be presented to the user. For example, information indicating a plurality of templates may be presented to the user via an interface. The user may be able to select one or more templates corresponding to the desired voice of the virtual character. For example, if the user wants the virtual character to have a female voice, the user may select a "female" template.

[0058] At 706, at least one sound may be generated by applying one of the plurality of templates to the at least one source sound based on a user's selection of one of the plurality of templates. For example, if the user selects one or more templates to apply to the source sound, predefined parameters associated with the one or more selected templates may be automatically applied to a representation associated with the source sound. In some embodiments, the user may be able to further modify the parameters after applying the one or more templates to the representation.

[0059] Figure 8 An example process 800 is shown that is performed by one or more components shown in FIG. 100. For example, process 800 may be performed at least in part by server 102 and / or client device 104. Process 800 may be performed to generate speech for a virtual character. Figure 4 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, remove, reorder, or modify the operations depicted.

[0060] As described above, a user can customize parameter modification by providing two different voices or speech. At 802, at least two different voices selected by the user can be analyzed. At 804, a set of parameters for generating a new voice for the virtual character can be generated based on the at least two different voices. For example, the two inputs can be analyzed and optimal parameters can be recommended for voice variation. The user can be able to further modify the parameters. At 806, the new voice can be modified based on the user's input of at least one parameter in the set of parameters.

[0061] Fig. 9 A computing device that can be used in various aspects such as Figure 1 to Figure 2 The services, networks, modules and / or devices depicted in . Fig. 9 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet computer, network appliance, PDA, e-reader, digital cellular phone or other computing node and can be used to perform any aspect of the computer described herein, such as to implement the methods described herein.

[0062] The computing device 900 may include a baseboard or "motherboard," which is a printed circuit board to which a number of components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 904 may operate in conjunction with a chipset 906. The CPU 904 may be a standard programmable processor that performs the arithmetic and logic operations required for the operation of the computing device 900.

[0063] The CPU 904 can perform necessary operations by switching from one discrete physical state to the next by manipulating switch elements that distinguish and change these states. Switch elements can typically include electronic circuits that maintain one of two binary states (such as flip-flops), and electronic circuits that provide output states based on logical combinations of the states of one or more other switch elements (such as logic gates). These basic switch elements can be combined to create more complex logic circuits, including registers, adders, subtractors, arithmetic logic units, floating point units, etc.

[0064] CPU 904 may be enhanced or replaced with other processing units, such as GPU 905. GPU 905 may include a processing unit specialized for, but not necessarily limited to, highly parallel computations, such as graphics and other visualization-related processing.

[0065] The chipset 906 may provide an interface between the CPU 904 and the remaining components and devices on the substrate. The chipset 906 may provide an interface to a random access memory (RAM) 908 used as main memory in the computing device 900. The chipset 906 may also provide an interface to a computer-readable storage medium such as a read-only memory (ROM) 920 or a non-volatile RAM (NVRAM) (not shown) for storing basic routines that may help boot the computing device 900 and transfer information between various components and devices. The ROM 920 or NVRAM may also store other software components required for the operation of the computing device 900 according to aspects described herein.

[0066] The computing device 900 may operate in a network environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 906 may include functionality for providing network connectivity through a network interface controller (NIC) 922, such as a Gigabit Ethernet adapter. NIC 922 may be capable of connecting the computing device 900 to other computing nodes through network 916. It should be understood that multiple NICs 922 may be present in the computing device 900 to connect the computing device to other types of networks and remote computer systems.

[0067] The computing device 900 may be connected to a mass storage device 928 that provides non-volatile storage for the computer. The mass storage device 928 may store system programs, application programs, other program modules, and data that have been described in detail herein. The mass storage device 928 may be connected to the computing device 900 via a storage controller 924 connected to the chipset 906. The mass storage device 928 may be composed of one or more physical storage units. The mass storage device 928 may include a management component 99. The storage controller 924 may interface with the physical storage unit via a serial attached SCSI (SAS) interface, a serial advanced technology connection (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between a computer and a physical storage unit.

[0068] The computing device 900 can store data on the mass storage device 928 by transforming the physical state of the physical storage unit to reflect the stored information. The specific transformation of the physical state may depend on various factors and different implementations of the present specification. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 928 is characterized as primary storage or secondary storage, etc.

[0069] For example, the computing device 900 may store information to the mass storage device 928 by issuing instructions through the storage controller 924 to change the magnetic properties of a specific location in a disk drive unit, the reflective or refractive properties of a specific location in an optical storage unit, or the electrical properties of a specific capacitor, transistor, or other discrete component in a solid-state storage unit. Other transformations of the physical medium are possible without departing from the scope and spirit of the present specification, and the above examples are provided only for the convenience of the present specification. The computing device 900 may also read information from the mass storage device 928 by detecting the physical state or properties of one or more specific locations in the physical storage unit.

[0070] In addition to the mass storage device 928 described above, the computing device 900 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. It should be understood by those skilled in the art that computer-readable storage media may be any available media that provides storage of non-transitory data and can be accessed by the computing device 900.

[0071] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, transient computer-readable storage media and non-transient computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state storage technology, compact disk ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), BLU-RAY or other optical storage, cassettes, magnetic tape, disk storage, other magnetic storage devices, or any other medium that can be used to store the desired information in a non-transitory manner.

[0072] Mass storage devices such as Fig. 9 The mass storage device 928 depicted in the figure can store an operating system for controlling the operation of the computing device 900. The operating system may include a version of the LINUX operating system. The operating system may include a version of the WINDOWS SERVER operating system from MICROSOFT. According to a further aspect, the operating system may include a version of the UNIX operating system. Various mobile phone operating systems, such as IOS and ANDROID, may also be used. It should be understood that other operating systems may also be used. The mass storage device 928 may store other systems or applications and data used by the computing device 900.

[0073] The mass storage device 928 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 900, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing the aspects described herein. As described above, these computer-executable instructions transform the computing device 900 by specifying how the CPU 904 switches between states. The computing device 900 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 900, may perform the methods described herein.

[0074] Computing devices such as Fig. 9 The computing device 900 depicted in FIG. 1 may also include an input / output controller 932 for receiving and processing input from a plurality of input devices, such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 932 may provide output to a display, such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It should be understood that the computing device 900 may not include Fig. 9 All components shown may include those not shown in Fig. 9 Other components explicitly shown in the Fig. 9 Completely different architecture as shown.

[0075] As described herein, a computing device may be a physical computing device such as Fig. 9 The computing device 900 of the present invention may also include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0076] It should be understood that the methods and systems are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is only for the purpose of describing specific embodiments and is not intended to be limiting.

[0077] As used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural referents unless the context clearly dictates otherwise. Ranges may be expressed herein as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from the one particular value and / or to the other particular value. Similarly, when a value is expressed as an approximation by using the antecedent "about", it should be understood that the particular value forms another embodiment. It should be further understood that the endpoints of each range in the range are significant relative to the other endpoint, and independent of the other endpoint.

[0078] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not occur.

[0079] Throughout the description and claims of this specification, the word "comprise" and variations of the word, such as "comprising / comprises" means "including but not limited to", and is not intended to exclude, for example, other components, integers or steps. "Exemplary" means "an example of..." and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is not used in a limiting sense, but is used for explanation purposes.

[0080] Components that can be used to perform the described methods and systems are described. When describing combinations, subsets, interactions, groups, etc. of these components, it should be understood that although specific references to each of the various individual and collective combinations and arrangements of these components may not be explicitly described, for all methods and systems, each is specifically contemplated and described herein. This applies to all aspects of the application, including but not limited to the operations in the described methods. Therefore, if there are various additional operations that can be performed, it should be understood that each of these additional operations can be performed with any specific embodiment or combination of embodiments of the described methods.

[0081] The present method and system may be understood more readily by reference to the following detailed description of the preferred embodiment and examples included therein and to the accompanying drawings and their description.

[0082] As will be appreciated by those skilled in the art, the methods and systems may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. In addition, the methods and systems may take the form of a computer program product on a computer-readable storage medium, the computer program product having computer-readable program instructions (e.g., computer software) embodied in the storage medium. More specifically, the methods and systems of the present invention may take the form of network-implemented computer software. Any suitable computer-readable storage medium may be used, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0083] Embodiments of the method and system will be described below with reference to block diagrams and flow charts of the method, system, apparatus, and computer program product. It should be understood that each block of the block diagram and flow chart, and the combination of blocks in the block diagram and flow chart, can be implemented by computer program instructions, respectively. These computer program instructions can be loaded on a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create means for implementing the functions specified in the flow chart blocks.

[0084] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner so that the instructions stored in the computer-readable memory produce a manufactured product that includes computer-readable instructions for implementing the functions specified in the flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to produce a computer-implemented process so that the instructions executed on the computer or other programmable device provide steps for implementing the functions performed in the flowchart blocks.

[0085] The various features and processes described above can be used independently of each other or in combination in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. In addition, in some specific implementations, certain method or process blocks may be omitted. The methods and processes described herein are not limited to any particular sequence, and the blocks or states associated therewith may be executed in other appropriate sequences.

[0086] For example, the described boxes or states may be performed in an order different from that specifically described, or multiple boxes or states may be combined in a single box or state. Example boxes or states may be performed serially, in parallel, or in some other manner. Boxes or states may be added to or removed from the described exemplary embodiments. The exemplary systems and components described herein may be configured differently than described. For example, elements may be added, removed, or rearranged compared to the described exemplary embodiments.

[0087] It should also be understood that the items are shown as being stored in or on memory when in use, and that these items or portions thereof may be transferred between memory and other storage devices for purposes of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via computer-to-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially implemented and provided in firmware and / or hardware, including but not limited to one or more application specific integrated circuits ("ASICs"), standard integrated circuits, controllers (e.g., by executing appropriate instructions, and including microcontrollers and / or embedded controllers), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), etc.

[0088] Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium, such as a hard disk, memory, network, or portable media item, so as to be read by an appropriate device or via an appropriate connection. Systems, modules, and data structures may also be transmitted as generated data signals (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media, including wireless-based and wired / cable-based media, and may take various forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital packets or frames). In other embodiments, such a computer program product may also take other forms. Therefore, the present invention may be practiced with other computer system configurations.

[0089] Although the method and system have been described in conjunction with preferred embodiments and specific examples, the scope is not limited to the specific embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0090] Unless expressly stated otherwise, any method set forth herein should not be construed as requiring that its operations be performed in a specific order. Therefore, in the event that a method claim does not actually list the order in which its operations are to be performed, or in the event that the operations are not otherwise expressly stated in the claim or description to be limited to a specific order, no order is intended to be inferred in any respect. This applies to any possible non-express basis for interpretation, including: matters of logic regarding arrangement of steps or flow of operations; ordinary meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0091] It will be apparent to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art by considering the specification and practice described herein. It is intended that the specification and example drawings be considered merely exemplary, with the true scope and spirit being indicated by the appended claims.

Claims

1. A method for generating speech for a virtual character, comprising: receiving a plurality of source sounds, wherein the plurality of source sounds correspond to a plurality of frames of a video, the video including a virtual character; converting the plurality of source sounds into a plurality of representations in a latent space using a first model, wherein each of the plurality of representations comprises a plurality of parameters, and the plurality of parameters correspond to a plurality of sound features; as well as A plurality of sounds are generated in real time for the virtual character in the video based at least in part on modifying at least one of the plurality of parameters of each representation.

2. The method according to claim 1, further comprising: encoding the plurality of source sounds into the plurality of representations, each representation of the plurality of representations comprising the plurality of parameters; presenting information indicative of the plurality of parameters to a user; as well as A plurality of modified representations are generated by modifying at least one of the plurality of parameters of each representation based on user input.

3. The method according to claim 2, further comprising: generating the plurality of sounds by decoding the plurality of modified representations into sound signals; as well as The plurality of sounds of the virtual character are output in a manner that the virtual character emits the plurality of sounds in the plurality of frames.

4. The method according to claim 1, further comprising: The plurality of source sounds corresponding to the plurality of frames are processed at a predetermined rate. The method of claim 4 , wherein the predetermined rate is 30 frames per second.

6. The method according to claim 1, further comprising: generating a plurality of templates, each template in the plurality of templates comprising a set of predefined parameters, wherein each template in the plurality of templates corresponds to a particular type of sound; presenting information indicative of the plurality of templates to a user; as well as At least one of the plurality of sounds is generated by applying the one of the plurality of templates based on a selection of the one of the plurality of templates by the user.

7. The method according to claim 1, further comprising: analyzing at least two different sounds selected by a user; as well as A set of parameters for generating a new voice for the virtual character is recommended based on the at least two different voices.

8. The method according to claim 7, further comprising: The new sound is modified based on user input to at least one parameter of the set of parameters.

9. The method of claim 1, wherein the plurality of sound features include pitch, tone, loudness, and duration.

10. The method according to claim 1, further comprising: Animation of the virtual character in the video is improved by using the plurality of source sounds as additional input to a second model, the second model being configured to control the animation of the virtual character in the video.

11. A system for generating speech for a virtual character, comprising: at least one processor; as well as at least one memory including computer readable instructions that, when executed by the at least one processor, cause the system to perform operations comprising: receiving a plurality of source sounds, wherein the plurality of source sounds correspond to a plurality of frames of a video, the video including a virtual character; converting the plurality of source sounds into a plurality of representations in a latent space using a first model, wherein each of the plurality of representations comprises a plurality of parameters, and the plurality of parameters correspond to a plurality of sound features; and The plurality of sounds are generated in real time for the virtual character in the video based at least in part on modifying at least one of a plurality of parameters of each representation.

12. The system of claim 11, the operations further comprising: encoding the plurality of source sounds into the plurality of representations, each representation of the plurality of representations comprising the plurality of parameters; presenting information indicative of the plurality of parameters to a user; as well as A plurality of modified representations are generated by modifying at least one of the plurality of parameters of each representation based on user input.

13. The system of claim 12, the operations further comprising: generating the plurality of sounds by decoding the plurality of modified representations into sound signals; as well as The plurality of sounds of the virtual character are output in a manner that the virtual character emits the plurality of sounds in the plurality of frames.

14. The system of claim 11, the operations further comprising: The plurality of source sounds corresponding to the plurality of frames are processed at a predetermined rate.

15. The system of claim 11, the operations further comprising: generating a plurality of templates, each template in the plurality of templates comprising a set of predefined parameters, wherein each template in the plurality of templates corresponds to a particular type of sound; presenting information indicative of the plurality of templates to a user; as well as At least one of the plurality of sounds is generated by applying the one of the plurality of templates based on a selection of the one of the plurality of templates by the user.

16. The system of claim 11, the operations further comprising: analyzing at least two different sounds selected by a user; as well as recommending a set of parameters for generating a new voice for the virtual character based on the at least two different voices; as well as The new sound is modified based on user input to at least one parameter of the set of parameters.

17. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: receiving a plurality of source sounds, wherein the plurality of source sounds correspond to a plurality of frames of a video, the video including a virtual character; converting the plurality of source sounds into a plurality of representations in a latent space using a first model, wherein each of the plurality of representations comprises a plurality of parameters, and the plurality of parameters correspond to a plurality of sound features; as well as A plurality of sounds are generated in real time for the virtual character in the video based at least in part on modifying at least one of a plurality of parameters of each representation.

18. The non-transitory computer readable storage medium of claim 16, the operations further comprising: encoding the plurality of source sounds into the plurality of representations, each representation of the plurality of representations comprising the plurality of parameters; presenting information indicative of the plurality of parameters to a user; as well as A plurality of modified representations are generated by modifying at least one of the plurality of parameters of each representation based on user input.

19. The non-transitory computer readable storage medium of claim 18, the operations further comprising: generating the plurality of sounds by decoding the plurality of modified representations into sound signals; as well as The plurality of sounds of the virtual character are output in a manner that the virtual character emits the plurality of sounds in the plurality of frames.

20. The non-transitory computer readable storage medium of claim 17, the operations further comprising: generating a plurality of templates, each template in the plurality of templates comprising a set of predefined parameters, wherein each template in the plurality of templates corresponds to a particular type of sound; presenting information indicative of the plurality of templates to a user; as well as At least one of the plurality of sounds is generated by applying the one of the plurality of templates based on a selection of the one of the plurality of templates by the user.

Images