Synthetic narration generation

Abstract

A system for synthetic voice generation may include a computer system having at least one processor and at least one memory storing executable machine instructions, wherein the computer system is configured to execute machine instructions to perform operations including: receive one or more recorded voice samples, each voice sample corresponding to a different human speaker voicing a training text with one or more voice characteristics different from voice characteristics of another human speaker; generate a structured training dataset based on the received recorded voice samples; train an artificial intelligence (AI) synthetic voice generation model using the generated structured training dataset; provide input corresponding to a target text to a deployed generative AI process to generate a narration track; and combine the synthetic narration track with a predetermined target video segment to generate a narrated audiovisual segment.

Description

Attorney Docket No.48501-0007WO1 SYNTHETIC NARRATION GENERATION TECHNICAL FIELD

[0001] This document describes data processing and network technology for synthetic voice generation, e.g., to provide narration either for a target video segment, or to be otherwise used for any of various purposes. BACKGROUND

[0002] A computer network can include a set of computers sharing resources located on or provided by network nodes. Computers use common communication protocols over digital interconnections to communicate with each other. These interconnections are made up of telecommunication network technologies based on physically wired, optical, and wireless radio-frequency methods that may be arranged in a variety of network topologies. One type of computing technology, artificial intelligence (“AI”), is an increasingly important technology in which the intelligence of machines and / or software, as opposed to the intelligence of humans, are used to achieve a task. Generative AI is a type of AI that can generate content such as text, images, or other media. SUMMARY

[0003] This document describes technology designed, among other things, to employ a generative AI model for generating synthetic audio tracks (e.g., narrations) to accompany visual media including, but not limited to, video books or the like.

[0004] AI-based content creation or enhancement technologies can be applied to video and / or audio segments. In the synthetic audio generation context, for example, a generative AI model can be trained using data representative of existingAttorney Docket No.48501-0007WO1 audio content (e.g., a narration recording corresponding to a training text segment) to enable the trained AI model to generate new, synthetic narration for other text segments different from the training text segment. And, in the context of AI-based enhancement, voice recordings can be processed to clean up the recording (e.g., remove background noise), improve the voicing captured by the recoding, e.g., by removing fillers (um, uh, er, like, you know), correct speech disorders (stutter, lisp), or essentially any other enhancement desired.

[0005] Data for this process can be structured and organized into formats and files that provide specific advantages due to how AI models and computer networks work, how data is handled, and how data is transmitted.

[0006] Implementations can include any, all, or none of the following features and / or advantages. By using an AI model to generate narration for a target video segment, the content creator (e.g., production companies of video books, movies, TV, songs, book publisher, etc.) can easily re-use a deployed AI model to generate synthetic narration to accompany essentially any suitable video segment without the time and expense of hiring one or more human narrators. In addition, because the AI model may be tuned to produce different vocal characteristics, it can be used to generate synthetic narration that varies in vocal characteristics such as volume, tone, inflection, pace, and the like.

[0007] In various aspects, a system or method for synthetic voice generation may include a computer system having at least one processor and at least one memory storing executable machine instructions, wherein the computer system is configured to execute machine instructions to perform operations including: receive one or more recorded voice samples, each voice sample corresponding to a different human speaker voicing a first training text with one or more voice characteristics differentAttorney Docket No.48501-0007WO1 from voice characteristics of another human speaker; generate a structured training dataset based on the received recorded voice samples; train an artificial intelligence (AI) synthetic voice generation model using the generated structured training dataset; provide input corresponding to a target text to a deployed generative AI process to generate a narration track; and combine the synthetic narration track with a predetermined target video segment to generate a narrated audiovisual segment. The system may further present the synthetically narrated audiovisual segment to an end- user. The system of claim 1 wherein one or more of the human speakers is one or more of a narrator, a singer, a poet, a lecturer, or a family member.

[0008] In the system or method, the training text may include one or more of prose, poetry, song lyrics, or lecture text. Generating the structured training dataset may include: processing the recorded voice samples to determine a median training dataset of the different vocal characteristics; and using the median training dataset in training the AI synthetic voice generation model. Prior to the combining, the system or method may iteratively adjusting one or more of the vocal characteristics of the generated narration track. The iterative adjustment may be automated, or use human input, or both. The vocal characteristics may include one or more of the following: volume, pitch, tone, register, pace, rate, inflection, breathiness, rhythm, fluency, emphasis, articulation, pronunciation, enunciation, intensity, range, or cadence. The recorded voice samples may be purpose-generated or based on one or more legacy voice recordings, or both. The recorded voice samples may be generated based at least in part on voicing directions provided to the one or more human speakers. The generated structured training dataset may include metadata to provide contextual guidance as to how AI model should process the structured training dataset. TheAttorney Docket No.48501-0007WO1 metadata may be in the form of tags inserted at desired points within the structured training dataset.

[0009] The system or method may be configured for one or more of the following purposes: create a personalized video book in which the synthetically generated narration includes words or phrases that are unique to a specific end-user; create a video book in which the synthetically generated narration includes words or phrases that sound like a deceased person’s voice; create a video book in which the synthetically generated narration includes words or phrases that sound like a non- verbal person’s voice; create a video book in which the synthetically generated narration includes words or phrases used for instructing children; and / or create a video book in which the synthetically generated narration includes words or phrases that sound different from the voice(s) of the one or more human speakers.

[0010] The system or method may produce words or phrases that sound different have been converted to be in a different language or dialect, wherein for example: the words or phrases that sound different have been converted to use a different accent; the words or phrases that sound different have been converted to a voice that sounds older or younger than the original one or more recorded voice samples; and / or the words or phrases that sound different have been converted to a voice that sounds older or younger than the original one or more recorded voice samples.

[0011] In the system or method, the original one or more recorded voice samples can be modified in one or more of the following manners: to remove defects arising from ambient noise; and / or to remove speech fillers.Attorney Docket No.48501-0007WO1

[0012] The system or method may be used to capture an elderly person’s voice to later generate narration for text yet to be generated and / or a child yet to be born; and / or for assisted reading improvement.

[0013] The technology described here can utilize data structures that provide technical advantages in computer networking. For example, client-server communications can be structured to avoid redundant or unnecessary data duplication and transmission. For example, a video template may be sent to a client device for narration-recording, but only the audio files of the narration need be returned in response. Similarly, if the narrator wants to verify their recordings and replace a portion, only a single file associated with that portion needs to be discarded and replaced instead of all the audio for the entire narration. With these techniques, the technology of data handling and network transmissions are enhanced.

[0014] This technology can be utilized to provide for the creation of narration to groups and users that would otherwise be unable to produce high-quality narration. For example, this technology can be used to create narration in the style of a person whose voice has been recorded, but who has passed away and is unable to record more training material. For example, an old recording of an ill grandparent can be utilized to train a new narration after the ill grandparent has passed away. Because the amount of recording needed is small (e.g., 30 seconds in some cases), found recordings such as telephone voice-mail or recorded public speaking at a local government meeting could be used by the family of the diseased to create new recordings for a grandchild that never met the grandparent. In another example, families of children with developmental limitations can benefit from the use of this technology. For example, a parent of a non-verbal child can record their voice once and generate a collection of different narrations. The non-verbal child can listen toAttorney Docket No.48501-0007WO1 dialog in the narration and identify a sentiment that they wish to communicate, but are incapable of voicing. The child can listen to the narration when their parent is unavailable, and listen to their parent narrating via live reading when the parent is available. In one example, an otherwise non-verbal child has learned to speak a message to their mother (“I love you, big truck.”) after listening to their mother and the synthetic narration (of a book about a family of cartoon trucks).

[0015] This technology provides a number of features can rely on the use of the machinery and hardware used to provide the functionality described herein. For example, a low-quality historic recording may be used as training data. As will be appreciated, corpuses of archival verbal recordings exist, and many of those recordings are on analog media that introduces auditory artifacts (e.g., a distinctive “pop” or “hiss”) or digitally with lossy compression artifacts (e.g., a hollow or tinny quality to the voice). However, due to the way that the technology maps features of text and speech data – instead of sound itself – these artifacts can be minimized or removed from the final synthesized narration. Because typical, spoken words in the “real world” are not data constructs the way that recordings of sound are, the use of computer processing can be applied to those data recordings to produce a machinery that can produce new sounds not otherwise creatable.

[0016] In some instances, this technology can be used in systems that spread computing resources over different devices, and can coordinate those devices so that sufficient resources are available for the most computationally intensive processes. For example, a recording can be performed by a mobile phone or laptop computer that has a microphone, but relatively weak processor and memory resources. As will be appreciated, these devices have the advantage of being portable, connecting to different network points, and used for a wide variety of tasks. Then, the data can beAttorney Docket No.48501-0007WO1 moved to relatively powerful – but large and immobile – data centers for processing. In this way, particular arrangements of hardware (e.g., a phone, a data network, and a server farm) can work together to provide the user with a client that is portable and flexible, while still taking advantage of computational resources that are only available in large, heavy, expensive devices.

[0017] Other features, aspects and potential advantages will be apparent from the accompanying description and figures. DESCRIPTION OF DRAWINGS

[0018] FIG.1 is a block diagram of a pipeline 100 configured to employ a generative AI model to generate a synthetic narration track for a target video segment.

[0019] FIG.2 is a framework for producing an audiovisual segment having a synthetically generated narration track.

[0020] FIG.3 shows a process 300 for producing an audiovisual segment having a synthetic narration track generated using generative AI.

[0021] FIG.4 is a schematic diagram showing an example of a computing device and a mobile computing device.

[0022] Like reference symbols in the various drawings indicate like elements DETAILED DESCRIPTION

[0023] An AI-based computer system is used to generate synthesized narration (including singing and other forms of voicing) of any arbitrary text, and then, optionally, combine the synthetic narration with a target video segment to create a synthetically narrated audiovisual segment, for example, a video representation of a book. Another option is to use the synthetic narration on its own, e.g., in an audio- only use case. The training text can be essentially any suitable text in myriadAttorney Docket No.48501-0007WO1 different genres, for example, poems, graphic novels, song lyrics, libretti, plays, children’s books, novels, non-fiction books, movie or TV scripts, speeches, lectures, newspapers, magazines, websites, blogs, social media or the like. Similarly, for configurations in which the synthetic narration track is to be combined with a target video segment, that video segment can be essentially any suitable video recording that needs or would benefit from narration. Details of such an AI-based computer system follow.

[0024] FIG.1 is a block diagram of a pipeline 100 configured to train and deploy a generative AI model to generate a synthetic narration track for a target video segment. As used herein, the term “narration” may refer to essentially any voicing by a human being, including not only reading of literature but also singing and / or other sounds orally generated by a human. In a configuration, the pipeline 100 may employ literature-type specific training in which different types or genres of literature may be identified, e.g., based on input from appropriate experts, and used to inform which vocal characteristics should be used or emphasized to complement any particular type or genre of training text to be voiced. For example, one set of vocal characteristics may be used to complement a Shakespearean play, while another set of vocal characteristics may be used to complement a book of nursery rhymes. In addition, different narrators having differing vocal characteristics, as well as voicing instructions for the narrators, may similarly be used to complement the particular type or genre of training text.

[0025] The first phase in generating synthetic narration, Sample Collection 202, involves creating or identifying voice recordings to be used for training a generative AI model. Such recordings may include not only standard reading of text but also singing, rapping, making sound effects or the like. Also, the recordings mayAttorney Docket No.48501-0007WO1 either be created anew from the voices of one or more narrators 102 and / or may be legacy recordings 104 that were obtained from public or private domains. The one or more narrators 102 may include either experienced or professional speakers (potentially with expertise in a particular type or genre of literature, or may be one or more laypersons having a desired vocal presentation (e.g., a nursery schoolteacher used to speaking to young children, or a western rancher used to giving instructions to ranch hands). In either case, the one or more narrators may be given directions for how to voice the training text. For example, the training text may be a 30-60 second script accompanied by directions to read the script in a normal, flat voice, or an excited voice, a happy voice, a sad voice, and so on. The directions provided may be either a single direction for the entire script or different directions for different portions of the script.

[0026] However obtained, the training recordings are collected and processed at stage 106, for example, to turn the recording into a training dataset suitable for training a generative AI model 110. To generate a training dataset using the recorded narration recordings, each sample must either be in digital form, or converted into digital form. In the context of obtaining voicing samples from multiple different narrators, the process of creating a training dataset may also involve finding a median voicing for one or more features (e.g., volume, tone, inflection, etc.) of the recorded samples from the different narrators. In addition, processing of the samples may also involve data modeling (identifying relationships, variables, and constraints to be represented in the dataset), and an annotation process in which the training data is labeled to make it easier for the AI model to interpret. In addition, regardless of the genre or type of literature being used for the sampling text, the resulting training setsAttorney Docket No.48501-0007WO1 would generally rely on the same set of variables, e.g., tone, inflection, pacing, rhyming, emphasis, and the like.

[0027] Once generated, in the Training stage 204, the training dataset 108 is provided to the generative AI model 110 for training 112. In general, training 112 an AI model involves four steps: model selection, initial training, training validation, and model testing. Model selection involves balancing various different factors including(i) whether the model is generative or predictive, (ii) the degree of complexity inproviding the desired functionality, (iii) the structure and size of the training dataset,(iv) the degree of accuracy desired, and (v) the computational resources available.Generating synthetic narration generally requires using a generative AI model, for example, a Variational Autoencoding (VAE) model, considered by some to be among the best models for generating synthetic audio, video, and imagery. However, essentially any suitable training algorithm may be used depending on design objectives.

[0028] Training the AI model may also involve tuning, which is usually a trial-and-error process involving changing hyperparameters (for example, the number of trees in a tree-based algorithm or the value of alpha in a linear algorithm), running the training algorithm on the data again, then comparing its performance in order to determine which set of hyperparameters results in the most accurate model. Once trained, the model is deployed as a generative AI tool 114. Optionally, the deployed AI tool 114 may be continuously trained and tested, among other reasons, to improve output results.

[0029] The deployed generative AI tool 114 uses the vocal characteristics learned from the training dataset 108 to generate a synthetic narration track 118 based on an input text 116. Optionally, the synthetic narration track 118 can be iterativelyAttorney Docket No.48501-0007WO1 adjusted 122 by varying one or more vocal characteristics at specified locations in the synthetic narration track 118, and providing corresponding feedback 120 to the AI tool 114. The iterative adjustment 122 may be accomplished, for example, using a graphic user interface (GUI) 121 to receive input from a human user. The GUI 121 may include, e.g., widgets such as sliders, knobs, buttons, and / or a wizard to enable the human user to vary vocal characteristics such as tone, volume, emphasis, inflection and the like. The iterative adjustment 122 could be used, for example, to account for differences between the input text 116 and training text used to generate the training dataset 108. For example, the training recording samples may come from the audio track of a public domain version of the animated film Sleeping Beauty to generate synthetic narration 124 for the audio track of a different animated film such as Peter and the Wolf, which may have a similar feel in some respects but different in others such as tone, mood, or the like. Optionally, the iterative adjustments could be used not only to alter the narration track 124 but also to fine-tune the generative AI model 110, and thus the deployed generative AI tool 114.

[0030] After the iterative adjustment stage 122, the final narration 124 is used as at least a portion of the combination stage 126, which combines the narration track 124 (and potentially one or more other audio tracks) with a target video segment 127 using a production tool designed for that purpose. The output of the combination stage 126 is a synthetically narrated audiovisual segment, for example, a video book or the like.

[0031] FIG.2 shows a framework 200 for using a generative AI model to generate, using text input such as a script, a synthetic narration track for a target video segment, such as a video rendition of a book. Consistent with the pipeline 100 of FIG.1, the framework 200 has four stages: Sample Collection 202, AI Model TrainingAttorney Docket No.48501-0007WO1 204, Production 206, and Presentation 208. Each will be described in turn at a relatively high level, followed by more detailed explanations.

[0032] Sample Collection 202 involves obtaining recorded narration samples from either multiple narrators or a single narrator, which will be used as a training dataset for a generative AI Model. The recorded narration samples can either be newly generated for the purpose of generating narration for new scripts, or the recorded samples can be preexisting, e.g., legacy narration recordings from public and / or private domains. In general, each narration sample corresponds to the same text or script as voiced by a person having expertise in narration, public speaking, voice acting, or the like, although in some use cases voice recordings from untrained speakers may be desirable.

[0033] Sample Collection 202 also involves dataset preparation – that is, generating a training dataset, which will be used to train the AI model. To generate a training dataset using the recorded narration samples, each sample must either be in digital form, or converted into digital form. In the context of obtaining voicing samples from multiple different narrators, the process of creating a training dataset may also involve finding a median voicing for one or more features (e.g., volume, tone, inflection, etc.) of the recorded samples from the different narrators. In addition, Sample Collection 202 may also involve data modeling (identifying relationships, variables, and constraints to be represented in the dataset), and an annotation process in which the training data is labeled to make it easier for the AI model to interpret.

[0034] Typically, a training dataset will be comprised of a relatively large quantity of data points for each of a predetermined number of different voicing features. In the case of voice recordings, such features can include pitch, intonation, pronunciation, rhythm, inflection, volume, and the like. Generally speaking, aAttorney Docket No.48501-0007WO1 training dataset is formatted as a type of table, with features corresponding to the columns, and data points corresponding to the rows. A rule of thumb is that the training dataset should include at least 10 times as many rows (data points) as there are columns (voicing features), although any other suitable ratio may be used depending on design objectives.

[0035] AI Model Training 204 involves using the training dataset generated and / or collected during the Sample Collection phase 202 to train a generative AI model, which when trained and deployed, can be used to generate new, synthetic narration for any arbitrary script or text.

[0036] Production 206 is the process of using a deployed, generative AI tool to produce new, synthetic narration for any arbitrary script or other text. In one configuration, Production 206 can generally be divided into three different sub- phases.

[0037] In the first sub-phase 206a, a script or other text for which synthetic narration is desired is used as input to the AI tool, which generates an initial synthetic narration track.

[0038] Next, in second sub-phase 206b, which is optional, a human operator may provide the AI tool with iterative feedback, e.g., via a user interface such as a GUI, to adjust one or more different vocal characteristics of the initial synthetic narration track as desired. For example, the feedback may be to adjust the inflection or emphasis or tone at certain points in the track. This iterative process is repeated until the synthetic narration sounds as desired, at which point a final narration track is generated by the AI tool.

[0039] Lastly, in the third phase 206c of Production 206, the final narration track (and potentially one or more other audio tracks) is combined as desired with theAttorney Docket No.48501-0007WO1 target video segment, preferably by an automated process. Optionally, combining the narration track and the video can be accomplished by using a production tool designed for that purpose, e.g., one having functionality similar to ADOBE PREMIERE®. Or the combining could be accomplished using both automation and human input. Regardless of which method is used, the output of sub-phase 206 is a synthetically narrated audiovisual segment, for example, a video book or the like.

[0040] The framework 200 concludes at the Presentation phase 208, in which the synthetically narrated audiovisual segment is presented to one or more end-users for consumption, e.g., on a browser window showing the production company’s website, in a dedicated app running on a general-purpose computing device, or on a special-purpose device designed to present audiovisual segments.

[0041] FIG.3 shows a process 300 for producing an audiovisual segment having a synthetic narration track generated using generative AI. First, at 302, the process 300 receives multiple recorded narration samples, each sample corresponding to a different human narrator voicing a predetermined script or other text corpus. In the case of music, the narrator – i.e., the person(s) voicing the source text – can be a singer, either professional or otherwise. Each human narrator potentially has one or more vocal and / or reading characteristics that differ from other narrators. Such characteristics may include volume, pitch, tone, register, pace, rate, inflection, breathiness, rhythm, fluency, emphasis, articulation, pronunciation, enunciation, intensity, range, and cadence, among others. Suitable narrators can be persons having expertise in narration, voice acting, singing, or the like.

[0042] The narrations may be recorded specifically for the purposes of generating synthetic narration, or they be existing (aka legacy) narration recordings, for example, from the public domain or a private collection. Regarding the former,Attorney Docket No.48501-0007WO1 the narrators may be provided specific directions about how to voice a script, including instructions about when to sound quiet, loud, scary, happy, sad, professorial, casual, or the like. The instructions may also specify how the script should be voiced to follow a desired emotional arc of the story, for example, a storyteller may be instructed that the voicing should be quieter for scary parts of the story, or louder for exciting parts of the story. One scenario for collecting the recorded narration samples to be used for training the AI model is to hire one or more speakers having desired or differing vocal or reading characteristics.

[0043] The scripts corresponding to the recorded narration samples may come from different styles or genres of texts, for example, poems, graphic novels, instruction manuals, nursery rhymes, children’s books, plays, lectures, songs (in which the narration would be in the form of singing), or otherwise any writings or text that are suitable for narration purposes. Different genres may be treated differently to reflect their unique characteristics. For example, a graphic novel (i.e., a story told by sequential art that is presented in comic-strip format) may be treated differently than, say, a prose novel given its sequential and partitioned format. Similarly, different types of poetry (Shakespearean Sonnets written in iambic pentameter versus nursery rhymes) may be treated differently in terms of cadence and rhyming scheme, among other aspects.

[0044] Next, at 304, the collected voice recordings are processed to generate a structured training dataset, e.g., similar to a JavaScript Object Notation (JSON) file, with metadata (e.g., flags) inserted at specified locations within the training dataset. The metadata, which provides instructions for training the generative AI model 110 regarding how the training text should be read, e.g., locations at which specified vocal characteristics (e.g., tone, inflection) should be used in generating the syntheticAttorney Docket No.48501-0007WO1 narration, may serve to map, e.g., emotional or contextual guidance about how to process the structured dataset. For example, the metadata flags could indicate when the training text (corresponding to the structured training dataset) is too be read in a “happy” voice or a “sad” voice, each of which would map to specific vocal characteristics that impart the desired mood of the narration. The metadata potentially could be automatically generated during processing of the training dataset, and / or by manually specified by a human user.

[0045] Next, at 306, the generative AI model 110 is trained in the manner described above with regard to FIGs.1 and 2.

[0046] Next, at 308, a target script or other input text 116 is input into a deployed AI tool 114, which uses it to generate a corresponding synthetic narration track 118 of the input text 116 as described above. Depending on implementation details, the synthetic narration track 118 can serve either as an initial synthetic narration track to be iteratively adjusted 122 in the next step, or as the final synthetic narration track 124 if the optional iterative adjustment is not implemented.

[0047] Next, at 310, depending on the desired implementation, the narration track 118 can be iteratively adjusted, for example, using a GUI that receives adjustment input, with corresponding adjustment feedback provided to the generative AI tool 114, in the manner described above, to generate a final narration track 124 in the manner described above.

[0048] Next, at 312, the synthetic narration track 124 is optionally combined (potentially with one or more other audio tracks such as music or sound effects) with a target video segment to produce a synthetically narrated audiovisual segment 128, encoded in any suitable manner such as .MP4 or .MOV encoding. Optionally, theAttorney Docket No.48501-0007WO1 synthetic narrow track 124 may be used without combining it with a video segment depending on the desired use case, examples of which are described below.

[0049] Finally, in the presentation stage 208 at 314, the synthetically narrated audiovisual segment is presented to an end-user. Optionally, the audiovisual segment provided to the end-user may itself have functionality that enables the end-user to tweak vocal characteristics of the synthetic narration track, although such end-user adjustments typically would not be used to fine tune the generative AI model 110.

[0050] Exemplary Use Cases:

[0051] The following non-exhaustive descriptions provide exemplary use cases for the synthetic narration generation technology described above.

[0052] Personalized Audio Books – Synthetic narration generation as described above may be used to personalize the content provided to the end-user, e.g., by inserting voicing corresponding to the end-user’s name or essentially any other item of information personal to the end-user (e.g., birthday, favorite color, names of other family members, street address, hobbies, accomplishments, snippets of favorite songs) at specified locations within the audio book. For example, various names of family members may be interspersed with the target text in an audio book to make the book more entertaining, memorable, important, or treasured by the end-user. Privacy laws and other rules identifying any items of personal information that should not be discovered or inserted into the audio book should be followed strictly.

[0053] To generate a personalized audio book, the fixed text corresponding to the structured training dataset input to the generative AI model would be modified to include placeholders (such as the metadata flags described above) for alterable text used for personalization purposes. The alterable, personalization text to be inserted at the placeholders in the structured dataset could come from essentially any internalAttorney Docket No.48501-0007WO1 (i.e., internal to the company producing the audio book) or external, publicly available source such as may be identified using a standard Internet search engine. A source of internal information could come from subscription data (e.g., the end-user’s email address) provided by the end-user at the time of signing up for an audio book service.

[0054] One way to introduce personalization into the synthetic narration generation process would be to use a template, which would have alterable fields corresponding to specified items of personal information. The alterable fields could be filled in either by the end-user (or, as appropriate, another person desiring to gift the audio book to the end-user), or based on internal information and / or search engine search results, either automatically or manually by personnel involved in the production of the audio book. Typically, personalization of an audio book would occur before providing the training dataset to the AI model, e.g., by inserting the personalization information at appropriate places in the structured dataset, e.g., using flags or other metadata as described above. Doing so would have the benefit of being able to use the same dataset structure whether personalization is being used or not.

[0055] Synthetic generation of a deceased person’s voice – another potential use case is to generate synthetic narration for a person who is deceased or otherwise unable to speak. For example, legacy or found voice recordings (e.g., from old family videos, voice-mail recordings, recorded speeches, or the like) of a deceased person’s speaking can be used to train a generative AI model to generate synthetic narration for the deceased person. Then, using the deployed AI tool, new narration for the deceased person (e.g., a grandmother) can be synthetically generated to be used to accompany a video presentation (e.g., a video book), or used in a standalone configuration. That way, a young child just learning to read can enjoy a video bookAttorney Docket No.48501-0007WO1 with the voice of his / her grandmother who, at this point, is long since deceased, and perhaps never even met the grandchild.

[0056] Synthetic generation of a (mostly) non-verbal person’s voice – another potential use case is to synthetically generate the voice of a generally non-verbal person (but with hearing ability) in order to generate new narration using one or more voice recordings of that non-verbal person’s speech to train a generative AI model. This use case may be potentially helpful in a therapeutic context to aid a non-verbal person in speaking new words or phrases in addition to those the non-verbal person has already voiced in a recording. For example, assume a non-verbal child who wants to express love to his mother but is unable to do so verbally because of a speech disorder. Further, the child has watched a narrated video book titled “I Love You, Little Truck” one or more times has recognized that, in the story, the Little Truck represents himself, while the Big Truck represents his mother, and has further learned that a way to verbally express love to his mother is to voice a line from the story, namely, “I love you, Big Truck.” If his speech can be recorded, then it can be used to train a generative AI model to generate synthetic speech in that child’s voice. In that way, the otherwise non-verbal child may be able to learn new words or phrases that the child has not previously voiced. Although typically training an AI voice model is based on voice recordings of 30 or 60 seconds, shorter snippets may also be used for that purpose, potentially with diminished accuracy, e.g., by using automated extrapolation and / or interpolation to synthesize a full vocabulary including phonemes that may be missing from the recorded voice sample used for AI training.

[0057] Synthetic voice generation for providing instructions to others – another potential use case is to use appropriate input recordings to generate synthetic voice generation for others. For example, voice recordings of a speaker who mayAttorney Docket No.48501-0007WO1 soon die (e.g., a grandparent) or otherwise be incapacitated such that the speaker can no longer articulate comprehensible words (e.g., an advanced ALS victim) could be used as input training samples to generate synthetic voice output for any arbitrary text input to the generative AI tool. As described above, the voice input recording(s) could be obtained either from new recordings made by the speaker pre-incapacitation specifically for the purpose of subsequently being able to generate synthetic voicing, or from legacy recordings, e.g., taken from an existing source such old home movies or the like.

[0058] Once the speaker’s voice has been used to train the generative AI model, several potential uses would be possible even after the speaker can no longer generate new voicings. For example, if the speaker is the parent or caretaker of a minor child, and no longer has voicing capability, standard instructions for proceeding successfully in daily life situations (e.g., wake up, get dressed, eat breakfast, do your homework, take a bath, eat your vegetables) could be synthetically generated in the exact voice of the incapacitated speaker. Due to its familiarity with the speaker’s authentic voice, such a synthetic duplicate voice may be more authoritative, soothing, or evoke fond memories. The standard instructions optionally could be stored in a library for ongoing use as needed.

[0059] In implementing a synthetic voice instruction capability as described above, certain differences in the process to follow may arise depending on whether a speaker’s voice recording is obtained from (i) a newly generated voice recording purposely made during the training data collection phase, or (ii) from legacy recordings of the speaker’s voice. For example, when generating new voice recordings, typically a 30-60 second script could be used along with instructions concerning the specific vocal characteristics to be used by the speaker. In contrast,Attorney Docket No.48501-0007WO1 when using legacy recordings, providing instructions or other ways to for the speaker to produce variations of his / her vocal characteristics is possible. Rather, varying vocal characteristics in the legacy recording can be accomplished only by (i) relying on metadata inserted into the training dataset, and / or (ii) varying the vocal characteristics during the iterative adjustment phase, as described above. A potential upside to using the foregoing techniques to vary the vocal characteristics of a legacy recording is that these techniques, when applied to one voice recording, may also provide the benefit of helping to improve the generative AI model for other voice recordings on an ongoing basis.

[0060] Changing miscellaneous aspects of original voice recordings – various examples include (i) changing the apparent age of the speaker in the voice recording, for example, by modifying the vocal characteristics to make the synthetic voice deeper and gravelly for older voices or higher for younger voice; (ii) aiding a SLP (speech language pathologist) to improve the voicing of a patients suffering from various speech disorders such as stuttering or lisping such that the impaired speaker can listen to his / her voice (generated synthetically) speaking without the speech disorder; (iii) altering a voicing spoken with a certain accent (e.g., deep south) to a voicing without that accent or with a different accent (e.g., Long Island or Italian);(iv) altering the pronunciation of certain words arising from regional disparities (e.g.,“wash” versus “warsh” or “watter” versus “wadda”); (v) altering an original voicing in one language or dialect to a different language or dialect (e.g., English-to-French, or Spanish-to-Castilian); (v) altering a speaker’s voice for one character to another character (e.g., in an animated TV show); (vi) altering a speaker’s voice to adopt an archetypical voice (e.g., generate a synthetic version of a speaker’s voice to adopt the vernacular of a pirate, “arr, matey”; (vii) altering a speaker’s pronunciation of wordsAttorney Docket No.48501-0007WO1 based on age (e.g., change a speaker’s pronunciation of words in his / her adult voice to his / her pronunciation (potentially incorrect) of words in the speaker’s child voice; (viii) cleaning up legacy audio recordings, e.g., to eliminate background noise, hisses, crackling, etc.; (ix) removing unwanted fillers (e.g., um, er, you know, like) in a voice recording; (x) converting an adult book (e.g., a sophisticated history book) into a kids’ version (simplified to the level of a child’s understanding) of that same book; (xi) altering a voice recording to insert or otherwise implement a voicing technique such as inserting a pause corresponding to the page turns of a book being voiced; (xii) capture an elderly person’s voice to later generate narration for text yet to be generated and / or a child yet to be born, and after the elderly person has lost the capacity for intelligible speed; (xiii) obtain an audiovisual recording of an elderly person, and use the captured voicing and imagery to generate a new audiovisual segment of the elderly person having altered synthetic voicing and altered video corresponding to a new text, and potentially displaying a picture-in-picture of the synthesized elderly person reading the text of a new video book; (xiv) assisted reading improvement, e.g., creating a synthetic audiovisual segment of a speaker having difficulty pronouncing unfamiliar words (and / or voicing a phrase or syllable) so that the speaker can watch his / her eyes and facial expressions and / or suggest other appropriate learning materials (e.g., other video books) for the speaker to experience in the hope that his / her reading will improve; (xv) further to (xiv), use the techniques described herein to track where a child has trouble with a word or phrase or syllable and suggest other titles that may help overcome speech disorders, or suggest specific training material and / or suggest titles as the appropriate reading level; (xvi) also further to (xiv), determine the best speed at which to present a video book to enhance reading accuracy, produce a report for parents and / or teachers on which reading issuesAttorney Docket No.48501-0007WO1 should be the subject of ongoing remediation, and / or determine which titles the child is enjoying and / or reading well, and suggest other titles that may have a similar effect; (xvii) AI-assisted selection of titles to suggest to enhance reading ability and / or enjoyment, e.g., based on end-user-indicated “likes,” frequency of play, potentially coupled with an analysis of which titles the child is enjoying to determine which future books would make sense to convert to a video book

[0061] FIG.4 shows an example of a computing device 400 and an example of a mobile computing device that can be used to implement the techniques described here. The computing device 400 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the inventions described and / or claimed in this document.

[0062] The computing device 400 includes a processor 402, a memory 404, a storage device 406, a high-speed interface 408 connecting to the memory 404 and multiple high-speed expansion ports 410, and a low-speed interface 412 connecting to a low-speed expansion port 414 and the storage device 406. Each of the processor 402, the memory 404, the storage device 406, the high-speed interface 408, the high- speed expansion ports 410, and the low-speed interface 412, are interconnected using various busses, and can be mounted on a common motherboard or in other manners as appropriate. The processor 402 can process instructions for execution within the computing device 400, including instructions stored in the memory 404 or on theAttorney Docket No.48501-0007WO1 storage device 406 to display graphical information for a GUI on an external input / output device, such as a display 416 coupled to the high-speed interface 408. In other implementations, multiple processors and / or multiple buses can be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices can be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi- processor system).

[0063] The memory 404 stores information within the computing device 400. In some implementations, the memory 404 is a volatile memory unit or units. In some implementations, the memory 404 is a non-volatile memory unit or units. The memory 404 can also be another form of computer-readable medium, such as a magnetic or optical disk.

[0064] The storage device 406 is capable of providing mass storage for the computing device 400. In some implementations, the storage device 406 can be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. A computer program product can be tangibly embodied in an information carrier. The computer program product can also contain instructions that, when executed, perform one or more methods, such as those described above. The computer program product can also be tangibly embodied in a computer- or machine-readable medium, such as the memory 404, the storage device 406, or memory on the processor 402.

[0065] The high-speed interface 408 manages bandwidth-intensive operations for the computing device 400, while the low-speed interface 412 manages lowerAttorney Docket No.48501-0007WO1 bandwidth-intensive operations. Such allocation of functions is exemplary only. In some implementations, the high-speed interface 408 is coupled to the memory 404, the display 416 (e.g., through a graphics processor or accelerator), and to the high- speed expansion ports 410, which can accept various expansion cards (not shown). In the implementation, the low-speed interface 412 is coupled to the storage device 406 and the low-speed expansion port 414. The low-speed expansion port 414, which can include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) can be coupled to one or more input / output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

[0066] The computing device 400 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a standard server 420, or multiple times in a group of such servers. In addition, it can be implemented in a personal computer such as a laptop computer 422. It can also be implemented as part of a rack server system 424. Alternatively, components from the computing device 400 can be combined with other components in a mobile device (not shown), such as a mobile computing device 450. Each of such devices can contain one or more of the computing device 400 and the mobile computing device 450, and an entire system can be made up of multiple computing devices communicating with each other.

[0067] The mobile computing device 450 includes a processor 452, a memory 464, an input / output device such as a display 454, a communication interface 466, and a transceiver 468, among other components. The mobile computing device 450 can also be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 452, the memory 464, the displayAttorney Docket No.48501-0007WO1 454, the communication interface 466, and the transceiver 468, are interconnected using various buses, and several of the components can be mounted on a common motherboard or in other manners as appropriate.

[0068] The processor 452 can execute instructions within the mobile computing device 450, including instructions stored in the memory 464. The processor 452 can be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 452 can provide, for example, for coordination of the other components of the mobile computing device 450, such as control of user interfaces, applications run by the mobile computing device 450, and wireless communication by the mobile computing device 450.

[0069] The processor 452 can communicate with a user through a control interface 458 and a display interface 456 coupled to the display 454. The display 454 can be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 456 can comprise appropriate circuitry for driving the display 454 to present graphical and other information to a user. The control interface 458 can receive commands from a user and convert them for submission to the processor 452. In addition, an external interface 462 can provide communication with the processor 452, so as to enable near area communication of the mobile computing device 450 with other devices. The external interface 462 can provide, for example, for wired communication in some implementations, or for wireless communication in other implementations, and multiple interfaces can also be used.

[0070] The memory 464 stores information within the mobile computing device 450. The memory 464 can be implemented as one or more of a computer- readable medium or media, a volatile memory unit or units, or a non-volatile memoryAttorney Docket No.48501-0007WO1 unit or units. An expansion memory 474 can also be provided and connected to the mobile computing device 450 through an expansion interface 472, which can include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 474 can provide extra storage space for the mobile computing device 450, or can also store applications or other information for the mobile computing device 450. Specifically, the expansion memory 474 can include instructions to carry out or supplement the processes described above, and can include secure information also. Thus, for example, the expansion memory 474 can be provide as a security module for the mobile computing device 450, and can be programmed with instructions that permit secure use of the mobile computing device 450. In addition, secure applications can be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

[0071] The memory can include, for example, flash memory and / or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, a computer program product is tangibly embodied in an information carrier. The computer program product contains instructions that, when executed, perform one or more methods, such as those described above. The computer program product can be a computer- or machine-readable medium, such as the memory 464, the expansion memory 474, or memory on the processor 452. In some implementations, the computer program product can be received in a propagated signal, for example, over the transceiver 468 or the external interface 462.

[0072] The mobile computing device 450 can communicate wirelessly through the communication interface 466, which can include digital signal processing circuitry where necessary. The communication interface 466 can provide forAttorney Docket No.48501-0007WO1 communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication can occur, for example, through the transceiver 468 using a radio- frequency. In addition, short-range communication can occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 470 can provide additional navigation- and location-related wireless data to the mobile computing device 450, which can be used as appropriate by applications running on the mobile computing device 450.

[0073] The mobile computing device 450 can also communicate audibly using an audio codec 460, which can receive spoken information from a user and convert it to usable digital information. The audio codec 460 can likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 450. Such sound can include sound from voice telephone calls, can include recorded sound (e.g., voice messages, music files, etc.) and can also include sound generated by applications operating on the mobile computing device 450.

[0074] The mobile computing device 450 can be implemented in a number of different forms, as shown in the figure. For example, it can be implemented as a cellular telephone 480. It can also be implemented as part of a smart-phone 482, personal digital assistant, or other similar mobile device.

[0075] Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designedAttorney Docket No.48501-0007WO1 ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0076] These computer programs (also known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms machine- readable medium and computer-readable medium refer to any computer program product, apparatus and / or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine- readable signal refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0077] To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visualAttorney Docket No.48501-0007WO1 feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0078] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0079] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

[0080] While this specification contains many specific implementation details, these should not be construed as limitations on the scope of the disclosed technology or of what may be claimed, but rather as descriptions of features that may be specific to particular embodiments of particular disclosed technologies. Certain features that are described in this specification in the context of separate embodiments can also be implemented in combination in a single embodiment in part or in whole. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.Attorney Docket No.48501-0007WO1 Moreover, although features may be described herein as acting in certain combinations and / or initially claimed as such, one or more features from a claimed combination can in some cases be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination. Similarly, while operations may be described in a particular order, this should not be understood as requiring that such operations be performed in the particular order or in sequential order, or that all operations be performed, to achieve desirable results. Particular embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims.

Claims

Attorney Docket No.48501-0007WO1 WHAT IS CLAIMED IS:

1. A system for synthetic voice generation, the system comprising: a computer system comprising at least one processor and at least one memory storing executable machine instructions; wherein the computer system is configured to execute machine instructions to perform operations comprising: receive one or more recorded voice samples, each voice sample corresponding to a different human speaker voicing a training text with one or more voice characteristics different from voice characteristics of another human speaker; generate a structured training dataset based on the received recorded voice samples; train an artificial intelligence (AI) synthetic voice generation model using the generated structured training dataset; provide input corresponding to a target text to a deployed generative AI process to generate a narration track; and combine the synthetic narration track with a predetermined target video segment to generate a narrated audiovisual segment.

2. The system of claim 1 further comprising presenting the synthetically narrated audiovisual segment to an end-user.

3. The system of claim 1 wherein one or more of the human speakers is one or more of a narrator, a singer, a poet, a lecturer, or a family member.Attorney Docket No.48501-0007WO14. The system of claim 1 wherein the first training text comprises one or more ofprose, poetry, singing, rapping, or lecturing.

5. The system of claim 1 wherein generating the structured training datasetcomprises: processing the recorded voice samples to determine a median training dataset of the different vocal characteristics; and using the median training dataset in training the AI synthetic voice generation model.

6. The system of claim 1 further comprising, prior to the combining, iterativelyadjusting one or more of the vocal characteristics of the generated narration track.

7. The system of claim 6 wherein the iterative adjustment is automated, or usinghuman input, or both.

8. The system of claim 1 wherein the vocal characteristics include one or more ofthe following: volume, pitch, tone, register, pace, rate, inflection, breathiness, rhythm, fluency, emphasis, articulation, pronunciation, enunciation, intensity, range, or cadence.

9. The system of claim 1 wherein the recorded voice samples are purpose-generated or based on one or more legacy voice recordings, or both.Attorney Docket No.48501-0007WO1 10. The system of claim 1 wherein the recorded voice samples are generated based at least in part on voicing directions provided to the one or more human speakers.

11. The system of claim 1 wherein the generated structured training dataset includes metadata to provide contextual guidance as to how AI model should process the structured training dataset.

12. The system of claim 11 wherein the metadata comprises tags inserted at desired points within the structured training dataset.

13. The system of claim 1 wherein the system is used to create a personalized video book in which the synthetically generated narration includes words or phrases that are unique to a specific end-user.

14. The system of claim 1 wherein the system is used to create a video book in which the synthetically generated narration includes words or phrases that sound like a deceased person’s voice.

15. The system of claim 1 wherein the system is used to create a video book in which the synthetically generated narration includes words or phrases that sound like a non-verbal person’s voice.

16. The system of claim 1 wherein the system is used to create a video book in which the synthetically generated narration includes words or phrases used for instructing children.Attorney Docket No.48501-0007WO1 17. The system of claim 1 wherein the system is used to create a video book in which the synthetically generated narration includes words or phrases that sound different from the voice(s) of the one or more human speakers.

18. The system of claim 17 wherein the words or phrases that sound different have been converted to be in a different language or dialect.

19. The system of claim 17 wherein the words or phrases that sound different have been converted to use a different accent.

20. The system of claim 17 wherein the words or phrases that sound different have been converted to a voice that sounds older or younger than the original one or more recorded voice samples.

21. The system of claim 17 wherein the words or phrases that sound different have been converted to a voice that sounds older or younger than the original one or more recorded voice samples.

22. The system of claim 1 wherein the original one or more recorded voice samples are modified to remove defects arising from ambient noise.

23. The system of claim 1 wherein the original one or more recorded voice samples are modified to remove speech fillers.Attorney Docket No.48501-0007WO1 24. The system of claim 1 wherein the system is used to capture an elderly person’s voice to later generate narration for text yet to be generated and / or a child yet to be born.

25. The system of claim 1 wherein the system is used for assisted reading improvement.

26. A computer-implemented method for synthetically generating a voice track, comprising: receiving one or more recorded voice samples, each voice sample corresponding to a different human speaker voicing a first training text with one or more voice characteristics different from voice characteristics of another human speaker; generating a structured training dataset based on the received recorded voice samples; training an artificial intelligence (AI) synthetic voice generation model using the generated structured training dataset; providing input corresponding to a target text to a deployed generative AI process to generate a narration track; and combining the synthetic narration track with a predetermined target video segment to generate a narrated audiovisual segment.

27. The method of claim 26 further comprising any of claims 2-25.

Images