Improved method of using multiple large language models to create an animated conversation

Abstract

An input may be accepted by the computer application. If the input matches if the input matches a stored pre-spoken phrase the system and method may then determine if the phrase has an inflection. If the determination is that the phrase has an inflection similar to a phrase and inflection stored in a memory, then the stored phrase and inflection may be spoken as required by the text. Instead of having to attempt to pronounce every phrase and not include any inflection, the stored phrase may save processing power as already being stored as a voice file. Similarly, the spoken phrase will have the appropriate inflection.

Description

IMPROVED METHOD OF USING MULTIPLE LARGE LANGUAGE MODELS TO CREATE AN ANIMATED CONVERSATIONCROSS REFERENCE TO RELATED APPLICATIONS

[0001] This application claims the benefit of U.S. Provisional Application No. 63 / 538,693, entitled “IMPROVED METHOD OF USING MULTIPLE LARGE LANGUAGE MODELS TO CREATE AN ANIMATED CONVERSATION”, filed on September 15, 2023, reference of which is hereby incorporated herein in its entirety.BACKGROUND

[0002] Creating speech or conversations using a large language model can be computationally intensive and may cause a lag in responsiveness of the system making the speech sound choppy. Computing power has increased but there still is a noticeable lag when the input is analyzed by a large language model and an appropriate response is formulated. As a result, animation presenting the conversation can look disjointed and may not flow smoothly.SUMMARY OF THE INVENTION

[0003] An improved method and system of computer-generated animated speech may be described. A speech request may be received. The speech request may be analyzed to determine a prepared response that correlates to the speech request. The system and method may respond with the prepared response that correlates to the speech request. The system and method may create a discussion for a first character and a second character using a trained large language model where a first character uses a first large language model and a second character uses a second large language model. A speech animation process may be used to animate the speech in the discussion for the first character and the second character. The animation of the first and second character may be displayed delivering the appropriate section of the speech for each of the first and the second character.BRIEF DESCRIPTION OF THE DRAWINGS

[0004] Fig. 1 may illustrate a sample machine learning illustration;

[0005] Fig. 2 may illustrate a method in accordance with the claims;

[0006] Fig. 3 may illustrate a method in accordance with the claims;

[0007] Fig. 4 may illustrate a sample computing system.

[0008] Persons of ordinary skill in the art will appreciate that elements in the figures may be illustrated for simplicity and clarity so not all connections and options have been shown toavoid obscuring the inventive aspects. For example, common but well-understood elements that are useful or necessary in a commercially feasible embodiment are not often depicted in order to facilitate a less obstructed view of these various embodiments of the present disclosure. It may be further appreciated that certain actions and / or steps may be described or depicted in a particular order of occurrence while those skilled in the art will understand that such specificity with respect to sequence is not actually required. It may also be understood that the terms and expressions used herein are to be defined with respect to their corresponding respective areas of inquiry and study except where specific meanings have otherwise been set forth herein. All dimensions specified in this disclosure may be by way of example only and are not intended to be limiting. Further, the proportions shown in these Figures may not be necessarily to scale. As may be understood, the actual dimensions and proportions of any system, any device or part of a system or device disclosed in this disclosure may be determined by its intended use.SPECIFICATION

[0009] Creating speech or conversations using a large language model can be computationally intensive and may cause a lag in responsiveness of the system making the speech sound choppy. Computing power has increased but there still is a noticeable lag when the input is analyzed by a large language model and an appropriate response is formulated. As a result, animation presenting the conversations can look disjointed and may not flow smoothly.

[0010] An improved method of computer-generated animated speech may be described. A speech request may be received. The speech request may be analyzed to determine a prepared response that correlates to the speech request. The system and method may respond with the prepared response that correlates to the speech request. The system may create a discussion for a first character and a second character using a trained large language model where a first character uses a first large language model and a second character uses a second large language model. A speech animation process may be used to animate the speech in the discussion for the first character and the second character. The animation of the first and second character may be displayed delivering the appropriate section of the speech for each of the first and the second character.

[0011] Methods and devices that may implement the embodiments of the various features of the invention may now be described with reference to the drawings. The drawings and the associated descriptions may be provided to illustrate embodiments of the invention and not tolimit the scope of the invention. Reference in the specification to “one embodiment” or “an embodiment” may be intended to indicate that a particular feature, structure, or characteristic described in connection with the embodiment may be included in at least an embodiment of the invention. The appearances of the phrase “in one embodiment” or “an embodiment” in various places in the specification may not necessarily be referring to the same embodiment.

[0012] Throughout the drawings, reference numbers may be re-used to indicate correspondence between referenced elements. As used in this disclosure, except where the context requires otherwise, the term “comprise” and variations of the term, such as “comprising”, “comprises” and “comprised” may not be intended to exclude other additives, components, integers or steps.

[0013] In the following description, specific details may be given to provide a thorough understanding of the embodiments. However, it may be understood by one of ordinary skill in the art that the embodiments may be practiced without these specific details. Well-known circuits, structures and techniques may not be shown in detail in order not to obscure the embodiments. For example, circuits may be shown in block diagrams in order not to obscure the embodiments in unnecessary detail.

[0014] Also, it is noted that the embodiments may be described as a process that is depicted as a flowchart, a flow diagram, a structure diagram, or a block diagram. The flowcharts and block diagrams in the figures may illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer programs according to various embodiments disclosed. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, that may include one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures.

[0015] Although a flowchart may describe the operations as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. A process may be terminated when its operations are completed. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or the main function. Additionally, each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, may be implemented by special purposehardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0016] Moreover, a storage may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic disk storage mediums, optical storage mediums, flash memory devices and / or other non-transitory machine readable mediums for storing information. The term "machine readable medium" may include, but is not limited to portable or fixed storage devices, optical storage devices, wireless channels and various other non-transitory mediums capable of storing, comprising, containing, executing or carrying instruction(s) and / or data.

[0017] Furthermore, embodiments may be implemented by hardware, software, firmware, middleware, microcode, or a combination thereof. When implemented in software, firmware, middleware or microcode, the program code or code segments to perform the necessary tasks may be stored in a machine-readable medium such as a storage medium or other storage(s). One or more than one processor may perform the necessary tasks in series, distributed, concurrently or in parallel. A code segment may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or a combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or transmitted through a suitable means including memory sharing, message passing, token passing, network transmission, etc. and are also referred to as an interface, where the interface is the point of interaction with software, or computer hardware, or with peripheral devices.

[0018] Referring to Fig. 2, a method of computer-generated animated speech may be illustrated. At block 200, a speech request may be accepted. The purpose of the request may be many and varied. It may be a question such as a question to a chat bot. It may be a request to create an entertaining dialog. The request may be in to form of text, of music or any other form including brain scans. The request may also have a theme. The speech request may be translated into a standard form that may be understood by the system and method. For example, in some embodiments, the request may be for the system and method to create a conversation between a first animated character and additional animated characters. The length of the conversation may also be specified as may be the theme. For example, a user may request a 20 minute humorous conversation of two people on a first day of college.

[0019] At block 205, the system and method may analyze the speech request to determine a prepared response that correlates to the speech request. The purpose of the analysis may be to create speech data that may be compared to stored speech data. The prepared response may be brief speech that fills time while the large language model creates the intended conversation. However, the response may be more than just a time filler. It also may be related to the request. For example, if a user requests a documentary type of discussion of the merits of solar power over other types power, the prepared response may be a serious response that the system and method “needs to carefully consider the response” while the animated character may be illustrated as thinking hard. As another example, the request may be for a comparison of the comedic careers of John Belushi and John Candy and the response may be more humorous such as “that is a laughable request!” while the animated character smiles, laughs and may act out some of the famous bits of the two Johns.

[0020] The responses may be pre-made and may be stored in a memory. As a result, the access to the pre-made speeches may have virtually no time lag and may seem appropriate in response to the request. Further, the illustration of the animated character may also be saved in the memory which will save processor cycles in comparison to creating the animated speech from a text file.

[0021] At block 210, the system and method may respond with the prepared response that correlates to the speech request. The purpose of the response may be to keep the attention of the user while the system and works to prepare a response. The system and method may retrieve a video file to be displayed from a memory. The video file make contain an animated character making the prepare response, including the desired inflection which may be serious or may be humorous or any appropriate inflection. The inflection also may have levels such as extreme humor or somewhat serious and the video file may be appropriate to the received request. There may be a significant number of stored responses. The system and method may learn from requests and may create additional stored responses to be used in the future.

[0022] In addition, the prepared response length may be proportional to the time needed to create the animation and the speech. For example, if the request asks for a two hour movie involving multiple characters having conversations back and forth and doing actions, the large language model may take a longer time to create the conversation that a request for a knock-knock joke between two characters. Thus, the length of the response may need to be longer in the two hour conversation request. The stored response may have indicated lengths.In addition, some stored response may be stitched together to further fill the needed time before the desired conversation is complete.

[0023] The speech request may also have a theme. A theme may be a topic of conversation among the characters. Themes may be very high level such a scenes in the old west or may be more specific such as stories about famous people that have eaten at a specific restaurant during a certain period of time. The theme may be used to assist in selecting the response and may be used in creating the discussion.

[0024] At block 215, trained large language model may be used to create a discussion for a first character and a second character wherein a first character uses a first large language model and a second character uses a second large language model. The purpose of two large language models may be to improve the speed of a response and to create responses that are different from using a single large language model. Large language models may be created by different entities and each may have strengths and weaknesses and each may produce different results from the same inputs.

[0025] A framework for machine learning algorithm like a large language model may involve a combination of one or more components, sometimes three components: (1) representation, (2) evaluation, and (3) optimization components. Representation components refer to computing units that perform steps to represent knowledge in different ways, including but not limited to as one or more decision trees, sets of rules, instances, graphical models, neural networks, support vector machines, model ensembles, and / or others. Evaluation components refer to computing units that perform steps to represent the way hypotheses (e.g., candidate programs) are evaluated, including but not limited to as accuracy, prediction and recall, squared error, likelihood, posterior probability, cost, margin, entropy k-L divergence, and / or others. Optimization components refer to computing units that perform steps that generate candidate programs in different ways, including but not limited to combinatorial optimization, convex optimization, constrained optimization, and / or others. In some embodiments, other components and / or sub-components of the aforementioned components may be present in the system to further enhance and supplement the aforementioned machine learning functionality.

[0026] Machine learning algorithms sometimes rely on unique computing system structures. Machine learning algorithms may leverage neural networks, which are systems that approximate biological neural networks (e.g., the human brain). Such structures, while significantly more complex than conventional computer systems, are beneficial in implementing machine learning. For example, an artificial neural network may be comprisedof a large set of nodes which, like neurons in the brain, may be dynamically configured to effectuate learning and decision-making.

[0027] Machine learning tasks are sometimes broadly categorized as either unsupervised learning or supervised learning. In unsupervised learning, a machine learning algorithm is left to generate any output (e.g., to label as desired) without feedback. The machine learning algorithm may teach itself (e.g., observe past output), but otherwise operates without (or mostly without) feedback from, for example, a human administrator. Meanwhile, in supervised learning, a machine learning algorithm is provided feedback on its output. Feedback may be provided in a variety of ways, including via active learning, semisupervised learning, and / or reinforcement learning. In active learning, a machine learning algorithm is allowed to query answers from an administrator. For example, the machine learning algorithm may make a guess in a face detection algorithm, ask an administrator to identify the photo in the picture, and compare the guess and the administrator's response. In semi-supervised learning, a machine learning algorithm is provided a set of example labels along with unlabeled data. For example, the machine learning algorithm may be provided a data set of 100 photos with labeled human faces and 10,000 random, unlabeled photos. In reinforcement learning, a machine learning algorithm is rewarded for correct labels, allowing it to iteratively observe conditions until rewards are consistently earned. For example, for every face correctly identified, the machine learning algorithm may be given a point and / or a score (e.g., “75% correct”). An embodiment involving supervised machine learning is described herein.

[0028] As elaborated herein, in practice, machine learning systems and their underlying components are tuned by data scientists to perform numerous steps to perfect machine learning systems. The process is sometimes iterative and may entail looping through a series of steps: (1) understanding the domain, prior knowledge, and goals; (2) data integration, selection, cleaning, and pre-processing; (3) learning models; (4) interpreting results; and / or (5) consolidating and deploying discovered knowledge. This may further include conferring with domain experts to refine the goals and make the goals more clear, given the nearly infinite number of variables that can possible be optimized in the machine learning system. Meanwhile, one or more of data integration, selection, cleaning, and / or pre-processing steps can sometimes be the most time consuming because the old adage, “garbage in, garbage out,” also reigns true in machine learning systems.

[0029] By way of example, FIG. 1 illustrates a simplified example of an artificial neural network 100 on which a machine learning algorithm may be executed. FIG. 1 is merely anexample of nonlinear processing using an artificial neural network; other forms of nonlinear processing may be used to implement a machine learning algorithm in accordance with features described herein.

[0030] In FIG. 1, each of input nodes 110 a-n is connected to a first set of processing nodes 120 a-n. Each of the first set of processing nodes 120 a-n is connected to each of a second set of processing nodes 130 a-n. Each of the second set of processing nodes 130 a-n is connected to each of output nodes 140 a-n. Though only two sets of processing nodes are shown, any number of processing nodes may be implemented. Similarly, though only four input nodes, five processing nodes, and two output nodes per set are shown in FIG. 1, any number of nodes may be implemented per set. Data flows in FIG. 1 are depicted from left to right: data may be input into an input node, may flow through one or more processing nodes, and may be output by an output node. Input into the input nodes 110 a-n may originate from an external source 160. Output may be sent to a feedback system 150 and / or to storage 170. The feedback system 150 may send output to the input nodes 110 a-n for successive processing iterations with the same or different input data.

[0031] In one illustrative method using feedback system 150, the system may use machine learning to determine an output. The output may include anomaly scores, heat scores / values, confidence values, and / or classification output. The system may use any machine learning model including xgboosted decision trees, auto-encoders, perceptron, decision trees, support vector machines, regression, and / or a neural network. The neural network may be any type of neural network including a feed forward network, radial basis network, recurrent neural network, long / short term memory, gated recurrent unit, auto encoder, variational autoencoder, convolutional network, residual network, Kohonen network, and / or other type. In one example, the output data in the machine learning system may be represented as multidimensional arrays, an extension of two-dimensional tables (such as matrices) to data with higher dimensionality.

[0032] The neural network may include an input layer, a number of intermediate layers, and an output layer. Each layer may have its own weights. The input layer may be configured to receive as input one or more feature vectors described herein. The intermediate layers may be convolutional layers, pooling layers, dense (fully connected) layers, and / or other types. The input layer may pass inputs to the intermediate layers. In one example, each intermediate layer may process the output from the previous layer and then pass output to the next intermediate layer. The output layer may be configured to output a classification or a real value. In one example, the layers in the neural network may use an activation function suchas a sigmoid function, a Tan h function, a ReLu function, and / or other functions. Moreover, the neural network may include a loss function. A loss function may, in some examples, measure a number of missed positives; alternatively, it may also measure a number of false positives. The loss function may be used to determine error when comparing an output value and a target value. For example, when training the neural network the output of the output layer may be used as a prediction and may be compared with a target value of a training instance to determine an error. The error may be used to update weights in each layer of the neural network.

[0033] In one example, the neural network may include a technique for updating the weights in one or more of the layers based on the error. The neural network may use gradient descent to update weights. Alternatively, the neural network may use an optimizer to update weights in each layer. For example, the optimizer may use various techniques, or combination of techniques, to update weights in each layer. When appropriate, the neural network may include a mechanism to prevent overfitting — regularization (such as LI or L2), dropout, and / or other techniques. The neural network may also increase the amount of training data used to prevent overfitting.

[0034] Once data for machine learning has been created, an optimization process may be used to transform the machine learning model. The optimization process may include (1) training the data to predict an outcome, (2) defining a loss function that serves as an accurate measure to evaluate the machine learning model's performance, (3) minimizing the loss function, such as through a gradient descent algorithm or other algorithms, and / or (4) optimizing a sampling method, such as using a stochastic gradient descent (SGD) method where instead of feeding an entire dataset to the machine learning algorithm for the computation of each step, a subset of data is sampled sequentially. In one example, optimization comprises minimizing the number of false positives to maximize a user's experience. Alternatively, an optimization function may minimize the number of missed positives to optimize minimization of losses from exploits.

[0035] In one example, FIG. 1 depicts nodes that may perform various types of processing, such as discrete computations, computer programs, and / or mathematical functions implemented by a computing device. For example, the input nodes 110 a-n may comprise logical inputs of different data sources, such as one or more data servers. The processing nodes 120 a-n may comprise parallel processes executing on multiple servers in a data center. And, the output nodes 140 a-n may be the logical outputs that ultimately are stored in results data stores, such as the same or different data servers as for the input nodes 110 a-n. Notably,the nodes need not be distinct. For example, two nodes in any two sets may perform the exact same processing. The same node may be repeated for the same or different sets.

[0036] Each of the nodes may be connected to one or more other nodes. The connections may connect the output of a node to the input of another node. A connection may be correlated with a weighting value. For example, one connection may be weighted as more important or significant than another, thereby influencing the degree of further processing as input traverses across the artificial neural network. Such connections may be modified such that the artificial neural network 100 may learn and / or be dynamically reconfigured. Though nodes are depicted as having connections only to successive nodes in FIG. 1, connections may be formed between any nodes. For example, one processing node may be configured to send output to a previous processing node.

[0037] Input received in the input nodes 110 a-n may be processed through processing nodes, such as the first set of processing nodes 120 a-n and the second set of processing nodes 130 a- n. The processing may result in output in output nodes 140 a-n. As depicted by the connections from the first set of processing nodes 120 a-n and the second set of processing nodes 130 a-n, processing may comprise multiple steps or sequences. For example, the first set of processing nodes 120 a-n may be a rough data filter, whereas the second set of processing nodes 130 a-n may be a more detailed data filter.

[0038] The artificial neural network 100 may be configured to effectuate decision-making. As a simplified example for the purposes of explanation, the artificial neural network 100 may be configured to detect faces in photographs. The input nodes 110 a-n may be provided with a digital copy of a photograph. The first set of processing nodes 120 a-n may be each configured to perform specific steps to remove non-facial content, such as large contiguous sections of the color red. The second set of processing nodes 130 a-n may be each configured to look for rough approximations of faces, such as facial shapes and skin tones. Multiple subsequent sets may further refine this processing, each looking for further more specific tasks, with each node performing some form of processing which need not necessarily operate in the furtherance of that task. The artificial neural network 100 may then predict the location on the face. The prediction may be correct or incorrect.

[0039] The feedback system 150 may be configured to determine whether or not the artificial neural network 100 made a correct decision. Feedback may comprise an indication of a correct answer and / or an indication of an incorrect answer and / or a degree of correctness (e.g., a percentage). For example, in the facial recognition example provided above, the feedback system 150 may be configured to determine if the face was correctly identified and,if so, what percentage of the face was correctly identified. The feedback system 150 may already know a correct answer, such that the feedback system may train the artificial neural network 100 by indicating whether it made a correct decision. The feedback system 150 may comprise human input, such as an administrator telling the artificial neural network 100 whether it made a correct decision. The feedback system may provide feedback (e.g., an indication of whether the previous output was correct or incorrect) to the artificial neural network 100 via input nodes 110 a-n or may transmit such information to one or more nodes. The feedback system 150 may additionally or alternatively be coupled to the storage 170 such that output is stored. The feedback system may not have correct answers at all, but instead base feedback on further processing: for example, the feedback system may comprise a system programmed to identify faces, such that the feedback allows the artificial neural network 100 to compare its results to that of a manually programmed system.

[0040] The artificial neural network 100 may be dynamically modified to learn and provide better input. Based on, for example, previous input and output and feedback from the feedback system 150, the artificial neural network 100 may modify itself. For example, processing in nodes may change and / or connections may be weighted differently. Following on the example provided previously, the facial prediction may have been incorrect because the photos provided to the algorithm were tinted in a manner which made all faces look red. As such, the node which excluded sections of photos containing large contiguous sections of the color red could be considered unreliable, and the connections to that node may be weighted significantly less. Additionally or alternatively, the node may be reconfigured to process photos differently. The modifications may be predictions and / or guesses by the artificial neural network 100, such that the artificial neural network 100 may vary its nodes and connections to test hypotheses.

[0041] The artificial neural network 100 need not have a set number of processing nodes or number of sets of processing nodes, but may increase or decrease its complexity. For example, the artificial neural network 100 may determine that one or more processing nodes are unnecessary or should be repurposed, and either discard or reconfigure the processing nodes on that basis. As another example, the artificial neural network 100 may determine that further processing of all or part of the input is required and add additional processing nodes and / or sets of processing nodes on that basis.

[0042] The feedback provided by the feedback system 150 may be mere reinforcement (e.g., providing an indication that output is correct or incorrect, awarding the machine learning algorithm a number of points, or the like) or may be specific (e.g., providing the correctoutput). For example, the machine learning algorithm 100 may be asked to detect faces in photographs. Based on an output, the feedback system 150 may indicate a score (e.g., 75% accuracy, an indication that the guess was accurate, or the like) or a specific response (e.g., specifically identifying where the face was located).

[0043] The artificial neural network 100 may be supported or replaced by other forms of machine learning. For example, one or more of the nodes of artificial neural network 100 may implement a decision tree, associational rule set, logic programming, regression model, cluster analysis mechanisms, Bayesian network, propositional formulae, generative models, and / or other algorithms or forms of decision-making. The artificial neural network 100 may effectuate deep learning.

[0044] A large language model may be a language model characterized by its large size. Their size is enabled by Al accelerators, which are able to process vast amounts of text data, mostly scraped from the Internet. The artificial neural networks which are built can contain from tens of millions and up to billions of weights and are (pre-)trained using self-supervised learning and semi-supervised learning. Transformer architecture contributed to faster training.

[0045] As language models, they work by taking an input text and repeatedly predicting the next token or word. Up to 2020, fine tuning was the only way a model could be adapted to be able to accomplish specific tasks. Larger sized models, such as GPT-3, however, can be prompt-engineered to achieve similar results. They are thought to acquire embodied knowledge about syntax, semantics and "ontology" inherent in human language corpora large language models are trained using self-supervised learning or semi-supervised learning. This means that they are trained on large amounts of unlabeled text, large language models can adjust their internal parameters and learn from new inputs from users over time.

[0046] Large language models are trained to predict the next word in a sentence based on the previous input sentence. This is a self-supervised learning task because you are not defining separate output labels. The process is repeated until the model reaches an acceptable level of accuracy. Some large language models, like InstructGPT and ChatGPT, use both supervised learning and reinforcement learning. The combination of the two is crucial for optimal performance.

[0047] Referring again to block 225, the large language model for each character may be used to create a conversation around the request. The purpose of the conversation may be many. The purpose may be to amuse a user or inform a user or answer questions from a user. For example, a user may request the characters discuss the quarterbacks of the Chicago Bearsthroughout history. The first character may consult the first large language model and may note “that there have been an abnormally large number of starting quarterbacks for the Bears in comparison to other teams.” This quote from the first large language model may be communicated to the second large language model which may respond “you would think with that many quarterbacks the Bears would find a good one every decade or so!” The quote from the second large language model may then be communicated back to the first large language model which then may formulate a response and the discussion may continue.

[0048] In some embodiments, each party in the discussion may have a computer generated personality. Some characters may be humorous and some may be serious while some may be old while others may be young. These personality traits may be taken into account by the various large language model and may be used to formulate appropriate response for each character. Related, the computer generated personality may include computer generated animations of the characters that reflect the personality of the characters. For example, young personalities may be illustrated as being young and older personalities may be illustrated as being older.

[0049] At block 230, a speech animation process may be used to animate the speech in the discussion for the first character and the second character. The purpose of animating the speech may be to make the speech more interesting to a user. Figure 3 may illustrate a method of determining an expression for a character.

[0050] Referring briefly to Fig. 3, at block 300, a large language model may be used to determine a physical expression for sections of the speech. The purpose of the large language model is to analyze the input and make a prediction of the appropriate physical expression. As noted, large language models excel at responding with logical responses to textual inputs. By asking the large language model “what is the emotion or inflection for a character that just said ‘quote’” an appropriate inflection or emotion may be received from the large language model.

[0051] At block 310, the physical expression may be used to determine the animation for the sections of the speech. The purpose of the physical expression is to make the animation even more interesting and appropriate in view of the speech. The received inflection or emotion may be used to illustrate the animated character. The inflection or emotion may have levels. For example, a small level of happiness may be illustrated differently than an extreme level of happiness. In response, an animated character may have a slight smile or may be illustrated dance in the clouds with a huge smile to illustrate extreme happiness.

[0052] At block 320, the animation may be displayed with the determined physical expression. The purpose of the display may be to create a visually interesting display. Again, some characters may have personalities and the personalities may have specific characteristics. As an example of a personality characteristic, Bugs Bunny would often say “What’s up” at moments of conflict. The characters may have their own special phrases that are stated at different moments and in different circumstances which may be dictated by the large language model and the personality. Similarly, the large language model may be used to determine an appropriate animation for the characters given the statement from the character.

[0053] Referring again to Fig. 2, at block 225 the animation of the first and second character may be displayed delivering the appropriate section of the speech for each of the first and the second character. The purpose of the display may be to create a visually interesting display. There may be more characters and the additional characters may be displayed at appropriate time depending on the conversation that has been created by the various large language models. Some of the characters may use the same large language model but at a minimum, two different large language model may be used to ensure there is variety to the conversation and that the conversation is not predictable and repeatable. For example, a large language model may be inclined to talk about Mike Ditka when discussing the Chicago Bears while another model may be inclined to only talk about players that played defense for the Chicago Bears.

[0054] In timing, the conversation may take turns in communicating as described in relation to create the conversation in block 225. It should be noted there may be times when the characters speak over each other intentionally such as when a debate becomes heated or each party is aggressively trying to make a point. Further, the display may be of just one character or may contain many characters all of which are part of the scene.

[0055] The technical problems address by the system and method are many. The technical problem of waiting on a large language model in real time is addressed and results in a smother and more natural conversation between animated characters. In one aspect, by using two large language models, delays are reduced significantly due to using two models rather than a single model and real time conversations between animated characters are significantly smoother. In addition, the system may use pre-saved animated responses while the model is working while again saves processing time and creates a smoother conversation.

[0056] Computing devices are used through the method and system. As shown in Fig. 4, the computing device 401 that executes the method may include a processor 402 that is coupledto an interconnection bus. The processor 402 may include a register set or register space 404, which is depicted in Fig. 4 as being entirely on-chip, but which could alternatively be located entirely or partially off-chip and directly coupled to the processor 402 via dedicated electrical connections and / or via the interconnection bus. The processor 402 may be any suitable processor, processing unit or microprocessor. Although not shown in Fig. 4, the computing device 401 may be a multi-processor device and, thus, may include one or more additional processors that are identical or similar to the processor 402 and that are communicatively coupled to the interconnection bus.

[0057] The processor 402 of Fig. 4 may be coupled to a chipset 406, which includes a memory controller 408 and a peripheral input / output (I / O) controller 410. As is well known, a chipset may typically provide VO and memory management functions as well as a plurality of general purpose and / or special purpose registers, timers, etc. that are accessible or used by one or more processors coupled to the chipset 406. The memory controller 408 may perform functions that enable the processor 402 (or processors if there are multiple processors) to access a system memory 412 and a mass storage memory 414, that may include either or both of an in-memory cache (e.g., a cache within the memory 412) or an on-disk cache (e.g., a cache within the mass storage memory 414).

[0058] The system memory 412 may include any desired type of volatile and / or non-volatile memory such as, for example, static random access memory (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc. The mass storage memory 414 may include any desired type of mass storage device. For example, the computing device 401 may be used to implement a module 416 (e.g., the various modules as herein described). The mass storage memory 414 may include a hard disk drive, an optical drive, a tape storage device, a solid-state memory (e.g., a flash memory, a RAM memory, etc.), a magnetic memory (e.g., a hard drive), or any other memory suitable for mass storage. As used herein, the terms module, block, function, operation, procedure, routine, step, and method refer to tangible computer program logic or tangible computer executable instructions that provide the specified functionality to the computing device 401, the systems and methods described herein. Thus, a module, block, function, operation, procedure, routine, step, and method can be implemented in hardware, firmware, and / or software.

[0059] In one embodiment, program modules and routines may be stored in mass storage memory 414, loaded into system memory 412, and executed by a processor 402 or may be provided from computer program products that are stored in tangible computer-readable storage mediums (e.g. RAM, hard disk, optical / magnetic media, etc.).

[0060] The peripheral I / O controller 410 may perform functions that enable the processor 402 to communicate with a peripheral input / output (I / O) device 424, a network interface 426, a local network transceiver 428, (via the network interface 426) via a peripheral I / O bus. The I / O device 424 may be any desired type of I / O device such as, for example, a keyboard, a display (e.g., a liquid crystal display (LCD), a cathode ray tube (CRT) display, etc.), a navigation device (e.g., a mouse, a trackball, a capacitive touch pad, a joystick, etc.), etc. The I / O device 424 may be used with the module 416, etc., to receive data from the transceiver 428, send the data to the components of the system 100, and perform any operations related to the methods as described herein. The local network transceiver 428 may include support for a Wi-Fi network, Bluetooth, Infrared, cellular, or other wireless data transmission protocols. In other embodiments, one element may simultaneously support each of the various wireless protocols employed by the computing device 401. For example, a software- defined radio may be able to support multiple protocols via downloadable instructions. In operation, the computing device 401 may be able to periodically poll for visible wireless network transmitters (both cellular and local network) on a periodic basis. Such polling may be possible even while normal wireless traffic is being supported on the computing device 401. The network interface 426 may be, for example, an Ethernet device, an asynchronous transfer mode (ATM) device, an 802.11 wireless interface device, a DSL modem, a cable modem, a cellular modem, etc., that enables the system 100 to communicate with another computer system having at least the elements described in relation to the system 100.

[0061] While the memory controller 408 and the I / O controller 410 are depicted in Fig. 4 as separate functional blocks within the chipset 406, the functions performed by these blocks may be integrated within a single integrated circuit or may be implemented using two or more separate integrated circuits. The computing environment 400 may also implement the module 416 on a remote computing device 430. The remote computing device 430 may communicate with the computing device 401 over an Ethernet link 432. In some embodiments, the module 416 may be retrieved by the computing device 401 from a cloud computing server 434 via the Internet 436. When using the cloud computing server 434, the retrieved module 416 may be programmatically linked with the computing device 401. The module 416 may be a collection of various software playgrounds including artificial intelligence software and document creation software or may also be a Java® applet executing within a Java® Virtual Machine (JVM) environment resident in the computing device 401 or the remote computing device 430. The module 416 may also be a “plug-in” adapted to execute in a web-browser located on the computing devices 401 and 430. In someembodiments, the module 416 may communicate with back end components 438 via the Internet 436.

[0062] The system 400 may include but is not limited to any combination of a LAN, a MAN, a WAN, a mobile, a wired or wireless network, a private network, or a virtual private network. Moreover, while only one remote computing device 430 is illustrated in Fig. 6 to simplify and clarify the description, it is understood that any number of client computers may be supported and may be in communication within the system 400.

[0063] Additionally, certain embodiments may be described herein as including logic or a number of components, modules, blocks, or mechanisms. Modules and method blocks may constitute either software modules (e.g., code or instructions embodied on a machine- readable medium or in a transmission signal, wherein the code is executed by a processor) or hardware modules. A hardware module may be a tangible unit capable of performing certain operations and may be configured or arranged in a certain manner. In example embodiments, one or more computer systems (e.g., a standalone, client or server computer system) or one or more hardware modules of a computer system (e.g., a processor or a group of processors) may be configured by software (e.g., an application or application portion) as a hardware module that operates to perform certain operations as described herein.

[0064] In various embodiments, a hardware module may be implemented mechanically or electronically. For example, a hardware module may comprise dedicated circuitry or logic that is permanently configured (e.g., as a special-purpose processor, such as a field programmable gate array (FPGA) or an application-specific integrated circuit (ASIC)) to perform certain operations. A hardware module may also comprise programmable logic or circuitry (e.g., as encompassed within a processor or other programmable processor) that is temporarily configured by software to perform certain operations. It will be appreciated that the decision to implement a hardware module mechanically, in dedicated and permanently configured circuitry, or in temporarily configured circuitry (e.g., configured by software) may be driven by cost and time considerations.

[0065] Accordingly, the term “hardware module” may be understood to encompass a tangible entity, be that an entity that is physically constructed, permanently configured (e.g., hardwired), or temporarily configured (e.g., programmed) to operate in a certain manner or to perform certain operations described herein. As used herein, “hardware-implemented module” may refer to a hardware module. Considering embodiments in which hardware modules are temporarily configured (e.g., programmed), each of the hardware modules need not be configured or instantiated at any one instance in time. For example, where thehardware modules include a processor configured using software, the processor may be configured as respective different hardware modules at different times. Software may accordingly configure a processor, for example, to constitute a particular hardware module at one instance of time and to constitute a different hardware module at a different instance of time.

[0066] Hardware modules may provide information to, and receive information from, other hardware modules. Accordingly, the described hardware modules may be regarded as being communicatively coupled. Where multiple of such hardware modules exist contemporaneously, communications may be achieved through signal transmission (e.g., over appropriate circuits and buses) that connect the hardware modules. In embodiments in which multiple hardware modules are configured or instantiated at different times, communications between such hardware modules may be achieved, for example, through the storage and retrieval of information in memory structures to which the multiple hardware modules have access. For example, one hardware module may perform an operation and store the output of that operation in a memory device to which it is communicatively coupled. A further hardware module may then, at a later time, access the memory device to retrieve and process the stored output. Hardware modules may also initiate communications with input or output devices, and can operate on a resource (e.g., a collection of information).

[0067] The various operations of example methods described herein may be performed, at least partially, by one or more processors that are temporarily configured (e.g., by software) or permanently configured to perform the relevant operations. Whether temporarily or permanently configured, such processors may constitute processor-implemented modules that operate to perform one or more operations or functions. The modules referred to herein may, in some example embodiments, comprise processor-implemented modules.

[0068] The methods or routines described herein may be at least partially processor- implemented . For example, at least some of the operations of a method may be performed by one or processors or processor-implemented hardware modules. The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines . In some example embodiments, the processor or processors may be located in a single location (e.g., within a home environment, an office environment or as a server farm), while in other embodiments the processors may be distributed across a number of locations.

[0069] The one or more processors may also operate to support performance of the relevant operations in a “cloud computing” environment or as a “software as a service” (SaaS). Forexample, at least some of the operations may be performed by a group of computers (as examples of machines including processors), these operations being accessible via a network (e.g., the Internet) and via one or more appropriate interfaces (e.g., application program interfaces (APIs).)

[0070] The performance of certain of the operations may be distributed among the one or more processors, not only residing within a single machine, but deployed across a number of machines. In some example embodiments, the one or more processors or processor- implemented modules may be located in a single geographic location (e.g., within a home environment, an office environment, or a server farm). In other example embodiments, the one or more processors or processor-implemented modules may be distributed across a number of geographic locations.

[0071] Some portions of this specification may be presented in terms of algorithms or symbolic representations of operations on data stored as bits or binary digital signals within a machine memory (e.g., a computer memory) . These algorithms or symbolic representations may be examples of techniques used by those of ordinary skill in the data processing arts to convey the substance of their work to others skilled in the art. As used herein, an “algorithm” may be a self-consi stent sequence of operations or similar processing leading to a desired result. In this context, algorithms and operations may involve physical manipulation of physical quantities. Typically, but not necessarily, such quantities may take the form of electrical, magnetic, or optical signals capable of being stored, accessed, transferred, combined, compared, or otherwise manipulated by a machine. It is convenient at times, principally for reasons of common usage, to refer to such signals using words such as “data,” “content,” “bits,” “values,” “elements,” “symbols,” “characters,” “terms,” “numbers,” “numerals,” or the like. These words, however, may be merely convenient labels and are to be associated with appropriate physical quantities.

[0072] Unless specifically stated otherwise, discussions herein using words such as “processing,” “computing,” “calculating,” “determining,” “presenting,” “displaying,” or the like may refer to actions or processes of a machine (e.g., a computer) that manipulates or transforms data represented as physical (e.g., electronic, magnetic, or optical) quantities within one or more memories (e.g., volatile memory, non-volatile memory, or a combination thereof), registers, or other machine components that receive, store, transmit, or display information.

[0073] As used herein any reference to “embodiments,” “some embodiments” or “an embodiment” or “teaching” may mean that a particular element, feature, structure, orcharacteristic described in connection with the embodiment is included in at least one embodiment. The appearances of the phrase “in some embodiments” or “teachings” in various places in the specification may not necessarily all be referring to the same embodiment.

[0074] Some embodiments may be described using the expression “coupled” and “connected” along with their derivatives. For example, some embodiments may be described using the term “coupled” to indicate that two or more elements are in direct physical or electrical contact. The term “coupled,” however, may also mean that two or more elements are not in direct contact with each other, but yet still co-operate or interact with each other. The embodiments may not be limited in this context.

[0075] Further, the figures depict preferred embodiments for purposes of illustration only. One skilled in the art may be readily recognize from the following discussion that alternative embodiments of the structures and methods illustrated herein may be employed without departing from the principles described herein.

[0076] Upon reading this disclosure, those of skill in the art may appreciate still additional alternative structural and functional designs for the systems and methods described herein through the disclosed principles herein. Thus, while particular embodiments and applications have been illustrated and described, it is to be understood that the disclosed embodiments may not be limited to the precise construction and components disclosed herein. Various modifications, changes and variations, which may be apparent to those skilled in the art, may be made in the arrangement, operation and details of the systems and methods disclosed herein without departing from the spirit and scope defined in any appended claims.

Claims

CLAIMS1. A method of computer generated animated speech comprising: accepting a speech request; analyzing the speech request to determine a prepared response that correlates to the speech request; responding with the prepared response that correlates to the speech request; using trained large language model to create a discussion for a first character and a second character wherein a first character uses a first large language model and a second character uses a second large language model; using a speech animation process to animate the speech in the discussion for the first character and the second character; displaying the animation of the first and second character delivering the appropriate section of the speech for each of the first and the second character.

2. The method of claim 1, further comprising: using a large language model to determine an expression for sections of the speech; using the expression to determine the animation for the sections of the speech; and displaying the animation with the expression.

3. The method of claim 1, wherein the animation and playing the speech are submitted to a large language model to ensure the animation and speech are appropriate for the speech request and the characters.

4. The method of claim 1, wherein the speech request contains a theme.

5. The method of claim 1, wherein each parties has a computer generated personality.

6. The method of claim 5, wherein the computer generated personality comprises computer generated animations.

7. The method of claim 1 wherein the prepared response length is proportional to the time need to create the animation and the speech.

8. The method of claim 7, wherein the prepared response is selected from a plurality of prepared responses.

9. A computer system for generated animated speech comprising: a processor physically configured according to computer executable instructions; a memory physically configured to store computer executable instructions: an input-output circuit; the computer executable instructions comprising instructions for: accepting a speech request; analyzing the speech request to determine a prepared response that correlates to the speech request; responding with the prepared response that correlates to the speech request; using trained large language model to create a discussion for a first character and a second character wherein a first character uses a first large language model and a second character uses a second large language model; using a speech animation process to animate the speech in the discussion for the first character and the second character; displaying the animation of the first and second character delivering the appropriate section of the speech for each of the first and the second character.

10. The computer system of claim 9, further comprising computer instructions for: using a large language model to determine an expression for sections of the speech; using the expression to determine the animation for the sections of the speech; and displaying the animation with the expression.

11. The computer system of claim 9, wherein the animation and playing the speech are submitted to a large language model to ensure the animation and speech are appropriate for the speech request and the characters.

12. The computer system of claim 9, wherein the speech request contains a theme.

13. The computer system of claim 9, wherein each parties has a computer generated personality.

14. The computer system of claim 9, wherein the prepared response length is proportional to the time need to create the animation and the speech.

15. The computer system of claim 14, wherein the prepared response is selected from a plurality of prepared responses.

Images