Character model training engine

Abstract

A computer implemented interaction system for a trained character model comprising a set of character weights and a set of base weights in which the base and character weights are operable to be merge

Description

Field The present invention relates to training large language models. More particularly, the present invention relates to training large language models in a way that allows resource efficient deployment on local computing devices in order to provide realistic interaction with characters such as computer game characters. Background Realistic simulation of characters, such as non-playable characters in computer games, that allows real time and believable interactions with human users has been desired for a long time. Initially, in early computer games, non-playable characters were provided with limited scripts for interaction with human players such that human players could only select from a pre-defined list of interactions with a non-playable character and the non-playable character would then provide a pre-programmed response. The disadvantage to this arrangement is that the human player was aware of the very limited interactions possible with non-playable characters, as there was a limited level of player freedom within these games and limited responsiveness to the player’s actions, but also that the programmers had to hand write code for all possible interactions with non-playable characters. Increased interaction was possible, but this had to be hand coded by the programmers, which took an increased amount of time and human players were still aware of the very limited behaviour possible by non-playable characters. Computer games and other applications started to develop more complex non-playable characters with a basic level of interaction possible with human players, typically using deterministic decision tree systems to make characters appear more responsive or autonomous, but these solutions were still very limited when it came to interactive dialogue until quite recently. Very recently, large language models have become available that allow humans to interact in a conversational way with computer models. The models that are generally available are generalist models, which tend to be very large, state-of-the-art models created using extremely large amounts of data. These generalist models also tend to go through a process termed preference tuning to enable the models to generalise to as many use cases as possible. However, these generalist models are often kept proprietary and closed source and are only accessed by third parties via paid subscription application programming interfaces which allow inference (generation of tokens) on a paid “per token” basis in order to recoup the very significant costs incurred in building the large language model. Some large language models have been made available on an open-source basis with highly permissive licences, which can allow users to use these models as base models and built upon these base models using further training on specific data in a process termed “fine-tuning”. Until recently, fine-tuning models was prohibitive to most users due to the required hardware and running costs. There are still many problems remaining when trying to integrate current large language models into applications such as computer games in order to generate more realistic dialogue with characters modelled by the computer. For example, there is a lack of creative involvement in large language model development, as there is no established way for a creative team to work on an artificial intelligence system due to only big technology companies or research institutions being of a scale to be able to train large language models. These big technology companies and research institutions only make available the inference endpoints to third parties, which only allows third parties to create experiences that are fully dependent on the generalist product and services trained by another entity. The generalist models available are often bland and trained for helpfulness as a general assistant rather than for specific use cases that third parties might want to provide, limiting the possibility of creative implementations giving the models unique traits or behaviours. There are currently no easy ways for small creative teams to create and curate data and then directly train their own model, due to the technical difficulty in doing so and also the resources required to do so. Another example is that current approaches appear over-reliant on training large generalist models that are incredibly resource intensive to train, maintain and host for inference. Further, these generalist models are also limited in terms of the user experience they can offer due particularly to their generalised nature. For example, using these models to provide characters to interact with requires a very large “context length” (i.e. the amount of text in tokens that the model processes / considers / remembers per query) as an input into the model in order to provide a response in keeping with the character being interacted with, because the input each time needs to include details of the role being played by the generalist model along with all of the previous conversation in order to carry on the interaction - this can become a bottleneck if the context length of the model is unable to include all of this information and / or the token cost will be very high for these model responses due to the long context length - and in any case this relies heavily on the reasoning capabilities of the model. Memory requirements scale quadratically as context length grows, and as the actual conversation and history of the conversation must be provided within the context length in order to preserve realistic character interaction, over time a very large amount of compute is consumed during character modelling using a generalist model. Reducing the information provided to reduce the context length typically results in the model not acting as might be intended or unreliably, but this might also occur once the context length increases over a length at which the model can take into account all of the input information / context (which might vary depending on the complexity of the input rather than simply the length). As a result, creativity is restricted when using general models due to the limited control over the model output that a third-party developer has, when using a proprietary closed-source model. Further, the generalist model may be trained in a way that improves safety compliance and to keep the model general, but which conflicts with the generalist model providing realistic character responses. A further problem presented by the use of large infrastructure to provide large language models on a pay per token basis is that the use of large language models for example to model realistic conversations in video games would require ongoing use of the large language model and therefore ongoing purchase of tokens for the use of the large language model, which is not always desired by either or both the video game creators nor the video game players. The video game industry, to a large extent, is modelled on up-front work to create a game and an up-front cost to purchase a game but minimal to zero ongoing cost to either developer or player, which does not integrate well with ongoing token costs to enable realistic character models nor with the possible replacement, alteration or even discontinuation of commercial large language models accessed on a paid-per-token basis. Use of a large language model on a subscription basis would also potentially risk the game being discontinued should player numbers dwindle sufficiently that continuous support and maintenance of the large language model integration becomes uneconomical for the developer, in addition to the upfront costs for the developer to create a game and the expertise needed by the developer to deploy their game via a server infrastructure supporting interaction with the selected large language model via the provided API. Trying to deploy large models locally by using a technique termed “quantisation” also results in problems. Quantisation is the practice of reducing the number of bits of parameters, thus reducing numerical precision. This lowers the quality but also lowers the memory and computation workload as there are fewer bits held in memory and used in computation. Typically, models are trained in 32-bit floating-point data type and then run inference at 16-bit floating-point or 16-bit brain-float data type with negligible degradation in performance. Quantisation can take models to 8-bit, 4-bit or even 2-bit representations, effectively halving the requirements each time. However, it has been found that the performance drop off from quantisation produces diminishing returns. The performance degradation can be counteracted via "quantisation-aware training" on task specific data, however, by relying on generalist models, developers do not have task specific data as they are relying on high level reasoning capabilities from the massive swathes of data from pre-training, but this high-level reasoning ability deteriorates with quantisation for generalist models. Summary of Invention Aspects and / or embodiments seek to provide a more creative artificial intelligence data collection, training and deployment process for substantially improved characters driven by large language models, where the trained large language models can be run locally on consumer-grade hardware rather than via paid-for tokens from a cloud service. According to a first aspect, there is provided a computer-implemented method of interacting with at least one trained character model, comprising: receiving at least one set of character weights, wherein each trained character model is associated with a set of character weights; receiving a base model comprising a set of base weights, wherein the base weights are operable to be merged with the set of character weights; receiving a game state, wherein the game state is operable to select the set of character weights to merge with the base weights; receiving an interaction with the at least one trained character model; processing the interaction using the base model and the merged base weights and selected character weights to generate a character response; outputting a character response. Providing only character weights that can be merged with base model weights can allow multiple different models to be run efficiently using a single base model to provide realistic simulated character interactions, because the information about the character and their intended behaviour is encoded into the character weights (when merged with the base model weights) rather than achieved through statements within the context length provided to the model. Optionally game state can refer to any simulation or interaction state, not only video games. Optionally, the at least one trained character model is hosted locally and the interaction with the at least one trained character model is local. Local deployment of specialised models at scales small enough to deploy to consumer hardware with no need to set up server infrastructure to provide models can allow much more efficient deployment of models, reducing ongoing bandwidth usage that would otherwise be used to communicate with a large model hosted on a remote system or cloud infrastructure, and also can reduce power consumption and compute requirements as locally-run models only use power when in memory and in use, and only at the scale of consumer hardware compute and power requirements, whereas large generalist models running on remote servers or cloud infrastructure need to be ready for immediate response even if user demand is low, with large scale power and compute requirements. Optionally, processing the interaction further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines. Constrained decoding focussed training can allow models to behave effectively within complex finite state machines that can be integrated into products to deliver mechanically rich system functionality. Being able to use a large language model as finite state machines can allow their integration into a program in a meaningful way (beyond a basic chat interface) as a finite set of responses can be integrated into the programme code to change the programme state in a standardised way that works within the game / programme structure. Optionally, the at least one set of character weights and / or the set of base weights comprise low rank adaptors. Optionally, the interaction and / or character response are stored as further training data. Collecting further data from the interaction with deployed models can allow further training data to be collected to improve the locally hosted model. Optionally, on receiving a game state signal to change character, a new set of trained character weights are received and merged with the base weights. Being able to change character upon receiving a game state signal to do so, and to be able to do by changing character weights rather than the entire model (which is typically already loaded into memory and active), can allow much more efficient use of computational resources to produce different character modelling as required. According to a second aspect, there is a provided a computer-implemented method of training at least one character model for interaction with a user, comprising: receiving collected training data, the training data comprising paired prompt and response data for one or more tasks performed by a character; training character weights using the collected training data, the character weights operable for use with a base model to merge with the base weights and cause the base model to replicate the character behaviour demonstrated in the collected training data; outputting the character weights. Optionally, the collected training data comprises data gathered from an interaction between a human respondent and a human character actor. Training using a method for bespoke model development where the data collection system structures the creative process and uses actors as experts to create the training data within the intended experience can allow higher levels of creative input, freedom and possibilities and higher quality outputs through the collection of data pertaining to specific tasks, and the data collected can be a high volume of training data collected at high speed. In contrast, without this bespoke model development approach, a programmer would have to imagine the possible player actions and write responses to those actions and permutations of these, which is a slow process, and which limits the interactivity to what is thought up in advance and risks limiting the interactivity through omitting to think of a possible but likely player action or actions. Training using interactions within a game design mock-up, which allows the game experience to be used as context to collecting the training data, can provide a gameplay experience that can take the form of a performance in its own right, incentivising participation from prospective character actors and respondents alike. The mock-up mirrors or duplicates the final game experience where the player would interact with the model but, instead of the model a trained actor controls that which the Al model would be expected to control, interacting and responding in place of the model. These interactions are then recorded in a structured manner. The small cost and expertise needed to develop a bespoke character model or models, gather and train a group of character actors can democratise the creation of high-quality model training data and thereby model development in general. Optionally, the collected training data further comprises preference training data. Optionally, the method further comprises receiving playtest data and cleaning the playtest data and / or converting the playtest data into preference data. Collecting preference training data and / or playtest data can be simpler than collecting paired data and therefore can allow larger scale data collection and can assist with model refinement / training. Optionally, training character weights further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines. Constrained decoding focussed training can allow models to behave effectively within complex finite state machines that can be integrated into products to deliver mechanically rich system functionality. Optionally, the character weights and / or the set of base weights comprise low rank adaptors. Optionally, the method further comprises cleaning the collected training data. Cleaning the training data can significantly improve the resulting trained model. Optionally, training character weights comprises any or any combination of: parameter efficient fine-tuning; supervised fine-tuning; preference training; proximal policy optimisation; direct preference optimisation; exploratory preference optimisation; preference data based fine-tuning. Optionally, the base model comprises a large language model. According to a third aspect, there is provided a computer-implemented method of collecting data for training at least one character model comprising: inputting one or more interaction mechanics of each model; defining the structure of the input data and the response data; defining the types of input data and associated interaction mechanics; recording one or more demonstrations of one or more tasks in a structured data format operable for use to train weights for a character model. The data collection process can be carried out with minimal model training expertise and no server or infrastructure expertise, including the following training procedure involving both supervised and preference training steps. The data collection process can allow the collection of data at the volume / scale needed to train character model weights. Optionally, the method further comprises outputting the structured data format data. Optionally, the structured data format data is operable to be used for supervised fine-tuning training. Optionally, the method further comprises collecting any or any combination of: play data; preference data. Optionally, the method further comprises cleaning the collected structured data and / or preference data and / or play data. Optionally, playdata is converted to preference data. By converting playdata to preference data, training can be performed on increasing amounts of data, allowing the resulting model / weights to be improved and thus allowing less training data to be collected before testing the initial character models and / or speeding up or decreasing the cost / time involved in collecting initial training data. Brief Description of Drawings Embodiments will now be described, by way of example only and with reference to the accompanying drawings having like-reference numerals, in which: Figure 1 shows a data collection process according to an embodiment; Figure 2 shows a character model training process according to an embodiment; Figure 3 shows a game integration process according to an embodiment; and Figure 4 shows an engine deployment process according to an embodiment. Specific Description Referring to Figure 1, a data collection process 100 according to an embodiment will now be described in detail below. The data collection process 100 requires a respondent 120 who interacts 150 with a server 110 and a character-actor 130 who interacts 150 with the respondent 120 via the server 110. In this embodiment, the interaction is performed in a representation of the final software / game to be published (without character models yet integrated). In other embodiments, a replica of the software / game interactions but not the complete software / game (without character models) is used. The server 110 mimics the situation in which a character needs to be modelled, for example the computer game in which the character will be placed in order to be interacted with by a player of the computer game. To do this, a game state 160 is maintained in the server 110 which can provide context 162 and receive status updates 164 from the interaction 150 between the respondent 120 and character-actor 130. The game state 160 needs to be able to replicate the intended gameplay mechanics of the model(s) (or scenario if not a game) and also what structure of data will be provided to the model and what data is expected in return along with any inputs available to the player other than text input and how this needs to be captured by the character model(s) and presented to the character-actor(s) and / or character model(s). In this embodiment, what the character-actors are shown is the same as what will be inputted to the character model in the final software / game. As the character model will receive the same information as the character-actor does during data collection, the trained model can allow the character model to make the same decision / output as the character-actor. To create the game state 160 requires a significant amount of development work, in this embodiment the world / character planning and writing, game design, art design, music creation, data design, actor training and data collector development needs to be completed in order to collect the data needed for training the character models. The character-actor 130 receives training 140 to allow them to act as the character to be modelled, prior to their interaction 150 with the respondent 120 via the server 110. The character-actor receives context 134 either from the interaction with the respondent 124 or from the game state 160, and provides an intended Al output 132 to the server 110. Characteractors 130 can be actors or writers or skilled people in the role of the character or a mechanic (i.e. a mechanic within the software / game such as whether the answer to a question or puzzle was correct). The training 140 provided to the character-actors 130 must provide the same information that would otherwise be used to train the model, taking into account needing to give the desired range of outputs and / or ability to act. In embodiments, a variety of software / game mechanics can be incorporated into the training process. For an example embodiment of a game where the model is used to simulate the change in behaviour of a populace of a city, such as in a city building simulator, where for example the populace dislike the new tram system and go on strike, data needs to be collected from a character-actor but not response data from playing a specific character but instead data pertaining the mechanic within the system such as whether the populace goes on strike or not. For an example embodiment of an open-ended word puzzle game, the character-actor may need to input whether a player's written solution to an open-ended word puzzle was correct. For an example embodiment of a court / lawyer game, the character-actor will need to input the jury decides whether the defendant is guilty or not. For an example embodiment of a story-based game that has a finite set of endings, the game will have a mechanic that looks at the player's choices and current situation and determine which of the finite endings best fits their choices and the state of their in-game world, which will be trained by the character-actor inputting which finite ending to select. For an example embodiment of a doctor's triage system, the model may be provided with a selection of images and a conversation with the patient, and the character-actor (in this case most likely a medically trained professional) will decide whether to recommend they see a doctor or provide the user with a link to online resources. In embodiments, the OCC (Orthony, Clore and Collins) model for understanding and classifying emotions is used during the training process by the character-actors. This model can be used to train the character model to act more emotionally / realistically. During the training process, the character-actors are trained on how the OCC model works and under what conditions certain emotions are created relative to how their characters represent the world. During data collection, the character actors provide an emotional reaction output to represent what emotions the character felt during each interaction or as each interaction progressed, which was included in the structured data for their response. This allows the character model to be trained to provide an output via constrained decoding including OCC emotional tokens (with intensity) before their full response. An example output is: John T felt: Joy 8 Pride 10 John T pulled a: Joy face John T said: "9999? 50,000! What a workout! Oh, hey there." The interaction 150 is recorded 172 in a structured way and stored as a structured game state dataset 170, which in turn can be used for artificial intelligence model training 180. In embodiments, character responses can be text and / or action responses and / or emotional responses. In this embodiment, human users acting as “play testers” (who can be developers or real players or a mix of human users) function as the respondent 120 to interact with the game or other scenario via the server 110 as they would in the final product / service / game but, instead of acting with character models powered by a large language model, they interact with human character-actors 130 trained in the role they are meant to be playing / modelling. In this embodiment, the character actors 130 learn a task (i.e. how would a character act in situation X) and then demonstrate the task while the task demonstration is recorded 172. In an example embodiment, a graphical user interface is presented to the characteractor along with relevant data such as the in-game date, why they are meeting the player, who invited who to the meeting, where they are meeting in the fictional world. The character-actor would then enter their character’s responses to what the player says, using the graphical interface. Previous conversations are set out in the graphical user interface and the characteractor, when ready to respond, and in some embodiments the character-actor would press the record button before entering each response to demark their time considering their response from the actual response. The game state includes all of the information on previous context and saves new interactions in a structured format (for example: in-game date; in-game location; situation; previous interactions). To the character-actor, when recording the training data, the experience is like playing the game but with the system recording their interactions in order to create content for the training process. For example, in the final product / service, a conversation the character model will see the context 134 for an event and give a response 132, so if the conversation has taken more than a single turn, or the character model has previous interactions which are deemed as relevant (i.e. previous conversations with the same character), then that information must also be available to the model. This information is therefore also presented to the character-actor 130 within the data collection process 100 but typically character-actors 130 can often remember information from previous events as they progress through the data collection process with the player / respondent 120 in a training event / task. As long as the information the actor 130 is recollected from previous interactions during the collection process 100, and is utilised in the formulation of their response 132, was recorded 172 so it can be modelled as relevant in the context of the response 132 then their natural recollection will become training data 170 for the Al model 180. As all interactions are recorded 172, this should always be the case. However, the structuring of this is important as the Al models cannot hold infinite amounts of information and presenting it in a logical manner can help the trained Al models be more performant. A simple example is grouping data by character-actor 130 and respondent 120, by in-game or real-world date / time, by in-game place, or by plotline. These can then be reconstructed into pure-text contextual prompts within the process and data that can be used for training. In this embodiment, the user interface within the data collection process 100 allows all inputs from either the respondent 120 or character-actor 130 to be recorded and for the range of actions and their effect on the game state by either the respondent 120 or character-actor 130 to be recorded (and converted into worded context). For example, discrete inputs (i.e. non-vocal and non-textual inputs) can be turned into action sequences (e.g. up, up, left, left) or drop-down menus and tick boxes (e.g. which emotion is the character feeling right now: joy I distress I reproach I admiration ... etc). These discrete actions are converted to text regular expressions and / or finite states (for the finite state machine) so that the model behaviour can be constrained to act within those same restrictions. These discrete actions will be formatted within the data to match the defined regular expression so that the resulting models have optimised performance and understanding within the restricted state space. By collecting the data 172 in this way, it can be formatted into supervised fine-tuning data for model training. Referring now to Figure 2, a character model training process 200 according to an embodiment will now be described in detail below. Once the structured data 180 has been collected, it is added to the training dataset 210. Optionally, the data 210 can be reviewed and manually cleaned by a human user 205. If further data is required, further actor training 140 can be provided to an actor or producer 295 to provide either further training data using the collection process 100 to add further structured data 180 to the training data 210 or preference data 205 can be added to the training dataset 210. The tool for reviewing and cleaning the data allows human users to format the collected data into prompt / response pairs for use in supervised training and to clean the data. The review / cleaning tool also provides a user interface for the exporting of data into textual prompts and ties the exporting of responses in the prompt / response training samples to the regular expression output formatting defined within the constrained decoding to ensure matching state spaces and ease the optimisation of performance under those inference scenarios. In embodiments, training data that has been collected can be less consistent than desired or can have missing elements due to delays between the player and character-actor responses waiting for each other to respond. To clean this data, the response timing might need to be adjusted to remove unintended delays (where timing is important between responses), or to correct mistakes entered due to the character-actors feeling time pressure to respond to the player promptly to keep the game flowing, or (in embodiments where emotion data is collected) the data might need to be augmented to add emotional data using the OCC model, or the training data might need to be edited or rewritten to increase the comprehensiveness of replies or general quality of the replies to a consistent standard. The training data 210 is exported / split to provide data for supervised training 225 and / or for preference training 235. By using fine-tuning processes for base large language models such as “parameter efficient fine tuning”, consumer-grade hardware can be used to train custom large language models locally. An example of a parameter efficient fine-tuning technique is Low Rank Adaptors (or “LoRA”). In this embodiment, two fine tuning training approaches are used which can be described using the umbrella terms: “supervised fine-tuning” and “preference” training. Supervised fine-tuning training 225 takes demonstration input / output pairs of correct behaviour and uses these to train a base model by generating weights 245 that modify the base weights 230 of a base model 240. Supervised fine-tuning data is relatively costly and time-consuming and can need to be highly cleaned in order to substantially maximise effectiveness, especially with small amounts of supervised fine-tuning data. Preference training 235 takes example inputs (for example from playtesters working with prototype or partially trained character-models) and generates a series of output pairs which are ranked in terms of their adherence to a desired model behaviour. In embodiments, this ranked data is then used in one of a number of preference training regimes such as Proximal Policy Optimisation (PPO), Direct Preference Optimisation (DPO), or exploratory Preference Optimisation (XPO). Preference modelling can be used on a broad spectrum of non-labelled prompts (such as play data from a game) to align performance to the developer’s intended performance. Human users review the outputs and rank the effectiveness of the response according to a set of quality criteria, for example using the cleaning tool (which is suitable for non-technical users). In some embodiments, the initial character weights can be trained on groups of data before being further trained on specific task data. This can improve the training due where using pools of data that share similarities, which can increase performance of the model (especially where data quantity is limited). To create the character-specific weights 245 that cause the model 240 to behave as a character model based on the training data 210, the base adaptor weights 230 are used within a constrained-decoding aware training process to understand the state space on a larger data pool and fine-tuning can then be performed upon the base adaptor using characterspecific data in order to make the character behaviour more consistent with the character. In some embodiments, the character-specific adaptor can be trained without using a base adaptor training process, should there be sufficient data to do so. In some embodiments only one character is required and so only one set of character weights / model need to be trained for one character. Following generation of the character specific weights 245, the character model can be tested within the game / scenario 250 using constrained decoding 255 to ensure correct behaviour within the confines of the game / scenario, also using the regular expression formatting 260 provided from the export 215 of the training data 210. Constrained decoding 255 is an inference method that uses masks on the language modelling head of large language models to reduce the possible range of outputs a model can give. This allows developers to ensure the structure and behaviour from their model’s outputs through “regular expressions” (which define a structured output pattern with high specificity, limiting the model output to only those sequences that match the defined pattern) that equate to “finite state machines” (these define valid states and transitions between states, ensuring the generated output follows a predetermined structure or format, guiding the language model by restricting token choices to only those that lead to valid next states). There are a number of libraries that implement this methodology such as outlines, guidance, and SGLang. Constrained decoding is a technique that manipulates a generative model's token generation process to constrain its next-token predictions by "masking" out (i.e. making impossible) outputs that do not follow a preset structure (i.e. “regex” or defining expected regular expressions) effectively allowing them to operate as neurally-operated finite state machines. This technique is also resource effective as it does not require significant additional computation. Being able to use a large language model as finite state machines is important in order to integrate these models into a program / game in a meaningful way (beyond just basic chat interface) as the software / game needs a finite set of predefined responses in order to trigger changes to the software / game state, rather than a natural language response which will not trigger any change of state other than provide text responses to a player. By using constrained decoding, the character model is able to integrate in an expected way to the game structure, rather than function simply as an encased chatbot. If, following play testing 250, the character model is deemed ready 270 then it can be deployed 280, else if not ready 285 the playtest data 290 is sent to an actor / producer 295 for cleaning or for further preference data to be generated 205. Optionally, in future real playtest data 290 might also be collected from the game that is shipped to users which is used in further training 200 to improve the models for future deployment (for example in a new version of the game or in update to the deployed game). This can provide a method for collecting large quantities of contextual prompts in the desired structure that the model expects, accompanied with noisy model responses. Training directly on these model responses is likely to cause a dramatic performance nosedive, however, so different model versions (i.e. models trained using a different base model or checkpoints taken from different points in the same training run) can be used to generate alternate responses to the exact same prompt. These alternates can then be ranked within the review / cleaning tool 205 resulting in preference data. This preference data can be used in preference training techniques 235 to further improve and widen the robustness (i.e. broaden the diversity of inputs it will perform well in response to) of the model 245. In some embodiments, direct preference training methods such as DPO are easy to use due to their stability and do not require immense amounts of data to see results. However, the ease at which playdata can be collected is especially useful if attempting PPO training which requires a large amount of preference data to train a separate reward model and an even larger amount of unlabelled playdata (prompts-only) used within the fine-tuning of the model itself using the reward model. This means that, if the reward model is reliable and robust enough, the model can be suitably aligned with desired performance across a far larger range of situations and inputs at a scale likely to be infeasible from the manual data collection process 100 alone. Without the specific character response data collection from the manual data collection process 100, a character model can’t be trained or partially trained in order to start collecting playtest data to perform preference training to improve the character model. Referring now to Figure 3, a game integration (i.e. deployment) process 300 according to an embodiment will now be described in detail below. To deploy the model, the deliverable game 305 will have the generated character model(s) 315 integrated into the game 305 as installed on the player device 310. The training process 200 outputs the model via a deployment process 400 to the model 315 integrated in the game 305. In use on the player device 310, when the game 305 is loaded 345, the game state 325 provides the context to any interaction 320 between the player 330 and the character module 315 and receives state updates in order to inform the gameplay based on the game state 325. The player 330 provides input 320 via a game interface 360, which provides the gameplay experience to the player 330. View updates are provided to the game interface 360 by the game state 325. In this embodiment, the game interface 360 is a programmed graphical user interface that creates a visual representation of the game state, which can include text output from the Al model 315. In some embodiments, not all game state objects have a visual representation and not all interaction between the Al model 315 and the game state 325 include the player 330 - for example, more than one Al model can interact with another, or an Al model might interact with a feature of the game. The game interface 360 provides player input to a player parser 370. The player parser 370 can structure the player input as new high-level game state objects, or updates to existing high-level game state objects (when structuring is required). Such player inputs can be text or other inputs. For example, players might input movement within a simulated world but the Al model might need to receive this as images of the movement or as descriptions of the player movement within the world, but the game state 325 needs to record a high-level game state object and can then provide this to the Al model 315 using the game state formatter 365 in order to provide the correctly formatted context to the Al module 315 (which is dependent on how the model has been trained). The game state 325 uses and maintains high level game state objects that are intermediate representations of the game’s state that can be parsed from player / model input and converted into formatted text context for the model 315. In this embodiment, the intermediate representations are the same format as the data collected during training and the formatted game state data is the same format as the data used for training the models. For example, if a player 330 starts interacting 320 with a different character, a state update will be sent to the game state 325 triggering a change of character signal to be sent from the game state 325 to the character module / AI model 315 which changes the character weights to simulate a different character within the game. This can be an efficient way to represent multiple different characters using one large language model module 315 that can be reconfigured to represent different characters by changing the character weights used to those of the character to be modelled, thus reducing the computational resource required to run multiple models simultaneously when no interaction is occurring with all but one (or some) of the models. The Al module 315 is configured to use constrained decoding, as per the model training, in order to output only expected responses. The game state 325 provides high-level game state objects to a game state formatter 365. The game state formatter 365 turns the high-level game state into formatted input that the model 315 has been trained to deal with (via the character weights). In some embodiments, the game state formatter 365 also limits token length of the model input such that it can be consistently capped in length to ensure it can fit within a specific amount of memory. The formatted context is provided from the game state formatter 365 to the model 315. The Al model 315 provides natural language output to an Al parser 355, which also has the same regular expression output that the Al model 315 is constrained to. The Al parser 355 is needed because the model 315 outputs streams of text, and the Al parser 355 is configured to receive the constrained output expected from the model 315 thus can parse the stream of output text into high-level game state objects. The play data is recorded 350 and stored in a structured data set 335, which functions both as a way to save the game state 340 but also allows further training of the character weights using the playdata / diagnostics 290 when sent to the training process 200 to be used as further training data. Referring now to Figure 4, an engine deployment 400 according to an embodiment will now be described in detail below. When training the model in a tool such as PyTorch 405, one or more character low rank adaptors / weights 410 will be generated to work with a base model 420 using constrained decoding 415. For deployment, in this embodiment, a common inference mechanism can be used to allow cross-platform inference capability such as the “open neural network exchange” (ONNX) format 430 for representing machine learning models, which defines a common set of operators and common file format to enable a standard runtime to be deployed to the player’s hardware 450 while maintaining the base model 445 used in training and the trained character LoRA (low rank adaptors) / weights 435 to be used along with the constrained decoding schema 440. This is achieved using a ON NX export process 425 to create a version of the model and its adaptors that are compatible with the runtime. In other embodiment, alternative software or approaches to ON NX can be used. The use of a generalist model 445 which can switch LoRA weights 435 can allow a more efficient way to deploy character models to a local machine without needing a very large generalist model hosted remotely but while allowing the locally deployed model to be trained to substantially provide a realistic set of characters that can be simulated locally. Video game consumers typically have the exact hardware necessary to run a locally deployed model, namely graphical processing units (on a graphics card), so by ensuring the base model is sufficiently capable but works with typical consumer hardware, this can be leveraged to provide substantially realistic character models locally. By providing a base model merged with LoRA weights per character, and switching between LoRA weights to change characters as needed depending on game state, the context windows do not require the briefing for the character nor a copy of all conversations to date in order to preserve a generalist model’s performance in simulating a character, making the locally deployed character model substantially efficient in providing character modelling. Previous conversations still need to be provided to the character model, but this is much less context length compared to having to input all of the briefing to the language model to represent the character as well within the context length. In embodiments, retrieval-augmented-generation techniques can be used to further reduce the context length by storing at least some previous conversation data in a vector database or other data structure, that can be used by the character model when generating responses. By not relying on generalisation and high-performance reasoning capabilities in large language models, and instead using character weights, smaller models become performant enough and require sufficiently low computational performance for local deployment in embodiments. Because the models have been trained to perform under constrained decoding, the models are now effective at working as finite state machines that can be easily parsed and utilised by game state and can be implemented easily into any game engine to create wholly new kinds of video game experiences (or other experiences involving interactions with one or 5 more characters). Any system feature as described herein may also be provided as a method feature, and vice versa. As used herein, means plus function features may be expressed alternatively in terms of their corresponding structure. Any feature in one aspect may be applied to other aspects, in any appropriate 10 combination. In particular, method aspects may be applied to system aspects, and vice versa. Furthermore, any, some and / or all features in one aspect can be applied to any, some and / or all features in any other aspect, in any appropriate combination. It should also be appreciated that particular combinations of the various features described and defined in any aspects can be implemented and / or supplied and / or used is independently.

Claims

1. A computer-implemented method of interacting with at least one trained character model, comprising:receiving at least one set of character weights, wherein each trained character model is associated with a set of character weights;receiving a base model comprising a set of base weights, wherein the base weights are operable to be merged with the set of character weights;receiving a game state, wherein the game state is operable to select the set of character weights to merge with the base weights;receiving an interaction with the at least one trained character model;processing the interaction using the base model and the merged base weights and selected character weights to generate a character response;outputting a character response.

2. The method of claim 1 wherein the at least one trained character model is hosted locally and the interaction with the at least one trained character model is local.

3. The method of any preceding claim wherein processing the interaction further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines.

4. The method of any preceding claim wherein the at least one set of character weights and / or the set of base weights comprise low rank adaptors.

5. The method of any preceding claim wherein the interaction and / or character response are stored as further training data.

6. The method of any preceding claim wherein on receiving a game state signal to change character, a new set of trained character weights are received and merged with the base weights.

7. A computer-implemented method of training at least one character model forinteraction with a user, comprising:receiving collected training data, the training data comprising paired prompt and response data for one or more tasks performed by a character;training character weights using the collected training data, the character weights operable for use with a base model to merge with the base weights and cause the base model to replicate the character behaviour demonstrated in the collected training data;outputting the character weights.

8. The method of claim 7 wherein the collected training data further comprises preference training data.

9. The method of claims 7 or 8 wherein training character weights further comprises using any or any combination of: constrained decoding; regular expressions; finite state machines.

10. The method of any of claims 7 to 9 wherein the character weights and / or the set of base weights comprise low rank adaptors.

11. The method of any of claims 7 to 10 wherein the collected training data comprises data gathered from an interaction between a human respondent and a human character actor.

12. The method of any of claims 7 to 11 further comprising cleaning the collected training data.

13. The method of any of claims 7 to 12 further comprising receiving playtest data and cleaning the playtest data and / or converting the playtest data into preference data.

14. The method of any of claims 7 to 13 wherein training character weights comprises any or any combination of: parameter efficient fine-tuning; supervised fine-tuning; preference training; proximal policy optimisation; direct preference optimisation; exploratory preference optimisation; preference data based fine-tuning.

15. The method of any of claims 7 to 14 wherein the base model comprises a large language model.

16. A computer-implemented method of collecting data for training at least one character model comprising:inputting one or more interaction mechanics of each model;defining the structure of the input data and the response data;defining the types of input data and associated interaction mechanics;recording one or more demonstrations of one or more tasks in a structured data format operable for use to train weights for a character model.

17. The method of claim 16 further comprising outputting the structured data format data.

18. The method of claim 17 wherein the structured data format data is operable to be usedfor supervised fine-tuning training.

19. The method of any of claims 16 to 18 further comprising collecting any or any combination of: play data; preference data.

20. The method of any of claims 16 to 19 further comprising cleaning the collected structured data and / or preference data and / or play data.

21. The method of any of claims 16 to 20 wherein playdata is converted to preference data.

Images