Music data parsing and automated video game level generation

Abstract

Systems and methods are described for music notation level generation that can translate commonly used music notation into information that can be used within a game engine to create music based levels and games; an audio recognition engine capable of detecting individual pitches, chord tones, and multi-note sounds all simultaneously and in real-time, for game interactions, as well as identifying and isolating individual players' audio in order to create a real-time collaborative gaming experience; real-time audio to music notation parsing that can utilize real-time pitch detection and internal timing plus metronome features in the audio engine to hear what a player is playing on their instrument, then translate that to sheet notation and further into configuration files facilitating instant user-created levels in game; and machine-learning based content generation to generate personalized in-game content for players, for instance through the use of large language models.

Description

MUSIC DATA PARSING AND AUTOMATED VIDEO GAME LEVEL GENERATIONCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims benefit of U.S Provisional Patent Application No 63 / 514988, filed July 21, 2023, and titled “MUSIC NOTATION LEVEL GENERATION, AUDIO ENGINE, AUDIO TO NOTATION PARSING, AND RELATED ASPECTS.” The entire disclosure of each of the above items is hereby made part of this specification as if set forth fully herein and incorporated by reference for all purposes, for all that it contains.

[0002] Any and all applications for which a foreign or domestic priority claim is identified in the Application Data Sheet as filed with the present application are hereby incorporated by reference under 37 CFR 1.57 for all purposes and for all that they contain.TECHNICAL FIELD

[0003] Implementations of the present disclosure relate to computer technology, music- and / or audiobased video games, audio detection, and machine learning.BACKGROUND

[0004] The approaches described in this section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Therefore, unless otherwise indicated, it should not be assumed that any of the approaches described in this section qualify as prior art merely by virtue of their inclusion in this section.

[0005] Learning a new instrument can be a challenge for many individuals, and those learning a new instrument can lose interest over time when not engaged by the learning process. Gamifying the instrument learning process can engage and encourage learners, such that the learners may seek out new music to learn overtime or may generate their own new music.SUMMARY

[0006] The systems, methods, and devices described herein each have several aspects, no single one of which is solely responsible for its desirable attributes. Without limiting the scope of this disclosure, several nonlimiting features will now be described briefly.

[0007] Learning a new instrument can be a rewarding endeavor. Gamifying the learning process through music- and / or audio-based video games can induce players to stick it out when learning becomes challenging. Music-based video games can use input from live instruments as the controller. Thus, encouraging players to physically play their instrument to engage with the game, which can make practicing their instrument more fun.

[0008] Described herein are systems and methods that enable the development and execution of music- and / or audio-based video games. Development and execution of a music-based video game can have many components including (1) standardizing musical notation to use as input to generate a video game or video game levels; (2) audio recognition capable of detecting pitches, chords, and multi-note sounds at the same time, as well as identifying and isolating audio from a plurality of individuals; (3) real-time generation of musical notation from playergenerated audio using pitch detection and internal timing features; and (4) machine-learning based content generation to generate personalized in-game content for players.

[0009] Various combinations of the above and below recited features, implementations, and aspects are also disclosed and contemplated by the present disclosure

[0010] Additional implementations of the disclosure are described below in reference to the appended claims and / or clauses, which may serve as an additional summary of the disclosure.

[0011] In various implementations, systems and / or computer systems are disclosed that comprise one or more computer-readable storage mediums comprising program instructions, and one or more processors configured to execute the program instructions to cause the systems and / or computer systems to perform operations comprising one or more aspects of the above- and / or below-described implementations (including one or more aspects of the appended claims and / or clauses).

[0012] In various implementations, computer-implemented methods are disclosed in which, by one or more processors executing program instructions, one or more aspects of the above- and / or below-described implementations (including one or more aspects of the appended claims and / or clauses) are implemented and / or performed.

[0013] In various implementations, one or more computer-readable storage mediums, and / or computer program products comprising one or more computer-readable storage mediums, are disclosed, wherein the computer=readable storage medium(s) comprises program instructions, the program instructions executable by one or more processors to cause the one or more processors to perform operations comprising one or more aspects of the above- and / or below-described implementations (including one or more aspects of the appended claims and / or clauses).BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The following drawings and the associated descriptions are provided to illustrate implementations of the present disclosure and do not limit the scope of the claims Aspects and many of the attendant advantages of this disclosure will become more readily appreciated as the same become better understood by reference to the following detailed description, when taken in conjunction with the accompanying drawings, wherein:

[0015] Figure 1 is a block diagram illustrating an example computing environment, according to various implementations of the present disclosure;

[0016] Figure 2 is a block diagram illustrating an example user device, according to various implementations of the present disclosure;

[0017] Figure 3A illustrates an example schema for generating a configuration file, according to various implementations of the present disclosure;

[0018] Figure 3B illustrates an example configuration file for generating an interactive video game level, according to various implementations of the present disclosure;

[0019] Figure 4 illustrates an example interactive graphical user interface for uploading and / or modifying configuration files, according to various implementations of the present disclosure;

[0020] Figure 5 illustrates an implementation of a “platformer'’ video game level, according to various implementations of the present disclosure;

[0021] Figure 6 illustrates an implementation of an “RPG” video game level, according to various implementations of the present disclosure;

[0022] Figure 7 illustrates an implementation of a “whackanote” video game level, according to various implementations of the present disclosure;

[0023] Figure 8 is a flow chart illustrating an example process for automated music-based video game level generation, according to various implementations of the present disclosure;

[0024] Figure 9 is a block diagram illustrating an example audio system, according to various implementations of the present disclosure;

[0025] Figure 10 is a block diagram illustrating an example implementation of an audio engine, according to various implementations of the present disclosure;

[0026] Figure 11 is a diagram illustrating example functionality of a buffer of the audio engine, according to various implementations of the present disclosure;

[0027] Figure 12 illustrates an example of an audio engine detecting audio input from multiple players, according to various implementations of the present disclosure;

[0028] Figure 13 is a flow chart illustrating an example process for generating input controls based on audio data, according to various implementations of the present disclosure;

[0029] Figure 14 is a flow chart illustrating an example process for generating input controls based on audio data from multiple sources, according to various implementations of the present disclosure;

[0030] Figure 15 is a flow chart illustrating an example process for real-time audio to music notation parsing, according to various implementations of the present disclosure; and

[0031] Figure 16 is a flow chart illustrating an example process for generating level personalization using a language model, according to various implementations of the present disclosure.DETAILED DESCRIPTION

[0032] Although certain preferred implementations and examples are disclosed below, inventive subject matter extends beyond the specifically disclosed implementations to other alternative implementations and / or uses and to modifications and equivalents thereof. Thus, the scope of the claims appended hereto is not limited by any of the particular implementations described below. For example, in any method or process disclosed herein, the acts or operations of the method or process may be performed in any suitable sequence and are not necessarily limited to any particular disclosed sequence. Various operations may be described as multiple discrete operations in turn, in a manner that may be helpful in understanding certain implementations; however, the order of description should not be construed to imply that these operations are order dependent. Additionally, the structures, systems, and / or devices described herein may be embodied as integrated components or as separate components. For purposes of comparing various implementations, certain aspects and advantages of these implementations are described. Not necessarily all such aspects or advantages are achieved by any particular implementation. Thus, for example, various implementations may be carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other aspects or advantages as may also be taught or suggested herein.

[0033] Implementations of the present disclosure relate to computerized systems, methods, and software (generally collectively referred to herein as “the system”) that provide, for example: (1) music notation levelgeneration that can translate commonly used music notation into information that can be used within a game engine to create music based levels and games, including standardizing musical notation to use as input to generate a video game or video game levels; (2) an audio recognition engine that is capable of detecting individual pitches, chord tones, and multi-note sounds all simultaneously and in real-time, for game interactions; (3) a multi-user audio engine that can identify and isolate individual players' audio in order to create a real-time collaborative gaming experience; (4) realtime audio to music notation parsing that can utilize real-time pitch detection and internal timing plus metronome features in the audio engine to hear what a player is playing on their instrument, then translate that to sheet notation and further into Config files facilitating instant user-created levels in game; and (5) machine-learning based content generation to generate personalized in-game content for players, for instance through the use of large language models. Various combinations of the foregoing, and additional and related features are disclosed and / or contemplated.

[0034] To facilitate an understanding of the systems and methods discussed herein, several terms are described below. These terms, as well as other terms used herein, should be construed to include the provided descriptions, the ordinary and customary meanings of the terms, and / or any other implied meaning for the respective terms, wherein such construction is consistent with context of the term. Thus, the descriptions below do not limit the meaning of these terms, but only provide example descriptions.

[0035] The term “model,” as used in the present disclosure, can include any computer-based models of any type and of any level of complexity, such as any type of sequential, functional, or concurrent model. Models can further include various types of computational models, such as, for example, artificial neural networks (“NN”), language models (e.g., large language models (“LLMs”)), artificial intelligence (“Al”) models, machine learning (“ML”) models, multimodal models (e.g., models or combinations of models that can accept inputs of multiple modalities, such as images and text), and / or the like. A “nondeterministic model” as used in the present disclosure, is any model in which the output of the model is not determined solely based on an input to the model. Examples of nondeterministic models include language models such as LLMs, ML models, and the like

[0036] A Language Model is any algorithm, rule, model, and / or other programmatic instructions that can predict the probability of a sequence of words. A language model may, given a starting text string (e g., one or more words), predict the next word in the sequence. A language model may calculate the probability of different word combinations based on the patterns learned during training (based on a set of text data from books, articles, websites, audio files, and the like). A language model may generate many combinations of one or more next words (and / or sentences) that are coherent and contextually relevant. Thus, a language model can be an advanced artificial intelligence algorithm that has been trained to understand, generate, and manipulate language. A language model can be useful for natural language processing, including receiving natural language prompts and providing natural language responses based on the text on which the model is trained. A language model may include an n-gram, exponential, positional, neural network, and / or other type of model.

[0037] A Large Language Model (“LLM”) is any type of language model that has been trained on a larger data set and has a larger number of training parameters compared to a regular language model. An LLM can understand more intricate patterns and generate text that is more coherent and contextually relevant due to its extensive training. Thus, an LLM may perform well on a wide range of topics and tasks. LLMs may work by taking an input text and repeatedly predicting the next word or token (e g., a portion of a word, a combination of one or morewords or portions of words, punctuation, and / or any combination of the foregoing and / or the like). An LLM may be of any type, including a Question Answer (“QA”) LLM that may be optimized for generating answers from a context, a multimodal LLM / model, and / or the like. An LLM (and / or other models of the present disclosure) may include, for example, a NN trained using self-supervised learning and / or semi-supervised learning, a feedforward NN, a recurrent NN, and / or the like. An LLM (and / or other models of the present disclosure) may further include, for example, attentionbased and / or transformer architecture or functionality

[0038] While certain aspects and implementations are discussed herein with reference to use of a language model, LLM, and / or Al, those aspects and implementations may be performed by any other language model, LLM, Al model, generative Al model, generative model, ML model, NN, multimodal model, and / or other algorithmic processes. Similarly, while certain aspects and implementations are discussed herein with reference to use of a ML model, those aspects and implementations may be performed by any other Al model, generative Al model, generative model, NN, multimodal model, and / or other algorithmic processes.

[0039] In various implementations, the LLMs and / or other models (including ML models) of the present disclosure may be locally hosted, cloud managed, accessed via one or more Application Programming Interfaces (“APIs"), and / or any combination of the foregoing and / or the like. Additionally, in various implementations, the LLMs and / or other models (including ML models) of the present disclosure may be implemented in or by electronic hardware such application-specific processors (e.g., application-specific integrated circuits (“ASICs”)), programmable processors (e.g., field programmable gate arrays (“FPGAs”)), application-specific circuitry, and / or the like. Data that may be queried using the systems and methods of the present disclosure may include any type of electronic data, such as text, files, documents, books, manuals, emails, images, audio, video, databases, metadata, positional data (e.g., geocoordinates), geospatial data, sensor data, web pages, time series data, and / or any combination of the foregoing and / or the like. In various implementations, such data may comprise model inputs and / or outputs, model training data, modeled data, and / or the like.

[0040] Examples of models, language models, and / or LLMs that may be used in various implementations of the present disclosure include, for example, Bidirectional Encoder Representations from Transformers (BERT), LaMDA (Language Model for Dialogue Applications), PaLM (Pathways Language Model), PaLM2 (Pathways Language Model 2), Generative Pre-trained Transformed (GPT-2), Generative Pre-trained Transformer3 (GPT-3), Generative Pre-trained Transformer 4 (GPT-4), LLaMA (Large Language Model Meta Al), and BigScience Large Open-science Open-access Multilingual Language Model (BLOOM)System Overview

[0041] Figure 1 is a block diagram illustrating an example computing environment 100. In the illustrated example, computing environment 100 includes one or more user devices 110 (generally referred to as “user device 110), a network 120, and one or more server devices 130 (generally referred to as “server 130”). The user device 110 may be associated with one or more users 101 (generally referred to as “user 101”). The user device 110 and the server 130 may communicate with each other via the network 120. The network 120 may include a variety of method of communication between hardware devices. For example, the network 120 may include, but is not limited to, a local area network, wide area network (e.g., the internet), etc

[0042] The user device 110 may be a device configured to run a game application. For example, the user device 110 may be a mobile device, such as a phone or tablet, a personal computing device, or a video game console. The user device 110 may be configured to receive and respond to input from a user 101.

[0043] The server 130 may be a hardware device configured to execute instructions to provide one or more services (e.g., remote or cloud services) to the user device 110 over the network 120. The services may correspond to execution of a video game or other computing processes, as described herein The server 130 may include one or more databases.Game Application Overview

[0044] Figure 2 is a block diagram illustrating an example user device 110. In the illustrated example, the user device 110 comprises one or more cameras 202 and one or more microphones 204. A camera 202 may be any device capable of capturing an image or video. In some implementations, a camera 202 may be separate from the user device 110 but configured to interface with the user device 110. For example, a camera 202 may be a webcam connected to the user device 110 via a wired connection such as a USB cable, or via wireless connection such as Bluetooth.

[0045] A microphone 204 may be a device capable of capturing audio information. In some implementations, a microphone 204 may be separate from the user device 110 but configured to interface with the user device 110. For example, a microphone 204 may be configured to connect to the user device 110 via a wired connection such as a USB cable or auxiliary cable, or via wireless connection such as Bluetooth.

[0046] Additionally, in the illustrated example, the user device 110 is configured to run an application 210. The application 210 may comprise a parser 220, a fingering rules engine 222 (generally referred to as “rules engine 222”), an audio engine 224, a video game configuration engine 226 (generally referred to as “configuration engine 226”), video game engine 228 (generally referred to as “game engine 228"), a game session 230, and a Machine-Learning (ML) engine 232 (generally referred to therein as “model engine 232”). In some implementations, the application 210 may comprise, more, less, or different components. Additionally, in some implementations, one or more components may be combined or encompassed within other components. For example, in various implementations, the game engine 228 may comprise the parser 220, the rules engine 222, the audio engine 224, and / or the configuration engine 226.

[0047] The parser 220 comprises a software program configured to read an input music data file (e.g , a MusicXML file, or digital sheet music file) and parse out information from music notation (also referred to herein as notation string format) to a standardized notation format (e.g., JSON or “NoteyJSON”). In some implementations the standardized notation format can correspond to the format for a video game configuration. In some cases, the parser 220 may be hosted in a cloud-based capacity (e.g., on the server 130). For example, the parser may be configured to extract key information, such as note length and pitch, song beats per minute (BPM) and Key signature, and other information Instrument and playing related fingering information may also be included

[0048] In some implementations, information parsed from a music data file may comprise beat base, denominator, octave shift, key signature, divisions, bpm, and / or instrument mode. Beat base can correspond to the number of beats as found in the numerator of a time signature. Denominator can correspond to the beat unit as found in the denominator of a time signature. Octave shift can correspond to how many octaves to add to get from writtenpitch to sounding pitch. A key signature can correspond to the number of flats or sharps in a traditional key signature. Divisions can correspond to how many divisions per quarter note are used to indicate a note's duration. Bpm can correspond to a tempo information or beats per minute Instrument mode may be determined based on octave shift. For example, a particular octave shift may indicate that instrument mode should be set to guitar, and a different octave shift may indicate that instrument mode should be set to piano. Additional information added by the parser 220 can comprise a title, description, and video path corresponding to the music data file or a specific game configuration. In some implementations, the configuration engine 226 generates values for the title, description, and video path. Additionally, the parser may add a block note length. The block note length may be a preset length. Additional information can comprise time signature information or tolerance information.

[0049] Finally, the parser can generate a notation string from notes within a music data file. While a music data file may contain information to display sheet music in standard notation, standard notation itself often does not contain enough information to teach an individual how to play the notes. For example, on guitar, a middle C can be played on the fifth string 3rd fret, and also the sixth string Sth fret. Both fingering positions let the player play the exact same pitch, but due to physical limitations of the human hand, certain finger placements are superior to others in specific circumstances. Other situations include notation for a transposing instrument A transposing instrument may be a musical instrument for which music notation is not written at concert pitch (concert pitch is the pitch on a nontransposing instrument, such as the piano).

[0050] All of these situations mean more information may be required to holistically represent music in a game setting. The present system can handle this via the parser 220 generating a space delimited stringbased notation (generally referred to as a “notation string”) that contains the following information about musical notes as a sequence of triples in this pattern (although other similar arrangements / patterns / sequence lengths are contemplated):[PITCH]|[DURATION]|[ADDITIONAL FINGER INFORMATION] e.g. E_3|0.25|4,2,2In some implementations, the pitch can represent both a note and an octave (e.g , “E_3” may represent the note E in the 3rd octave). In some implementations, the duration may correspond to a period of time measured in seconds. Additional Finger notation can include, but is not limited to: fret position, which finger is to be used on the left and right hands, which string to play the note on, what location to place the hand, and any additional notation needed to play that note, such that the game engine 228 can generate a user interface to reasonably explain to a user how to play the note. Additional information for how to play a collection of notes may not be embedded in the music representation, but in other aspects of a game configuration, such as a component that explains accompanying chords or links to informational videos.

[0051] As an additional example, a notation string representing a plurality of notes may comprise the following:“E_4|2.0|1 ,0,0 E_4|2.0|1 ,0,0 B_3|2 0|2,0,0 B_3|2 0|2,0,0”Additionally, a notation string representing a chord may comprise the following:“C:Maj [G_4|.025|1,3,3 E_4|0.25| 1,0,0 C_4|0.25|2,1,1]|0.25”

[0052] In some cases, the parser 220 may act as a backend for a graphical user interface configured to accept or enable user input such as a music data file or other information. For example, the graphical user interface may be an interface (e.g. Website or in-game scene) that allows users to upload a music data file and provide additional information or create traditional sheet music in a custom interface which gets converted to a music data file (e.g , MusicXML) as a middleware before being converted to a game configuration file by the configuration engine 226. For example, the user interface can help a user, such as musicians used to having MusicXMLfiles available, translate from notation string format to a standardized notation format (e.g., video game configuration format) without the knowledge of the code or the underlying file formats.

[0053] The parser 220 may interface with the rules engine 222 when generating a notation string from a music data file. The rules engine 222 may comprise a set of rules for generating a notation string. For example, on a guitar, the same note may be played in more than one place. The rules can provide default fingering positions (also referred to herein as preferred fingerings) for different notes. The rules engine 222 may determine preferred fingerings for a plurality of notes based on relationships among the plurality of notes or additional information from the music data file.

[0054] In some implementations, the rules engine 222 may comprise a set of rules to enable translation from audio detection objects generated by the audio engine 224 to musical notation (e.g., MusicXML) to generate music data files, further discussion of musical detection objects can be found herein with reference to Figure 15. In some implementations, the rules may set forth a standard schema for generating music data files, and standard formatting for depicting notes in a music data file.

[0055] The audio engine 224 may comprise a software program configured to analyze audio input and generate input controls. For example, to control a player character in an interactive video game or interactive video game level generated by the game engine 228, a player can play an instrument, such as a guitar, violin, piano, etc. The audio engine can detect notes or chords and translate the audio into input controls for the game Further description of the audio engine 224 can be found herein with reference to Figure 9.

[0056] The configuration engine 226 comprises a software program capable of parsing standardized notation information generated by the parser 220 and generating a video game configuration (also referred to herein as a “configuration file”). Additionally, the configuration engine 226 may be configured to handle the loading and accessing of data from the configuration file. In some implementations, the configuration engine 226 may be an object within the game engine 228 that parses and safely distributes information to various components inside of the game.

[0057] The configuration file may comprise a passable representation of music from the music data file, as well as information for game objects to retrieve at runtime, for instance what game objects need to read what information In some implementations, the configuration file (e.g., “NoteyJSON”) can contain an ad-hoc number of parameters. For example, each key can represent an object in the game. Additionally, game code that requiresparameterized data can request its own game information via object name Further, scenes in a game may come pre- loaded with prefab-style objects containing code that will use parameters in the configuration file, and at runtime will request the required parameters from the configuration engine 226. In some implementations, the configuration engine 226 is expandable to automatically generate configuration files corresponding to a plurality of video game level types.

[0058] In some implementations, the configuration engine 226 can process a notation string generated by the parser 220 to generate one or more gameplay strings based on one or more game configuration rules. Gameplay strings can correspond to sequences of notes to be played during an interactive video game level. Some level types may utilize a single gameplay string and some level types may utilize a plurality of gameplay strings. For example, one level type can comprise a plurality of graphical platforms, wherein the number of platforms, the appearance of the platforms, and the relative on-screen locations of the platforms can correspond to the notes in a gameplay string (further description of this level type is provided herein with reference to Figure 5). For such a level, only a single gameplay string may be required. In such case, the gameplay string may be identical to the notation string generated by the parser 220. As an alternative example, another level type can comprise a multi-phase battle in which a player must successfully complete three different gameplay strings to win the game (further description of this level type is provided herein with reference to Figure 6). For such a level, since multiple gameplay strings are required, the configuration engine 226 may be required to determine how to split up the notation string into three parts. In some implementations, the configuration engine 226 may split up a notation string based on measures in a piece of music For instance, if the musical piece represented by the music data file has multiple measures, the configuration engine 226 may make each gameplay string a separate measure. In some implementations, the configuration engine 226 may split up a notation string based on chord progressions or some other division of phrases within the music data file. Additionally, the configuration engine 226 may generate checkpoint information based on the notation strings or gameplay strings. For instance, the configuration engine 226 may generate in-game player checkpoints based on chord progression, measures, or other division of phrases within the notation string. For instance, in a level that only uses a single gameplay string, the configuration engine 226 may use techniques similar to that used to split up a notation string into multiple gameplay strings, to generate multiple in-game checkpoints corresponding to different points in the notation string. In some implementations, the game engine 228 may be configured to generate background music for a video game level based on chords in a gameplay string. Further description of the configuration engine 226 and generation of configuration files is disclosed herein with reference to Figures 3A and 3B.

[0059] In some implementations, server 130 may comprise a hosting service (such as Firebase). In some implementations, the configuration engine 226 can connect to the server 130 to host configuration files. In some implementations, the configuration engine 226 may be encompassed within the parser 220 such that the parser 220 may connect with the server 130 to host configuration files.

[0060] In some implementations, the configuration engine 226 handles the loading and accessing of data from the configuration file, ensuring objects only access data they are meant to access For example, all data can, by default, be passed in with a 2 deep dictionary of dictionary of string, string pairs (For example, in C#: Dictionary<string, Dictionary<string, string»), with a dictionary of strings (the inner layer) containing information for 1 isolated game object. In some implementations, requests to the configuration engine 226 can (and, in some implementations, must) include the object’s in-scene ID, so that the configuration engine 226 can determine if theobject has parameters in the configuration file, as well as if the requesting object has the correct requirements to access that data This can allow configuration files to scale to any number of video game scenes, including several different styles of mini games in a single game.

[0061] The game engine 228 comprises a software program configured to manage a video game. In some implementations, the game engine 228 includes, among other things, a Tenderer, simulator, and stream layer. In some implementations, the game engine 228 uses game data (e.g., a configuration file, state data, render data, simulation data, audio data, and other data types of the like) to generate and / or render one or more outputs (e.g., visual output, audio output, and haptic output) for one or more hardware devices. In some implementations, the game engine 228 comprises a physics engine and an event system. The physics engine can manage physics for game objects as well as managing time-based controls. The event system can manage in-game events, such as the movement of game objects, or the timing to display a tutorial window.

[0062] As used herein in some implementations, a Tenderer is a graphics framework that manages the production of graphics corresponding to lighting, shadows, textures, user interfaces, and other effects or game assets of the like. As used herein in some implementations, a simulator refers to a framework that manages simulation aspects corresponding to physics and other corresponding mechanics used in part for animations and / or interactions of gameplay objects, entities, characters, lighting, gasses, and other game assets or effects of the like.

[0063] As used herein in some implementations, a stream layer is a software layer that allows a Tenderer and simulator to execute independently of one another by providing a common execution stream for renderings and simulations to be produced and / or synchronized (i.e., scheduled) at and / or during runtime. For example, a Tenderer and simulator of the game engine 228 may execute at different rates (e.g., ticks, clocks) and have their respective outputs synchronized accordingly by a stream layer

[0064] The game engine 228 may be configured to generate an interactive video game level based on a configuration file. In some implementations, the game engine 228 can generate an interactive video game level at runtime, in real time or substantially real time. For example, the game engine 228 may be configured to generate an interactive video game level based on user input during a game session 230. For instance, the game engine 228 may be configured to generate a screen or menu during a game session 230 that enables a player to select a particular level or level type to play. In some implementations, the game engine 228 may comprise game data that includes a mapping of game levels, such as a sequence of level types. The game engine 228 may generate a user interface to illustrate to the player the sequence or levels and enable user input for selection of levels. For instance, some level types may be targeted towards teaching a particular skill, other level types may be targeted towards practicing a particular skill, and other level types may be targeted towards testing a particular skill. Accordingly, a mapping of game levels may include a 3-level sequence of a teaching level type (e.g., a whackanote level as described with reference to Figure 7), followed by a practicing level type (e.g., a platformer level as described with reference to Figure 5), followed by a testing level type (e.g , an RPG level as described with reference to Figure 6). In some implementations, the sequence can comprise different level types, a different order of level types, or more or less levels of a particular level type. In some implementations, when a player completes one level in the sequence, the game engine 228 may automatically retrieve data regarding the next level in the sequence and retrieve an associated configuration file togenerate the next level In some cases, the player provides input to the game engine 228 to indicate they are ready to play the next level and that the game engine 228 should generate the level.

[0065] In some implementations, the game engine 228 may connect to the server 130 to retrieve a stored configuration file. In some implementations, the game engine 228 may receive a configuration file directly from the configuration engine 226 In some implementations, Music Notation Level Generation can take advantage of notation strings to ensure that a configuration file provides enough information to the game engine 228 to represent and teach how to play all notes present in a configuration file or music data file associated with a game session 230.

[0066] The game session 230 is an instance of one or more virtual interactive environments of the application 210. In some implementations, a virtual interactive environment includes one or more virtual levels and / or graphical user interfaces providing an interactive virtual area or virtual space for gameplay. For example, the game session 230 can be among a game level, which may include one or more player characters, non-player characters, quests, objectives, and other features, elements, or aspects known in the art The virtual interactive environment may have a topography and include one or more objects positioned within the topography that are capable of locomotion within the topography. In some instances, the topography may include a two-dimensional topography. In other instances, the topography may include a three-dimensional topography. In some implementations, the game session 230 is produced and / or maintained in part by the configuration engine 226 and the game engine 228, among other things; such as, for example, the server 130

[0067] As a non-limiting example, a first instance of a game session may be of a first version of a first virtual interactive environment, while a subsequent instance of a game session may be of a subsequent version of the first virtual interactive environment, such that there are one or more changes or differences among the first virtual interactive environment between the two instances of the game session.

[0068] The game session 230 may include a number of player characters and / or non-player characters. Player characters of the game session 230 can refer to controllable character models configured to facilitate or perform gameplay actions or commands. In some implementations, a user or player can control and / or direct one or more player characters in a virtual interactive environment of the game session 230. The term “non-player character” corresponds to character models that are not controlled and / or directed by players (commonly known as “NPCs”). An NPC can be configured with computer executable instructions to perform one or more tasks and / or actions among the gameplay of game session 230 (i.e., gameplay actions); such as with and / or without interaction with or from a player character.

[0069] The game session 230 may include a number of player objects. Player objects of the game session 230 can refer to controllable objects, or models, used to facilitate or enable gameplay or other in-game actions. Player objects may be, for example, vehicles, vessels, aircraft, ships, tiles, cards, dice, pawns, and other in-game items of the like known to those of skill in the art. In some implementations, a user or player can control or direct one or more player objects in the game session 230, including, in some instances, by controlling player characters which in turn causes the objects to be controlled.

[0070] For simplicity, player characters and player objects are collectively referred to herein as player characters in some implementations. It should be understood that, as used herein, “controllable” refers to the characteristic of being able and / or configured to be controlled and / or directed (e.g., moved, modified, etc.) by a playeror user through one or more input means, such as audio input, a controller, or other input device. As known to a person of ordinary skill in the art, player characters include character models configured to receive input.

[0071] In some implementations, the application 210 can include the model engine 232. The model engine 232 can provide various machine-learning related functionality for the application 210. For example, in some implementations, the model engine 232 can provide various LLM-related functionality for the application 210. The model engine 232 may, for example, receive inputs to, and provide outputs from, one or more internal or external LLMs for various LLM functionality of the application 210, as described herein. In various implementations, the model engine 232, and / or one or more LLMs accessible via the model engine 232, may be locally hosted, cloud managed, accessed via one or more APIs, and / or any combination of the foregoing and / or the like.

[0072] In some implementations, the model engine 232 may comprise a model, such as an LLM, to generate notes for a custom interactive game level For example, gameplay data may be collected for a player during a game session 230 by the game engine 228. Gameplay data may comprise information related to how well a player played a set of notes or a particular song (referred to herein as “play results”). For example, the play results may demonstrate that a player consistently struggled to correctly play a particular note or chord, or alternatively, that a player consistently played a particular note or chord correctly For example, the play results may comprise a play score for a song, such that incorrect notes result in a lower play score. The play results may indicate a lower play score for a portion of a song a player struggled with In some implementations, play results may comprise notes played correctly or incorrectly, note played at the wrong time, notes missed, concepts learned, game scores, etc.

[0073] In some implementations, the game engine 228 can generate a prompt for an LLM that comprises a portion of a song with a low play score (e.g., a portion of the gameplay string or corresponding notation string), an example configuration file, and a request to generate a new configuration file based on the portion of the song with the low play score. The game engine 228 may communicate this prompt to the model engine 232, which can accept the prompt as an input for an associated LLM. The model engine 232 may provide the prompt to the LLM and receive the corresponding output, i.e , the generated configuration file. The model engine 232 can then communicate the generated configuration file to the game engine 228 for level generation. In some implementations, the game engine 228 may generate a prompt for an LLM to generate only a portion of a configuration file, such as a new gameplay string or tutorialization information. For instance, the game engine 228 may generate a prompt for a new full-length gameplay string based on the portion of a gameplay sting with the player struggled, without prompting for an entire configuration file. Tutorialization information can include a variety of information or supportive text. For example, tutorial information may comprise information that informs or instructs a player about how to perform actions in-game and external to the game. For example tutorial information can comprise instruction about how to play a particular note such as which string to play, which fret to press, fingering information, or other play instruction. In some implementations, tutorial information can comprise information that informs or instructs a player about how to play a particular level or part of a level. For example, tutorial information can explain to a player that they need to play certain notes at certain times and / or can explain how a player can tell which notes to play or when to play the notes based on information on a rendered graphical user interface. In some implementations, supportive text can comprise text or other icons indicating that a player is successfully completing a level. For example, supportive text may comprise “great job!” or “half way there!” Supportive text may be displayed after certain notes are played or a certain percentage of a gameplay stringis completed. Supportive text may comprise emoticons or other visual icons to demonstrate support to a player. In some implementations, tutorialization information can be displayed at the beginning of a gameplay level, during a gameplay level, or at the end of a gameplay level. For example, tutorialization information at the end of a gameplay level may instruct a player about how to navigate to the next level, or may comprise supportive text regarding the player’s completion of the level, such as “well done!”.

[0074] In some implementations, the model engine 232 may be implemented on the server 130 or may be hosted by a 3rd party service provider. In some implementations, the model engine 232 may be implemented by another component of the user device 110, such as the configuration engine 226 For example, the configuration engine 226 may implement the model engine 232 to generate tutorial screens and text for a particular gameplay string.

[0075] Although illustrated as being implemented on the user device 110, it will be understood that one or more components of the application 210 may be implemented on the server 130, or in another location, such as a third-party hosting service. For example, in some implementations, the parser 220, the rules engine 222, and / or the configuration engine 226 may be implemented on the server 130. For example, the user device 110 may be configured to transmit a music data file to the server 130 for parsing and configuration generation. The server may then transmit a resultant configuration to the user device 110. In some implementations, the server 130 may be configured to receive music data files directly. In some implementations, the parser 220 and rules engine 222 may be implemented on the server 130, while the configuration engine 226 is implemented on the user device 110. For example, the user device 110 may be configured to transmit a music data file to the server 130 for parsing, the parser 220 may generate a standardized notation format file, and the server 130 may transmit the standardized notation format file to the user device 110 for configuration generation. In some implementations, the audio engine 224 may be implemented on the server 130. For example, audio input may be received by the user device 110 and transmitted to the server 130 for processing by the audio engine 224. The processed data may be transmitted back to the user device 110. Additionally, in some implementations, the application 210 may comprise more, less, or different components.Music Notation Level Generation

[0076] The system can translate commonly used music notation (e.g., MusicXML notation) into information that can be used within a game engine (e.g., Unity Engine and / or the like) to create music-based games. Such information can include, but is not limited to: (1) Level Title and Description, (2) Location in a larger story progression, (3) musical pitch information, (4) information on how to play both an instrument and a game, (5) custom information for individual game elements within a Game Engine Scene, and / or combinations of the foregoing and / or the like. A Game Engine can then use this data to generate unique levels at run time, leading to a scalable system that allows for a near infinite number of levels to be created from a single game.

[0077] Figure 3A illustrates an example schema for generating a configuration file Data generated by the parser 220 can be input into the schema to generate a configuration file, such as that discussed with reference to Figure 3B. In some implementations, a particular schema is associated with a particular video game level type. In some implementations, a particular video game level type may be associated with a plurality of schemas. For instance, a particular video game level type may have a tutorial mode and a standard mode such that each mode is associated with a unique schema. In some implementations, a particular video game level types may be associated with a “master schema” as well as “child schemas.” For example, a “master schema" may be a schema used whenever generating aconfiguration file for a particular video game level type, and “child schemas” may be used for variations of the particular video game level type, such as tutorial or standard modes, or different difficulty levels.

[0078] In some implementations, the schema may comprise a plurality of key-value pairs. Some of the values may be preset to ensure consistency across configuration files associated with the same schema or video game level type. Some of the values may correspond to references such that the configuration engine 226 can replace the references with values from an object output by the parser 220.

[0079] In the illustrated example, the schema represents a template for a configuration file for a particular type of interactive video game level. The schema comprises categories of information that the game engine 228 can use to generate an interactive video game level. In the illustrated example, the categories comprise application data 302, Start Game View 308, Pitch Volume Engine 314, BPM Counter 318, and Playlist Manager 320. In some implementations, the schema can comprise more, less, or different categories. Application data 302 comprises information such as the title 304 (the title of the game or game level) and the level type 306. The system can support a plurality of level types, some of which are described in further detail below with reference to Figures 5-7 Start Game View 308 corresponds to an introduction view for the level, for example there may be a screen displayed with the title and a description of the level generated based on the title 304 and the description 310. Additionally, in some implementations, the Start Game View 308 may comprise a video, such as an animation. A reference path to the video may be accessed based on the video path 312. The Pitch Volume Engine 314 can comprise information for the audio engine 224. Such as the instrument mode 316. The BPM Counter 318 can comprise additional information for the audio engine such as bpm, beat base, denominator, and tolerance. The Playlist Manager 320 can comprise information about various gameplay strings. In the illustrated example, the interactive video game level type is associated with a singular gameplay string 322. Some level types can be associated with multiple gameplay strings.

[0080] Since the schema acts as a template, some of the values for the information act as references rather than hard-coded values. For example, title 304 comprises a key “Title” and a value “{{ title }}.” The value for title 304 is a reference to a title attribute. The title attribute may be from an object generated by the parser 220. In some implementations, the configuration engine 226 can receive an object having key-value pairs from the parser 220, such that the keys of the object correspond to the references in the schema. For example, the parser may pass an object with the following structure to the configuration engine 226:“title”: “I am a title”“description": “I am a description”“videoPath: “”“instrumentMode”:“bpm”: 56“beatBase”: 4“denominator”: 3“playlist”: "E_4|2.0| 1,0,0 E_4|2.0| 1,0,0 B_3|2.0|2,0,0The keys in the key-value pairs in the object correspond to the references in the schema. The configuration engine 226 can use the reference value in the schema as a key to extract the associated value from the parser object keyvalue pair. Accordingly, in the configuration file, the resulting key value pair would comprise the value extracted from the parser object.

[0081] Figure 3B illustrates an example configuration file for generating an interactive video game level. In some implementations, a configuration file contains information that the game engine 228 can use to generate a game level. In some implementations, a game level is equivalent to an entire music game or mini game.

[0082] The configuration engine 226 can use a schema, such as that illustrated and discussed with reference to Figure 3A to generate the configuration file. For example, title 304A corresponds to title 304, but the value is no longer a reference, rather it has a value extracted from an object passed to the configuration engine 226, based on the reference in title 304. The same is true for gameplay string 322A which no longer contains a reference like gameplay string 322 but has a gameplay string corresponding to notes parsed from a music data file by the parser 220 as described with reference to figure 2.

[0083] As explained above with reference to Figure 2, the configuration file can comprise a plurality of parameters (key-value pairs). Each key can represent an in-game object. Some game objects are parameterized, meaning that the game engine 228 can retrieve information from the configuration file to determine how to render the parameterized game objects. For example, a particular game object may be a platform and the number of platforms generated may correspond to the number of notes in the gameplay string 322A. The game engine 228 may read the gameplay string 322A to determine how many notes the string comprises and then generate a corresponding number of platforms. Additionally, the game engine 228 may retrieve information from the configuration file to determine a size and relative location in which to render a platform. For example, the game engine 228 may extract a first note from the gameplay string 322A and extract a duration and pitch from the first note. The size of the platform may correspond to the duration of the first note and the relative location of the platform may correspond to the pitch of the first note.

[0084] The illustrated schema and configuration file correspond to a particular video game level type involving a player character moving along a plurality of platform objects. Different schemas and configuration files may be used for different video game level types. For example, video game level types may comprise those described with reference to figures 5-7 (“platformer,” “RPG," and “whackanote,”). Additionally, other video game level types (e.g , puzzle, racing, multi-player battle, storytelling, tower defense, sports, trivia, etc.) may be represented by different schemas and configuration files.

[0085] In some implementations, a configuration file can comprise information for an in-game tutorial, such as that in the tutorial manager category 324. The tutorial manager category 324 can comprise information for providing different tutorial text or screens before certain notes, wherein the key value corresponds to the note. For example, the tutorial manager category 324 comprises a key-value pair with a key of “0”. This indicates that the game engine 228 should generate a tutorial screen before the first note. Additional keys comprising a decimal value are associated with a corresponding integer key. For example, the key-value pairs with keys “0.1” and “03” can provide additional information for the tutorial generated based on the key-value pair with key “0”. Decimal key-value pairs can also correspond to additional tutorial text or screens with decimal values indicating the sequence in which they shouldbe displayed. For instance, a tutorial window generated based on the “0.1” key-value pair would be displayed before a tutorial window generated based on the “0.3" key-value pair.

[0086] Figure 4 illustrates an example interactive graphical user interface for uploading and / or modifying configuration files. In some implementations, the interface enables a user to upload a music data file and generate a corresponding configuration file. In some implementations, the interface enables a user to upload a pregenerated configuration file for updates or modification. In some implementations, the interface enables a user to upload a generated or modified configuration file to the server 130 or other third-party hosting service.

[0087] In the illustrated example, section 402 corresponds to a plurality of inputs for identifying a configuration file or music data file. Input 404 corresponds to a user input for executing computer code to generate one or more configuration files based on the identified configuration file or music data file in section 402. Section 406 corresponds to windows for modifying one or more configuration files. For example, a user may desire to add to or modify tutorial text or the description of a game level. Input 408 corresponds to a user input for executing computer code to upload the one or more configuration files to the server 130 or a third-party hosting service.Example Video Game Level Implementations

[0088] Figure 5 illustrates an implementation of a “platformer” video game level (also referred to herein as a “platformer level”) As illustrated, a platformer level comprises a sequence of a plurality of graphical platforms in which a player character walks along platforms 504 and jumps over gaps between the platforms 504 based on user input.

[0089] The game engine 228 may generate the platforms 504 based on a gameplay string extracted from a configuration file. For example, a sequence of graphical platforms can be used to represent musical notes from the gameplay string. For example, the number of platforms 504 can correspond to the number of notes, the size of individual of the platforms 504 can correspond to the duration of the note associated with the individual platform 504, and relative locations of the platforms 504 can correspond to the pitch of notes. For example, a platform 504 corresponding to an E note may be displayed higher than a platform 504 corresponding to a B note. Additionally, a platform 504 corresponding to a note having a duration of 2 seconds may have a larger size than a platform 504 corresponding to a note having a duration of 1 second. The size of a platform 504 may correspond to height, length, or depth. For example, notes of longer duration may correspond to platforms of a longer length than notes of shorter duration. The platforms 504 may additionally comprise note identifiers 508 at a first end and action indicators 506 at a second end. Note identifiers 508 can identify to a player the note that they must play to cross onto a particular platform 504. Action indicators 506 can indicate to a player the timing of when the player should play a particular note to make the player character jump across gaps to particular platforms 504 or an end zone 516.

[0090] Additionally, as illustrated, a platformer level comprises a graphical representation of musical notes as an instrumental notation 510, such as standard (staff) notation or tablature, in a portion of the rendered display. In some implementations, the instrumental notation 510 may be in a bottom, top, or middle portion the rendered display. In some implementations, the instrumental notation 510 may be displayed in an embedded window. Individual or sequences of graphical representations of musical notes (e.g., icons or note indicators 514) may scroll along the instrumental notation 510. For example, the instrumental notation 510 may comprise a plurality of staffs (or strings) and a notation type indicator (e.g., “TAB”, a treble clef, etc.) displayed at one end of the plurality of staffs, along withan action indicator line 512 proximate the notation type indicator and spanning the staffs. Note indicators 514 may appear at the opposite end of a staff from the notation type indicator and move along the staff over time towards the action indicator line. One or more notes may be displayed on the instrumental notation 510 at a particular time. For example, to indicate a chord, multiple notes may be displayed vertically in line with each other along the staffs.

[0091] Note indicators 514 may indicate information about how to play the note. For example, a note indicator may be located on a particular staff and comprise a particular color and number indicator. For example, for a guitar, a note indicator 514 may appear on the first staff and be a blue circle with the number 0 indicating to strum the E string without pressing any frets.

[0092] As illustrated at T=1 , two platforms are displayed, a first platform 504A and a second platform 504B with a gap in between the platforms. The player character 502 is walking along the first platform 504A towards the second platform 504B. Additionally, the player character 502 is approaching the concluding end of the first platform 504A. An action indicator 506A is located at concluding end of the first platform 504A. In the illustrated implementation, the action indicator 506A looks like a button or disk. Additionally, the action indicator 506A is rendered with light emanating from it to indicate that the player is approaching the action indicator 506A. In some implementations, action indicators may have a different appearance (e.g. , such as a flag, or a coil spring), be smaller or larger, or be a different color. A note identifier 508A is located at the starting end of the second platform 504B, identifying that the player should play an E note to cross the gap between the first platform 504A and the second platform 504B.

[0093] Additionally, at T=1, a note indicator 514A is approaching the action indicator line 512. This indicates to a player that they should play the note corresponding to the note indicator 514A when the note indicator 514A reaches the action indicator line 512 in order to make the player character 502 jump to the second platform 504B.

[0094] As illustrated at T=2, a third platform 504C is displayed along with the first platform 504A and second platform 504B. Between T=1 and T= 2, the player played an E note at the correct time, and the player character 502 has successfully crossed from the first platform 504A to the second platform 504B and is walking towards the third platform 504C. Additionally, the length of the second platform 504B corresponds to the duration of the E note played between T=1 and T=2. Accordingly, the E note played by the player is still being detected by the system, enabling the player character 502 to continue forward along the second platform 504B.

[0095] The third platform 504C has a note identifier 508B at its starting end indicating that a B note must be played to cross the gap from the second platform 504B to the third platform 504C. Consequently, the third platform 504C is rendered lower than the first platform 504A and second platform 504B which were associated with E notes which correspond rather than B notes. This provides a visual representation of the corresponding staffs in guitar tablature

[0096] The second platform 504B has an action indicator 506B at its concluding end indicating when the player should play the B note to cross the gap between platforms. Additionally, in the illustrated example, the player character 502 is not close enough to the concluding end of the second platform 504B to trigger a note indicator on the instrumental notation 510 to indicate to the player how to play the B note to cross the gap between the second platform 504B and the third platform 504C.

[0097] As illustrated at T=3, the second platform 504B and third platform 504C are displayed along with an end zone 516. The player character 502 is approaching the action indicator 506B on the second platform 504B. Theplayer character 502 is close enough to the action indicator 506B that the action indicator 506B is rendered with light emanating from it. Additionally, a note indicator 514B has appeared on the instrumentation notation 510 and is approaching the action indicator line 512. A note identifier 508C is displayed in the end zone 516.

[0098] Figure 6 illustrates an implementation of an "RPG” video game level (also referred to herein as an “RPG level”). As illustrated, an RPG level comprises two game characters, a player character 602 and a battle character 604, a plurality of game character positions represented by player status platforms 606 and battle character status platforms 608, an instrumentation notation 510, an action indicator line 512, and a plurality of note indicators 514 (in the illustrated example, note indicators 514D and 514E

[0099] During an RPG level, a player attempts to survive several phases and defeat a battle character 604. The RPG level in the illustrated example comprises three phases. An RPG level may have more or less phases. A phase of a game can be represented by the player and / or battle character’s position on the player status platforms 606 or battle character status platforms 608, the player character’s position may be based on the player’s success or failure of completing one or more of a plurality of gameplay string. For instance, if a player successfully completes a gameplay string the battle character 604 moves to a battle character status platform 608 that is adjacent to the currently active battle character status platform 608 and which is located closer to the battle edge of the battle ground 616. However, if a player fails to successfully complete a gameplay string, the player character 602 moves to a player status platform 606 that is adjacent to the currently active player status platform 606 and which is located closer to the player edge of the battle ground 616.

[0100] For each phase, the gameplay string that a player attempts to complete can correspond to a gameplay string in the configuration file used to execute the RPG level. The configuration file for an RPG level can contain a plurality of gameplay strings. Each of the plurality of gameplay strings can correspond to a phase of game play In some implementations, each gameplay string may comprise an individual song or other form of musical piece. In some implementations, each gameplay string may comprise a portion of a song or other form of musical piece with the combination of the plurality of gameplay strings comprises a complete song or other form of musical piece.

[0101] Figure 7 illustrates an implementation of a “whackanote” video game level (also referred to herein as a “whackanote level”). As illustrated, a whackanote video game level comprises a player character 702, a plurality of strings 708 corresponding to notes 706, and a plurality of frets 704.

[0102] During a whackanote level, at certain times, a particular string 708 may be highlighted to indicate to a player that the player should strum that string on their instrument. Additionally, at other times, a note indicator 710 may be displayed to indicate to a player how to play a particular note. For example, a note indicator may indicate that the player should place a particular finger on a particular fret, or in between two frets, on a particular string, and strum all strings, or a particular string on their instrument. As described above, different instruments may be taught using the same video game level types. In the illustrated example, the game is set up to teach guitar, but the game could be set up to teach a variety of other instruments such as violin or piano For example, instead of strings and frets, a whackanote level for a piano may have several piano keys displayed to a user that are highlighted over time.

[0103] Over time, different strings 708 may be highlighted or different note indicators 710 may be displayed. In the illustrated example, the instrument being taught is guitar. At T=1 , the “G” string 708A is highlighted and there are no note indicators 710 on the screen. This indicates to a player that the player should strum the “G” stringwithout placing any fingers on any of the frets. At T=2, a note indicator 710 is displayed on the screen on the second fret and on the “G” string. This indicates to a player that the player should place their finger on the “G” string on the second fret and strum the “G” string. The order in which different strings 708 or different note indicators 710 may be highlighted or displayed can be based on one or more gameplay strings in the configuration file for the whackanote level.

[0104] Figures 5-7 illustrate various example implementations of video game levels. In some implementations, the system may be configured to implement other video game level types with various different types of functionalities and play characteristics tied to audio-based user inputs. For example, a video game level may comprise putting a puzzle together such that notes in a gameplay string correspond to pieces in a puzzle such that the player can complete the puzzle by playing the correct notes or playing notes in a certain order. As another example, a video game level may comprise racing around a track such that notes in a gameplay string are represented by icons on the track, and players can play certain notes to move a player character forward. Additional video game level types may comprise multi-player battle, storytelling, tower defense, sports, trivia, etc.Example Process for Music Notation Level Generation

[0105] Figure 8 is a flow chart illustrating an example process 800 for automated music-based video game level generation. Although steps are illustrated in a particular order, steps may be performed multiple times, the order of the steps may be changed, and / or one or more steps may be performed concurrently. Additionally, fewer, more, or different steps may be performed. Further, the process 800, in whole or in part can be implemented by a user device 110, server 130, or other computing system. Although any number of systems, in whole or in part, can implement the process 800, to simplify discussion the process 800 will be described with respect to the user device 110.

[0106] At block 802, during a game session initiated on the user device 110, the parser 220 can receive a music data file. The music data file may be received from a server 130, or a player associated with the user device 110. In some implementations, the game session 230 can provide an input interface, such as that described with reference to Figure 2, to enable a user to upload a music data file to the parser 220. As explained above, a music data file may comprise any file containing parseable music notation, such as a MusicXML file.

[0107] At block 804, the parser 220 can parse the music data file received at block 802 to generate an object passable to the configuration engine 226 comprising information for generating a configuration file, specifically including a notation string. As described with reference to Figure 2, the parser 220 can generate the notation string based on a plurality of music notes in the music data file, such that the notation string comprises a pitch, duration, and any additional finger information for each note. The parser 220 can then pass the generated passable object, which contains the notation string, to the configuration engine 226.

[0108] At block 806, the configuration engine 226 can generate a configuration file based on the object received from the parser 220. As described with reference to Figures 2, 3A, and 3B, the configuration engine 226 can retrieve a schema corresponding to a particular interactive video game level type and generate a configuration file based on the schema and the object received from the parser 220, such that the resulting configuration file comprises a first video game level type and a first gameplay string.

[0109] In some implementations, the configuration engine 226 can generate tutorial information, such as that in the tutorial manager category 324 described with reference to Figure 3B, using the model engine 232. Forexample, the configuration engine 226 can generate a prompt for an LLM comprising a gameplay string, example tutorial information, such as that in the tutorial manager category 324, and a request for a tutorial information object based on the gameplay string. The configuration engine 226 may communicate the prompt to the model engine 232 which may provide the prompt as an input to an LLM. The model engine 232 may then receive generated tutorial information as output from the LLM and communicate that output to the configuration engine 226 for incorporation in a configuration file. In some implementations, the LLM can accept a plurality of gameplay strings as input and then generate and output tutorial information for each of the plurality of gameplay strings. In some implementations, the LLM may be configured or trained to generate output in object format such as that of the tutorial manager category 324.

[0110] In some implementations, the configuration engine 226 can pass the generated configuration file to the game engine 228 for video game level generation. In some implementations, the configuration engine 226 can upload the generated configuration file to the server 130 or some other hosting service for later retrieval by the game engine 228.

[0111] At block 808, during a gameplay session, in real-time or substantially real-time the game engine 228 can generate an interactive video game level based on the configuration file generated at block 806. As described with reference to Figures 2, 3A, and 3 B, the game engine 228 may access the generated configuration file to generate an interactive video game level. The game engine 228 can then generate the interactive video game level based on the accessed configuration file. The game engine 228 may render a plurality of parameterized and non-parameterized game objects For example, the game engine 228 may generate background imagery for a platformer level based on a computer code common to all platformer levels but may generate a variety of differently sized and located platforms based on the configuration file. Additionally, the game engine 228 may generate a plurality of tutorial windows for display before certain notes based on the configuration file.Example Audio Engine-Related Features and Functionality

[0112] The system can include an audio recognition engine (herein referred to as the “audio engine 224”) that is capable of detecting individual pitches, chord tones, and multi-note sounds all simultaneously and in realtime, along with keeping track of rhythmic information, for game interactions. This audio engine 224 can be capable of being the primary interaction method for an entire game and can be used as the primary input controller for a musicbased game. The audio engine can take in audio in real-time using a dedicated microphone or built-in microphone on the device and use that information to determine the above information all in real time. In some implementations, the audio engine 224 can be built into or made native to the game engine 228. In some implementations, the audio engine 224 can be optimized to recognize pitch at a frequency of once a frame for a game (e.g., a mobile game) running at greater than 60 fps to provide accurate values with very minimal latency

[0113] Figure 9 is a block diagram illustrating an example audio system 900. Although illustrated as being implemented on the user device 110. In some implementations, one or more components of the audio system 900 may be implemented on the server 130 or in another location, such as a third-party hosting service. Additionally, in some implementations, the audio system 900 can comprise more, less, or different components. In the illustrated example, the audio system 900 comprises a microphone 204 and a game engine 228 as described with reference to Figure 2, as well as an audio engine 224 as described with reference to Figure 2 and further described herein. Theaudio engine 224 comprises a microphone input manager 920, a sound detection manager 922, a rhythmic information control 924, an action identifier 926, and a broadcaster / receiver 928.

[0114] The microphone input manager 920 manages a real-time or substantially real-time connection to the input of the microphone 204. The microphone input manager 920 can read and store real-time audio data as it is picked up by the microphone 204. The microphone input manager 920 may comprise a plurality of settings to manage microphone input. The setting can include but are not limited to: (1 ) selected microphone name, (2) number of samples, (3) sample rate, (4) channels, and (5) microphone sensitivity. In some implementations, these settings can be adjusted based via a user interface generated during a game session 230 to enable a user to adjust the microphone settings. In some implementations, the settings can be adjusted for an individual game session 230 or can be adjusted for a particular player or user. For example, a game level may be generated with an input to access a settings menu during the game level. Alternatively, or additionally, the game engine 228 may be configured to generate a home screen or main menu for a player to access microphone settings during a game session 230 when no interactive video game level is active. Users may adjust settings based on personal preferences, such as microphone sensitivity or which microphone to use on a device with multiple microphone support. The microphone input manager 920 can transmit the audio data to the sound detection manager 922.

[0115] The sound detection manager 922 can manage components for detecting pitch, volume, and / or chords based on audio received via the microphone 204 and the microphone input manager 920. The sound detection manager 922 via its various components, can determine the pitch, volume, and potential chord of audio received at a given instance or during a sampling period. Sampling periods may be periods of time having durations of multiple seconds or a fraction of a second (e.g., 0.25 seconds, 0.5 seconds, 1 second, 2 seconds, etc.). In some implementations, the sound detection manager 922 may analyze audio received over a sampling period rather than a single point in time to determine pitch, volume, or other audio information.

[0116] The sound detection manager 922 can use a combination of signal processing techniques (Fast Fourier Transform (FFT), harmonic product spectrum (HPS) analysis, spectrogram analysis) in combination with machine learning (ML) based techniques (including, for example, one or more models such as ML models, convolutional neural networks (“CNNs”), and / or the like) to determine the pitch of individual notes and / or multiple notes simultaneously. The sound detection manager 922 can choose which technique to use or use an adaptive polling approach to combine the individual techniques to best determine how to represent the incoming sounds.

[0117] The sound detection manager 922 can comprise a volume detector 930, a single note pitch detector 932, a chord detector 934, and a multi-note pitch detector 936. The volume detector 930 may be configured to determine the volume, as measured in decibels (or some other unit of measure), of a piece of audio during a sampling period. For example the volume detector 930 may measure the amplitude of a waveform.

[0118] The single note pitch detector 932 may be configured to determine the pitch of a single note during a sampling period. In some implementations, the single note pitch detector 932 can utilize HPS analysis to determine the pitch of an audio signal or waveform at a given instance or during a sampling period. The chord detector 934 may be configured to detect and determine a chord from an audio signal or waveform at a given instance or during a sampling period In some implementations, the chord detector 934 may use spectrogram analysis. The multi-note pitch detector 936 may be configured to detect multiple notes from an audio signal or waveform at a given instance orduring a sampling period. In some implementations, the multi-note pitch detector 936 may use one or more CNNs to detect and determine notes. The sound detection manager 922 is further discussed herein with reference to Figures 10 and 11. Information from the sound detection manager 922 may be transmitted to the action identifier 926 and / or the broadcaster / receiver 928. In some implementations the data from the components of the sound detection manager 922 may be aggregated before being transmitted to other components of the audio engine 224.

[0119] The rhythmic information control 924 can manage rhythmic information For example, the rhythmic information control 924 can keep track of rhythmic information, keep track of an expected BPM for a given situation, send signals to components in a music-based game, keep track of notes being played on beat, early, or late (in relation to the expected playing time at the given BPM), and connect notes being played to a larger measure, section, phrase, or melody. For example, the rhythmic information control 924 can identify that a note is being played partway through a measure or at the beginning of a new measure. In some implementations, the rhythmic information control 924 may be configured to implement incrementing timer(s) and several listeners at different intervals, being able to handle rhythmic subdivisions such as a full measure, a singular beat (quarter note), eighth note, sixteenth notes, and more. In some implementations, the timer(s) can be updated through the physics engine of the game engine 228, which may have tighter time-based controls allowing for much more accurate and consistent rhythm control.

[0120] The rhythmic information control 924 can receive information from the game engine 228. As explained above with reference to Figures 3A and 3B, a configuration file for a video game level may comprise rhythm information such as bpm, beat base, denominator, and tolerance. The game engine 228 may pass this information to the rhythmic information control 924. In some implementations, the rhythmic information control 924 can request this information from the game engine 228 when needed. The rhythmic information control 924 can transmit rhythmic information to the action identifier 926 to enable or aid in action identification.

[0121] The action identifier 926 can manage the use of notes as game controls. In some implementations, the action identifier 926 can identify a singular action by a player from an audio signal or wave form. In order to correctly handle ‘game action’ style interactions, the audio engine 224 can advantageously define a single music action (playing / singing a note or pitch) beyond just the current pitch at a given time. Doing so can involve knowledge of when a note has started to be played, or if the current pitch is a continuation of the same note being played. This becomes more complex as different instruments are introduced, having different natural note decays, both in terms of duration and acoustic properties. For example, guitar has a short decay with the peak amplitude at the start of the note’s lifespan Violins have varying peak amplitudes (usually later in the notes lifespan) but contain clear breaks between individual notes played. Advantageously, the audio engine 224 via the action identifier 926 can identify these individual music actions (e.g. playing a new note) and translate them into game actions. The action identifier 926 can do this in several ways, including but not limited to: (1) keeping track of peak amplitude to detect the start of new notes, (2) keeping track of silences and using them as dividers between individual music actions, (3) using a buffer of the previous audio data, such as the previous several pitches and amplitudes heard in order to understand the waveform shape over the past few milliseconds, and / or (4) a ML model designed to identify the unique shape of an individual action based on instrument and current audio being read by the microphone 204. The action identifier 926 can transmit action information to the broadcaster / receiver 928 for communication to the game engine 228. In someimplementations, the action identifier 926 can transmit action information directly to the game engine 228. Additional disclosure related to the action identifier 926 is discussed with reference to Figures 10 and 11.

[0122] The broadcaster / receiver 928 can integrate with the game engine 228 to broadcast and receive information The audio engine 224 via the broadcaster / receiver 928 can broadcast the information it collects in a way that lets it tightly integrate with existing game engines. Ways it can do this include but are not limited to (1 ) broadcasting through the event system of the game engine 228, allowing other game objects in a given scene to subscribe to high level state information and get notified in real time as the state of the audio engine 224 state changes (e.g. a new note is played); (2) connecting to the physics engine of the game engine 228 to better control speed and pacing of a level; (3) safely enabling references by other elements in the game engine 228 to access specific real time data e.g. a microphone Ul element which displays the waveform to display to the user the audio entering the game in real-time; and (4) allowing for input from the game to modify settings or simulate audio data from in-game sources to allow for closer control of the audio engine 224. In some implementations, the audio engine 224 can be plugged into an existing game and replace some or all necessary user input controls with audio-based signals.

[0123] In some implementations, the audio engine 224 can run natively on the game engine 228 and can take advantage of the event system of the game engine 228 to transmit data outward. The audio engine 224 may be configured to transmit all data needed to play the required pitches on an instrument and can contain modifiable buffers to tune to individual players’ play style. In an implementation, the audio engine is a platform native ML + Signal Processing solution for entirely audio controlled video games. In such an implementation, the game engine 228 can advantageously take advantage of a native game engine for tight integration with the internal physics engine, which provides improved timekeeping for Rhythmic information and tighter integration with Events and other one-to-many broadcasting solutions included inside of modern game engines (e.g. Unity Engine).

[0124] Figure 10 is a block diagram illustrating an example implementation of the audio engine 224. In the illustrated implementations, the microphone 204 receives audio signals as the waveform 1002. Overtime, the waveform 1002 is transmitted to the single note pitch detector 932, chord detector 934 and multi-note pitch detector 936. The output of the single note pitch detector 932 may be sent though a buffer 1010 to help structure the output of the single note pitch detector 932 and help enable the action identifier 926 to detect identify a single music action. Buffers can compile audio data over a period of time to help with pattern recognition and enable detection of note starts, note stops, and note durations. Accordingly, buffers can comprise audio data previous to that currently being received by the microphone 204. Similarly, the output of the chord detector 934 may be sent through a buffer 1020 to help structure the output of the chord detector 934 and help enable the action identifier 926 to detect identify a single music action, such as the playing of a chord. In some implementations, the buffer 1010 and buffer 1020 function in the same manner. Additionally, in some implementations, the output of the multi-note pitch detector 936 may be sent through a buffer 1030 similar to the buffer 1010 and the buffer 1020. The output of the buffers can be provided to the action identifier 926. In some implementations, the buffer 1010 may be a component of the single note pitch detector 932. In some implementations, the buffer 1020 may be a component of the chord detector 934. In some implementations, the buffer 1030 may be a component of the multi-note pitch detector 936. In some implementations, one or more of the buffer 1010, the buffer 1020, or the buffer 1030 may be a component of the sound detection manager 922. In some implementations, one or more of the buffer 1010, the buffer 1020, or the buffer 1030 may be a componentof the action identifier 926. In some implementations, one or more of the buffer 1010, the buffer 1020, or the buffer 1030 may receive information from the volume detector 930. Further detail regarding the buffers is disclosed with reference to Figure 11.

[0125] In the illustrated implementation, the action identifier 926 comprises a new note estimator 1040. The new note estimator 1040 can receive information from the game engine 228, such as the game level type, the skills being tested, or the type of note expected (e.g., single note, multiple notes, or chord). The new note estimator 1040 can process the information received from the buffer 1010 and the buffer 1020 (and in some implementations, the buffer 1030) to determine which note was heard by the audio engine 224 and whether the note is a new note. The new note estimator 1040 may determine which note was heard by selecting from and / or averaging the data from the buffers and / or the volume detector 930, the single note pitch detector 932, the chord detector 934, and / or the multinote pitch detector 936 For example, if the single note pitch detector 932 detects a pitch of B3and the multi-note pitch detector 936 detects a pitch of B3, but the chord detector 934 detects a G Minor chord, the new note estimator 1040 may determine that the note heard was B3based on the determination that the note was detected by the single note pitch detector 932 and the multi-note pitch detector 936, and the chord detected by the chord detector 934 comprises that note

[0126] The new note estimator 1040 may determine whether a note is a new note by tracking notes heard by the audio engine 224 over time. Additionally, the new note estimator 1040 may determine whether a note is a new note based on the volume or amplitude of the note to determine whether it corresponds to a peak amplitude. For instance, a note may be loudest (i.e., have a peak amplitude) when initially played and then fade in volume over time. Accordingly, if a player is strumming notes with the same pitch, the system may detect the same pitch at various volumes over various times. For instance, at T=1 , the pitch may be B3with a volume of -21 dB. At T=2, the pitch may be B3with a volume of -18dB. The new note estimator 1040 can determine that the player has played a new note at T=2, since the volume of the pitch was louder at T=2 than at T=1. Similarly, the new note estimator 1040 can determine that the player has not yet played a new note if the volume of a successive pitch is the same or lower than the previously detected pitch. The action identifier 926 may transmit its determination of a new note action to the broadcaster / receiver 928 for transmittal to the game engine 228 to trigger an in-game action.

[0127] Figure 11 is a diagram illustrating example functionality of a buffer of the audio engine 224. The functionality of the example may apply to any buffer, such as the buffer 1010, the buffer 1020, or the buffer 1030. For simplicity, the illustrated example is implemented with the buffer 1010. In the illustrated implementation, a player plays notes 1102 in succession based on a set bpm and rhythm. The audio of these notes is detected by the microphone 204 as the waveform 1104. The waveform 1104 is transmitted via the microphone 204 to the microphone input manager 920 and to the sound detection manager 922. During a sampling period the single note pitch detector 932 detects a pitch and the volume detector 930 detects a volume for the sampling period. The single note pitch detector 932 communicates the pitch information to the buffer 1010, and the volume detector 930 communicates the volume information to the buffer 1010. The buffer 1010 may compile this information in a table over time to enable pattern recognition such as the start, duration, and end of a note.

[0128] In the illustrated example, notes 1102 are both B4notes. During the sampling period between T=1 and T=2, the single note pitch detector 932 detects a pitch of B3and the volume detector 930 detects a volume of-38d B. This indicates that the audio engine 224 has not yet detected a new B4 note. Subsequently, during the sampling period between T=2 and T=3, the single note pitch detector 932 detects a pitch of B4 and the volume detector 930 detects a volume of -16dB. The change in pitch and the increase in volume signal the start of a new note. Accordingly, the buffer 1010 may identify a new note detection 1110A. In some implementations, the buffer information is passed to the action identifier 926 which identifies the new note detection 1110A Further, this indicates the start of a new note range 1106A. Note ranges 1106 can be a plurality of pitch-volume pairs comprising a single note. This can enable the sound detection manager 922 to track the duration of notes as well as determine the timing of the peak amplitude for a note.

[0129] During the sampling period between T=3 and T=4, the single note pitch detector 932 detects a pitch of B4and the volume detector 930 detects a volume of -21 dB. The consistency in pitch and decrease in volume indicates the continuance of the same note detected during the previous sampling period. This is similarly true for the sampling period between T=4 and T=5 in which the same pitch is detected as in the previous period, but at a lower volume (-31dB).

[0130] During the sampling period between T=4 and T=5, the single note pitch detector 932 detects a pitch of B4and the volume detector 930 detects a volume of -14dB. While the pitch is the same as that of the previously sampling period, the volume is louder than the previous sampling period indicating a new note, and the buffer 1010 may identify a new note detection 1110B and a new note range 1106B. Further, this can indicate that the previous sampling period is the end of the note range 1106A.

[0131] Overtime, a player may pause after playing a note. The last note played may fade and the microphone 204 may detect white noise or other background noise. The single note pitch detector 932 and volume detector 930 may still determine a pitch and volume for this noise The audio engine 224 can track that the sounds are inconsistent overtime and at a low volume to determine that the audio is just background noise and not the player attempting to play an action soundExample Multi-User Audio Engine-Related Features and Functionality

[0132] In some implementations, the audio engine 224 can be configured as a multi-user audio engine. The multi-user audio engine can be used in group and classroom settings and can identify and isolate individual players' audio in order to create a real-time collaborative gaming experience. The system can use an individual or multiple microphones (depending on the number of devices that are in the vicinity), as well as optional camera data for a multimodal decision-making process. Accordingly, the system can advantageously identify the number of unique users in a session and isolate the audio information per user in order to translate audio to game actions in real time on a perplayer basis.

[0133] In one implementation, the audio engine 224 can be configured as a multi-user audio engine based on a multi-modal solution. For example, the audio engine 224 can accept and utilize multiple modalities in order to gather enough information to both detect music information such as individual pitches, chord tones, and multi-note sounds all simultaneously and in real-time (such as that described herein with reference to Figures 9, 10, and 11), in addition to assigning music information to one or more players all situated around one or more devices, such as the camera 202 and / or the microphone 204. The different modalities can be used in conjunction with a multi-modal modelto identify the number of individual players, as well as which players are currently actively playing and assign audio information to a player respectively.

[0134] In one implementation, a solution to utilize multiple modalities can include but is not limited to (1) a computer vision ML model which identifies individuals through a camera 202, and then uses pose estimation on each hand to determine which user is playing at a given time. For example, the ML model can determine that a player has their hands in a certain position on their instrument, such as having their fingers on certain frets of a guitar. As another example, the ML model can determine that a player is strumming a guitar based on the movement of their hands. As another example, the ML model can determine that a player is playing a violin based on the movement of the players hands with the movement of the bow. As another example, the ML model can determine that a player is playing a piano based on detecting that certain keys are depressed. In some implementations a plurality of cameras 202 may be utilized to get additional views of a group of players.

[0135] The camera information can then be combined with music information from the audio engine 224 to both detect music as well as which player is playing. Unique game actions can then be taken depending on which player is playing at a given time. For instance, two players could attempt to race each other across a set of platforms. The system could detect the notes from each player to determine whether either one of the two players correctly played a note at the correct time to cross from one platform to another. In another example, two players could face off in a game level such as the RPG level type, such that instead of a battle character, there are two player characters, and the system can track which player completed a greater percentage of a gameplay string successfully.

[0136] In another implementation, the audio engine 224 can be configured as a multi-user audio engine based on a multi-model solution. For example, the audio engine 224 can comprise two audio frequency-based models, one to detect music information such as individual pitches, chord tones, and multi-note sounds all simultaneously and in real-time (such as that described herein with reference to Figures 9, 10, and 11), and one which uses unique audio frequency identifiers to determine different players’ playing.

[0137] In one implementation, a solution to utilize multiple models can include a model which reads the same audio data as the first model which detects music information (as described above) and uses categorization to identify different instruments playing. The model can then identify different users playing together, provided they are playing different instruments. This model can use several audio processing ML model techniques including but not limited to Convolutional Neural Networks, Attention Models, Large Language Models. For example, in such an implementation, a model may be able to detect that one player is playing a guitar and another player is playing a bass, and the model may be able to detect the notes from each instrument.

[0138] In another implementation, a solution to utilize multiple models can include several models that receive data from multiple microphones on multiple devices. The models can then use the correlating volume data to determine how many people in a space are playing, and which device they are closest to. This closeness or proximity determination can be used to identify which individuals are playing which notes. The audio engine 224 can then communicate this information to the game engine 228 to identify and separate each players’ actions. In some implementations, the action identifier 926 is utilized to identify and separate each players’ actions. In some implementations, the model engine 232 is utilized to identify and separate each players’ actions. For example, the audio engine 224 may communicate the audio information to a model implemented with the model engine 232 toidentify separate note ranges and timing. In some implementations, the model may utilize information from a configuration file received from the configuration engine 226 or the game engine 228. For example, the model may receive the gameplay string for each player along with the audio input to help identify each players’ notes and actions.

[0139] Figure 12 illustrates an example of the audio engine 224 detecting audio input from multiple players. In the illustrated example, the audio engine 224 can receive the raw audio 1202. The audio engine 224 can then identify and differentiate between individual players’ audio using one or more of the methods described above. For example, the audio engine 224 may input the raw audio 1202 into a model and receive segment information corresponding to different players from the model. For example, the model may indicate a first audio segment 1206A and a second audio segment 1206B correspond to a first player, and a third audio segment 1206C and a fourth audio segment 1206D correspond to a second player. In some cases, the notes of multiple players can be overlapping musical notes. For instance, in the illustrated implementation, the second audio segment 1206B and the fourth audio segment 1206D overlap. In some implementations, the audio engine 224 can aggregate the segments for each player such as the illustrated player 1 audio 1204A, and the player 2 audio 1204B.

[0140] This may be because the players played the same notes at the same time. The audio engine 224 may simultaneously or serially communicate the audio segments to the sound detection manager 922 for note detection and action identification such as that described with reference to Figures 10 and 11.Example Features Related to Generating Input Controls Based on Audio Data

[0141] Figure 13 is a flow chart illustrating an example process 1300 for generating input controls based on audio data. Although steps are illustrated in a particular order, steps may be performed multiple times, the order of the steps may be changed, and / or one or more steps may be performed concurrently. Additionally, fewer, more, or different steps may be performed. Further, the process 1300, in whole or in part can be implemented by a user device 110, server 130, or other computing system. Although any number of systems, in whole or in part, can implement the process 1300, to simplify discussion the process 1300 will be described with respect to the user device 110.

[0142] At block 1302, the system can receive audio data. The audio data can correspond to audio from an instrument played by a player. The audio data may be received by the microphone 204, which can communicate the received audio data to the audio engine 224 for processing. As described herein with reference Figure 9, the audio data can be received via the microphone input manager 920 and communicated to the sound detection manager 922 for analysis.

[0143] At block 1304, the audio engine 224 can analyze the audio data to determine a pitch of one or more notes. For example, as described herein with reference to Figures 9, 10, and 11, the audio data can be communicated to the single note pitch detector 932, the chord detector 934, and / or the multi-note pitch detector 936. These components may output a detected pitch based on the received audio data to an action identifier 926.

[0144] At block 1306, the audio engine 224 can analyze the audio data to determine a peak amplitude of one or more notes. For example, as described herein with reference to Figures 9, 10, and 11 , the audio data can be communicated to the volume detector 930. The volume detector 930 may then output a detected volume for one or more notes to the action identifier 926. In some cases, the volume detector 930 may provide output to a buffer 1010, which can also receive information from the single note pitch detector 932. The buffer 1010 may then use the correlatedvolume and pitch information to determine a peak amplitude. The volume detector 930 may simultaneously provide output volume information to a buffer 1020 or buffer 1030. In some cases, multiple buffers may determine amplitude information which may be communicated to the action identifier 926, which can determine a peak amplitude via the new note estimator 1040.

[0145] At block 1308, the action identifier 926 can determine a user action based on the peak amplitude determined at block 1306 In some implementations, the action identifier 926 can determine a single user action based on a change in amplitude of the audio data over time, a rate of change in amplitude of the audio data over time, a relative difference between the peak amplitude and another amplitude of the audio data at a different point in time, or a point in time associated with the peak amplitude or another indicator in the audio data. For example, as described herein with respect to Figures 9, 10, and 11 , action identifier 926 can generate input control information based on the peak amplitude. For instance, the identification of a new note can be used to generate a game control to be used by the game engine 228 to cause a player character to jump across a platform. Accordingly, the audio engine 224 can communicate the game control to the game engine 228 via the broadcaster / receiver 928

[0146] At block 1310, the game engine 228 utilizes the received game control to perform an action within the interactive video game level. For example, as described herein with reference to Figures 5 and 9, the game engine 228 can use the game control received from the action identifier 926 to cause a player character to jump across a platform.Example Features Related to Generating Input Controls Based on Audio Data from Multiple Sources

[0147] Figure 14 is a flow chart illustrating an example process 1400 for generating input controls based on audio data from multiple sources. Although steps are illustrated in a particular order, steps may be performed multiple times, the order of the steps may be changed, and / or one or more steps may be performed concurrently. Additionally, fewer, more, or different steps may be performed. Further, the process 1400, in whole or in part can be implemented by a user device 110, server 130, or other computing system. Although any number of systems, in whole or in part, can implement the process 1400, to simplify discussion the process 1400 will be described with respect to the user device 110.

[0148] At block 1402, the application 210 can receive audio data. The audio data can correspond to audio from multiple instruments played by multiple players. The audio data may be received by one or more microphones 204, which can communicate the received audio data to the audio engine 224 for processing. As described herein, the audio engine 224 can be configured as a multi-user audio engine and can be configured to receive audio data via the microphone input manager 920, which can communicate the audio data to the sound detection manager 922 for analysis.

[0149] At block 1404, the application 210 can receive image data The image data can correspond to images or video captured by one or more cameras 202. The image data may be used to capture pose data for a plurality of players, such as positioning of a player’s hand on their instrument, or the movement of components of a player's instrument, such as the strings on a guitar, the keys of a piano, or the bow of a violin. In some implementations, the image data is communicated to the audio engine audio engine 224 for analysis along with the audio data.

[0150] At block 1406, the audio engine 224 can analyze the audio data in conjunction with the image data to determine individual audio segments for each of a plurality of players. For example, as described herein withreference to Figure 12, the audio engine 224 can determine different audio segments corresponding to different players from the audio data received at block 1402. Additionally, as described above the audio engine 224 can utilize machine learning methods on the image data such as pose estimation to determine which user is playing at a given time. This can help the audio engine 224 correlate audio segments with particular players

[0151] At block 1408, the audio engine 224 can determine user actions for some or all of the plurality of players (e.g., at least two players). For instance, the audio engine 224 can determine when each player played a new note. As another example, the audio engine 224 can determine the duration of each player’s note. These can lead to the generation of game controls to be communicated to the game engine 228. For example, for a platformer level as discussed herein with reference to Figure 5, a game control may correspond to jumping over a platform gap or a game control may correspond to walking forward along a platform. These game controls can be generated based on the detection of a new note, or the continuation of an already recognized note. The audio engine 224 can communicate the generated controls to the game engine 228.

[0152] At block 1410, the game engine 228 utilizes the received game controls to perform actions within the interactive video game level. For example, for a first player character corresponding to a first player, a game control may correspond to a jump action, and the game engine 228 can render the level such that the first player character takes the jump action. For a second playercharactercorresponding to a second player, a game control may correspond to a walking action, and the game engine 228 can render the level such that the second player character takes a walking action.Example Features and Functionality Related to Real-Time Audio to Music Notation Parsing

[0153] In some implementations, the system can utilize real-time pitch detection and internal timing plus metronome features in the audio engine 224 to hear what a player is playing on their instrument, then translate that to standard sheet notation and game configuration files in order to facilitate instant user-created levels in game.

[0154] In some implementations, the system can utilize a few preset rules and then directly translate to sheet notation (either TAB or Standard Notation) in real time or substantially real time. This music notation can then be used in-game by a game level by being converted into a configuration file or be exported to a music notation format (e.g., MusicXML) to be read by external sheet music apps Once the audio has been parsed into a computer readable format, the system can then translate this to a configuration file and upload the configuration file to the server 130 or some other database (e.g., a NoSQL database) for storage and distribution. In some implementations, a configuration file can contain information to generate a playable level, so once the configuration file is uploaded, the system can distribute the configuration file to any other player. Accordingly, the system can advantageously unlock frictionless user generated content for a music-based games (e.g , Notey’s World), allowing for an infinite number of levels to be created, limited only by the creativity of the players of the game.

[0155] Figure 15 is a flow chart illustrating an example process 1500 for real-time audio to music notation parsing. Although steps are illustrated in a particular order, steps may be performed multiple times, the order of the steps may be changed, and / or one or more steps may be performed concurrently. Additionally, fewer, more, or different steps may be performed. Further, the process 1500, in whole or in part can be implemented by a user device 110, server 130, or other computing system Although any number of systems, in whole or in part, can implement the process 1500, to simplify discussion the process 1500 will be described with respect to the user device 110.

[0156] At block 1502, the system can receive audio data. The audio data can correspond to audio from an instrument played by a player. The audio data may be received by the microphone 204, which can communicate the received audio data to the audio engine 224 for processing. As described herein with reference Figure 9, the audio data can be received via the microphone input manager 920 and communicated to the sound detection manager 922 for analysis.

[0157] At block 1504, the audio engine 224 can analyze the audio data to determine a pitch of one or more notes. For example, as described herein with reference to Figures 9, 10, and 11, the audio data can be communicated to the single note pitch detector 932, the chord detector 934, and / or the multi-note pitch detector 936. In some implementations, the data from the volume detector 930 can be fed into one or more of the buffer 1010, the buffer 1020, or the buffer 1030. Additionally, the data from the single note pitch detector 932 can be fed into the buffer 1010, the data from the chord detector 934 can be fed into the buffer 1020, and / or the data from the multi-note pitch detector 936 can be fed into the buffer 1030. In some implementations, the sound detection manager 922 can average the data from the buffers and / or the volume detector 930, the single note pitch detector 932, the chord detector 934, and the multi-note pitch detector 936. For example, if the single note pitch detector 932 detects a pitch of B3and the multi-note pitch detector 936 detects a pitch of B3, but the chord detector 934 detects a G Minor chord, the sound detection manager 922 may determine that the note heard was B3based on the determination that the note was detected by the single note pitch detector 932 and the multi-note pitch detector 936, and the chord detected by the chord detector 934 comprises that note. Further, the information collected by the sound detection manager 922 may comprise the length of the note, for instance based on a note range 1106, as described with refence to Figure 11 .

[0158] In some implementations, the sound detection manager 922 may be configured to generate an audio detection object comprising information about the note detected, such as the pitch and duration. In some implementations, the audio detection object may comprise a sequence of notes, for example notes may be stored in an array. In some implementations, audio detection objects can comprise additional information such as the instrument being played, the bpm, or the key. The audio detection object(s) can be provided to the rules engine 222 to translate the audio into musical notation (e.g., MusicXML) and generate a music data file The generated music data file can then be transmitted to the parser 220.

[0159] At block 1506, the system can perform the same actions as described herein with reference to block 804 of Figure 8.

[0160] At block 1508, the system can perform the same actions as described herein with reference to block 806 of Figure 8.

[0161] At block 1510, the system can perform the same actions described herein with reference to block 808 of Figure 8.Example Features and Functionality Related to Language Model Based Level Personalization Generator

[0162] Figure 16 is a flow chart illustrating an example process 1600 for generating level personalization using a language model. Although steps are illustrated in a particular order, steps may be performed multiple times, the order of the steps may be changed, and / or one or more steps may be performed concurrently. Additionally, fewer, more, or different steps may be performed. Further, the process 1600, in whole or in part can be implemented by a user device 110, server 130, or other computing system. Although any number of systems, in wholeor in part, can implement the process 1600, to simplify discussion the process 1600 will be described with respect to the user device 110.

[0163] At block 1602, the system can obtain play results. The game engine 228 can collect play results during a game session game session 230, as described herein with reference to Figure 2. In some implementations, the play results can correspond to results of a single video game level or can correspond to aggregated results of many levels.

[0164] At block 1604, the game engine 228 can determine a portion of a song associated with a lower play score, as discussed herein with reference to Figure 2. In some implementations, the play results can include the results of each note in a gameplay string. For example, if a gameplay string consists of 10 notes, and the player correctly plays notes, 1 , 2, 3, 5, 9, and 10, but incorrectly plays the other notes, the game engine 228 can determine that the player struggled with a middle portion of the gameplay string. Accordingly, the game engine 228 may generate a resultant string comprising notes, 4, 5,6, 7, and 8 as being associated with a lower play score. The game engine 228 may generate this note string even though the player got note 5 correct, since the player got the notes around note 5 wrong. A note may be determined to be incorrect if timed incorrectly. For instance, even if the player got the pitch correct, if the note was played out of rhythm with the other notes, the game engine 228 may mark it as an incorrect note with a lower score. In some instances, a note may be evaluated on several aspects. For instance, rather than a binary correct or incorrect, each note may be evaluated based on whether a correct pitch was played at a correct time such that a percentage value may be provided to each note. In such case, the game engine 228 may look for a portion of a gameplay string associated with low percentage values.

[0165] At block 1606, the game engine 228 may generate a prompt for an LLM. In some implementations, such as the illustrated example, the prompt may comprise a request for a custom video game configuration file aimed at helping a player learn the portion of the song associated with a lower play score identified at block 1604 as described herein with reference to Figure 2. For example, the prompt may comprise a portion of a song with a low play score (e.g., a portion of the gameplay string or corresponding notation string), an example configuration file, and a request to generate a new configuration file based on the portion of the song with the low play score. In some implementations, the prompt can indicate a video game level type for the configuration file. In some implementations, the prompt can indicate that one or more complete gameplay strings should be generated based off the gameplay string portion provided. In some embodiments, the prompt can request that tutorialization information be generated with the configuration file as described herein with reference Figure 2 and block 806 of Figure 8. In some implementations, the prompt may comprise requests for portions of a configuration file, for one or more gameplay strings, for tutorialization information for one or more gameplay strings, for a music data file, etc. In some embodiments, the prompt can include different information, such as example gameplay strings, example tutorial information, example configuration files, example music data files, etc.

[0166] At block 1608, the prompt is provided to the LLM as described herein with reference to Figure 2. In some implementations, the game engine 228 may directly provide the prompt as input to the LLM. In some implementations, the game engine 228 may communicate the prompt to the model engine 232 which can provide the prompt as input to the LLM.

[0167] At block 1610, the system can receive output from the LLM as described herein with reference to Figure 2. In some implementations, the game engine 228 may receive output directly from the LLM. In some implementations, the model engine 232 can receive output from the LLM and communicate the output to the game engine 228 The output can be in response to the prompt provided at block 1608, which in the illustrated example is a configuration file. For example, if the prompt was for a gameplay string, the output can be a gameplay string or a plurality of gameplay strings.

[0168] At optional block 1612, the system can generate tutorialization information using the LLM as described herein with reference to Figure 2 and block 806 of Figure 8. In some implementations, the system can use the same LLM as was used at block 1608. In some implementations, a different LLM may be used. For example, the system may comprise a plurality of LLMs trained to generate different outputs. For example, system may comprise an LLM to generate configuration files, an LLM to generate gameplay strings, an LLM to generate tutorial information, an LLM to generate music data files, etc. In some implementations, the model engine 232 may manage the plurality of LLMs and prompts for the LLMs. For example, as described herein with reference to block 1606, the game engine 228 may generate a prompt for a configuration file based on a portion of a gameplay string and communicate the prompt to the model engine 232. The model engine 232 may parse the prompt into components and generate prompts for each of a plurality of LLMs based on the prompt components. For example, the model engine 232 may parse the prompt for a new configuration file into the following portions: a prompt for one or more new gameplay strings, a prompt for tutorialization information, and a prompt for a full configuration file The model engine 232 may execute these portions in sequence such that a first LLM generates a new gameplay string. The newly generated gameplay string is then used as an input for a second LLM to generate tutorialization information based off the gameplay string generated by the first LLM. The newly generated tutorialization information and gameplay string can then be used as input for a third LLM to generate a complete configuration file comprising the gameplay string and tutorialization information.

[0169] At optional block 1614, the system can perform actions such as that described herein with reference to block 808 of Figure 8 using a configuration file generated, in whole or in part, by an LLM during the earlier blocks of process 1600.Additional Implementation Details and Embodiments

[0170] Various implementations of the present disclosure may be a system, a method, and / or a computer program product at any possible technical detail level of integration. The computer program product may include a computer readable storage medium (or mediums) having computer readable program instructions thereon for causing a processor to carry out aspects of the present disclosure.

[0171] For example, the functionality described herein may be performed as software instructions are executed by, and / or in response to software instructions being executed by, one or more hardware processors and / or any other suitable computing devices. The software instructions and / or other executable code may be read from a computer readable storage medium (or mediums).

[0172] The computer readable storage medium can be a tangible device that can retain and store data and / or instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device (including any volatile and / or non-volatile electronic storage devices), a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductorstorage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a solid state drive, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD- ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punchcards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

[0173] Computer readable program instructions described herein can be downloaded to respective computing / processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and / or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing / processing device.

[0174] Computer readable program instructions (as also referred to herein as, for example, “code,” “instructions,” “module,” “application,” “software application,” and / or the like) for carrying out operations of the present disclosure may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state-setting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Java, C++, or the like, and procedural programming languages, such as “h” "C" programming language or similar programming languages. Computer readable program instructions may be callable from other instructions or from itself, and / or may be invoked in response to detected events or interrupts. Computer readable program instructions configured for execution on computing devices may be provided on a computer readable storage medium, and / or as a digital download (and may be originally stored in a compressed or installable format that requires installation, decompression, or decryption prior to execution) that may then be stored on a computer readable storage medium. Such computer readable program instructions may be stored, partially or fully, on a memory device (e.g., a computer readable storage medium) of the executing computing device, for execution by the computing device. The computer readable program instructions may execute entirely on a ’ser's computer (e.g , the executing computing device), partly on the 'ser's computer, as a stand-alone software package, partly on the 'ser's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the 'ser's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some implementations, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) mayexecute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present disclosure

[0175] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to implementations of the disclosure It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer readable program instructions.

[0176] These computer readable program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart(s) and / or block diagram(s) block or blocks.

[0177] The computer readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus or other device to produce a computer implemented process, such that the instructions which execute on the computer, other programmable apparatus, or other device implement the functions / acts specified in the flowchart and / or block diagram block or blocks. For example, the instructions may initially be carried on a magnetic disk or solid state drive of a remote computer. The remote computer may load the instructions and / or modules into its dynamic memory and send the instructions over a telephone, cable, or optical line using a modem. A modem local to a server computing system may receive the data on the telephone / cable / optical line and use a converter device including the appropriate circuitry to place the data on a bus The bus may carry the data to a memory, from which a processor may retrieve and execute the instructions. The instructions received by the memory may optionally be stored on a storage device (e.g., a solid state drive) either before or after execution by the computer processor.

[0178] The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various implementations of the present disclosure In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the blocks may occur out of the order noted in the Figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. In addition, certain blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states relating thereto can be performed in other sequences that are appropriate.

[0179] It will also be noted that each block of the block diagrams and / or flowchart illustration, and combinations of blocks in the block diagrams and / or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions. For example, any of the processes, methods, algorithms, elements, blocks, applications, or other functionality (or portions of functionality) described in the preceding sections may be embodied in, and / or fully or partially automated via, electronic hardware such application-specific processors (e.g., applicationspecific integrated circuits (ASICs)), programmable processors (e.g., field programmable gate arrays (FPGAs)), application-specific circuitry, and / or the like (any of which may also combine custom hard-wired logic, logic circuits, ASICs, FPGAs, and / or the like, with custom programming / execution of software instructions to accomplish the techniques).

[0180] Any of the above-mentioned processors, and / or devices incorporating any of the above- mentioned processors, may be referred to herein as, for example, “computers,” “computer devices,” “computing devices,” “hardware computing devices,” “hardware processors,” “processing units,” and / or the like. Computing devices of the above-implementations may generally (but not necessarily) be controlled and / or coordinated by operating system software, such as Mac os, iOS, Android, Chrome OS, Windows OS (e.g., Windows XP, Windows Vista, Windows 7, Windows 8, Windows 10, Windows Server, and / or the like), Windows CE, Unix, Linux, SunOS, Solaris, Blackberry OS, VxWorks, or other suitable operating systems. In other implementations, the computing devices may be controlled by a proprietary operating system Conventional operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, I / O services, and provide a user interface functionality, such as a graphical user interface (“GUI”), among other things.

[0181] As described above, in various implementations certain functionality may be accessible by a user through a web-based viewer (such as a web browser), or other suitable software program. In such implementations, the user interface may be generated by a server computing system and transmitted to a web browser of the user (e.g., running on the user’s computing system). Alternatively, data (e.g., user interface data) necessary for generating the user interface may be provided by the server computing system to the browser, where the user interface may be generated (e.g , the user interface data may be executed by a browser accessing a web service and may be configured to render the user interfaces based on the user interface data). The user may then interact with the user interface through the web-browser. User interfaces of certain implementations may be accessible through one or more dedicated software applications. In certain implementations, one or more of the computing devices and / or systems of the disclosure may include mobile computing devices, and user interfaces may be accessible through such mobile computing devices (for example, smartphones and / or tablets).

[0182] Many variations and modifications may be made to the above-described implementations, the elements of which are to be understood as being among other acceptable examples. All such modifications and variations are intended to be included herein within the scope of this disclosure. The foregoing description details certain implementations. It will be appreciated, however, that no matter how detailed the foregoing appears in text, the systems and methods can be practiced in many ways. As is also stated above, it should be noted that the use of particular terminology when describing certain features or aspects of the systems and methods should not be taken toimply that the terminology is being re-defined herein to be restricted to including any specific characteristics of the features or aspects of the systems and methods with which that terminology is associated.

[0183] Conditional language, such as, among others, “can,” “could,” “might,” or “may," unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain implementations include, while other implementations may not include, certain features, elements, and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more implementations or that one or more implementations necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular implementation.

[0184] The term “substantially” when used in conjunction with the term “real-time” forms a phrase that will be readily understood by a person of ordinary skill in the art. For example, it is readily understood that such language will include speeds in which no or little delay or waiting is discernible, or where such delay is sufficiently short so as not to be disruptive, irritating, or otherwise vexing to a user.

[0185] Conjunctive language such as the phrase “at least one of X, Y, and Z,” or “at least one of X, Y, or Z,” unless specifically stated otherwise, is to be understood with the context as used in general to convey that an item, term, and / or the like, may be either X, Y, or Z, or a combination thereof. For example, the term “or” is used in its inclusive sense (and not in its exclusive sense) so that when used, for example, to connect a list of elements, the term “or” means one, some, or all of the elements in the list. Thus, such conjunctive language is not generally intended to imply that certain implementations require at least one of X, at least one of Y, and at least one of Z to each be present.

[0186] The term “a” as used herein should be given an inclusive rather than exclusive interpretation. For example, unless specifically noted, the term “a” should not be understood to mean “exactly one” or “one and only one”; instead, the term “a” means “one or more” or “at least one,” whether used in the claims or elsewhere in the specification and regardless of uses of quantifiers such as “at least one,” “one or more,” or “a plurality” elsewhere in the claims or specification.

[0187] The term “comprising” as used herein should be given an inclusive rather than exclusive interpretation. For example, a general purpose computer comprising one or more processors should not be interpreted as excluding other computer components, and may possibly include such components as memory, input / output devices, and / or network interfaces, among others.

[0188] While the above detailed description has shown, described, and pointed out novel features as applied to various implementations, it may be understood that various omissions, substitutions, and changes in the form and details of the devices or processes illustrated may be made without departing from the spirit of the disclosure. As may be recognized, certain implementations of the inventions described herein may be embodied within a form that does not provide all of the features and benefits set forth herein, as some features may be used or practiced separately from others. The scope of certain inventions disclosed herein is indicated by the appended claims rather than by the foregoing description. All changes which come within the meaning and range of equivalency of the claims are to be embraced within their scope.Example Clauses

[0189] Examples of the implementations of the present disclosure can be described in view of the following example clauses. The features recited in the below example implementations can be combined with additional features disclosed herein. Furthermore, additional inventive combinations of features are disclosed herein, which are not specifically recited in the below example implementations, and which do not include the same features as the specific implementations below. For sake of brevity, the below example implementations do not identify every inventive aspect of this disclosure. The below example implementations are not intended to identify key features or essential features of any subject matter described herein. Any of the example clauses below, or any features of the example clauses, can be combined with any one or more other example clauses, or features of the example clauses or other features of the present disclosure.

[0190] Clause 1. A computer-implemented method to automate video game level generation, the computer-implemented method comprising, by one or more hardware processors executing program instructions: receiving a music data file; parsing the music data file to generate a notation string comprising, for each of a plurality of musical notes, at least: pitch, duration, and additional finger information; executing a video game configuration engine on at least the notation string to generate a first video game configuration comprising at least: a first video game level type, and a first gameplay string; and executing a video game engine configured to automatically generate a first interactive video game level based on the first video game configuration, wherein the first interactive video game level is of the first video game level type, wherein the first interactive video game level comprises a sequence of graphical representations of musical notes, and wherein the sequence of graphical representations of musical notes is displayed according to the first gameplay string.

[0191] Clause 2 The computer-implemented method of Clause 1, wherein the music data file comprises a digital sheet music file.

[0192] Clause 3. The computer-implemented method of any of Clauses 1-2, wherein the notation string comprises a sequence of triples which represent the plurality of musical notes.

[0193] Clause 4 The computer-implemented method of any of Clauses 1-3 further comprising: executing a fingering rules engine to generate the additional finger information for each of the plurality of musical notes, wherein the fingering rules engine determines preferred fingerings for each of the plurality of musical notes based on at least one of: relationships among the plurality of musical notes, or additional information of the music data file.

[0194] Clause 5. The computer-implemented method of any of Clauses 1-4 wherein the additional finger information includes at least: string, fret position, or finger.

[0195] Clause 6. The computer-implemented method of any of Clauses 1-5 further comprising: further parsing the music data file to generate at least one of: tempo information, time signature information, or tolerance information

[0196] Clause 7. The computer-implemented method of any of Clauses 1-6, wherein the first gameplay string comprises at least portion of the notation string

[0197] Clause 8. The computer-implemented method of any of Clauses 1-7, wherein executing the video game configuration engine comprises: determining the first gameplay string based on one or more game configuration rules associated with the first video game level type.

[0198] Clause 9. The computer-implemented method of any of Clauses 1-7, wherein the first video game configuration further comprises a second gameplay string, and wherein executing the video game configuration engine comprises: determining, based on one or more game configuration rules associated with the first video game level type, a first portion of the notation string for generating the first gameplay string; and determining, based on the one or more game configuration rules associated with the first video game level type, a second portion of the notation string for generating the second gameplay string

[0199] Clause 10. The computer-implemented method of any of Clauses 1-9 further comprising: further executing the video game configuration engine on at least the notation string to generate a second game configuration comprising at least: a second video game level type, and a third gameplay string, wherein the third gameplay string is determined by the video game configuration engine based on one or more game configuration rules associated with the second video game level type.

[0200] Clause 11. The computer-implemented method of any of Clauses 1-10, wherein the video game configuration engine is expandable to automatically generate a plurality of video game configurations corresponding to a plurality of video game level types based on at least the notation string.

[0201] Clause 12. The computer-implemented method of any of Clauses 1-11, wherein: the first video game level type is platformer, the sequence of graphical representations of musical notes comprises a sequence of graphical platforms, sizes of the sequence of graphical platforms are determined by the video game engine based on durations associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, and relative locations of the sequence of graphical platforms are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string.

[0202] Clause 13. The computer-implemented method of Clause 12, wherein the sequence of graphical representations of musical notes further comprises: a visualization of a plurality of strings, and a sequence of icons that scroll along the plurality of strings, wherein: locations of the sequence of icons on the plurality of strings are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, colorings of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, and number indicators of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string.

[0203] Clause 14. The computer-implemented method of any of Clauses 1-11, wherein: the first video game level type is whackanote, the sequence of graphical representations of musical notes comprises a sequence of highlighted strings and / or icons, and colorings and / or number indicators of the sequence of highlighted strings and / or icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string

[0204] Clause 15. The computer-implemented method of any of Clauses 1-11, wherein: the first video game level type is RPG, the first video game configuration further comprises a plurality of gameplay strings including the first gameplay string, the sequence of graphical representations of musical notes comprises: a visualization of aplurality of strings, and a sequence of icons that scroll along the plurality of strings, and the first interactive video game level further comprises a game character and a plurality of game character positions.

[0205] Clause 16. The computer-implemented method of Clause 15, wherein: locations of the sequence of icons on the plurality of strings are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by at least one gameplay string of the plurality of gameplay strings, colorings of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the at least one gameplay string of the plurality of gameplay strings, and number indicators of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the at least one gameplay string of the plurality of gameplay strings.

[0206] Clause 17. The computer-implemented method of any of Clauses 15-16, wherein: the plurality of gameplay strings correspond to the plurality of game character positions, and the video game engine is configured to move the game character among the plurality of game character positions based on success or failure of a player at completing the plurality of gameplay strings.

[0207] Clause 18. The computer-implemented method of any of Clauses 1-17, wherein the video game engine is configured to determine, based on chords included in the first gameplay string, at least one of: background music for the first interactive video game level, or player checkpoints for the first interactive video game level.

[0208] Clause 19. A system comprising: one or more computer-readable storage mediums comprising program instructions; and one or more processors configured to execute the program instructions to cause the system to perform the computer-implemented method of any of Clauses 1-18.

[0209] Clause 20. One or more computer-readable storage mediums comprising program instructions, the program instructions executable by one or more processors to cause the one or more processors to perform the computer-implemented method of any of Clauses 1-18.

[0210] Clause 21. A computer-implemented method to detect audio information for controlling a video game, the computer-implemented method comprising, by one or more hardware processors executing program instructions: receiving audio data obtained via a microphone; analyzing the audio data over time to determine a pitch; analyzing the audio data over time to determine a peak amplitude; determining a single user action based at least in part on the peak amplitude; and executing a video game engine configured to generate an interactive video game level and provide an input control to the interactive video game level based on at least: the pitch, and the single user action.

[0211] Clause 22. The computer-implemented method of Clause 21, wherein the audio data comprises sounds produced by a musical instrument being played by a player of the interactive video game level.

[0212] Clause 23. The computer-implemented method of any of Clauses 21-22, wherein the single user action is determined further based on at least one of: a change in amplitude of the audio data over time, a rate of change in amplitude of the audio data over time, a relative difference between the peak amplitude and another amplitude of the audio data at a different point in time, or a point in time associated with the peak amplitude or another indicator in the audio data.

[0213] Clause 24. The computer-implemented method of any of Clauses 21-23, wherein determining the single user action includes determining a start of the single user actions and / or an end of the single user action.

[0214] Clause 25. The computer-implemented method of any of Clauses 21-24, wherein analyzing the audio data and / or determining the single user action is based on at least one of: Fast Fourier Transform, harmonic product spectrum analysis, spectrogram analysis, a machine learning model, or a convolutional neural network.

[0215] Clause 26. The computer-implemented method of any of Clauses 21-25, wherein analyzing the audio data over time to determine the pitch includes using a buffer of previous audio data and analyzing at least a portion of the previous audio data.

[0216] Clause 27. The computer-implemented method of any of Clauses 21-26 further comprising: analyzing the audio data over time to determine at least one of: a chord, or rhythmic information.

[0217] Clause 28. The computer-implemented method of any of Clauses 21-27 further comprising: analyzing the audio data over time to determine a series of overlapping musical notes comprising a series of user actions.

[0218] Clause 29. The computer-implemented method of any of Clauses 21-28 further comprising: analyzing the audio data over time to differentiate among multiple players of multiple musical instruments to provide inputs to the interactive video game level.

[0219] Clause 30. The computer-implemented method of Clause 29, wherein the audio data is obtained via multiple microphones associated with each of the multiple players.

[0220] Clause 31. The computer-implemented method of any of Clauses 21-28 further comprising: receiving image data obtained via a camera; and analyzing the image data in conjunction with analyzing audio data to determine user actions provided by at least two players of musical instruments to provide inputs to the interactive video game level.

[0221] Clause 32. The computer-implemented method of Clause 31 , wherein analyzing the image data in conjunction with analyzing audio data is performed, at least in part, using a multi-modal model.

[0222] Clause 33. A computer-implemented method to detect audio information for generating a video game, the computer-implemented method comprising, by one or more hardware processors executing program instructions: detecting audio-based information in accordance with one or more aspects of any of Clauses 21 -32 above; and automatically generating an interactive video game level based on the detected audio-based information in accordance with one or more aspects of any of Clauses 1-18 above.

[0223] Clause 34. The computer-implemented method of Clause 33, wherein the detected audio-based information comprises at least a sequence of detected musical notes, and wherein the interactive video game level comprises at least a sequence of graphical representations of the detected musical notes.

[0224] Clause 35. A computer-implemented method to automate video game level generation using a large language model, the computer-implemented method comprising, by one or more hardware processors executing program instructions: obtaining play results associated with a music-based video game played by a player; determining, based on the play results, a portion of a song associated with a lower play score; generating a prompt for a large language model, the prompt comprising at least: the portion of the song in a notation string format, an example of a video game configuration data format, and a request to generate a custom video game configuration in the video gameconfiguration format based on at least the portion of the song; providing the prompt to the large language model; and receiving an output from the large language model in response to the prompt, wherein the output comprises the custom video game configuration.

[0225] Clause 36. The computer-implemented method of Clause 35, wherein the play results comprise at least one of: notes played, concepts learned, notes played early, notes played late, notes missed, or scoring information

[0226] Clause 37. The computer-implemented method of any of Clauses 35-36 further comprising: executing a video game engine configured to automatically generate an interactive video game level based on the custom video game configuration.

[0227] Clause 38. The computer-implemented method of any of Clauses 35-37 further comprising: generating, using the large language model, tutorialization information associated with, or to include with, the custom video game configuration.

[0228] Clause 39. The computer-implemented method of any of Clauses 21-38, wherein the computer- implemented method is performed in real-time or substantially real-time.

[0229] Clause 40. A system comprising: one or more computer-readable storage mediums comprising program instructions; and one or more processors configured to execute the program instructions to cause the system to perform the computer-implemented method of any of Clauses 21-39.

[0230] Clause 41. One or more computer-readable storage mediums comprising program instructions, the program instructions executable by one or more processors to cause the one or more processors to perform the computer-implemented method of any of Clauses 21-39.

Claims

WHAT IS CLAIMED IS:

1. A computer-implemented method to automate video game level generation, the computer- implemented method comprising, by one or more hardware processors executing program instructions: receiving a music data file; parsing the music data file to generate a notation string comprising, for each of a plurality of musical notes, at least: pitch, duration, and additional finger information; executing a video game configuration engine on at least the notation string to generate a first video game configuration comprising at least: a first video game level type, and a first gameplay string; and executing a video game engine configured to automatically generate a first interactive video game level based on the first video game configuration, wherein the first interactive video game level is of the first video game level type, wherein the first interactive video game level comprises a sequence of graphical representations of musical notes, and wherein the sequence of graphical representations of musical notes is displayed according to the first gameplay string.

2. The computer-implemented method of Claim 1 , wherein the music data file comprises a digital sheet music file.

3. The computer-implemented method of Claim 1, wherein the notation string comprises a sequence of triples which represent the plurality of musical notes.

4. The computer-implemented method of Claim 1 further comprising: executing a fingering rules engine to generate the additional finger information for each of the plurality of musical notes, wherein the fingering rules engine determines preferred fingerings for each of the plurality of musical notes based on at least one of: relationships among the plurality of musical notes, or additional information of the music data file.

5. The computer-implemented method of Claim 1 wherein the additional finger information includes at least: string, fret position, or finger.

6. The computer-implemented method of Claim 1 further comprising: further parsing the music data file to generate at least one of: tempo information, time signature information, or tolerance information.

7. The computer-implemented method of Claim 1 , wherein the first gameplay string comprises at least portion of the notation string.

8. The computer-implemented method of Claim 1 , wherein executing the video game configuration engine comprises: determining the first gameplay string based on one or more game configuration rules associated with the first video game level type.

9. The computer-implemented method of Claim 1 , wherein the first video game configuration further comprises a second gameplay string, and wherein executing the video game configuration engine comprises: determining, based on one or more game configuration rules associated with the first video game level type, a first portion of the notation string for generating the first gameplay string; anddetermin i n g, based on the one or more game configuration rules associated with the first video game level type, a second portion of the notation string for generating the second gameplay string.

10. The computer-implemented method of Claim 1 further comprising: further executing the video game configuration engine on at least the notation string to generate a second game configuration comprising at least: a second video game level type, and a third gameplay string, wherein the third gameplay string is determined by the video game configuration engine based on one or more game configuration rules associated with the second video game level type.

11. The computer-implemented method of Claim 1 , wherein the video game configuration engine is expandable to automatically generate a plurality of video game configurations corresponding to a plurality of video game level types based on at least the notation string.

12. The computer-implemented method of Claim 1, wherein: the first video game level type is platformer, the sequence of graphical representations of musical notes comprises a sequence of graphical platforms, sizes of the sequence of graphical platforms are determined by the video game engine based on durations associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, and relative locations of the sequence of graphical platforms are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string.

13. The computer-implemented method of Claim 12, wherein the sequence of graphical representations of musical notes further comprises: a visualization of a plurality of strings, and a sequence of icons that scroll along the plurality of strings, wherein: locations of the sequence of icons on the plurality of strings are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, colorings of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string, and number indicators of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string.

14. The computer-implemented method of Claim 1, wherein: the first video game level type is whackanote, the sequence of graphical representations of musical notes comprises a sequence of highlighted strings and / or icons, andcolorings and / or number indicators of the sequence of highlighted strings and / or icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the first gameplay string.

15. The computer-implemented method of Claim 1, wherein: the first video game level type is RPG, the first video game configuration further comprises a plurality of gameplay strings including the first gameplay string, the sequence of graphical representations of musical notes comprises: a visualization of a plurality of strings, and a sequence of icons that scroll along the plurality of strings, and the first interactive video game level further comprises a game character and a plurality of game character positions.

16. The computer-implemented method of Claim 15, wherein: locations of the sequence of icons on the plurality of strings are determined by the video game engine based on pitches associated with respective musical notes of the plurality of musical notes as indicated by at least one gameplay string of the plurality of gameplay strings, colorings of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the at least one gameplay string of the plurality of gameplay strings, and number indicators of the sequence of icons are determined by the video game engine based on additional finger information associated with respective musical notes of the plurality of musical notes as indicated by the at least one gameplay string of the plurality of gameplay strings.

17. The computer-implemented method of Claim 15, wherein: the plurality of gameplay strings correspond to the plurality of game character positions, and the video game engine is configured to move the game character among the plurality of game character positions based on success or failure of a player at completing the plurality of gameplay strings.

18. The computer-implemented method of Claim 1 , wherein the video game engine is configured to determine, based on chords included in the first gameplay string, at least one of: background music for the first interactive video game level, or player checkpoints for the first interactive video game level19. A system comprising: one or more computer-readable storage mediums comprising program instructions; and one or more processors configured to execute the program instructions to cause the system to perform the computer-implemented method of Claim 1 .

20. One or more computer-readable storage mediums comprising program instructions, the program instructions executable by one or more processors to cause the one or more processors to perform the computer- implemented method of Claim 1.

Images