Dynamic music creation in games

Abstract

A method and system for dynamic music creation is disclosed. An emotion is assigned to one or more musical ideas and a game vector is associated with the emotion. Based on the emotion, the one or more musical ideas are mapped to the game vector. A music composition is generated based on the game vector and the desired emotion.

Description

Technical Field

[0001] The present disclosure relates to the fields of music composition, music arranging, machine learning, game design, and psychological mapping of emotions. Background Art

[0002] Games have always been dynamic pursuits, where gameplay responds to the player's actions. As games become more cinematic and immersive, the importance of music continues to grow. Currently, the majority of music in games is created from pre-written segments (often pre-recorded) that fit together like puzzle pieces. They are occasionally slowed down, sped up, pitch-shifted, and often layered on top of each other. Composers can guess at possible paths through gameplay, but because gameplay is interactive, much of it is unpredictable, and of course, the timing of most parts is rarely predictable.

[0003] In parallel, machine learning and artificial intelligence make it possible to generate content based on a training set of existing content labeled by human reviewers. In addition, there are large corpora of review data and sentiment mappings to various forms of artistic expression.

[0004] As another element, we are learning more and more about the players who are participating in the game. Players who have opted in can be tracked on social media, analysis of their personality can be made based on their behavior, and as more users have biometric devices that track them (galvanic skin activity, pulse and respiration, body temperature, blood pressure, brain wave activity, genetic predispositions, etc.), environmental customization can be applied to the music environment. Summary of the invention

[0005] The disclosure describes a mechanism for analyzing music, which separates music components (rhythm, time signature, melody structure, mode, harmony structure, harmony density, rhythm density and timbre density) so that the human viewpoints about concerts, records, etc. expressed based on the comments and social media released are mapped to emotional components individually and in combination. This is done on both the macroscopic and microscopic levels in the music work. Based on this training set, a fader (or a virtual fader in the software) is given emotional components such as tension, power, joy, surprise, softness, detachment, peace, nostalgia, sadness, synaesthesia, fear. These music components are mapped with reference to the music ideas (motif) created for each element / participant of the game, and the element / participant includes but is not limited to role (leader, partner, main enemy, wizard, etc.), activity type (fighting, resting, planning, hiding, etc.), region (forest, city, desert, etc.), the character of the person playing the game, etc. These music ideas can be melody, harmony, rhythm, etc. Once the composer has created musical ideas and assigned intended emotions to faders, a game simulation can be run where the composer selects musical idea combinations and mood mappings and applies them to various simulations. These simulations can be described a priori, or generated using similar algorithms to map the game to similar emotional environments. Actual physical faders (such as those used in computerized audio mixing consoles) may make the process of mapping emotions to situations more intuitive and instinctive, and the physicality will produce serendipitous results (e.g., even for a tense environment, raising the synaesthesia fader may have a better and more interesting effect than raising the tension fader). BRIEF DESCRIPTION OF THE DRAWINGS

[0006] The above and additional objects, features and advantages of the present invention will become apparent upon consideration of the following detailed description of certain specific embodiments of the present invention, particularly when taken in conjunction with the accompanying drawings, in which like reference numerals are used to designate like parts throughout the various drawings, and in which:

[0007] Figure 1 is a block diagram depicting an overview of a dynamic music creation architecture in accordance with aspects of the present disclosure.

[0008] Figure 2 is a block diagram illustrating the collection and analysis of music elements for generating a music element corpus in accordance with aspects of the present disclosure.

[0009] Figure 3 is a block diagram depicting the collection and analysis of reviews and comments for generating a corpus of annotated performance data in accordance with aspects of the present disclosure.

[0010] Figure 4 is a representative depiction of an implementation of mood parameters for music according to aspects of the present disclosure.

[0011] Figure 5 is a systematic view of using music and emotional commentary elements to train a model according to aspects of the present disclosure.

[0012] Figure 6 is a block diagram illustrating the creation of an annotated collection of game elements in accordance with aspects of the present disclosure.

[0013] Figure 7 is a block diagram illustrating the creation and mapping of music elements with reference to game elements using virtual or real faders in accordance with aspects of the present disclosure.

[0014] Figure 8 is a block diagram depicting the interaction of faders and switches with a game context playback engine in accordance with aspects of the present disclosure.

[0015] Fig. 9 is a block diagram illustrating how a prompt display may be used when mapping gaming and music elements by previewing upcoming events in accordance with aspects of the present disclosure.

[0016] Fig.10 is a block diagram illustrating how a prompt display may be used to enable foreshadowing to manage upcoming music and game elements in accordance with aspects of the present disclosure.

[0017] Fig.11 Depicted is a schematic block diagram of a system for dynamic music creation in games in accordance with aspects of the present disclosure.

[0018] Fig. 12A is a simplified node graph of a recursive neural network used in dynamic music creation in games in accordance with aspects of the present disclosure.

[0019] Fig. 12B is a simplified node graph of an unfolded recurrent neural network for use in dynamic music creation in games in accordance with aspects of the present disclosure.

[0020] Fig. 12C is a simplified diagram of a conventional neural network used in dynamic music creation in games in accordance with aspects of the present disclosure.

[0021] Fig.12D is a block diagram of a method for training a neural network in dynamic music creation in games in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0022] Although the following specific embodiments include many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are described without loss of generality and without implying limitations to the claimed invention.

[0023] Ingredients Overview

[0024] from Figure 1 As you can see from the diagram, there are many components that make up the complete system. The details are covered below, but it is useful to view the complete system at a high level. The components break down as follows:

[0025] First, music mood data must be constructed: Music data can be collected from a corpus of written and recorded music 101. This can be a score, a transcription created by a person, or a transcription created by an intelligent software system. Next, the system must break down the music data into its individual components - melody, harmony, rhythm, etc. 102, and these components must be stored as metadata associated with the work or part of the work. Now, in order to determine the emotional components or tags associated with each musical element, we will rely on the wisdom of the crowd. Obviously, this is accurate because it is the human emotion related to the music we are trying to capture. We will use reviews, blog posts, album annotations and other forms of comments to generate emotional metadata for the entire work and individual parts 103. The next step is to map the music metadata to emotional metadata 104. At this point, we will have a fairly large corpus of music data associated with emotional data, but it will never be complete. The next step is to use a convolutional neural network (CNN) to compare the actual music metadata with the analysis of the emotional components 105. This CNN can then be used to suggest emotional associations for music that the model has not been previously trained on. The accuracy can then be checked. Review and repeat – As iterations continue, accuracy will improve.

[0026] The next step is to analyze the game environment. The game's stages, component emotions, and features must first be collected and mapped. These scenes and components include things like locations, characters, environments, actions like combat, spellcasting, and so on 106. The next step is to map the game components to emotional metadata components 107.

[0027] Once the game environment is mapped out, it is time to write the musical idea 108. Ideas can be written for all key characters, scenes, moods, or any recurring elements of the game. Once the basic idea is written, faders (virtual or real) can be used to map the musical mood markers to game markers 109. The personality or behavioral components of the user (game player) can also be mapped as additional emotional characteristics to be considered 110. Next, continue testing, reviewing, and iterating, trying out the model on different situations in the game under development 111.

[0028] A more detailed description of the various processes follows.

[0029] Music component analysis

[0030] Several elements are needed to analyze the composition of music. Figure 2 . First, there is a large corpus of sheet music. Scores are available in printed form201. These printed scores can be scanned and analyzed to convert them into electronic versions of the scores. Any form of electronic notation can be used (MIDI, Sibelius, etc.). There are also scores available in digital form202, which can be added to the corpus. Finally, we can use machine learning to transcribe music into its musical elements (including the score)203. Machine learning methods can generate scores for music that may not be in notated form. This would include multiple versions of the same song by the same artist. The advantage of analyzing live and recorded performances is that they can be mapped in more detail, including things like changes in tempo and dynamics; and for improvised material, many other variables can be taken into account. We have to keep track of the analysis of different performances204 so that we know how performances differ, for example a performance of Beethoven’s 5th Symphony conducted by Leonard Bernstein with the New York Philharmonic in 1964 will be different from a performance of the same piece by Seiji Ozawa with the Boston Symphony Orchestra a few years later. These scores have to be analyzed for meter205, rhythmic components206, melody207, harmonic structure208, dynamics209, harmonic and improvisational elements210. All of this is now stored in a detailed corpus of musical performances211.

[0031] Once a larger corpus of music has been subjected to in-depth mechanical analysis, this analysis is then mapped to the reviews of these works using descriptions from the reviews and analysis of the literature about the works. This can be done in Figure 3306. The reviews and analyses may be in text, auditory, or visual form, and include, but are not limited to, articles 301, reviews 302, blogs 303, podcasts 304, album notes 305, and social media comments 306. From this larger corpus of music analysis 307, individual works and parts of works will be described 308. In addition, these analyses must have a context 309 within the period in which the works were written and performed. For example, one might look at the works of Mozart and read the descriptions in the literature of his works (which may be different periods, such as Mozart works performed in the 20th century). Next, known structural music analysis 310 is applied to determine which works and / or parts of works or melodic phrases are beautiful, heroic, melancholic, harsh, sad, solemn, powerful, expressive, etc. This may involve looking at the music literature and applying general principles of compositional development to the analysis. For example, a melodic leap and then a reversal of direction step by step is considered beautiful. Melodic intervals may be associated along a pleasant, unpleasant, or natural scale (e.g., a minor ninth jump is unpleasant, a fifth is heroic, and a pitch within a key symbol is natural). All of this performance and composition data may be used to map emotional keywords to performances and segmentations 311.

[0032] The “emotion” words might come from a process of digitally parsing the reviews, and could include, and any human analysis would almost certainly include, the following words: intense, powerful, joyful, surprising, soft, detached, peaceful, nostalgic, sad, and synaesthesia.

[0033] Note that for these keywords to work, it doesn't necessarily matter if they are accurate, just that they have a consistent effect. In actual use of the final system, it won't matter whether the faders associated with the word synaesthesia actually create more sensory music, as long as it is predictable and emotionally understandable to the composer. There are two reasons for this: 1) the labels can always be changed to more intuitive names; and 2) humans are very adaptable when processing music and sound (in the history of music synthesizers, despite meaningless labels, knobs, buttons and faders, such as going back to the Yamaha DX-7, did not in any way affect the sound that was intuitive on the basis of the name, but rather had the effect of becoming part of the musician's sensory memory and muscle memory, and therefore easy to use).

[0034] Detailed analysis of musical compositions and components is placed in a corpus of annotated performance data312.

[0035] Note that mapping musical elements to create a corpus of musical performances need not be and will not likely be a one-to-one mapping, but rather set on a scale of mood continuum, such as a set of musical vectors. For example, a piece might be an 8 out of 10 on the scale of synaesthesia, or a 4 out of 10 on the scale of sadness. Once the musical elements are mapped to these mood vectors, the model can be tested on titles that were not in the training dataset. One can listen to the results and fine-tune the model until it is more accurate. Eventually, the model will become very accurate at some point. Different models can be run in different time frames to create 50s style horror music or 21st century style horror music, keeping in mind that the actual descriptors are not as important as the categorical groupings (i.e., what is mellow for one composer may be boring for another).

[0036] Note that the model not only maps compositions and parts to mood vectors, but also maps the compositional melody, rhythm, and chord progression to those same mood vectors. At the same time, the machine learns melodic structure (e.g., a melodic leap up and then down a step is often considered beautiful, a leap of a minor ninth is often considered dissonant, etc.).

[0037] Structural music analysis 310 may involve multiple components. Figure 4 As shown, beauty, dissonance, and all subjective descriptive measures can be used as a guide for composition in due course. For example, what was considered dissonant in Bach's time (e.g., the major seventh interval was considered too dissonant to play unless it was moved into the seventh from a previous more harmonious chord) was considered beautiful in a later era (today, major seventh chords are considered normal and sometimes even artificially emotive).

[0038] In one existing music emotion classification method 401, the emotions of songs are classified according to the traditional emotion model of psychologist Robert Thayer. The model divides songs from happy to sad and from calm to energetic along the lines of energy and pressure, respectively. The eight categories created by Thayer's model include the extreme values ​​of the two lines and each of the possible intersections of the lines (e.g., happy-energetic or sad-calm).

[0039] This historical analysis may be of limited value, and the methods described herein may be more nuanced and flexible. Due to the size of the corpus, and therefore the size of the training data set, a richer and more nuanced analysis may be applied.

[0040] Some of the components to be analyzed 402 may include, but are not limited to, harmonic groupings, modes and scales, time signatures, beats, harmonic density, rhythmic density, melodic structure, phrase length (and structure), dynamics, phrasing and compositional techniques (modulation, inversion, retrograde, etc.), groove (including rhythms such as funk, lounge, Latin, reggae, swing, tango, merengue, salsa, fado, 60s disco, 70s disco, heavy metal, etc., there are hundreds of established rhythmic styles).

[0041] Training the model

[0042] Just as people label photos of peaches to train a convolutional neural network to recognize photos of peaches when it is shown photos of peaches it has never seen before, data about consumer and reviewer opinions can be used to train a machine learning model that can then be used to classify new material. Figure 5 An example of this process is shown in In the implementation shown, we can use the wisdom of the crowd (e.g., comments, posts, etc.) to train our model to identify the emotions associated with different music segments.

[0043] exist Figure 5 In the implementation shown, the corpus of music data is divided into three buckets. The first bucket, Corpus A 501, is a set of music data 502 for which collected emotional descriptions 503 have been mapped based on analysis of audience comment data. The second bucket, Corpus B 506, is a set of music data 507 for which its own set of collected emotional descriptions 508 have been mapped. The third bucket, Corpus C, is a set of music data 511 for which no collected emotional descriptions exist. The model is initially trained on Corpus A. Convolutional neural networks 504 are used to map the collected emotional descriptions 503 to the music data 502. The result of this mapping is a trained engine 505 that should be able to derive emotional descriptions from the music data. The music data 507 of Corpus B 506 is then fed into the trained engine 505. The resulting output is a predicted emotional description 509 for the music of Corpus B. These predictions are then compared to the collected descriptions 510 to determine the accuracy of the predictions. This prediction, comparison, and iteration process continues (e.g., by moving music elements from Corpus B into Corpus A and repeating) until the predicted mood descriptions 509 for Corpus B music closely match the actual results. This process can continue indefinitely, with the system becoming increasingly intelligent over time. After the trained engine 505 has gone through enough iterations, it can be used as a prediction engine 512 on Corpus C 511 to predict mood descriptions 513 for Corpus C music.

[0044] Mapping Game Elements

[0045] In order to match dynamic musical elements to different elements in the game, a corpus of elements that might require music or musical changes can be created. There are many different elements that can influence how the music should sound at that moment. These elements are used by the game developers, of course, and can be made available to composers and sound designers to make the music match. Figure 6 As can be seen in the 601 diagram, there are many game elements to track and map. These include, but are not limited to, game characters 601, game environments or locations 602, game moods or tones 603, and perhaps most importantly, game events 604. An event in a game can be anything from the appearance of an enemy or friend to a change of position, to a fight, to casting a spell, to a match (and many more). All of these game elements 605 and their triggers (inputs or outputs) and modifiers are collected. In addition, these can be collected as game vectors 606. They are called vectors because many of the components have multiple states. For example, a venue at night may be different from a venue during the day, and a venue in the rain may be different from a venue in the sun. The enemy's abilities may be variable, increasing or decreasing based on time, location, or level. From this perspective, any element can have an array of vectors. All game elements and their vectors have emotions associated with them. Emotions can change with the value of the vector, but despite this, emotions can be associated with game elements. Once a game element is mapped to its emotion 607, we can store this relationship in a corpus or collection of annotated game elements 608.

[0046] Gather the ingredients needed to feed the environment

[0047] Eventually, the composition tool could be used to create music from the gameplay itself, but this embodiment focuses on creating music from composition primitives. The composer would create musical elements or musical ideas and associate them with characters or elements in the game.

[0048] The use of tools here need not be used in isolation, but are often used to enhance traditional notation techniques, where pre-recorded music is combined with multiple components in sequence and in layers, which are mixed together to create a whole. That is, today, different layers may start at different times, and some layers will be superimposed on other layers, and still others may run independently of other layers. The idea here is to develop additional tools that can be used 1) as a supplement to existing techniques (e.g., on top of existing techniques), 2) in place of existing techniques in some or all places, or 3) as a mechanism for informing the use of existing techniques. In addition, these mechanisms can be used to create entirely new forms of interactive media (e.g., people trying to control their blood pressure or brain wave states can use musical feedback as a training tool, or even use biometric identification markers as a composition tool.

[0049] In traditional composition research, music idea refers to little melody segmentation usually.But, under this background, music idea can be melody segmentation, harmony structure, rhythm structure and / or specific tone.Music idea can be created for each element / participant of as many games as needed, and described element / participant includes but is not limited to character (leader, partner, main enemy, wizard etc.), activity type (fighting, resting, planning, hiding etc.), area (forest, city, desert etc.), character (young, old, male, female, introvert, extrovert etc.) of the people who play games.Music idea can be melody, harmony, rhythm etc.In addition, single element can have multiple music ideas, for example can be used alone or together with rhythmic pattern and melody pattern, or same role may have the sad music idea and happy music idea that use in different situations.

[0050] Once the composer has created the musical idea, it can be assigned to an element / character. This can be done by the composer or the sound designer, and can change as the game is developed, or even dynamically changed within the game after release.

[0051] Use tools to map musical components to game elements

[0052] The aforementioned aspects can be combined to map music components to game elements. Throughout the document, it should be assumed that references to faders and buttons can be real physical buttons or faders, as well as virtual buttons or faders. Intuitively, based on the experience of musicians and composers, it is expected that physical buttons and faders will be more intuitive to use and may have better results. Actual physical faders (such as those used in computerized audio mixing consoles) may, even more likely, make the process of mapping emotions to situations more intuitive and instinctive, and physicality will produce accidental results (for example, even for a tense environment, raising the synaesthesia fader may have a better and more interesting effect than raising the tension fader). However, for the purposes of this application, either one may work.

[0053] An example of a possible logical sequence of events is Figure 7 As shown, however any order will produce results, and the process will undoubtedly be iterative, recursive, and non-linear.

[0054] Figure 7 The logical sequence depicted in may begin with a collection 701 of annotated game elements, which may be obtained from Figure 6 , a collection of annotated game elements in 608 is transferred in. Then, at 702, game triggers are assigned to switches. These triggers can be the appearance of an enemy or obstacle or a level increase, among others. Next, a primary emotion vector 703 is selected, and a visual tag is assigned at 704 so that the element type can be displayed to the composer. Then, the game elements are assigned to the emotion elements in a multidimensional array, as shown at 705. Then, at 710, the multidimensional array is assigned to the faders and switches, which are used to map the music emotion tags to the game tags. Once a set of emotion tags is established for the game elements, music can be applied to the game elements. The composer writes the musical idea 707. As mentioned above, the musical idea can represent a character, an event, an area, or an emotion. Next, we select the baseline emotions of the musical idea 708. These are the default emotions. For example, whenever a wizard appears, there may be a default musical idea with its default emotions. These can vary based on the situation, but if no variables are applied, they will operate in their default mode. Now, we assign the emotion vector to the fader 709. Now that the game vector has been assigned and the music vector has been assigned, at 710, the music mood markers can be mapped to the game markers using the faders and switches. The faders do not have to correspond to a single mood. Because this is like fine-tuning the timbre of an instrument, the composer can be creative. Perhaps a fader of 80% heroic and 20% sad will produce an unexpected and pleasing result.

[0055] Next, the various themes could be mapped to a set of buttons (perhaps in a color matrix so it's easy to see many at once). Some of these buttons might be the same as game switches, but others would be music switches. These buttons could be grouped by character type (hero, villain, wizard, etc.), musical composition (e.g. groove on one side and melody on the other, with the key across the top), and setting (urban, rural, etc.).

[0056] Note that many of the game triggers that have been mapped to buttons or switches will ultimately be "driven" by the gameplay itself, but in the early stages it will be useful to be able to simulate things like the arrival of an enemy, or the sunrise, or impending doom.

[0057] Now, game simulations can be run live into early versions of the game, even if it's just a storyboard, or even without a storyboard just to write music for different situations. Composers can choose musical ideas and mood mappings and apply them to various simulations and test drive them. These situations can be fine-tuned as the game develops.

[0058] In fact, this can also be used as generalized composition, where a composer or performer can use a machine to create music from primitives.

[0059] Program your music with faders and switches

[0060] Figure 8 Examples of how game simulation and music programming can actually be used are shown. Figure 7 As shown in Figure 8As can be seen in the figure, music 801, game triggering factor 802 and game composition 803 are mapped to fader and switch 804. These music 801, game triggering factor 802 and game composition 803 mapped to fader and switch 804 are inserted in the game situation playback engine 805. Now, the composer plays music 806 while testing various game situations, and is recorded by the music scene / event mapping module 808 of record. In order to see what the element is, the composer can check the prompt display screen 808, which illustrates which elements are being used or will appear. If the fader is touched, the element that it is mapped to may be highlighted on the screen to provide visual feedback. The prompt display can illustrate the elements from the game situation playback engine 805, because these elements are being played out from real-time fader and switch. This is represented by the module 806 that plays music while playing games and comes from the music scene / event mapping module 807 of record. Just like updating faders when mixing with automation faders, all faders and switches can be updated and modified in real time at 809 and remembered for later playback. Additionally, musical ideas, compositions and triggers can be updated at 810 and remembered by the system. Of course, there is unlimited undo, and different performances can be saved as different versions that can be used at different times and in combination with each other.

[0061] Fig. 9 Additional details about how the prompt monitor 907 can be used according to various aspects of the present disclosure are shown. Upcoming events 901, characters 902, and locations 903 can all be seen on the prompt monitor (there may be multiple prompt monitors). Based on the expected parameters of the game play process, there may be default weights 904. These and the fader (and switch) mapping matrix 905 are all displayed on one or more prompt monitors 907. The player history / prediction matrix 906 is another input to the fader mapping matrix 905 and is visible on the prompt monitor. Based on the player's play style or other factors (time of day, minutes or hours in the current session, game state, etc.), this matrix can automatically change the music, or change the music based on parameters set by the composer or sound designer.

[0062] Music that heralds change

[0063] In film, music often precedes visual changes. This foreshadowing or foreshadowing process is important for both emotional connection to the piece and preparing the viewer / listener for an emotional change or otherwise (even if it is a false prediction and takes the viewer by surprise). Now, when we are writing music in the moment, we will want to be able to foreshadow change. This may be associated with approaching the completion of a level or signaling the entry of a new character or environment (or setting up the viewer for one change but actually surprising them with another).

[0064] How would our composition engine effectively foreshadow? We could use triggers that are known in advance, and use faders or dials to control the timing of the foreshadow and the slope of the foreshadow. For example, if a timer is running out on a level, we could set the foreshadow to start 30 seconds before the timer ends, and use an exponential curve (such as y=2 x ) to increase the intensity. Many foreshadowings are possible, from possible fear overtones to happy predictions. Again, because this is also designed to be instinctive, it is very likely that this feature will end up having unexpected results, some of which will be useful to program into the game. See Fig.10 , a set of possible next events 1001 can be seen on the prompt matrix 1003. There is also a next event timing matrix 1002, which can be used to set timing values ​​and ramp characteristics. Another factor that may be involved in prediction is the probabilistic emotion weighting engine 1004. This engine weights the expected emotional state of upcoming events and is used to validate the predictive audio. It can be seen on the prompt monitor and its weighting effect can be changed. The likelihood of an upcoming event having any particular emotion and the predicted strength of said emotion are mapped by the event to emotion likelihood mapping engine 1005. Predictions are also influenced by player history and the player history / prediction matrix 1007, which can be seen on the prompt monitor and also feeds into the predictive matrix engine 1006. Although Fig.10 All elements of the composition can be seen in the Cue Monitor, but since the Premonition Matrix Engine will use machine intelligence to surface the most likely scenarios, they may be relegated to the Auxiliary Cue Monitor. The visible components can be set or programmed as desired, but with material coming in quickly (although, of course, the composer can have many preparatory moves), it is best for the learning engine to automate as many of these components as possible.

[0065] Use outside of games

[0066] The use of this composition tool is not limited to gaming use. Even in non-gaming use, interactive VR environments can take advantage of these technologies. Additionally, this could be used as a composition tool for scoring traditional TV shows or movies. And, one end use could be to use it in pure music creation to create the basis for an album or a hit song, etc.

[0067] system

[0068] Fig.11 A system for dynamic music creation in games according to aspects of the present disclosure is depicted. The system may include a computing device 1100 coupled to a user input device 1102. The user input device 1102 may be a controller, a touch screen, a microphone, a keyboard, a mouse, a joystick, a fader pad, or other device that allows a user to input information including sound data into the system. The system may also be coupled to a biometric device 1123, which is configured to measure skin electrodermal activity, pulse and respiration, body temperature, blood pressure, or brain wave activity. The biometric device may be, for example, but not limited to, a thermal or infrared camera pointed at a user and configured to determine the user's respiration and heart rate based on thermal signatures, for more information see Chen et al.'s commonly owned patent No. 8,638,364, "USER INTERFACE SYSTEM AND METHOD USING THERMAL IMAGING," the contents of which are incorporated herein by reference. Alternatively, the biometric identification device may be other devices such as, but not limited to, a pulse oximeter, a blood pressure cuff, an electroencephalograph, an electrocardiograph, a wearable activity tracker, or a smart watch with biometric sensing.

[0069] The computing device 1100 may include one or more processor units 1103, which may be configured according to well-known architectures (e.g., single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, etc.). The computing device may also include one or more memory units 1104 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).

[0070] The processor unit 1103 may execute one or more programs, portions of which may be stored in the memory 1104, and the processor 1103 may be operably coupled to the memory (e.g., by accessing the memory via the data bus 1105). The programs may be configured to generate or create music using a vocal idea 1108 based on a game vector 1109 and an emotion vector 1110 of a video game. A vocal idea may be a short musical idea composed by a musician, a user, or a machine. In addition, the memory 1104 may contain programs that implement the training of the sound classification and classification NN 1121. The memory 1104 may also contain one or more databases 1122 of annotated performance data and emotional descriptions. A neural network module 1121, such as a convolutional neural network for associating musical ideas with emotions, may also be stored in the memory 1104. The memory 1104 may store a report 1110 that lists items that were not identified by the neural network module 1121 as being in the database 1122. The vocal imagery, game vectors, emotion vectors, neural network modules, and annotated performance data 1108, 1109, 1121, 1122 may also be stored as data 1118 in a mass storage area 1118 or at a server coupled to a network 1120 accessed via a network interface 1114. Additionally, data for the video game may be stored in memory 1104 as data in a database or elsewhere or as program 1117 or data 1118 in a mass storage area 1115.

[0071] The overall structure and probabilities of the NN may also be stored as data 1118 in the mass storage area 1115. The processor unit 1103 is also configured to execute one or more programs 1117 stored in the mass storage area 1115 or the memory 1104, which cause the processor to perform methods for dynamic music creation using musical ideas 1108, game vectors 1109, and emotion vectors 1110, as described herein. The music generated from the musical ideas may be stored in the database 1122. In addition, the processor may perform methods described herein for NN 1121 training and classification of musical ideas to emotions. The system 1100 may generate neural networks 1122 as part of the NN training process and store them in the memory 1104. The complete NN may be stored in the memory 1104 or in the mass storage area 1115 as data 1118. In addition, the NN 1121 may be trained using actual responses from a user, wherein the biometric recognition device 1123 is used to provide biofeedback from the user.

[0072] The computing device 1100 may also include well-known support circuits, such as input / output (I / O) 1107, circuits, power supply (P / S) 1111, clock (CLK) 1112, and cache 1113, which may communicate with other components of the system, for example, via bus 1105. The computing device may include a network interface 1114. The processor unit 1103 and the network interface 1114 may be configured to implement a local area network (LAN) or PAN via a suitable network protocol (e.g., Bluetooth) for a personal area network (PAN). The computing device may optionally include a mass storage device 1115 (such as a disk drive, CD-ROM drive, tape drive, flash memory, etc.), and the mass storage device may store programs and / or data. The computing device may also include a user interface 1116 to facilitate interaction between the system and the user. The user interface may include a monitor, a television screen, a speaker, headphones, or other devices that convey information to the user.

[0073] The computing device 1100 may include a network interface 1114 to facilitate communication via an electronic communications network 1120. The network interface 1114 may be configured to enable wired or wireless communication over local area networks and wide area networks such as the Internet. The device 1100 may send and receive data and / or requests for files via one or more message packets over the network 1120. Message packets sent over the network 1120 may be temporarily stored in a buffer 1109 in the memory 1104. Annotated performance data, vocal music ideas, and annotated game elements may be obtained over the network 1120 and stored in part in the memory 1104 for use.

[0074] Neural network training

[0075] Typically, a neural network for dynamic music generation may include one or more of several different types of neural networks and may have many different layers. By way of example and not limitation, a classification neural network may be composed of one or more convolutional neural networks (CNNs), recurrent neural networks (RNNs), and / or dynamic neural networks (DNNs).

[0076] Fig. 12A The basic form of an RNN is depicted with a layer of nodes 1220, each of which is characterized by an activation function S, an input weight U, a recursive hidden node transfer weight W, and an output transfer weight V. The activation function S can be any nonlinear function known in the art and is not limited to a hyperbolic tangent (tanh) function. For example, the activation function S can be a Sigmoid or ReLu function. Unlike other types of neural networks, an RNN has a set of activation functions and weights throughout the layer. As Fig. 12BAs shown, the RNN can be considered as a series of nodes 1220 with the same activation function moving in time T and T+1. Therefore, the RNN maintains historical information by feeding the results from the previous time T to the current time T+1.

[0077] In some embodiments, a convolutional RNN can be used. Another type of RNN that can be used is a long short-term memory (LSTM) neural network, which adds storage blocks in the RNN nodes with input gate activation functions, output gate activation functions, and forget gate activation functions to form a gated memory, which allows the network to retain some information for a longer period of time, as described in Hochreiter and Schmidhuber, "Long Short-term Memory" (Neural Computation 9(8): 1735-1780 (1997)), which is incorporated herein by reference.

[0078] Fig. 12C An exemplary layout of a convolutional neural network, such as a CRNN, according to aspects of the present disclosure is depicted. In this depiction, a convolutional neural network is generated to train data in the form of an array 1232 (e.g., having 4 rows and 4 columns, providing a total of 16 elements). The depicted convolutional neural network has filters 1233 of size 2 rows by 2 columns (whose skip value is 1) and channels 1236 of size 9. For clarity, the Fig. 12C Only the connection 1234 between the first column channel and its filter window is depicted. However, aspects of the present disclosure are not limited to such implementations. According to aspects of the present disclosure, the convolutional neural network implementing classification 1229 can have any number of additional neural network node layers 1231 and can include such layer types of any size such as additional convolutional layers, fully connected layers, pooling layers, maximum pooling layers, local contrast normalization layers, etc.

[0079] As in Fig.12D As seen in , training a neural network (NN) begins with initializing the NN's weights 1241. Typically, the initial weights should be randomly assigned. For example, a NN with a tanh activation function should have a distribution over and A random value between , where n is the number of inputs to the node.

[0080] After initialization, activation functions and optimization procedures are defined. The NN is then provided with a feature vector or input data set 1242. Each of the different feature vectors can be generated by the NN from an input with a known label. Similarly, the NN can be provided with a feature vector corresponding to an input with a known label or classification. The NN then predicts the label or classification of the feature or input 1243. The predicted label or class is compared with the known label or class (also called the ground truth), and the loss function measures the total error between the prediction and the ground truth over all training samples 1244. By way of example and not limitation, the loss function can be a cross entropy loss function, a quadratic cost, a triplet contrast function, an exponential cost, etc. A plurality of different loss functions can be used depending on the purpose. By way of example and not limitation, to train a classifier, a cross entropy loss function can be used, while to learn a pre-trained embedding, a triplet contrast function can be used. The NN is then optimized and trained 1245 using the results of the loss function and using known training methods for neural networks (such as back propagation with adaptive gradient descent, etc.). At each training epoch, the optimizer tries to pick the model parameters (i.e., weights) that minimize the training loss function (i.e., the total error). The data is divided into training samples, validation samples, and test samples.

[0081] During training, the optimizer minimizes the loss function on the training samples. After each training epoch, the model is evaluated on the validation samples by calculating the validation loss and accuracy. If there are no significant changes, the training can be stopped and the resulting trained model can be used to predict the labels of the test data.

[0082] Therefore, a neural network can be trained from inputs with known labels or classifications to recognize and classify those inputs.

[0083] Although the above is a complete description of the preferred embodiment of the present invention, it is possible to use various alternatives, modifications and equivalents. Therefore, the scope of the present invention should not be determined with reference to the above description, but should instead be determined with reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, Indefinite article "a" or "a kind" It refers to the quantity of one or more of the item following the article unless explicitly stated otherwise. The appended claims should not be read as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A method for dynamic music creation, comprising: a) assigning emotions to one or more musical ideas; b) associating a game vector with the emotion, wherein the game vector is determined based on a game element; c) mapping the one or more musical ideas to the game vector based on the emotion; d) Generating a music composition based on said game vector and the desired emotion.

2. The method of claim 1, wherein the emotions are assigned to the one or more musical ideas by a neural network trained to assign emotions to music based on content that expresses human opinions about the music.

3. The method of claim 1, wherein a rating is assigned to the emotion, and the mapping of the one or more musical notes to the game vector varies with the rating assigned to the emotion. The method of claim 3 , wherein an emotion rating slider is used to change the rating assigned to the emotion.

5. The method of claim 1, further comprising associating a user behavior with the emotion, and mapping the one or more musical ideas to the user behavior based on the emotion.

6. The method of claim 1, further comprising assigning the one or more musical ideas to the emotion based on a user's physical reaction to each of the one or more musical ideas.

7. The method of claim 1 further comprising assigning one of the one or more musical notations to a game vector as a default musical notation, and wherein the default musical notation is included in the musical composition whenever the game vector is present within a video game.

8. The method of claim 1, wherein the game vector comprises a game element and a trigger for the game element within a video game.

9. The method of claim 8, further comprising assigning one of the one or more musical notations to a game element as a default musical notation for the game element, and including the default musical notation in the musical composition whenever the game element is present within the video game.

10. The method of claim 8, further comprising assigning one of the one or more musical ideas to the trigger as a default musical idea for the trigger, and including the default musical idea in the musical composition whenever the trigger is activated within the video game.

11. The method of claim 1 , wherein generating a music composition comprises: A music theme of the one or more music themes is selected to be played while the video game is playing, wherein the music theme is played simultaneously with the video game.

12. The method of claim 1, further comprising generating an event cue for a video game, wherein the event cue comprises an upcoming event within the video game, and wherein the event cue is used during generation of the musical composition.

13. A method as claimed in claim 12, wherein the event prompt includes an event probability, an event timing, and an event emotion weight, wherein the event probability defines the likelihood that an event will occur within the event timing, and wherein the event emotion weight provides the emotion of the event. The method of claim 13 , wherein the event sentiment weight comprises a level of the sentiment.

15. A system for dynamic music creation, comprising: processor; a memory coupled to the processor; non-transitory instructions embedded in the memory, which when executed cause the processor to perform a method comprising: a) assigning emotions to one or more musical ideas; b) associating a game vector with the emotion, wherein the game vector is determined based on a game element; c) mapping one or more musical ideas to the game vector based on the emotion; d) Generating a music composition based on said game vector and the desired emotion.

16. The system of claim 15, wherein the emotions are assigned to the one or more musical ideas by a neural network trained to assign emotions to music based on content that expresses human opinions of the music.

17. The system of claim 15, wherein a rating is assigned to the emotion, and the mapping of the one or more musical notes to the game vector changes as a function of the rating assigned to the emotion.

18. The system of claim 17, further comprising a mood slider coupled to the processor, wherein the mood slider is configured to control the rating assigned to the mood.

19. The system of claim 15, further comprising assigning the one or more musical ideas to the emotion based on a user's physical reaction to each of the one or more musical ideas.

20. The system of claim 19, further comprising a biometric device coupled to the processor and configured to track the physical response of the user.

21. A non-transitory instruction, the non-transitory instruction being embedded in a computer-readable medium, the non-transitory instruction, when executed, causing a computer to perform a method comprising: a) assigning emotions to one or more musical ideas; b) associating a game vector with the emotion, wherein The game vector is determined based on the game element; c) mapping one or more musical ideas to the game vector based on the emotion; d) Generating a music composition based on said game vector and the desired emotion.

Images