Adaptive Audio Mixing

Abstract

A system, apparatus and method for performing adaptive audio mixing are disclosed. A trained neural network dynamically selects and mixes pre-recorded human-composed music stems arranged in a mutually compatible set. Stem and track selection, volume mixing, filtering, dynamic compression, acoustic / reverberation characteristics, segue, tempo, beat matching and cross-fade parameters generated by the neural network are inferred from game scene characteristics and other dynamically changing factors. The trained neural network selects pre-recorded stems of artists and mixes the stems in a unique way in real time to dynamically adjust and change background music based on factors such as the game scenario, the player's unique storyline, scene elements, player's profile, interests, performance, adjustments made to game controls (e.g., music volume), number of viewers, comments received, player popularity, player's native language, player's influence and / or other factors. The trained neural network generates unique music that dynamically changes according to real-time conditions.

Description

[Background technology]

[0001] Description of Related Art Video game music plays an important role in perceived quality and user engagement. Currently, commercially available video games use pre-recorded audio tracks composed by artists. However, these pre-recorded audio tracks result in a predictable audio experience that can become repetitive for users who play the same video game and traverse the same game scenes.

[0002] The advantages of the methods and mechanisms described herein may be better understood by referring to the following description in conjunction with the accompanying drawings. [Brief explanation of the drawings]

[0003] [Figure 1] FIG. 1 is a block diagram of one embodiment of a computing system. [Figure 2] FIG. 1 is a block diagram of one embodiment of a neural network. [Figure 3] FIG. 10 is a block diagram of another embodiment of a neural network. [Figure 4] FIG. 1 is a block diagram of one embodiment of an adaptive audio mixing neural network training system. [Figure 5] FIG. 1 is a block diagram of one embodiment of an adaptive audio mixing neural network training system. [Figure 6] FIG. 1 is a block diagram of an embodiment of an adaptive audio mixing system. [Figure 7] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for performing adaptive audio mixing. [Figure 8] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for dynamically generating audio output by a machine learning engine. [Figure 9]FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for training a neural network to generate dynamic non-deterministic audio mixes for video game applications. [Figure 10] FIG. 1 is a generalized flow diagram illustrating one embodiment of a method for influencing an adaptive audio mixing neural network based on measures of human emotion captured via user movements. DETAILED DESCRIPTION OF THE INVENTION

[0004] In the following description, numerous specific details are set forth to provide a thorough understanding of the methods and mechanisms presented herein. However, those skilled in the art should recognize that various embodiments may be practiced without these specific details. In some instances, well-known structures, components, signals, computer program instructions, and techniques have not been shown in detail to avoid obscuring the approaches described herein. It will be understood that for simplicity and clarity of illustration, elements shown in the figures have not necessarily been drawn to scale. For example, the dimensions of some elements may be exaggerated relative to other elements.

[0005] Various systems, devices, and methods for performing adaptive audio mixing are disclosed herein. In one embodiment, a trained neural network dynamically selects and mixes pre-recorded, human-composed music stems organized into mutually compatible sets. The stem and track selection, volume mixing, filtering, dynamic compression, acoustic / reverberation characteristics, segues, tempo, beat matching, and crossfade parameters generated by the neural network are inferred from game scene characteristics and other dynamically changing factors. The trained neural network selects pre-recorded tracks and / or stems from artists and mixes the stems in real time in a unique way to dynamically adjust and change background music based on factors such as the game scenario, the player's unique storyline, scene elements, player profile, interests, performance, adjustments made to game controls (e.g., music volume), number of viewers, comments received, player popularity, player's native language, player influence, and / or other factors. The trained neural network generates unique music that dynamically changes according to real-time conditions.

[0006] In one embodiment, the system generates a dynamic audio mix as the user plays the video game, and the mix is ​​not repetitive. In one embodiment, the system employs a heuristic algorithm that allows the composer to introduce random variations in how pre-recorded stems are mixed in different scenarios within the game. These different scenarios are based on what the user is doing in the game, scene elements, player profile, interests, how well the user is performing, whether the user is adjusting the volume, the number of viewers in the game, etc. These factors of how the user is interacting with the game are then used to vary the mixing parameters that generate the dynamic audio mix.

[0007] In one embodiment, an adaptive artificial intelligence (AI) engine receives pre-recorded, human-composed music stems arranged as a mutually compatible set. The AI ​​also receives metrics of user interaction with the game. The AI ​​engine performs stem and track selection, volume mixing / filtering, dynamic compression, and other steps to generate a dynamic audio mix. In one embodiment, the AI ​​engine is trained based on what the developer expects users to respond positively to. For example, if the beat track comes in a little too early, the tester may indicate that this was not the desired result. However, if the beat track comes in at the right time, the tester may indicate that this was a good result. This feedback is provided to the AI ​​engine, allowing it to adjust its settings and improve over time. This training method is a type of reinforcement learning.

[0008] 1 , a block diagram of one embodiment of computing system 100 is shown. In one embodiment, computing system 100 includes at least processors 105A-105N, input / output (I / O) interface 120, bus 125, memory controller 130, network interface 135, memory device 140, display controller 150, display 155, and speaker / headphones 160. In other embodiments, computing system 100 includes other components and / or is arranged differently. Processors 105A-105N represent any number of processors included in system 100.

[0009] In one embodiment, processor 105A is a general-purpose processor such as a central processing unit (CPU). In this embodiment, processor 105A executes drivers 110 (e.g., graphics drivers) for communicating with and / or controlling the operation of one or more of the other processors in system 100. Note that, depending on the embodiment, driver 110 may be implemented using any suitable combination of hardware, software, and / or firmware. In one embodiment, processor 105N is a data-parallel processor with a highly parallel architecture, such as a dedicated neural network accelerator or a graphics processing unit (GPU) that provides pixels to display controller 150 for transmission to display 155.

[0010] A GPU is a complex integrated circuit that performs graphics processing tasks. For example, a GPU performs graphics processing tasks required by end-user applications, such as video game applications. Increasingly, GPUs are also being used to perform other tasks unrelated to graphics. A GPU may be a separate device or may be included in the same device as another processor, such as a CPU. Other data-parallel processors that may be included in system 100 include digital signal processors (DSPs), field programmable gate arrays (FPGAs), and application-specific integrated circuits (ASICs). In some embodiments, processors 105A-105N include multiple data-parallel processors.

[0011] An emerging technology field is machine learning, and neural networks are a type of machine learning model. Neural networks have demonstrated superior performance in tasks such as handwritten digit classification and face detection. Other applications of neural networks include speech recognition, language modeling, sentiment analysis, text prediction, and the like. In one embodiment, processor 105N is a data-parallel processor programmed to execute a neural network application to generate a dynamically adaptive audio mix that is output on speakers / headphones 160 as part of a video game application. In this embodiment, the audio mix adapts to the game scenario by combining pre-recorded audio stems based on various real-time game scenarios and user parameters. As used herein, the term "audio stem" is defined as an individual audio track.

[0012] In one embodiment, real-time game scenario and user parameters are input to a trained neural network, which generates a mixing array of weights to apply to pre-recorded audio stems. Example network topologies for the trained neural network include generative-adversarial networks (GANs), attention models, transformer networks, recurrent neural networks (RNNs)-transducer networks, and their derivatives. The output of the trained neural network is a non-deterministic music mix that is integrated into the video game application in real time. In one embodiment, the trained neural network executes on one or more of processors 105A-105N. As used herein, the term "non-deterministic" is defined as having different characteristics in different iterations through the same scene in the video game application.

[0013] Memory controller 130 represents any number and type of memory controller accessible by processors 105A-105N. While memory controller 130 is shown as separate from processors 105A-105N, it should be understood that this represents merely one possible embodiment. In other embodiments, memory controller 130 may be embedded in one or more of processors 105A-105N and / or may be located on the same semiconductor die as one or more of processors 105A-105N. Memory controller 130 is coupled to any number and type of memory devices 140. Memory devices 140 represent any number and type of memory devices. For example, types of memory in memory devices 140 may include dynamic random access memory (DRAM), static random access memory (SRAM), NAND flash memory, NOR flash memory, ferroelectric random access memory (FeRAM), etc.

[0014] I / O interface 120 represents any number and type of I / O interface (e.g., a peripheral component interconnect (PCI) bus, a PCI-Extended (PCI-X), a PCI Express (PCIE) bus, a gigabit Ethernet (GBE) bus, or a universal serial bus (USB)). Various types of peripheral devices (not shown) are coupled to I / O interface 120. Such peripheral devices include, but are not limited to, displays, keyboards, mice, printers, scanners, joysticks, other types of game controllers, media recording devices, external storage devices, and the like. Network interface 135 can receive and transmit network messages over a network.

[0015] In various embodiments, computing system 100 is a computer, a laptop, a mobile device, a game console, a server, a streaming device, a wearable device, or any of a variety of other types of computing systems or devices. Note that the number of components of computing system 100 varies from embodiment to embodiment. For example, in other embodiments, more or fewer components are present than those shown in FIG. 1 . Note also that in other embodiments, computing system 100 includes other components not shown in FIG. 1 . Additionally, in other embodiments, computing system 100 is structured in a manner other than that shown in FIG. 1 .

[0016] Turning now to FIG. 2 , a block diagram of one embodiment of a portion of a neural network 200 is shown. The portion of neural network 200 shown in FIG. 2 includes convolutional layer 202, subsampling layer 204, convolutional layer 206, subsampling layer 208, and fully connected layer 210. Neural network 200 may include multiple groupings of layers similar to those shown, sandwiched together to create the overall structure of the network. Other groupings of layers that are part of neural network 200 may include other numbers and arrangements of layers other than those shown in FIG. 2 . Note that layers 202-210 are intended only as an example of a grouping of layers that may be implemented in a sequential manner in one particular embodiment. The arrangement of layers 202-210 shown in FIG. 2 does not exclude that other methods of stacking layers together may be used to create other types of neural networks.

[0017] When neural network 200 is implemented on a computing system (e.g., system 100 of FIG. 1), neural network 200 generates an adaptive non-deterministic audio mix using multiple audio stems based on real-time game scenario parameters and user-specific parameters of a user playing the video game application. The adaptive non-deterministic audio mix is ​​then integrated into the video game application and provided to the user via headphones, speakers, or other audio output devices.

[0018] Referring to FIG. 3, a block diagram of another embodiment of a neural network 300 is shown. The neural network 300 illustrates another example of a neural network that may be implemented on a computing system (e.g., system 100 of FIG. 1). In one embodiment, the neural network 300 is a recurrent neural network (RNN) and includes at least an input layer 310, a hidden layer 320, and an output layer 330. The hidden layer 320 may represent any number of hidden layers, each having any number of neurons. Neurons used in RNNs include long short-term memory (LSTM) and gated recurrent units (GRUs), among others. Also, any number and type of connections may exist between neurons in the hidden layers. Additionally, the number of backward connections between hidden layers 320 may vary from network to network. In other embodiments, the neural network 300 includes other arrangements of layers and / or other connections between layers that differ from those shown in FIG. 3. In some cases, neural network 300 may include any of the layers of neural network 200 (of FIG. 2). In other words, some or all of the convolutional neural networks (CNNs) may be combined with some or all of the RNNs to create a single neural network.

[0019] In one embodiment, neural network 300 processes an input data set to generate results data. In one embodiment, the input data set includes a plurality of real-time game scenario parameters and user-specific parameters of a user playing a video game. In this embodiment, the input data set includes a plurality of pre-recorded audio stems. In this embodiment, the results data indicates how to combine the pre-recorded audio stems to generate an adaptive non-deterministic audio mix that is played to the user while playing the video game. In other embodiments, the input data set and / or results data include any of a variety of other types of data.

[0020] 4, there is shown a block diagram of one embodiment of an adaptive audio mixing neural network training system 400. System 400 represents one example of a pre-deployment training system for use in generating a trained neural network from pre-deployment neural network 420. In other embodiments, other methods of generating a trained neural network may be employed.

[0021] In one embodiment, a game clip 410A is provided as input to a neural network 420, where the game clip 410A represents a real-life example of a user playing a video game or a simulation of a user playing a video game. The neural network 420 generates audio output 430 from audio stems 415A-415N based on the game scenario encountered in the game clip 410A. The audio output 430 is provided to a reinforcement learning engine 440, which generates feedback 450 that is provided to the neural network 420 as an indication of the quality of the audio output 430. In one embodiment, the reinforcement learning engine 440 includes a human tester (i.e., a musician, composer) who evaluates the audio output 430 generated by the neural network 420. In other embodiments, the reinforcement learning engine 440 may include any combination of human involvement and / or machine interpretation techniques, such as a trained discriminator or actor-critic used in a GAN to generate feedback 450.

[0022] In one embodiment, if the neural network 420 produces a relatively high-quality audio output 430, the reinforcement learning engine 440 will generate feedback 450 with a relatively high score, which will strengthen the existing parameters (i.e., weights) of the layers of the neural network 420. On the other hand, if the neural network 420 produces a relatively low-quality audio output 430, the feedback 450 will have a relatively low score, which will cause the neural network 420 to train the layers by adjusting the parameters to counteract the generated “error.” Subsequent game clips 410B-410N are similarly processed to continue training the neural network 420 by refining the parameters of the various layers. Training may occur over a series of epochs, where for each epoch, the entire or a subset of the training data set is repeated, often in random order of presentation, and the process of repeated training epochs continues until the network reaches a satisfactory level of accuracy. As used herein, an “epoch” is defined as one pass through the complete set of training data. Additionally, "subset" refers to the common practice of setting aside a portion of the training data to use for validation and test vectors.

[0023] Referring to FIG. 5, a block diagram of one embodiment of an adaptive audio mixing neural network training system 500 is shown. System 500 represents a real-time usage environment in which a neural network 520 is deployed as part of a video game application 540 in the field, continually adapting the layer weights of the neural network 520 to improve the adaptive audio mixing algorithm. These updated weights may be uploaded to the cloud, allowing these updates to be applied to other neural networks. Thus, after the neural network 520 is deployed, incremental training may continue to refine the characteristics of the neural network 520. This allows the neural network 520 to improve the production of audio output 530 to enhance the overall user experience.

[0024] Real-time parameters 510A-510N are those parameters collected in real time while a user is using video game application 540. Neural network 520 uses real-time parameters 510A-510N as inputs to its layers to generate audio output 530 from audio stems 515A-515N. Audio output 530 is then provided to video game application 540 and presented to the user. While the user is playing the video game, user interactions 550, such as the user's navigation of the video game application environment, the user's performance, any game setting changes the user makes, and the user's movements, are captured. These user interaction data sequences are provided to user engagement level measurement engine 555. In one embodiment, video game application 540 runs on game console 545. Game console 545 includes any of the components shown in system 100 (of FIG. 1) and other components not shown in system 100. In another embodiment, video game application 540 runs in the cloud as part of a cloud gaming scenario. Any of the other components shown in Figure 5 may be implemented locally on game console 545 or other computer hardware local to the user, and / or one or more of these components may be implemented in the cloud.

[0025] Engine 555 generates a measure of the level of user engagement, shown as level of engagement metric 560. In one embodiment, engine 555 is a trained neural network. Depending on the embodiment, the level of engagement metric may have a range of 0 to 1, 0 to 10, 0 to 63, or any other suitable range corresponding to the particular training algorithm. Level of engagement metric 560 is provided to neural network 520 as an indication of the measure of goodness of audio output 530. In various embodiments, level of engagement metric 560 is generated periodically and updates are provided to neural network 520.

[0026] The engagement level metric 560 is used to incrementally train the neural network 520 after deployment in the field. In one embodiment, the higher the value of the engagement level metric 560, the more positive feedback is provided to the neural network 520, indicating that the neural network 520 selected the correct combination of audio stems 515A-515N and the correct combination of processing techniques to produce the audio output 530. Also, in this embodiment, the lower the value of the engagement level metric 560, the more negative feedback is provided to the neural network 520, indicating that the neural network 520 did a poor job selecting the combination of audio stems 515A-515N and processing techniques to produce the audio output 530. This feedback, either positive or negative, which may vary throughout the time that the user is playing the video game application 540, will enable the neural network 520 to continue its training and perform better in future iterations when dynamically generating the audio output 530. In one embodiment, the learning rate of neural network 520 is kept within a programmable range to avoid making overly aggressive changes to the trained parameters in the field. The learning rate is a variable scaling factor that adjusts the amount of change applied to the trained parameters during these incremental training passes.

[0027] The neural network 520 may have different settings for different scenes, different video games, different players / users, and these settings may be preloaded based on where the user is navigating in the game, which video game the user is playing, etc. For example, during more dangerous portions of the game, the neural network 520 may have a first set of parameters for that layer. During less dangerous phases of the game, such as when the user is in a car, waiting at a restaurant or store, or reading a book, the neural network 520 may have a second set of parameters for that layer. The neural network 520 may have any number of different sets of parameters for individual games, which may be loaded and programmed into the layers in real time as different phases of the game are encountered. Each set of parameters is trained based on the level of engagement metric 560 received during the corresponding portion of the game, independently of how other sets of parameters are trained for each portion of the game. Also, based on these different parameters, different scenes may use different sets of stems from which the neural network 520 selects to generate the sound mix for that particular scene.

[0028] Turning now to FIG. 6, a block diagram of one embodiment of an adaptive audio mixing system 600 is shown. In one embodiment, the system 600 includes at least an audio mixing engine 620 and a trained neural network 640. FMOD® and WWise™ are commercially available examples of audio mixing engines used in games. The audio mixing engine 620 and the trained neural network 640 may be implemented using any combination of hardware and / or program instructions. For example, in one embodiment, the audio mixing engine 620 and the trained neural network 640 are implemented on one or more GPUs. In another embodiment, the audio mixing engine 620 and the trained neural network 640 are implemented on another type of parallel processor (e.g., FPGA, ASIC). In other embodiments, the audio mixing engine 620 and the trained neural network 640 may be implemented using other types of hardware devices (e.g., CPUs) and / or program instructions.

[0029] In one embodiment, the trained neural network 640 receives a plurality of game scenario parameters 630A-630N and a plurality of player profile parameters 635A-635N. The game scenario parameters 630A-630N are specific to the characteristics of the video game being played and the location of the user within the game. The game scenario parameters 630A-630N may include game scene characteristics such as the unique storyline experienced by the player, scene elements, etc. The player profile parameters 635A-635N may include characteristics such as the player's interests, the player's performance, the player's game control adjustments, the player's audience size, the type of comments the player is receiving, the player's popularity, the player's native language, and the player's influence. Some of the player profile parameters 635A-635N are static, while some of the player profile parameters 635A-635N change throughout the game.

[0030] A plurality of game scenario parameters 630A-630N and a plurality of player profile parameters 635A-635N are provided as inputs to a trained neural network 640. These inputs are processed by various layers of the trained neural network 640 to generate a set of audio mixing weights 645A-645N. The audio mixing weights 645A-645N represent any number of outputs generated by the trained neural network 640. In one embodiment, the audio mixing weights 645A-645N are arrays of values ​​that are applied to the audio stems 610A-610N. In this embodiment, there is an output 645A-645N corresponding to each audio stem 610A-610N that defines how these audio stems 610A-610N are mixed together by the audio mixing engine 620 to produce the audio output 650. In other embodiments, the weights 645A-645N are applied by the audio mixing engine 620 in other manners to generate the audio output 650 from the audio stems 610A-610N. In some embodiments, the weights 645A-645N include timing parameters that specify when the audio stems 610A-610N are mixed together to produce the audio output 650.

[0031] In one embodiment, the audio stems 610A-610N are pre-recorded audio tracks generated by a composer. The audio mixing engine 620 determines how to combine the audio stems 610A-610N based on weights 645A-645N generated by the trained neural network 640. For example, in one embodiment, each weight 645A-645N is applied to a corresponding audio stem 610A-610N to determine whether the audio stem is included in the mix. For example, in this embodiment, there is a one-to-one correspondence between the weights 645A-645N and the audio stems 610A-610N. Note that the audio mixing weights 645A-645N can change over time. In one embodiment, each weight 645A-645N is a binary value of 0 or 1, but in other embodiments, each weight 645A-645N can take on any value within a predetermined range. In other embodiments, audio mixing engine 620 applies weights 645A-645N to audio stems 610A-610N in other ways to produce audio output 650. Audio output 650 is then provided to a user via speakers, headphones, or another audio output device.

[0032] Referring to Figure 7, one embodiment of a method 700 for performing adaptive audio mixing is shown. For purposes of explanation, the steps in this embodiment and in Figures 8-10 are shown sequentially. However, it should be noted that in various embodiments of the described method, one or more of the described elements may be performed simultaneously, in a different order than shown, or omitted entirely. Other additional elements may also be performed as desired. Any of the various systems or devices described herein may be configured to perform method 700.

[0033] The system receives and stores pre-recorded audio stems (block 705). The system monitors video game states in real time (block 710). The system generates a numerical value representing one or more video game states being monitored in real time (block 715). For example, a user's performance can be monitored and converted into a predetermined range of scores. The predetermined range may be 0 to 63 in one embodiment, and a trained neural network converts the user's performance into the predetermined range, with 0 being the worst performance and 63 being the best performance. Other video game states can be converted into numerical values ​​in a similar manner.

[0034] The numerical values ​​are provided as inputs to a trained neural network (block 720). The trained neural network then processes the inputs to generate a mixing array of weights (block 725). The trained neural network then applies the mixing array of weights to the pre-recorded audio stems to generate an adaptive audio mix (block 730). The audio output is then played for the user as part of the video game experience (block 735). After block 735, method 700 returns to block 710, where the system continues to monitor the real-time video game state. The granularity at which updates to the adaptive audio mix are generated may vary from embodiment to embodiment. In one embodiment, if a change to the video game state greater than a threshold is detected, the adaptive audio mix is ​​updated by taking another pass through method 700.

[0035] In one example, the trained neural network may increase the volume level or clarity of stems containing percussion effects to reinforce an increased player excitement level determined by a numerically represented video game state. Or, in another scene, the mixing of string or woodwind sounds may be increased to reinforce a player's emotional response inferred from the numerical value. In another example, the trained neural network may reduce the volume when the numerical value is inferred to indicate that the player is distracted.

[0036] Turning now to FIG. 8 , one embodiment of a method 800 for dynamically generating audio output by a machine learning engine is shown. The machine learning engine receives or extracts one or more user-specific parameters (block 805). In one embodiment, the machine learning engine is a trained neural network. In other embodiments, the machine learning engine is implemented based on other types of artificial intelligence or reinforcement learning models. The user-specific parameters include, but are not limited to, a player's interests, a player's performance, a player's game control adjustments, a player's audience size, the type of comments the player is receiving, the player's popularity, the player's native language, and the player's influence. The machine learning model also receives or extracts one or more indicators of a real-time scenario of a user playing a video game application (block 810). The one or more indicators of a real-time scenario of a user playing a video game application include, but are not limited to, a unique storyline, scene elements, scene characters, and the overall level of a scene within a game structure experienced by the player.

[0037] The machine learning engine then generates a non-deterministic music mix by combining multiple pre-recorded music stems based on one or more user-specific parameters and one or more indicators of the real-time scenario (block 815). The host computing system or device then renders the non-deterministic music mix as audio output for the user playing the video game (block 820). After block 820, method 800 ends. Note that method 800 may be repeated periodically or at programmable intervals to update the non-deterministic music mix generated for the user. Also, the mix level may change over time. For example, the mix level may include a gradual increase or decrease in volume and / or equalization level for individual stems or the global mix.

[0038] Referring to FIG. 9 , one embodiment of a method 900 for training a neural network to generate a dynamic non-deterministic audio mix for a video game application is shown. The neural network receives a plurality of game clips from a user playing a video game (block 905). The neural network may include any number and types of layers, with the number and type of layers varying according to the embodiment. The game clips may be game clips recorded from past video game play or game clips received in real time from a user playing the video game. In some cases, a combination of past and real-time game clips may be received by the neural network. The neural network also receives and / or generates various game scenario parameters and user-specific parameters associated with the game clips (block 910). The neural network also receives a plurality of pre-recorded audio stems (block 915). In one embodiment, the pre-recorded audio stems are generated by a music composer and are used in conjunction with a particular video game application.

[0039] A forward propagation pass is then performed through the neural network to generate a dynamic non-deterministic audio mix by combining audio stems based on game scenario parameters and user-specific parameters (block 920). The neural network then receives feedback on the dynamic non-deterministic audio mix (block 925). In one embodiment, the feedback is generated by a tester, game composer, or other person who listens to and evaluates the dynamic non-deterministic audio mix. The feedback includes a score generated by the tester, who may adjust the score at different moments throughout the audio mix. In another embodiment, the feedback is generated in real time based on actions taken by a user playing the video game application. In this embodiment, the feedback is inferred based on user actions such as volume adjustments and performance improvements or declines. In a further embodiment, the discriminative network is pre-trained by a supervised method to recognize parameters used in an appropriate music mix for a scene to generate an initial starting point for the reinforcement learning path.

[0040] A backpropagation pass is then performed through the neural network to adjust the weights in the neural network layers based on the feedback (block 930). After block 930, method 900 ends. Note that method 900 may be repeated any number of times with different training sets to iteratively refine the weights of the neural network layers.

[0041] Turning now to FIG. 10 , one embodiment of a method 1000 for influencing an adaptive audio mixing neural network based on a measure of human emotion captured via user movements is shown. While a user is playing a video game, one or more sensors track the user's movements (block 1005). A user engagement level measurement engine (e.g., user engagement level measurement engine 555 of FIG. 5 ) then generates a measure of human emotion based on the tracked user movements (block 1010). For example, if the user's arms and / or legs are being tracked, the relative amount of arm and / or leg movement, compared to previous amounts of movement by the user or the amount of movement of other users, provides an indication of the user's emotion. In one embodiment, the user emotion measure is intended to represent how apathetic the user is at one end of the scale or how amused the user is at the other end of the scale.

[0042] The human emotion measure is then provided as an input to an adaptive audio mixing neural network (block 1015). The adaptive audio mixing neural network generates an adaptive audio mix based on the human emotion measure and one or more other input parameters (block 1020). For example, if the human emotion measure is greater than a threshold, the adaptive audio mixing neural network increases the tempo of the adaptive audio mix. Other modifications to the adaptive audio mix based on the human emotion measure and other input parameters may also be implemented. The other input parameters vary according to the embodiment and may be any of the scene- and / or user-specific parameters listed above. After block 1020, method 1000 ends. Note that method 1000 may be repeated during video game play to update the adaptive audio mix generated by the neural network.

[0043] In various embodiments, program instructions of a software application are used to implement the methods and / or mechanisms described herein. For example, program instructions executable by a general-purpose or special-purpose processor are contemplated. In various embodiments, such program instructions are expressed in a high-level programming language. In other embodiments, the program instructions are compiled from the high-level programming language into a binary, intermediate, or other format. Alternatively, program instructions that describe the behavior or design of hardware are written. Such program instructions are expressed in a high-level programming language such as C. Alternatively, a hardware design language (HDL) such as Verilog is used. In various embodiments, the program instructions are stored on any of a variety of non-transitory computer-readable storage media. The storage medium is accessible by a computing system during use to provide the program instructions to the computing system for program execution. Generally speaking, such a computing system includes at least one or more memories and one or more processors configured to execute the program instructions.

[0044] It should be emphasized that the above-described embodiments are merely non-limiting examples of embodiments. Numerous variations and modifications will become apparent to those skilled in the art once the above disclosure is fully appreciated. It is intended that the following claims be interpreted to encompass all such variations and modifications.

Claims

1. 1. An apparatus comprising: a memory for storing a plurality of pre-recorded music stems; a machine learning engine; The machine learning engine: receiving one or more user-specific parameters of a user playing the video game application; receiving one or more indicators of a real-time scenario of a user playing the video game application; generating a non-deterministic music mix by combining the plurality of pre-recorded music stems based on the one or more user-specific parameters and one or more indicators of the real-time scenario; providing the non-deterministic music mix as an audio output to a user playing the video game application in real time; configured to: Device.

2. the one or more user-specific parameters include the user's player profile, the user's player interests, the user's player performance, game control adjustments, audience size, the user's player popularity, and the user's native language; 10. The apparatus of claim 1.

3. the one or more indicators of the real-time scenario include a storyline of the user's player, one or more scene elements, and one or more game scene characteristics; 10. The apparatus of claim 1.

4. the machine learning engine is a trained neural network, and the trained neural network is updated based on one or more feedback indicators from the user; 10. The apparatus of claim 1.

5. the one or more feedback indicators include user adjustments to game settings, the user's interactions, and the user's performance; 5. The apparatus of claim 4.

6. an updated version of the trained neural network configured to generate an updated non-deterministic music mix in a subsequent iteration of the video game application; 6. The apparatus of claim 5.

7. the user interaction is determined by tracking user movements while the user is playing the video game application, and the trained neural network is configured to increase the tempo of the non-deterministic music mix in response to detecting that the user interaction is greater than a threshold.

6. The apparatus of claim 5.

8. 1. A method comprising: a machine learning engine receiving one or more user-specific parameters of a user playing the video game application; receiving one or more indicators of a real-time scenario of a user playing the video game application; generating a non-deterministic music mix by combining a plurality of pre-recorded music stems based on the one or more user-specific parameters and one or more indicators of the real-time scenario; and providing the non-deterministic music mix as an audio output to a user playing the video game application in real time. method.

9. the one or more user-specific parameters include the user's player profile, the user's player interests, the user's player performance, game control adjustments, audience size, the user's player popularity, and the user's native language; 9. The method of claim 8.

10. the one or more indicators of the real-time scenario include a storyline of the user's player, one or more scene elements, and one or more game scene characteristics; 9. The method of claim 8.

11. the machine learning engine is a trained neural network, and the trained neural network is updated based on one or more feedback indicators from the user; 9. The method of claim 8.

12. the one or more feedback indicators include user adjustments to game settings, the user's interactions, and the user's performance; The method of claim 11.

13. an updated version of the trained neural network generating an updated non-deterministic music mix in a subsequent iteration of the video game application; 13. The method of claim 12.

14. the user interaction is determined by tracking user movements while the user is playing the video game application, and the method includes increasing a tempo of the non-deterministic music mix in response to detecting that the user interaction is greater than a threshold.

13. The method of claim 12.

15. 1. A system comprising: Machine learning engine and an audio mixing engine; The machine learning engine: receiving one or more user-specific parameters of a user playing the video game application; receiving one or more indicators of a real-time scenario of a user playing the video game application; generating a mixing array of weights based on the one or more user-specific parameters and one or more indicators of the real-time scenario; and The audio mixing engine generating a non-deterministic music mix by applying said mixing array of weights to a plurality of pre-recorded music stems; providing the non-deterministic music mix as an audio output to a user playing the video game application in real time; configured to: system.

16. the one or more user-specific parameters include the user's player profile, the user's player interests, the user's player performance, game control adjustments, audience size, the user's player popularity, and the user's native language; 16. The system of claim 15.

17. the one or more indicators of the real-time scenario include a storyline of the user's player, one or more scene elements, and one or more game scene characteristics; 16. The system of claim 15.

18. the machine learning engine is a trained neural network, and the trained neural network is updated based on one or more feedback indicators from the user; 16. The system of claim 15.

19. the one or more feedback indicators include user adjustments to game settings, the user's interactions, and the user's performance; 20. The system of claim 18.

20. an updated version of the trained neural network configured to generate an updated non-deterministic music mix in a subsequent iteration of the video game application; 20. The system of claim 19.

Images