Audio mixing and balancing and detection of audio events in gaming system

Abstract

Systems disclosed in conjunction with the subject matter may include, for example, a gaming system that cooperates with a graphical user interface to enable user modification and augmentation of one or more audio streams associated with the gaming system. In embodiments, the audio streams may include a game audio stream, a chat audio stream of sessions between players of the video game, and a microphone audio stream of players of the video game. Additional embodiments are disclosed.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U.S. patent application No. 17 / 899,766 filed on August 31, 2022, U.S. patent application No. 17 / 507,985 filed on October 22, 2021, U.S. patent application No. 17 / 507,889 filed on October 22, 2021, U.S. patent application No. 17 / 507,941 filed on October 22, 2021, U.S. patent application No. 17 / 507,966 filed on October 22, 2021, U.S. patent application No. 17 / 508,014 filed on October 22, 2021, U.S. patent application No. 17 / 507,900 filed on October 22, 2021, and U.S. Provisional Application No. 63 / 240,004 filed on September 2, 2021, all portions of which are incorporated herein by reference in their entirety. Technical Field

[0003] The present disclosure generally relates to a method and apparatus for extracting specific sounds or classes of sounds from game audio of a gaming system. Background Art

[0004] Today, gamers often utilize more than one gaming accessory. This is especially true for gamers who play online games or competitive games in a team or individual configuration. A gamer may have accessories at their disposal, such as a keyboard, a universal game pad, a mouse, a game console controller, a headset to communicate with other players, a joystick, a computer console, or other common gaming accessories.

[0005] A player may frequently use a combination of these accessories (e.g., headset, keyboard, and mouse) in a single game. Effective management and utilization of these accessories may often affect a player's competitive ability.

[0006] Accessory management may have utility in other disciplines that may not involve gaming applications. Efficient use of accessories in these other disciplines may be important to other users.

[0007] In addition, players can play video games with other players, either as teammates or against opponents, all of whom are remotely connected via a communication network. Players can communicate with teammates via verbal audio, and players can hear game sounds generated by the video game. Further, the video game can monitor various aspects of the player's interaction with the video game so that player performance can be reviewed to improve future performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0009] Figure 1 depicts an illustrative embodiment of a graphical user interface (GUI) generated by an accessories management software (AMS) application in accordance with the present disclosure;

[0010] Figure 2-3 An illustrative embodiment for communicatively coupling a game controller to a computing device is depicted;

[0011] Figure 4 An illustrative embodiment of a communication device is depicted;

[0012] Figure 5 An illustrative embodiment of a first method utilized in the subject disclosure is depicted;

[0013] Figure 6 depicts an illustrative embodiment of a second method utilized in the subject disclosure;

[0014] Fig. 7A depicts an illustrative embodiment of a third method utilized in the subject disclosure;

[0015] Figure 7B depicts an illustrative embodiment of a fourth method utilized in the subject disclosure;

[0016] Figure 7C depicts an illustrative embodiment of a fifth method utilized in the subject disclosure;

[0017] Fig.7D depicts an illustrative embodiment of a sixth method utilized in the subject disclosure;

[0018] Figure 7E-7G An illustrative embodiment of an interface associated with gameplay of a game is depicted;

[0019] Figure 7H An illustrative embodiment of a method for automatically triggering a notification to one or more gamers based on detected sounds in accordance with aspects of the subject disclosure is depicted;

[0020] Fig.7I depicts an illustrative embodiment of an additional method for automatically triggering a notification to one or more gamers based on detected sounds in accordance with aspects of the subject disclosure;

[0021] Figure 7J and Figure 7K Schematically illustrates at least part of Figure 5 , Figure 6 , FIG. 7A to FIG. 7C and FIG. 7H to FIG. 7I A system operated by a method;

[0022] Figure 7L An illustrative embodiment of a block diagram depicting an audio processing system for a gaming system in accordance with aspects of the subject disclosure;

[0023] Figure 7L-1 and Figure 7L-2 An illustrative embodiment of an operating system control for a user interface for selecting device preferences is depicted;

[0024] Figure 7L-3 An illustrative embodiment of a process by which a listener selects a head-related transfer function (HRTF) profile for use by a user is depicted;

[0025] Figure 7M-7T depicts an illustrative embodiment of a graphical user interface display for an audio processing system of a gaming system in accordance with aspects of the subject disclosure;

[0026] Figure 7U and Figure 7V Illustrative embodiments of methods utilized in the subject disclosure are depicted.

[0027] Figure 7W and Figure 7X depicts illustrative embodiments of exemplary audio scenes and audio mixing;

[0028] Figure 7Y is a block diagram of a system for extracting individual sounds from an audio stream according to some aspects of the subject disclosure;

[0029] Figure 8 Describes at least in part Figure 5 to an illustrative embodiment of a system operating in accordance with the method of FIG. 7 ;

[0030] Fig. 9 Depicted by Fig.12 An illustrative embodiment of a communication flow diagram utilized by a system of; and

[0031] Fig.10 An illustrative diagrammatic representation of a machine in the form of a computer system is depicted within which a set of instructions, when executed, may cause the machine to perform any one or more of the methodologies disclosed herein.

[0032] Fig.11 Depicted are illustrative diagrammatic representations of sliding windows in accordance with aspects of the present disclosure.

[0033] Fig.12 A storage medium for storing a representation of a sliding window according to aspects of the present disclosure is depicted.

[0034] Fig.13A system for identifying a set of possible trigger clips according to aspects of the present disclosure is described.

[0035] Fig.14 A system for distributing a set of trigger clips to a group of players in accordance with aspects of the present disclosure is described.

[0036] Fig.15A -D depicts an illustrative embodiment of a method utilized in the subject disclosure. DETAILED DESCRIPTION

[0037] This subject disclosure particularly describes an illustrative embodiment of, for example, a gaming system that collaborates with a graphical user interface to implement user modification and enhancement of one or more audio streams associated with the gaming system. In an embodiment, an audio stream may include a game audio stream, a chat audio stream of a conversation between players of a video game, and a microphone audio stream of a player of a video game. Players may independently adjust the audio of the audio stream to improve clarity, reduce noise, and help identify in-game sounds such as footsteps and gunshots. A graphical user interface may be displayed on a display device of the player, and may be manually adjusted to a preferred setting, or a player profile or configuration file may be retrieved for each audio stream. In order to further enhance sound and audio quality for the player, a head-related transfer function may be selected from a set of predefined transfer functions. Audio may be provided in a surround sound arrangement to further enhance the audio experience of the user. Individual sounds within the game audio may be analyzed and identified, for example, by a neural network, and enhanced to give game players (game players) improved audio experience and competitive advantages when playing video games. In addition, individual sounds or sound categories may be extracted from the game audio. The extracted sounds may be provided to the competition individually or may be enhanced to improve the fun and competitiveness of the player in the game. Other embodiments are described in the subject disclosure.

[0038] One or more aspects of the subject disclosure include receiving an input audio stream from a gaming system, the input audio stream comprising game audio of a video game played by a game player, the input audio stream comprising a plurality of categories of sounds, providing the input audio stream to a neural network, extracting, by the neural network, sounds of a selected sound category from the plurality of categories of sounds, and providing a plurality of output audio streams, including providing a first audio stream comprising sounds of the selected sound category from the input audio stream and a second audio stream comprising remaining sounds from the input audio stream.

[0039] One or more aspects of the subject disclosure include receiving an input audio stream from a gaming system, the input audio stream comprising game audio of a video game played by a game player, the input audio stream comprising multiple categories of sounds, extracting sounds of a selected sound category from the multiple categories of sounds from the input audio stream, receiving audio processing control information from the game player, and providing multiple output audio streams to the game player based on the audio processing control information, including selectively providing one of a first audio stream comprising sounds of the selected sound category in the input audio stream and a second audio stream comprising remaining sounds in the input audio stream.

[0040] One or more aspects of the subject disclosure include receiving an input audio stream of a gaming system, the input audio stream comprising game audio of a video game played by a game player, the input audio stream comprising game sounds, the game sounds being organized into a plurality of categories of sounds, providing the input audio stream to a neural network trained to recognize at least one category of game sounds from the plurality of categories of sounds, receiving from the neural network extracted game sounds from at least one category of the plurality of categories of sounds, and providing to the game player a first output audio stream comprising the extracted game sounds and a second output audio stream comprising remaining sounds of the game audio of the video game.

[0041] One embodiment of the subject disclosure includes receiving a plurality of audio streams, the plurality of audio streams including a game output audio stream of a game application, a gamer chat audio stream generated by a plurality of gamers participating in the game application, and a gamer microphone audio stream generated by one of the plurality of gamers, processing the game output audio stream according to a first audio processing setting of the one gamer to form a processed game output audio stream, and processing the gamer chat audio stream according to a second audio processing setting of the one gamer to form a processed gamer chat audio stream. The embodiment also includes processing the gamer microphone audio stream according to a third audio processing setting of the one gamer to form a processed gamer microphone audio stream, providing the processed game output audio stream and the processed gamer chat audio stream to an audio device of the one gamer, and providing the processed gamer microphone audio stream to one or more of the plurality of gamers.

[0042] One embodiment of the subject disclosure includes receiving a live game output audio stream of a game application currently being played by multiple gamers, providing a graphical user interface on a display device of one of the multiple gamers, the graphical user interface including a spectrum analyzer interface for the one gamer to select one or more live audio processing settings for the live game output audio stream, receiving data defining the one or more live audio processing settings from the graphical user interface, and modifying the live game output audio stream of the game application according to the one or more live audio processing settings.

[0043] One embodiment of the subject disclosure includes receiving a live game output audio stream of a game application currently being played by multiple gamers, retrieving a stored configuration file including data defining stored live audio processing settings, and modifying the live game output audio stream of the game application based on the stored live audio processing settings.

[0044] One embodiment of the subject disclosure includes a processing system comprising a processor and a memory storing executable instructions that, when executed by the processing system, facilitates performing operations including a gaming application that generates gaming output information including a gaming output audio stream, and an audio processing module in data communication with the gaming application, the audio processing module operating to receive the gaming output audio stream from the gaming application and generate a plurality of gaming audio streams, the audio processing module further operating to apply a respective audio customization to each respective gaming audio stream of the plurality of gaming audio streams based on player input that defines the respective audio customization for each respective gaming audio stream.

[0045] One embodiment of the subject disclosure includes a device comprising a gaming application that generates gaming output information including a gaming output audio stream, a head-related transfer function (HRTF) module in data communication with the gaming application, the HRTF module operating to receive the gaming output audio stream from the gaming application and apply a selected head-related transfer function to the gaming output audio stream to generate a surround sound audio stream, the selected head-related transfer function being selected from a plurality of standardized head-related transfer functions.

[0046] One embodiment of the subject disclosure includes receiving a gaming system audio stream during a video game, detecting a current trigger in the gaming system audio stream during the video game, recording a trigger clip of gaming content, wherein the trigger clip includes a portion of the gaming content including the current trigger, and providing the clip to a group of gaming devices over a communications network, wherein each gaming device in the group of gaming devices is associated with a player of the video game.

[0047] One embodiment of the subject disclosure includes receiving a gaming system audio stream during a video game, the gaming system audio stream comprising a plurality of audio streams, the plurality of audio streams comprising a chat audio stream, a gaming audio stream, and a microphone audio stream, detecting, by the processing system, a predetermined event in the gaming system audio stream during the video game, modifying an audio spectrum of the gaming system audio stream to emphasize the predetermined event in the gaming system audio stream, generating an equalized audio stream, and providing the equalized audio stream to an audio device of a player of the video game.

[0048] One or more aspects of the subject disclosure include receiving a gaming system audio stream during a video game, the gaming system audio stream comprising a plurality of audio streams, the plurality of audio streams comprising a chat audio stream, a gaming audio stream, and a microphone audio stream, detecting a predetermined event in the gaming system audio stream during the video game, modifying an audio spectrum of the gaming system audio stream to emphasize the predetermined event in the gaming system audio stream, generating an equalized audio stream, and providing the equalized audio stream to an audio device of a player of the video game.

[0049] One or more aspects of the subject disclosure include receiving a gaming system audio stream from a gaming system and detecting a predetermined sound in the gaming system audio stream. Aspects of the subject disclosure further include selecting a predetermined audio profile in response to detecting a predetermined event in the gaming system audio stream, modifying the gaming system audio stream according to the predetermined audio profile to produce a modified audio stream, and providing the modified audio stream to an audio device of a player of the gaming system.

[0050] One or more aspects of the subject disclosure include receiving a game system audio stream during a video game and separating the game system audio stream into a plurality of audio streams including a chat audio stream, a game audio stream, and a microphone audio stream. Aspects of the subject disclosure also include processing the game audio stream to detect a predetermined audio event in the game audio stream. Aspects of the subject disclosure also include providing a visual indication of the detection of the predetermined audio event to a first player of the video game on a display device, modifying the game audio stream to enhance the audibility of the predetermined audio event to the first player of the video game, generating a modified audio stream, and providing the modified audio stream to an audio device of the first player of the video game.

[0051] Figure 1An illustrative embodiment of a graphical user interface (GUI) generated by an accessory management software (AMS) application according to the present disclosure is depicted. The AMS application can be executed by a computing device such as a desktop computer, a laptop computer, a tablet computer, a server, a mainframe computer, a game console, a game accessory, or any combination or portion thereof. The AMS application can also be executed by a portable computing device such as a cellular phone, a personal digital assistant, or a media player. The AMS application can be executed by any device with suitable computing and communication resources.

[0052] Figure 2 Several embodiments are shown for utilizing the game controller 115 with a computing device 206 in the form of a game console. Figure 2 In the illustration of , the game controller 115 may be communicatively coupled to the game console 206 using a tethered cable interface 202 such as a USB or proprietary cable or a wireless interface 204 such as Wi-Fi, Bluetooth, ZigBee, or a proprietary wireless communication protocol. The cable interface 202 provides a means for communication that may not be susceptible to electromagnetic interference. It should be understood that the game controller 115 may also include a headset 114 (with or without a microphone not shown) that is utilized by the gamer to communicate with teammates and / or listen to game sounds in high fidelity. Figure 2 In the illustration of , the AMS application may be executed in whole or in part by the game controller 115, the game console 206, or a combination thereof.

[0053] Figure 3 Several other embodiments for utilizing the game controller 115 with a computing device 206 are shown. In this embodiment, the game controller 115 includes a mouse and the computing device 206 includes a computer. The game controller 115 can be connected to the computing device 206 via a cable interface 202 (e.g., a USB cable or a proprietary cable) or a wireless interface 204. The cable interface 202 provides a means for communication that may not be susceptible to electromagnetic interference. It should be understood that the game controller 115 may also include a headset (with or without a microphone not shown) used by the gamer to communicate with teammates and / or listen to game sounds in high fidelity. Figure 3 In the illustration of , the AMS application may be executed in whole or in part by the game controller 115, the game console 206, or a combination thereof.

[0054] For purposes of this description, the terms game console 206 and computer 206 will be used interchangeably with the term computing device 206, understanding that computing device 206 may represent a number of other devices, such as a server, a tablet, a smart phone, etc. Thus, computing device 206 may represent any device having suitable computing resources to perform the methods described in the subject disclosure.

[0055] Figure 4 An illustrative embodiment of a communication device 400 is depicted. The communication device 400 may be used in whole or in part as an illustrative embodiment of the device described in the subject disclosure. The communication device 400 may include a wired and / or wireless transceiver 402 (here, transceiver 402), a user interface (UI) 404, a power supply 414, a proximity sensor 416, a motion sensor 418, an orientation sensor 420, and a controller 406 for managing its operation. The transceiver 402 may support short-range or long-range wireless access technologies, such as Bluetooth, Wi-Fi, digital enhanced cordless telecommunications (DECT), or cellular communication technologies, to name just a few. Cellular technologies may include, for example, CDMA-1X, UMTS / HSDPA, GSM / GPRS, TDMA / EDGE, EV / DO, WiMAX, software defined radio (SDR), long term evolution (LTE), and other emerging next generation wireless communication technologies. The transceiver 402 may also be adapted to support circuit-switched wired access technologies (eg, PSTN), packet-switched wired access technologies (eg, TCP / IP, VoIP, etc.), and combinations thereof.

[0056] The UI 404 can include a depressible or touch-sensitive keypad 408 coupled to a navigation mechanism, such as a scroll ball, joystick, mouse, or navigation pad, for manipulating the operation of the communication device 400. The keypad 408 can be an integral part of a housing assembly of the communication device 400, or a separate device operably coupled to the communication device via a tethered wired interface, such as a USB cable, or a wireless interface supporting, for example, Bluetooth. The keypad 408 can represent a numeric keypad and / or a QWERTY keypad with alphanumeric keys. The UI 404 can further include a display 410, such as a monochrome or color LCD (liquid crystal display), an OLED (organic light emitting diode), or other suitable display technology for conveying images to an end user of the communication device 400.

[0057] In an embodiment where the display 410 utilizes touch-sensitive technology, a portion or all of the keypad 408 can be presented via the display 410 with navigation features. As a touch screen display, the communication device 400 can be adapted to present a user interface with graphical user interface (GUI) elements that a user can select by the touch of a finger. The touch screen display 410 can be equipped with capacitive, resistive, or other forms of sensing technology to detect how much surface area of ​​a user's finger has been placed on a portion of the touch screen display. This sensed information can be used to control the manipulation of GUI elements.

[0058] UI 404 can also include an audio system 412 that utilizes common audio technology to transmit low volume audio (such as audio that can only be heard when close to the human ear) and high volume audio (such as a speaker phone, stereo or surround sound system for hands-free operation). Audio system 412 can also include a microphone for receiving audible signals from the end user. Audio system 412 can also be used for speech recognition applications. UI 404 can also include an image sensor 413, such as a charge coupled device (CCD) camera, for capturing still images or moving images and performing image recognition therefrom.

[0059] The power supply 414 can utilize common power management techniques, such as replaceable or rechargeable batteries, power conditioning techniques, and charging system techniques, for supplying energy to the components of the communication device 400 to facilitate long-range or short-range portable applications. Alternatively, the charging system can utilize an external power source, such as DC power supplied through a physical interface such as a USB port or through a power cord attached to a transformer that converts AC power to DC power.

[0060] The proximity sensor 416 can utilize proximity sensing technology, such as an electromagnetic sensor, a capacitive sensor, an inductive sensor, an image sensor, or a combination thereof. The motion sensor 418 can utilize motion sensing technology, such as an accelerometer, a gyroscope, or other suitable motion sensing technology, to detect the movement of the communication device 400 in three-dimensional space. The orientation sensor 420 can utilize orientation sensing technology, such as a magnetometer, to detect the orientation of the communication device 400 (north, south, west, east, combinations thereof, in degrees, minutes, or other suitable orientation metrics).

[0061] The communication device 400 may use the transceiver 402 to also determine proximity to a cellular, Wi-Fi, Bluetooth or other wireless access point by common sensing techniques such as utilizing a received signal strength indicator (RSSI) and / or signal time of arrival (TOA) or time of flight (TOF). The controller 406 may utilize computing technology such as a microprocessor, a digital signal processor (DSP) and / or a video processor with associated memory such as flash, ROM, RAM, SRAM, DRAM or other storage technology.

[0062] The communication device 400 described herein can be used with Figure 4 More or fewer of the components depicted in the drawings may operate together to suit the implementation of the devices described in the subject disclosure. These variant embodiments are contemplated by the subject disclosure.

[0063] Figure 5-7AMethods 500-700 describing an illustrative embodiment of an AMS application are depicted. Method 500 may begin at step 502, where an AMS application is invoked in a computing device. The computing device may be a remote server (not shown), Figure 2-3 The invocation step may result from a user selecting the AMS application from a menu or iconic symbol presented by the computing device 206, or when the user communicatively couples the game controller 115 or other form of accessory device to the computing device 206. At step 504, the AMS application may detect, through a software driver in the operating system (OS) of the computing device 206, a plurality of operationally distinct accessories communicatively coupled to the computing device 206. The accessories may be coupled to the computing device 206 via a tethered interface (e.g., a USB cable), a wireless interface (e.g., Bluetooth or Wireless Fidelity—Wi-Fi), or a combination thereof.

[0064] In the present context, an accessory may represent any type of device that can be communicatively coupled to computing device 206 (or that can be an integral part of the computing device) and that can control aspects of the OS and / or software applications operating from computing device 206. An accessory may represent, for example, a keyboard, a touch screen display, a game pad, a game controller, a mouse, a joystick, a microphone, or a headset with a microphone, to name just a few examples.

[0065] In step 506, the AMS application presents Figure 1 , which depicts operationally distinguishable accessories, such as a keyboard 108, and a game controller 115. The GUI 101 presents accessories 108-116 in a scrollable portion 117. One or more accessories can be selected by a user with a mouse pointer. In this illustration, the keyboard 108 and the game controller 115 are selected for customization. When the keyboard 108 and the game controller 115 are selected from the scrollable window of the portion 117, the AMS application presents the keyboard 108 and the game controller 115 in split windows 118, 120, respectively, to assist the user during the customization process.

[0066] In step 508, the AMS application may be programmed to detect a user's selection of a particular software application such as a video game. This step may be when the user enters the name of the game application in the quick search field 160 (e.g., World of Warcraft TMor WoW). Upon identifying a gaming application, the AMS application may retrieve gaming application actions from a remote or local database in step 510, which may be presented in a scrollable portion 139 of the GUI, represented as "actions" 130. Actions may be policy actions 132, communication actions 134, menu actions 136, and move actions 138, which may be used to invoke and manage features of the gaming application.

[0067] The actions presented descriptively in section 130 of the GUI may represent a range of accessory input functions that a user may actuate through button presses, navigation, or voice. For example, pressing the left button on mouse 110 may represent the tactical action "Reload"; while simultaneously pressing "Ctrl A" on the keyboard may represent the tactical action "Melee Attack". For ease of use, the "Actions" 130 section of the GUI is presented descriptively rather than through descriptions of the input function(s) of a particular accessory.

[0068] Any of the actions 130 can be associated with one or more input functions of the accessories customized in windows 118 and 120 by drag-and-drop actions or other customization options. For example, a user can select "Melee Attack" by placing the mouse pointer 133 on the icon symbol associated with the action; when doing so, the display symbol can be highlighted to indicate to the user that the icon is selectable. At this point, the user can select the icon by holding down the left mouse button and dragging the symbol to any input function (e.g., button) of the keyboard 108 or a selectable option of the game controller 115 to associate with the input function of one of these accessories. The action of one accessory can also be associated with another accessory belonging to a different category. For example, the key press "Ctrl A" of the keyboard 108 can be associated with one of the buttons of the game controller 115 (e.g., the left button 119).

[0069] In one embodiment, a melee attack action can be associated by dragging the action to the left button 119 or the right button 121 of the game controller 115. Thus, when the selected button is pressed, the actuation signal generated by the selected button of the game controller 115 can be replaced by the AMS application with a melee attack action. In another embodiment, the AMS application can be configured so that a melee action can be associated with a combination of key button presses (e.g., simultaneous presses of the left button 119 and the right button 121, or a sequence of button presses: two quick left button presses followed by a right button press).

[0070] In yet another embodiment, the AMS application may be configured such that melee actions may be associated with movement of the game controller 115, such as, for example, a quick movement or shake of the game controller 115. In another embodiment, the AMS application may be adapted to associate with two-dimensional or three-dimensional movement of the game controller 115 according to the game venue state. For example, suppose the player's avatar enters a fighter jet. In this game venue state, moving the left navigation knob forward may be associated by the AMS application with controlling the throttle of a jet engine. Quickly moving the game controller 115 downward may represent the release of ammunition, such as a bomb.

[0071] In a play state where the player avatar has entered a building, raising the game controller 115 above the first displacement threshold may be associated with rapid movement of the avatar to an upper level. In contrast to the first displacement threshold, the second displacement threshold may be associated with rapid movement of the avatar to a lower level. Alternatively, the second displacement threshold may be associated with a different action such as jumping between buildings when the avatar is on the roof of a building.

[0072] The AMS application can monitor the gaming venue state by analyzing captured images generated by the gaming application (e.g., one or more still images of a tank, or a video of an avatar entering a tank) and / or receiving messages from the gaming application via an application programming interface (API) to enable the AMS application to identify the occurrence of a particular gaming venue state.

[0073] The AMS application may also respond to the user's selection of a profile at step 512. The profile may be a device profile or a master profile invoked by selecting a GUI button 156 or 158, each of which may identify an association of a gaming action with an input function of one or more accessories. If a profile selection is detected at step 512, the AMS application may retrieve, at step 514, previous associations and / or macros defined by the profile. At step 516, the actions and / or macros defined in the profile may also be presented by the AMS application in the action portion 130 of the GUI 101 to modify an existing profile association or create a new association.

[0074] In step 518, the AMS application may also respond to the user selection to create a macro. A macro in this context may mean any actionable command that can be recorded by the AMS application. The actionable command may represent a sequence of stimuli generated by manipulating the input functionality of an accessory, a combination of actions in the action portion 130, an identification of a software application to be launched by the OS of the computing device 206, or any other recordable stimulus for launching, controlling, or manipulating a software application. For example, a macro may represent the identity of a software application (e.g., an instant messaging tool) that is to be launched by the OS when the user inputs a voice command that is detected by the AMS application using voice recognition technology.

[0075] A macro may also represent a recordable speech delivered by a microphone alone or in combination with headphones for detection by another software application through speech recognition or so that the recorded speech is delivered to other parties. In another embodiment, a macro may represent a recordable navigation of an accessory such as a joystick of a game controller 115, a recordable selection of a button of a game controller 115, and the like. A macro may also be a combination of the above description and an action selected from an action 130 menu. A macro may be created from the GUI 101 by selecting a "record macro" button 148. A macro may be given a name and a category in the user defined fields 140 and 142.

[0076] When the record macro button 148 is selected, the macro can be activated by selecting an input function on the accessory (e.g., Ctrl A, voice, movement of a navigation knob of a game controller 115, etc.) and / or by manual input in field 144 (e.g., typing the name and location of a software application such as an instant messaging application to be launched by the OS, keyboard input such as Ctrl A, etc.). Once the macro is created, it can be tested by selecting a button 150 that can repeat the sequence specified in field 144. If desired, a clone button 152 can be selected to copy the macro sequence. Field 152 can also present the timing characteristics of the stimulus sequence in the macro, with the ability to modify and thereby customize the timing of one or more stimuli in the stimulus sequence. Once the macro has been fully defined, button 154 is selected in step 520 to record the macro. The recording step can be combined with a step for adding the macro to the associable item action column 130, thereby providing a means for the user to associate the macro with an input function of the accessory (e.g., one or more keys of the keyboard 108, a button of the game controller 115, etc.).

[0077] In step 522, the AMS application can respond to the action and the drag-and-drop association of the input function of the keyboard 108 or the game controller 115. The association can also be performed based on the two-dimensional or three-dimensional movement of the game controller 115. If the user input indicates that the user is performing the association, the AMS application can proceed to step 524, where it can be determined whether a profile has been identified in step 512 to record the detected (one or more) associations. If the profile has been identified, the association is recorded / stored in the profile in step 526. If the profile is not identified in step 512, the AMS application can create a profile for recording the detected association in step 528. In the same step, the user can name the newly created profile as needed. In step 530, the newly created profile can also be associated with one or more game software applications for future reference. The AMS application can also record the association based on the game venue state in the profile in step 526. In this embodiment, the same stimulus generated by the game controller 115 can produce different substitutions based on the game venue state detected by the AMS application.

[0078] Referring back to step 526, once the association has been recorded in the profile, the AMS application may determine in step 532 Figure 1-3 108, game controller 115) is communicatively coupled to computing device 206 and determines that the accessory is capable of performing stimulus substitution locally, then the AMS application may proceed to Figure 5 Step 534, in which the profile and its contents are submitted to be stored in the accessory (eg Figure 2-3 Once an accessory (e.g., game controller 115) is programmed with a profile, the accessory can perform incentive substitution based on the associations recorded by the AMS application in the profile. Alternatively, the AMS application can store the profile in Figure 2-3 The computing device 206 of the AMS application and performing substitution of the stimulus provided by the game controller 115 according to the association recorded in the profile by the AMS application.

[0079] Rendered by AMS application Figure 1 GUI 101 can have other functions. For example, GUI 101 can present the layout of the accessories (button 122), how the accessories are illuminated when the association between input functions and actions is made (button 124), and configuration options for the accessories (button 126). The AMS application can adapt GUI 101 to present more than one functional GUI page. For example, by selecting button 102, the AMS application can adapt GUI 101 to present means for creating macros and associating actions to accessory input functions, such as Figure 1 Selecting button 104 may cause the AMS application to adapt GUI 101 to present statistical information of game play results and / or incentive information captured by the AMS application, as described in the subject disclosure. Selecting button 106 may also cause the AMS application to adapt GUI 101 to present promotional offers and software updates.

[0080] The steps of method 500 may be repeated in whole or in part until the desired pattern of association between the excitation signal generated by the accessory and the alternative excitation is achieved. It will be apparent to one of ordinary skill in the art that there may be many other methods to implement the embodiments described by method 500 or its variations. These undisclosed methods are contemplated by the subject disclosure.

[0081] Figure 6 A method 600 is depicted for illustrating additional operations of an AMS application. Figure 2-3As shown, the AMS application may be operated in whole or in part from the game controller 115, the game console 206, the computer 206, or a remote server (not shown). For purposes of illustration, it is assumed that the AMS application is operated from the game console 206. The method 600 may begin with the AMS application communicating with the game console 206 and a remote server such as the game console 206 in steps 602 and 604. Figure 1 14. The steps may represent, for example, a user launching an AMS application from a game console 206, and / or a user plugging a connector of a USB cable connected to a game controller 115 into a USB port of the game console 206, which invokes the AMS application. In step 606, the game controller 115 and / or the earbuds 114 may then provide one or more accessory IDs to the AMS application, or the user may provide a user identification via a keyboard or the game controller 115. Using the accessory ID or user input, the AMS application may identify a user account associated with the game controller 115 and / or the earbuds 114 in step 608. In step 610, the AMS application may retrieve one or more profiles associated with the user account.

[0082] In step 612, the profiles available for selection by the user may be presented to the user via a display coupled to the game console 206. If the user makes a selection, the AMS application proceeds to step 614, where the association(s) stored therein are retrieved from the selected profile. If no selection is made, the AMS application may proceed to step 616, where it may be determined whether a software game application (e.g., a video game) is being operated from the game console 206, or whether the game console 206 is being communicatively coupled to the game console 206 (e.g., presenting a game such as World of Warcraft). TM The AMS application communicates with the software game application via a remote system (e.g., an online game server) of the game controller 115. If a game software application is detected, the AMS application proceeds to step 617, where a profile matching the detected game application and the associations (one or more) contained in the profile are retrieved. As previously described, the associations (one or more) may represent auxiliary stimuli, navigation, voice, invocation of other software applications, macros, or other suitable associations that result in alternative stimuli. The auxiliary stimuli may be stimuli generated by the game controller 115, stimuli from other accessories (e.g., the headset 114), or a combination thereof.

[0083] Once the profile and its contents have been retrieved in step 614 or step 617, the AMS application can proceed to step 719 of FIG. 7, where changes in the state of the game venue are monitored based on the presentation made by the game application or the API message supplied by the game application. For example, at the start of the game, the state of the game venue can be immediately determined based on the game options selected by the player. The AMS application can determine the state of the game venue by tracking the game options selected by the player, receiving API instructions from the game application, or by performing image processing on the video presentation generated by the game application. For example, the AMS application can detect that the player has guided the avatar into a tank. The AMS application can retrieve the association in step 719 for the game controller 115 to use to control the tank.

[0084] The AMS application can process the forward, backward or sideways movement of the game controller 115 in two or three dimensions to control the movement of the tank. Similarly, rotating the game controller 115 or tilting the game controller 115 forward can cause the accelerometer, gyroscope or magnetometer of the game controller 115 to provide navigation data to the AMS application, which can be replaced by the action of turning and / or moving the tank forward. The profile retrieved by the AMS application can indicate that the greater the forward tilt of the game controller 115, the greater the speed at which the tank should move forward. Similarly, the backward tilt can generate the following navigation data, which is replaced by the reverse movement and / or deceleration of the forward movement to stop or slow down the tank. The three-dimensional lifting of the mouse can cause the tank to turn according to the three-dimensional navigation data provided by the game controller 115. For example, the navigation data associated with the combination of the forward tilt and right tilt of the game controller 115 can be replaced by the AMS application to increase the forward speed of the tank, and the AMS application determines the right turn according to the degree of the right tilt of the game controller 115. In the above embodiment, the three-dimensional navigation data allows the player to control any directional vector of the tank, including speed, direction, acceleration and deceleration.

[0085] In another illustration, as a result of the player directing the avatar to leave the tank and walk on foot, the AMS application can detect a new game field state. In step 719, the AMS application again retrieves associations associated with the game field state. In this embodiment, the AMS application can associate button selections of the game controller 115 with weapon selection, firing, reloading, etc. Movement of the game controller 115 in two or three dimensions can control the direction of the avatar and / or the selection or use of weapon equipment. Once the game field state is detected in step 719, the AMS application retrieves associations associated with the field state and can execute replacements of stimuli generated by the game controller 115 and / or voice commands received by the microphone of the headset 114.

[0086] In one embodiment, the AMS application may be configured to retrieve a profile in step 719 that provides alternative incentives for replacing certain incentives generated by the accessory. The associations recorded in the profile may be venue-independent. In another embodiment, the AMS application may retrieve a combination of profiles, where one or more profiles provide venue-dependent alternative incentives and one or more other profiles provide venue-independent alternative incentives.

[0087] The AMS application may monitor, in step 720, a stimulus generated by an accessory coupled to the game console 206. The stimulus may be generated by the player manipulating the game controller 115 and / or by generating a voice command detected by the microphone of the headset 114. If a stimulus is detected in step 720, the AMS application may determine in step 722 whether to forward the detected (one or more) stimulus to the operating system (OS) of the game console 206 or directly to the game application without replacement. This determination may be made by comparing the detected (one or more) stimulus with the corresponding association in one or more profiles retrieved by the AMS application. If the detected (one or more) stimulus matches the association, the AMS application proceeds to step 740, where (one or more) substitute stimulus is retrieved in the (one or more) profile. In step 742, the AMS application may replace the detected (one or more) stimulus with the substitute stimulus in the (one or more) profile.

[0088] In one embodiment, the AMS application may track the substitute incentives in step 744 by updating the incentives with a unique identifier, such as a globally unique identifier (GUID). In this embodiment, the AMS application may also add a timestamp to each substitute incentive to track when the substitution was performed. In another embodiment, the AMS application may track each substitute incentive based on the order in which the substitute incentives were submitted to the gaming application. For example, a sequence number may be generated for the substitute incentives to track the order in which they were submitted to the gaming application. In this embodiment, the substitute incentives do not need to be updated with a sequence number or identifier, as long as the order in which the gaming action results submitted by the gaming application to the AMS application remains the same order in which the substitute incentives were initially submitted.

[0089] For example, if the first stimulus sent by the AMS application to the gaming application is a command to shoot, and the second stimulus sent to the gaming application is a command to shoot again, then as long as the gaming application first provides a first game action result for the first shot, followed by the game action result for the second shot, then the alternate stimulus will not need to be updated with a sequence number because the game action results are reported in the order in which the stimuli were sent. On the other hand, if the game action results may be submitted out of order, then the stimuli will need to be updated with a sequence number or another suitable identifier so that the AMS application can properly track and associate the stimuli with the corresponding game action results.

[0090] Referring back to step 722, if the detected stimulus(s) do not match the association(s) in the profile(s), the AMS application proceeds to one of steps 744 or 746 to track the stimulus for the accessory as described above. In another embodiment, the AMS application may skip steps 744 or 746 and proceed to Figure 7B Step 770 is used to bypass the tracking of the original stimulus or the alternative stimulus.

[0091] Once the incentive received in step 720 has been replaced with another incentive in response to a detected association, or has remained unchanged in response to no detected association with the replacement incentive, in step 742, and (optionally) the AMS application has selected an appropriate tracking method for associating the game action outcome with the incentive, the AMS application may proceed to Figure 7B Step 770. Reference Figure 7B At step 770, the AMS application may obtain an identification of an action to be monitored during gameplay associated with the game. The identification of the action may include a specification of a volume level associated with a user (e.g., a player). The identification of the action may include a specification of a number of user inputs that exceed a threshold. The number of user inputs may include a number of messages submitted, an identification of the content of the message, an identification of an emoticon, or a combination thereof. The identification of the action may include a game action provided by the game, see Figure 8-9 and the accompanying description.

[0092] At step 772, the AMS application may store a representation of the sliding window of the game process in a first storage medium (e.g., Fig.12 The storage of step 772 may occur in real time during the game process. The representation of the sliding window of the game process may include a video, an image, an audio track, or a combination thereof. The first storage medium may include a buffer of a graphics card, a random access memory, or a combination thereof.

[0093] The sliding window may have a substantially fixed duration such that the sliding window advances as the user / gamer continues to play the game. For example, and with brief reference to Fig.11 , a sliding window 1100 is shown (as a function of time t). As the game progresses, new / supplementary representations of the game process may be added, as shown via reference character / dashed line portion 1102. To accommodate the storage of portion 1102, another portion 1104 may be deleted / overwritten. Fig.11 In the illustrated embodiment, the portion 1104 to be deleted / overwritten corresponds to the oldest / earliest portion of the window 1100. In some embodiments, a portion other than the oldest portion or a portion other than the oldest portion may be identified as being deleted / overwritten. Further, in some embodiments, the sliding window 1100 may have a variable duration. For example, the duration / length of the sliding window may be a function of network traffic, the capabilities of the device (e.g., storage capacity), user / player input, etc.

[0094] Return to reference Figure 7B , at step 774, the AMS application may monitor recognition of actions during the game play.

[0095] In step 776, the AMS application may detect an identifier during the game process in response to monitoring. In some embodiments, whether an event has occurred as reflected by the detection of step 776 may be based on a comparison of the game action to one or more thresholds. Such thresholds may be specified by a user / gamer (e.g., based on user input / preferences), may be predetermined based on one or more rules / configurations associated with the game, and the like.

[0096] In step 778, the AMS application may store at least a portion of the representation of the sliding window of the game process in a second storage medium. The second storage medium may be the same as the first storage medium or different. The second storage medium may include a server associated with the social media platform, a server associated with the virtual machine, a memory contained in a common housing with the first storage medium, a network element (e.g., a router, a gateway, a switch, etc.), or a combination thereof.

[0097] The storing of step 778 may include storing a video of the gamer, an image of the gamer (eg, a thumbnail or icon representation of the gamer), an audio track of the gamer, or a combination thereof.

[0098] The storing of step 778 may include presenting a prompt (potentially in response to the monitoring of step 774), placing a copy of the representation of the sliding window of the gameplay in a third storage medium, which may be different from the first storage medium and / or the second storage medium, receiving user input in response to the prompt, and storing the copy in the second storage medium in response to the user input.

[0099] Placing the representation / copy of the sliding window of the game process in the third storage medium can free up / lighten the first storage medium so that the first storage medium can continue to capture the game process / action as the game process continues after the detection of step 776. Moreover, placing the representation / copy of the sliding window of the game process in the third storage medium can save the user / player from having to commit to placing the representation / copy of the sliding window of the game process into a more permanent memory (e.g., the second storage medium). For example, placement in the third storage medium can help to edit or review the representation / copy of the sliding window before uploading it to the second storage medium.

[0100] In some embodiments, placing a copy of the representation of the sliding window of the game process in a third storage medium may include starting a timer to store a second sliding window of the representation after detecting the action, thereby generating an updated representation of the sliding window of the game process. In response to detecting the expiration of the timer, the updated representation may be stored in the third storage medium. The length of the timer may be based on user input.

[0101] In some embodiments, the storing of step 778 may include storing a new representation of the sliding window of the gameplay during the gameplay in the first storage medium after the copy is placed in the third storage medium; in some embodiments, the storing of the new representation may be consistent with a step separate from step 778.

[0102] In step 780, the AMS application may present (e.g., simultaneously present) a representation of the sliding window and / or a video, image, and / or audio track of the player, or a combination thereof. In some embodiments, the user / player may generate media that may be shared on one or more platforms (e.g., a social media platform) while the game is taking place, wherein the media may include a representation of the sliding window and / or a video, image, and / or audio track of the player, or a combination thereof. Alternatively, the user / player may generate media after the game is over in order to avoid distractions during the game.

[0103] Combination Figure 7B One or more of the steps shown may be performed more than once. For example, after the representation of the sliding window of gameplay is stored in the second storage medium as part of step 778, a second representation of the sliding window of gameplay may be stored in the first storage medium (as part of a second execution of step 772). The storage of the second representation of the sliding window of gameplay may overwrite at least a portion of the representation of the sliding window of gameplay in the first storage medium as described above.

[0104] Figure 7C Shows that it can be combined Fig. 7A Another embodiment of the method of executing the process shown in FIG. Figure 7C As shown, in step 7170, the AMS application may obtain an identification of actions to be monitored during a game process associated with the game. The identification of the actions may include a specification of a number of actions per unit time.

[0105] In step 7172, the AMS application may store a representation of a portion of the gaming session in a storage medium.

[0106] In step 7174, the AMS application may monitor the gameplay for identification of actions.

[0107] In step 7176, the AMS application may apply a tag to the representation of the portion of the gameplay in the storage medium in response to the monitoring.

[0108] The representation of a portion of a game play may include a first video clip that occurs before the action occurs and a second video clip that occurs after the action occurs. The first duration of the first video clip, the first resolution of the first video clip, the second duration of the second video clip, and the second resolution of the second video clip may be based on one or more user preferences, network traffic, device capabilities, etc.

[0109] The representation of the portion of the game play may include a video clip. The tag may include a watermark applied to the video clip. The watermark may include an identification of the action. The tag may include metadata associated with the video clip. The metadata may be searchable via a search engine. The metadata may include a selectable link that, when selected, causes the client device to obtain the video clip.

[0110] Fig.7D Shows that it can be combined Fig. 7A Another embodiment of the method of executing the process shown in FIG. Fig.7D As shown, in step 7270, the AMS application may monitor identification of actions during the game process.

[0111] In step 7272, the AMS application may detect identification of actions during the game play in response to monitoring.

[0112] In step 7274, the AMS application may present a prompt in response to the detection.

[0113] In step 7276, the AMS application can store a representation of a portion of the gameplay, a representation of the player controlling the gameplay, or a combination thereof in a storage medium based on the user-generated input associated with the prompt.

[0114] In some embodiments, machine learning / artificial intelligence may be applied to identify portions of the game process that are memorable or interesting to the user (e.g., player). For example, in response to the user-generated input associated with the prompt in step 7276, the AMS application may monitor a second identification of the action (or alternative action) in step 7278.

[0115] In step 7280, the AMS application may detect a second indicator of an action (or an alternative action) during the game play in response to monitoring the second indicator.

[0116] In step 7282, the AMS application may store a second representation of the second portion of the gameplay, a second representation of the player, or a combination thereof in the storage medium without presenting the second prompt.

[0117] Once the AMS application provides the OS of the computing device 206 with the Figure 7B methods, Figure 7C methods, Fig.7D , or a combination thereof, the AMS application may proceed to step 734. The game action provided to the OS in step 748 may be the pure "original" game action of step 720, or a substitute game action generated by step 744 or 746. In step 734, the OS determines whether to call the software application identified in the stimulus(s) in step 736 (e.g., the gamer says "turn on team chat," which calls the chat application), whether to forward the received stimulus(s) to the gaming software application in step 738, or a combination thereof.

[0118] In parallel with the above-described embodiment, the AMS application can monitor the game action results provided by the game application via the previously described API message in step 750. For example, assume that the incentive sent to the game application in step 738 is a command to shoot a pistol. The game application can determine whether the result of firing the shot is a miss or a hit. The game application can respond with a message submitted to the AMS application via the API indicating whether the result of firing the shot is a miss or a hit. If an ID such as a GUID is sent with each incentive, the game application can submit the game action results together with their corresponding GUIDs so that the AMS application can associate the game action results with incentives having the same GUID.

[0119] For example, if the shooting command includes the ID "1234", the game action result indicating a miss will include the ID "1234", so that the AMS application associates the game action result with the incentive with the same ID in step 752. If, on the other hand, the order of the game action results can remain consistent with the order of the incentives, the AMS application can associate the incentives with the game action results in step 754 in the order in which the incentives are submitted and the order in which the game action results are received. In step 756, the AMS application can catalog the incentives and the game action results. In another embodiment, the AMS application may be adapted to catalog the incentives in step 760. In this embodiment, step 760 can be performed as an alternative to steps 750 to 756. In another embodiment, step 760 can be performed in combination with steps 750 to 756 to generate a directory of incentives and a directory of game action results associated with the incentives.

[0120] Fig. 7E An interface that can be used to present at least a portion of a game process associated with a game is shown. Various controls / commands such as, for example, VCR-type / style controls / commands can be presented as part of the interface to facilitate recording or capturing of one or more portions of the game process.

[0121] Figure 7F An interface is shown that can provide control over recording or sharing of one or more representations (e.g., clips) of gameplay associated with a game. Various controls, such as, for example, a "share" button, can be provided to enable a user (e.g., a player) to publish or otherwise share the representation(s). In some embodiments, editing controls can be provided to allow the user to customize the representation before or after sharing the representation.

[0122] In some embodiments, the user / gamer may be able to supplement the representation of the gameplay with commentary that describes, for example, what the user's thought process was during the captured / represented portion of the gameplay. In this regard, and assuming that the user / gamer is viewed or otherwise characterized as an expert in the game, the sharing of the representation of the gameplay may serve as a tutorial for novice users.

[0123] Figure 7G An interface is illustrated in which a tag 702g (e.g., a watermark and / or metadata) associated with a representation of a game process may be presented. The tag 702g may include data acquired / obtained during the game process, such as, for example, a statement or other indication of the results obtained by the player during the game process. This statement or other indication may be received via, for example, a microphone, a keyboard, a mobile device, a computing / gaming console, etc.

[0124] The methods described herein (e.g., Figures 7A-7D The described method may include additional aspects. For example, in some embodiments, clips may be generated based on user defined keybinds (on a keyboard, mouse, or controller). The keybind that triggers the buffer clipping to be saved to the local file system may be customizable (e.g., may be based on user preferences). The gamer will be able to select: the actual key bound to the action, and the time slice that is saved (N seconds before and N' seconds after).

[0125] In some embodiments, a clip may be automatically generated based on some event, such as, for example, a detected event, an audible input (e.g., a scream), a message associated with a chat client, etc. In some embodiments, default settings may be provided, and those settings may be at least partially overridden / replaced based on affirmative user input and / or artificial intelligence / machine learning based user preferences.

[0126] In some embodiments, one or more filtering techniques may be applied to remove uninteresting content from the representation of the game process. This filtering may be based on one or more user inputs / preferences, which may be learned over time via machine learning / artificial intelligence, etc. When a plurality of monitored events / actions occur within a threshold amount of time (which may be consistent with the buffer time), the event / action endpoint may be extended to create a long slice / representation of the game process. Alternatively, a separate representation may be generated in some embodiments.

[0127] In some embodiments, a marker (e.g., a watermark) may be overlaid on a representation (e.g., a video) of the gameplay. The watermark may have a given level of transparency associated with it to avoid obscuring / obscuring the representation of the gameplay. One or more logos may be applied as part of the marker. In some embodiments, the watermark may pulse or otherwise fade in and out. In this regard, a dynamic watermark may be used. The use of a dynamic watermark may be used to draw additional / incremental attention to the watermark, which may be used for promotional / marketing / branding purposes.

[0128] Aspects of the representation of the shared game process can be controlled via one or more control parameters. Such control parameters can regulate sharing in terms of the size of the representation (e.g., video length), the content of the representation (e.g., control can be present to limit the spread of the representation in view of intellectual property or other rights). In some embodiments, the sharing of the representation of the game process can be limited to users authorized by the player (or other entity). For example, sharing can be based on identifying the contacts (e.g., friends) of the player in one or more applications (e.g., phone applications, email applications, text messaging applications, social media applications, etc.).

[0129] In some embodiments, the audio from the game can be monitored and analyzed in real time to generate game events. In addition, the sounds that occur during the game can be analyzed to generate notifications to one or more game participants; these notifications may include live clips of the game process (sometimes referred to as moment clips), and / or statistical information related to the performance, equipment, health status, etc. of one or more players.

[0130] refer to Figure 7H In step 7320, the AMS application may obtain one or more audio profiles (e.g., spectral profiles) of sounds that may be received from a game software application at a player station during a game process associated with the game. The sounds may correspond to game actions (e.g., machine gun firing, explosives detonating, etc.) or may be generated by a player (e.g., a player speaking a particular word during a game session). Game action sounds may be based on a player's action (e.g., a player shooting) or based on another player's action (e.g., another player detonating explosives). In one embodiment, the audio profiles are stored locally at the player station. In other embodiments, the audio profiles may be stored on a server or game console that executes a game software application associated with the game.

[0131] The AMS application monitors sounds generated at the player station (step 7322). In one embodiment, the AMS application can distinguish between sounds emitted by the gaming software application and audibly rendered by the player station and sounds generated in the player's environment. (e.g., Figure 7J In a player station or equipment 7411, if a sound 7419 detected via a microphone 7414 has a similar profile to a sound 7418 emitted from a speaker 7413 and occurs at a large extent at the same time, the sound is understood to be a game sound; otherwise, it is a player environment sound, which may be a sound generated by a player.) In a particular embodiment, the player station includes a spectral filter that can be tuned to detect player environment sounds with known characteristics.

[0132] When a sound is detected (step 7324), its spectrum profile is analyzed and compared with the audio profile. In one embodiment, monitoring may include comparison to determine correlation; the correlation may be based on a threshold of similarity between the features of the detected sound and the features of the reference sound in the audio profile. If the profile of the detected sound matches the stored profile (step 7326), a game event (e.g., an explosion is detected) may be identified. In one embodiment, a game event may be distributed as a notification corresponding to the detected sound, which is distributed to other participants in the game session (step 7328). In another embodiment, a game event may trigger the detection of an image on the display 7415 of a player station to further analyze the importance of the game event and generate further information that may be useful to other players (e.g., an explosion is detected, but the health of players near the explosion is not affected). This enhanced information may be delivered to other players in a notification. In a further embodiment, a notification may be triggered when a displayed image and / or a sound generated by a game is detected, combined with the player's voice, and the player's voice may be conveyed together with the notification. In other embodiments, AMS game events may cause the AMS to generate game actions that may be sent to the game and / or players (e.g., activate shield protection for the affected player, send a message to teammates that the player's shield has been activated due to the explosion). Game actions can be pre-stored and sent to the game and / or players via Figure 1 The AMS application interface 100 is associated with the detected sound and / or image recognition when the sound occurs.

[0133] According to an embodiment of the present disclosure, the notifications mentioned earlier can take several forms. The AMS application can send a text message, such as "Player A opened fire" or "Bomb detonated at Player A". In one embodiment, the message can be displayed on the display of the player station, superimposed on the game video being presented; in another embodiment, the message can be sent to different player devices. The AMS application can also analyze the current state of the game action seen by the player and provide additional information (e.g., statistical information) about the game action associated with the detected sound; for example, "The bomb detonated near Player A, Player A's health has been reduced by x, the bomb type is b, and Player A is exchanging fire with n opponents."

[0134] In other embodiments, game events detected by sound (or a combination of sound and imaging) can trigger Figures 7A-7GIn some embodiments, the AMS application may include a recording of video clips before and after a game event as described in the embodiments of the present invention. The notification may be included in the video clip, indicating the game process occurring at the player station when the sound is detected (i.e., a live clip about the detected sound). In some embodiments, a live clip is automatically generated when the sound is detected. The AMS application may distribute the live clip as part of the notification (e.g., text "Bomb near player A" superimposed on the live clip). In one embodiment, the AMS application generates a text notification by image processing the live clip.

[0135] The notification may also include a sequence of two or more video clips (e.g., a live clip combined with a clip of game action starting 30 seconds earlier). In one embodiment, the triggering sound and / or image is analyzed to determine which video clips are to be combined and distributed to other participants.

[0136] In further embodiments, the AMS application shares a live clip of the player (and / or additional clips in a sequence) with another player upon request. Alternatively, the AMS application may select one or more other players to share a video clip (multiple video clips) based on analysis of a triggering sound (or combination of sound / image / voice) and statistical information about the player and other players. For example, a sound may trigger analysis of a player's health status, equipment status (e.g., remaining ammunition), and location in relation to corresponding data about other players; a notification may then be sent to the nearest player with resources that may be shared.

[0137] In additional embodiments, the AMS application can analyze the game action shown in the live clip along with the recent movement of the player avatar, and provide suggestions associated with the detected game action (e.g., "Run east 100 yards").

[0138] refer to Fig.7I , the AMS application may maintain a log of sounds detected during the gaming session. If the detected sound does not have a profile that matches a stored audio profile (step 7376 / N), but has been reproduced during the gaming session (step 7382 / Y), the AMS application may add the profile of the sound to the stored audio profiles (step 7384). In one embodiment, the profile of the reproducing sound is added after the reproducing sound has been detected a specified number of times. The AMS application may analyze the clip corresponding to the reproducing sound to identify the sound, so that the profile of the sound may be stored with a text tag identifying the sound.

[0139] Figure 7J and Figure 7K Schematically illustrates at least part of Figure 5 , Figure 6 and Figures 7A-7CA system operated by a method such as Figure 7J As shown, a player 7410 uses a device 7411 (commonly referred to as a player station) including a local processing system to play a game; in this embodiment, an AMS application 7412 and a game application 7425 are executed on the player station. The game application 7425 can be downloaded from a game server 7405 via a network 7420. The player 7410 can use any of a number of accessories, including, for example, a mouse 7416, to participate in the game action.

[0140] In this embodiment, the player station stores audio profiles 7430 for game action sounds (e.g., sounds 7418 from speakers 7413) and player-generated sounds 7419 detected by microphone 7414 (e.g., words spoken by the player). Figure 7K As shown, a notification 7510 about the detected sound can be distributed to the equipment of other players 7511, 7512, 7513 participating in the game session via the network 7420. In one embodiment, the notification includes a live clip associated with the occurrence of the sound. In a further embodiment, in response to a request (e.g., a request 7515 from the equipment of the player 7511), the live clip is sent to the equipment of the player.

[0141] Figure 7L An illustrative embodiment of a block diagram of an audio processing system 7600 for a gaming system in accordance with aspects of the subject disclosure is depicted. The audio processing system 7600 can cooperate with the gaming system to provide audio mixing and equalization for multiple audio streams associated with the gaming system. In embodiments, the gaming system can be implemented as an accessory, such as a game controller 115 or a computing device 206 ( Figure 2 , Figure 3 ). The computing device 206 operates according to an operating system (OS), which is used to control various functions of the computing device 206, including the input of data and other information and the output of data or other information. The audio processing system 7600 further cooperates with the computing device's OS to control audio processing, including mixing and equalization of audio streams associated with the computing device 206.

[0142] The audio processing system 7600 includes a plurality of audio input channels, which in this embodiment include a game audio input channel 7602, a chat audio input channel 7604, and a capture audio channel 7606. In addition, the audio processing system 7600 includes one or more audio output devices 7608, a graphical user interface (GUI) 7610, and a remote audio device 7612. Other embodiments may include additional or alternative features and functionality.

[0143] The game audio input channel 7602 operates under the control of the GUI 7610 to select one or more audio streams from the game audio source 7614 and control the audio processing of the selected audio streams in the game audio processing module 7618. In addition, the game audio input channel 7602 operates under the control of the GUI 7610 to route the processed audio of the game audio input channel 7602 to the output device 7608. The game audio input channel 7602 includes a game audio input source 7614, a game virtual audio device 7616, and an audio processing module 7618. Other embodiments may include additional or alternative features or functions. The game audio source 7614 can be, for example, a gaming system, one or more applications running on a device such as the computing device 206, and other sources such as network sources accessed by a user of the gaming system. In the illustrated embodiment, the game audio source 7614 can be an application or online source, such as a game software and application. (available online at steampowered.com) or Or Counter-Strike: Global Offensive(CS GO) The audio from the gaming system may include game sounds associated with the video displayed on the video display device of the computing device 206. In the case of a war simulation game, for example, the game audio input stream 7602 may include the sounds of gunfire, footsteps of teammates or opponents, explosions, vehicle noises, and the like.

[0144] Game audio input stream 7602 may also include audio from other sources operating as game audio sources 7614. Some applications generate audio and may be active on computing device 206 along with the gaming system. For example, a gamer may listen to music while playing a game on the gaming system. Game audio input stream 7602 may include a music audio stream from a music application. Additionally, audio streams may be received from a remote location over a network such as the public Internet. For example, a gamer may be watching a video game hosted by a computer such as Or movies provided by online providers from other sources. Game audio input stream 7602 may include audio received over the network.

[0145] In some applications, the game audio input stream 7602 includes multi-channel audio or surround sound. Multi-channel audio can include stereo audio with sound separated into right (R) and left (L) channels. The stereo audio can be easily provided to speakers built into a gaming device such as the computing device 206. The stereo audio can also be easily provided to headphones worn by a gamer or other user.

[0146] Some applications such as games or videos generate surround sound audio. For example, surround sound audio may include technology for enriching the fidelity and depth of sound reproduction by using multiple audio channels from multiple speakers surrounding the listener, which are referred to as surround channels. Surround sound audio may use speakers placed in front of and behind the listener and four to seven or more independent audio channels so that the sound surrounds the listener. Surround sound audio may be generated according to a standard that specifies the encoding and decoding of audio data. An example of a surround sound standard is called 5.1 surround sound. The 5.1 standard requires six speakers, including a center (C) in front of the listener; a left front (FL) and a right front (FR) located in front of the listener and at an angle of 30 degrees off center to the listener; a left surround (Ls) and a right surround (Rs) at an angle of 100 to 120 degrees off center to the listener; and a subwoofer, the position of which is not specified. The 5.1 surround sound standard was published by the International Telecommunication Union (ITU) as Recommendation ITU-R BS.775-3 in August 2012. Another example of surround sound is known as 7.1 surround sound. 7.1 surround sound is similar to 5.1 surround sound, but has four surround channels instead of two, rear left (RL) and rear right (RR) at an angle of 135 to 150 degrees from the listener to the center; left side (SL) and right side (SR) at an angle of 90 to 110 degrees from the listener. The 7.1 surround sound standard was published by the ITU as Report ITU-R BS.2159-7 on February 2, 2015.

[0147] Surround sound audio is typically directed to the listener's position where the audio effect is best. Surround sound encoding presents a fixed or forward perspective of the sound field to the listener at that position. The technology enhances the perception of sound spatialization by utilizing sound localization. Sound localization refers to the listener's ability to identify the location or origin of detected sounds in terms of direction and distance. In a surround sound system, sound localization is achieved by using multiple discrete audio channels routed to a speaker array. Each respective audio channel is routed to a respective speaker.

[0148] Multi-channel or surround sound audio can enhance the listener's enjoyment. This is especially true for listeners enjoying videos, movies, or audio such as concerts in a home theater experience. This is also especially true for gamers playing games using a gaming system on a computing device 206.

[0149] The game virtual audio device 7616 provides device routing for the game audio source 7614. The game virtual audio device 7616 can form an audio logic device. The game virtual audio device 7616 can be, for example, a device for recording received audio data or for sending received audio data on a network. In contrast to the virtual audio device, a real audio device is a physical device such as a laptop audio system or a Bluetooth speaker. On a computer such as a laptop forming the computing device 206, the operating system will expose the digital device. The operating system treats the digital device as an addressable endpoint. There is no real physical system behind the addressable endpoint, including no hardware and no firmware. The game virtual audio device 7616 is only a device in the sense of the operating system. The game virtual audio device 7616 can present itself as a sound card or other physical device to the operating system of the computing device 206. It can be just an address to which the processing system writes audio data under the control of the operating system. The audio data can then be read and processed by other devices. However, the game virtual audio device 7616 is a virtual device because no physical device is associated with it.

[0150] In one embodiment, the game virtual audio device 7616 can be configured as a portion of a code stored in a memory for performing appropriate functions such as controlling a processing system. The game virtual audio device 7616 can receive multiple audio streams of a game audio source 7614 from a game audio input stream 7602 and combine them to form a target audio stream. The game virtual audio device 7616 is configured to route the target audio stream to an audio processing module 7618.

[0151] In one embodiment, the game virtual audio device 7616 includes source code that embeds functionality capable of routing a target audio stream to the audio processing module 7618. In practice, the game virtual audio device 7616 retransmits the audio data received on its audio receiver endpoint in the form of a plurality of first audio streams to be read by the audio processing module 7618 on an audio transmitter endpoint. In some embodiments in conjunction with a Windows operating system, the audio endpoint receiver of the game virtual audio device 7616 typically corresponds to a so-called Windows operating system audio endpoint receiver, and the audio endpoint transmitter of the game virtual audio device 7616 typically corresponds to a so-called Windows operating system audio endpoint source.

[0152] The audio processing module 7618 can implement custom digital signal processing effects. In some embodiments, the audio processing module 7618 can be embodied as a Windows audio processing object (APO). In the operating system, the Audio Processing Object (APO) provides software-based digital signal processing (DSP) for the Windows audio stream. APOs are COM host objects that contain algorithms that provide specific DSP effects called audio effects. Examples of APOs include graphic equalizers, reverb, vibrato, acoustic echo cancellation (AEC), and automatic gain control (AGC). APOs are COM-based, real-time, in-process objects. APOs provide programmable audio processing.

[0153] The chat audio input stream 7604 includes one or more chat audio sources 7620, a chat virtual audio device 7621, and a chat neural network effects service 7622. The chat audio input stream 7604 operates under the control of the GUI 7610 to select one or more audio streams from the chat audio sources 7620 and control audio processing of the selected audio streams in the chat neural network effects service 7622. In addition, the chat audio input stream 7604 operates under the control of the GUI 7610 to route the processed audio of the chat audio input stream 7604 to the output device 7608.

[0154] According to various aspects described herein, a player participating in a game or gaming application using computing device 206 can use a chat channel to verbally communicate with one or more teammates or others. For example, teammates can use a Voice over Internet Protocol (VoIP) application to communicate with each other. Examples of suitable applications include Applications and Application. In other examples, a gaming application may include a built-in voice communication channel. A gamer or other user may access the GUI 7610 or the operating system of the computing device 206 to select one or more chat audio sources 7620 for communication. Audio from the chat audio sources 7620 is provided to the audio processing system 7600 at the chat audio input stream 7604 and may include voice conversations of the gamer's teammates or other users associated with the audio processing system 7600.

[0155] The chat virtual audio device 7621 operates according to principles similar to those described herein for the game audio device 7616. The chat virtual audio device 7621 may be embodied as a module comprising software code. The chat virtual audio device 7621 may operate in response to control input from the GUI 7610 to select one or more of the plurality of chat audio sources 7620. The chat virtual audio device 7621 routes audio streams from the plurality of chat audio sources 7620 to a selected destination such as a neural network effect service 7622 in response to control data from the GUI 7610. In some embodiments, multiple instances of the chat virtual audio device 7621 may be used to access and route multiple respective audio streams from multiple respective chat audio sources 7620.

[0156] Neural network effects service 7622 provides audio processing for a user audio stream from chat audio input stream 7604. In some embodiments, neural network effects service 7624 implements an artificial intelligence routine for customizing one or more audio streams, such as an audio stream from chat audio source 7620. For example, neural network effects service 7624 may implement artificial intelligence-based noise cancellation for a user audio stream from microphone 7623 that operates to remove background noise originating from the environment of the user or gamer. Neural network effects service 7622 responds to control information from GUI 7610 to customize audio processing in an audio stream received from chat virtual audio device 7621. FIG. 7M to FIG. 7T An illustrative embodiment of a GUI is depicted that may embody GUI 7610. Additionally, GUI 7610 provides control information to route an audio stream from neural network effects service 7622 to one or more output devices 7608.

[0157] Capture audio stream 7606 includes microphone 7623, neural network effect service 7624, and microphone virtual audio device 7625. Capture audio stream 7606 corresponds to audio originating from a player or user interacting locally with a gaming system on computing device 206. For example, a user may interact with one or more gaming accessories such as a gaming console to participate in a game. As noted, the user may access other audio sources, such as music and video including audio from a local application or through a network such as the Internet. As part of the interaction, the user provides speech audio input that may be shared, for example, with a gaming teammate in a chat. Chat or other conversations from teammates are received at chat audio input stream 7604. Similarly, a reverse channel with a chat or other conversation from a player to a teammate is provided on capture audio stream 7606.

[0158] Microphone 7623 can include any microphone or other device for capturing audio locally at the user. In an example, the user wears headphones or a headset equipped with a microphone. In another example, a laptop or other computing device 206 includes a microphone for capturing user audio. The user audio stream from microphone 7623 is routed and processed according to the operating system of the computing device 206.

[0159] Neural network effects service 7624 provides audio processing for a user audio stream from microphone 7623. In some embodiments, neural network effects service 7624 implements an artificial intelligence routine for customizing one or more audio streams, such as a user audio stream. For example, neural network effects service 7624 may implement artificial intelligence-based noise cancellation for a user audio stream from microphone 7623 that operates to remove background noise originating from the environment of the user or gamer. In addition, neural network effects service 7624 responds to control information from GUI 7610 to customize audio processing in a user audio stream received from microphone 7623. FIG. 7M to FIG. 7T An illustrative embodiment of a GUI in which GUI 7610 may be embodied is depicted.

[0160] The user may wish to adjust how the user sounds on a team chat with teammates or other people who may hear the user. GUI 7610 can be used to adjust the processing by the neural network impact service 7624 or other audio processing system to adjust how the user sounds to others. For example, some users want their voice to sound a little deeper than others. Some users may want their voice to sound a little clearer than others, such as by adding noise reduction to the microphone audio stream. In another example, the user can select a pre-set audio processing configuration, such as a broadcaster preset, which modifies the user's voice to sound like a professional broadcast audio stream. In yet other examples, the user's voice can be completely modified to sound different, such as a male user who wants to sound female or a very young user who wants to sound old. In another example, a female user can modify her voice to sound male to avoid harassment that may occur online. The user can choose to add effects such as compression to remove relatively high frequencies or relatively low frequencies. In addition, the user can choose to apply parametric equalization to the user's voice audio stream to further customize the user's output audio. This can improve the intelligibility of the user's voice. In another example, a user may add an accent or remove an accent, such as adding an Australian accent to sound more charming or intelligent to an American teammate. As indicated, the user may be provided with a plurality of selectable preset audio modifications that may be applied by neural network effects service 7624. The selectable preset audio modifications may be accessed and selected by actuation of GUI 7610.

[0161] The microphone virtual audio device 7625 operates similarly to those described herein for the game virtual audio device 7616. The microphone virtual audio device 7625 may be embodied as a module including software code. The microphone virtual audio device 7625 may operate in response to control input from the GUI 7610 to select one or more of the further destinations of the audio stream from the neural network effect service 7624.

[0162] Thus, GUI 7610 provides control information to route the audio stream from neural network effect service 7622 to remote audio device 7612. In this example, remote audio device 7612 may include any of chat audio sources 7620, such as a VoIP application. Examples of suitable VoIP applications include Applications and Apply. Others can also be specified.

[0163] The audio processing system 7600 can create virtual audio devices, such as a game virtual audio device 7616, a chat virtual audio device 7621, and a microphone virtual audio device 7625. Such virtual audio devices can be accessed in the Windows operating system. Applications can be assigned to virtual audio devices. Virtual audio devices can be assigned names or other titles and can be actively selected by the user or gamer. Figure 7L In the example of , a single virtual audio device 7616 is illustrated for processing audio from a game such as Stream, and from a video such as Live video streaming services or sports video programming video source, and audio streams from sources such as The audio stream of the audio source of the audio stream service. In other embodiments, a corresponding virtual audio device can be established for each corresponding audio stream processed by the audio processing system 7600.

[0164] In an embodiment, the audio processing system can be run on a device such as a computing device 206, and processes the audio of some or all of the applications opened on the device. For example, if five applications are currently running on the device. If three applications in the five opened applications are routed to identical virtual audio device 7616 and audio processing module 7618, then these three applications will be subjected to the same effect, such as volume and equalization and filtering. Similarly, if the other two applications are routed to different virtual audio devices similar to virtual audio device 7616 and different audio processing modules similar to audio processing module 7618, then the other two applications will be subjected to the same effect. Therefore, virtual audio device 7616 and audio processing module 7618 and similar devices for other audio streams can form a switching module for mapping or routing the audio stream from an application or other source to any audio destination, such as audio output device 7608, graphical user interface (GUI) 7610 and remote audio device 7612.

[0165] Figure 7L-1 and Figure 7L-2 shows a user interface for selecting device preferences Operating system controls. In particular, Figure 7L-1 and Figure 7L-2 2 shows the control of audio routing within a Windows device such as computing device 206. A user can access the Windows Control Panel to control audio stream routing. Figure 7L-1 and Figure 7L-2 , various audio sources are illustrated, including in this example System sounds, Discord, Microsoft Teams, OBS 27.0.1, and Slack. System sounds refer to audio generated internally by the Windows operating system. Discord, Microsoft Teams, Open Broadcaster Software (OBS), and Slack are applications for sharing audio and video within computing device 206 and between computing device 206 and other devices on a network. For example, in a gaming environment, teammates can use features of the game to chat or communicate verbally, or they can use third-party applications such as Discord or Discord.

[0166] Figure 7L-1 and Figure 7L-2 The Windows Control Panel allows users to control the association of audio sources and audio destinations. Users can select input devices and output devices for each application. Figure 7L-1 In the example, the audio stream from the Discord application is selected as the default input. Figure 7L-2In the example of , the audio stream from the Discord application is selected to be routed to a destination designated as a SoundStage chat capture device (SteelSeries SoundStage device). In other examples, Figure 7L-1 and Figure 7L-2 The apps listed on the left side of can include additional apps for music, streaming video, broadcast TV, and other sources.

[0167] The operating system of the computing device 206, in this example, The operating system can therefore be used to map audio streams from audio sources to audio destinations. Audio sources include games, applications, network locations, and any other source of audio to the computing device. Mapping audio streams can further include mapping or routing specific audio streams to specific audio processing modules 7618, such as APO or neural network effect service 7622.

[0168] In some embodiments, the audio processing system 7600 can automatically characterize audio events that occur in a game audio stream, a chat audio stream, a microphone audio stream, or some combination of these audio streams. In some embodiments, machine learning techniques can be used to characterize or outline audio events in an audio stream to identify a predetermined audio event associated with another player, such as footsteps or gunshots. In an example, a live game audio stream can be provided to a neural network. Any suitable data such as audio from other games or conversations can be used to train the neural network. The training data can be marked, for example, to identify predetermined audio events such as gunshots or footsteps. Different features can be further trained and identified, such as footsteps on wet pavement or gunshots of ricochet attacks. Once trained and provided with live audio, a neural network or other processing module can generate an indication when a predetermined audio event has occurred. In some embodiments, the indication is a value corresponding to the probability that a predetermined audio event has occurred. If the probability exceeds a threshold, such as 75% or 95%, the audio processing system can conclude that a predetermined event has been detected.

[0169] In response to detecting a predetermined audio event, the audio processing system can take further steps to enhance or adjust the audio corresponding to the predetermined audio event. In an example, the audio processing system can automatically adjust the parametric equalizer to strengthen the spectral components of the predetermined audio event. For example, if footsteps are detected, the spectrum can be adjusted in the parametric equalizer to strengthen the footsteps to the listener, the player. In addition, if the player is using a headset or other audio device that provides directivity or other surround sound effects, the audio can be automatically adjusted to strengthen the origin direction of the predetermined event. For example, if the footsteps in the game come from the back of the player, the audio of the footsteps in the surround sound arrangement will strengthen the direction, making the audio from the back louder or clearer, and the footsteps are more obvious. Other sounds from this area can be suppressed to strengthen footprints. In the video game where the player has an associated avatar, the directionality of the sound can be processed according to the position and orientation of the avatar in the game.

[0170] In other embodiments, a graphical user interface or other visual interface may be activated to provide a visual indication to the player of a detected predetermined audio event. For example, the graphical user interface may flash text stating "Footsteps behind," "Possible gunshot detected to the right." The nature of the visual indication may be based on the nature of the detected audio event and the probability or likelihood of detection, plus any other information that may be derived from the audio stream.

[0171] Further, in some embodiments, GUI 7610 may be used to control the association of audio sources and audio destinations. Figure 7M and Figure 7M-1 An example is shown in which the GUI 7610 of the audio processing system 7600 can be used to access the Windows Control Panel to manage audio stream configurations. In the illustrated embodiment, a Windows Control Panel image is generated on a display screen displaying the GUI 7610. In other embodiments, a custom graphical user interface display (not shown) with appropriate controls can be generated that enables a user to access and modify the mapping or routing of a particular audio stream. Any suitable interface controls can be provided, including mouse-based controls, touch-based controls on a touch-sensitive screen, and voice-based controls.

[0172] In some embodiments, the audio processing system 7600 can automatically detect one or more audio sources and one or more audio destinations, and automatically route audio streams from the sources to the destinations. In some embodiments, the audio processing system 7600 can pre-populate a routing matrix, which can then be manually modified by a user, such as by accessing a program similar to Figure 7L-1 and Figure 7L-2 As shown in or similar to Figure 7M-1 or Figure 7M-27610. The audio processing system 7600 may select a set of audio applications known to be the most popular applications and pre-populate the menu with those applications. For example, the most popular Voice over Internet Protocol (VoIP) application may be selected by default. In another example, if it is known that the user has a preference for an application, the audio processing system 7600 may select the preferred application by default. The user may access the GUI 7610 and modify the selection as needed. The default selection for audio routing and audio processing is automatically selected for the user when the game system or other application is started to simplify and streamline the user's process. The user starts the game and automatically establishes his preferred audio settings. In addition, if the user has different preferences for different games or different types of games, or for different types of applications such as music, video, and streaming video, the audio processing system 7600 automatically detects the type of game or the type of application and selects the user's preferred settings for each audio stream. The information defining the user's preferred settings may be stored at any convenient location, such as locally stored at the computing device 206 or remotely stored at a server or other storage location accessible through a network such as the public Internet. The user's preferred settings may be part of the user's profile or configuration file.

[0173] In some examples, the audio processing system 7600 can apply a head-related transfer function (HRTF). HRTF is a method for generating virtual surround sound. It can be used to simulate virtual objects in a virtual surround environment using two speakers. Human sound positioning can depend on HRTF in the frequency domain and head-related impulse response (HRIR) in the time domain. HRTF involves the acoustic footprint from the position in space to the ear of a person. Due to the acoustic reflection and diffraction phenomena of the listener's body, shoulders, head and auricle, the sound waves propagating from the position in space to the ear of a person will be modified. HRTF contains so-called positioning clues, which are information used by the human brain to decode the direction of sound events. Hearing is affected by interaural time difference (ITD) and interaural level difference (ILD). These are mainly caused by the acoustic head shadow effect. They enable the listener to perform lateral positioning (left / right), but they cannot provide enough information to accurately locate the sound source.

[0174] HRTFs include various spectral clues used by the user to locate sounds. These include notches and peaks in the audio spectrum. These are primarily due to reflections on the listener's shoulders (low frequencies) and reflections or diffractions off the listener's pinnae (high frequencies). Such spectral clues provide information used by the listener's brain to decode the front / back and top / bottom locations of sound sources.

[0175] Spatial sound on headphones uses HRTFs to simulate sound sources at precise locations around the listener. Most conventional spatial sound or virtual surround sound solutions use a unique generic HRTF profile that is developed from a unique person or a virtual head. In an example, if S(f) is the signal of the sound source, the stereo signal played back by the headphones is obtained by filtering it with the appropriate left and right HRTFs. This can be described as follows for the left (L) ear or channel and the right (R) ear or channel, respectively:

[0176] S L (f) = S(f)*HRTF L (f,direction person)

[0177] S R (f) = S(f)*HRTF R (f,direction person).

[0178] Generic HRTF profiles cannot fit every listener, and positioning cues will be correct for some listeners and not for others. Positioning cues depend on the listener's anatomy. Typical disadvantages of using generic HRTF profiles are front / back and up / down positioning distortions. For example, a sound intended to be heard in front of the listener is heard by the listener behind the listener. Obtaining a personal HRTF for a user is cumbersome, expensive, and requires a lot of time and equipment, and is unrealistic in most cases. Some conventional methods for HRTF customization include using pictures of the listener's ears to derive an adapted HRTF profile, or allowing the user to choose from fine-tuned versions of a single HRTF profile.

[0179] Figure 7L-3 The process of a listener selecting an HRTF profile for user use is shown. Figure 7L-3 The technology uses a version of A / B testing to allow the user to select the HRTF that sounds best to the listener. Figure 7L-3 In the process illustrated in FIG. 1 , several HRTF profiles are presented to the user. The user is guided through a series of A / B comparison tests to select the best or most appropriate HRTF for the user. The HRTF is then applied to an audio processing system such as Figure 7L Audio Processing System 7600.

[0180] In the example embodiment, a series of eight HRTF profiles are presented to the listener. The eight profiles are arbitrarily selected. Any suitable number may be used. Figure 7L-3In the example embodiment, the HRTF profiles are labeled A1, B1, A2, B2, A3, B3, A4, and B4. Each HRTF proposal in the embodiment is based on a different HRTF developed for another person and is relevant to people of different morphologies, or with different anatomical structures that will affect the HRTF. In the example embodiment, eight HRTF profiles represent multiple morphologies or people.

[0181] Figure 7L-3 The process is a guided selection by the user of the best or optimal or closest fitting HRTF profile. The process can be described as a tournament of several A / B listening tests. Therefore, in the first round 7626 of the tournament, a listening test and comparison between HRTF profile A1 and HRTF profile B1 is presented to the user. Further, in the first round 7626, a listening test and comparison between HRTF profile A2 and HRTF profile B2 is presented to the user. Further in the first round 7626, a listening test and comparison between HRTF profile A3 and HRTF profile B3, as well as a listening test and comparison between HRTF profile A4 and HRTF profile B4 are presented to the user.

[0182] Any suitable listening test may be provided. In some embodiments, an audio test file is presented to the listener that highlights common perceptual defects, such as difficulty in front / back discrimination. In an example, the listener first hears one or more audio test files as modified by HRTF profile A1 and the same one or more audio test files as modified by HRTF profile B1. The user selects or indicates whether profile A1 or profile B1 sounds better or more realistic or more natural to the listener, preferably focusing on the listener's ability to spatially locate the sounds played in the one or more audio test files. The selected profile A1 or B1 is judged as the winner of the first round 7626. The user continues with other A / B tests in the first round, including profile A2 versus profile B2; profile A3 versus profile B3; and profile A4 versus profile B4. For each A / B test in the first round 7626, the winner advances to the second round 7627.

[0183] The A / B testing process is repeated in a second round 7627. The four winners from the first round 7626 are subjected to two A / B tests, and the two winners of the second round 7627 advance to the final round. Once again, the winner of each A / B test is the HRTF profile that is judged by the listener to be the best for that listener, focusing on the spatial location of the sounds of the audio test file. A final round 7628 is conducted with similar A / B testing, and the final winner HRTF profile is selected as the best or optimal HRTF profile for the listener among the eight HRTF profiles initially selected.

[0184] exist Figure 7L-3Throughout the process shown in the figure, users can interact with Figure 7L The graphical user interface may provide written instructions or audible instructions to the listener to guide the listener through each A / B test and through each round including the first round 7626, the second round 7627, and the final round 7628. The instructions and guidance may be as detailed as the listener desires. Typically, Figure 7L-3 The process needs to be performed once to select the best HRTF profile for the listener.

[0185] Using a tournament format for selecting the best HRTF profile for a listener simplifies the process of selecting the best fit or best HRTF profile for the listener. The tournament format reduces the number of A / B listening tests required for the user to identify the best fit for the listener. The listening test allows the listener to make the HRTF profile selection that best fits the listener by directly comparing it to another HRTF profile. As the tournament proceeds through the first round 7626, the second round 7627, and the final round 7628, the HRTF profile should generally be closer to the actual ideal HRTF profile for that particular listener. The winner of the final round 7628 is unlikely to be exactly the ideal HRTF profile for the listener, but is likely to be a good fit and the best fit from a wide selection of eight pre-selected HRTF profiles.

[0186] Once, for example, Figure 7L-3 The process shown in the figure has established the user's HRTF profile, then Figure 7L The HRTF profile is used in the audio processing system 7600 of the game audio processing system 7600. In some embodiments, the game audio input stream 7602 includes audio processing to establish HRTF processing. HRTF processing can be applied to audio streams encoded with 5.1 surround sound or 7.1 surround sound, for example, to provide a virtual surround sound effect when the user listens through two stereo speakers. For example, the audio processing module 7618 may include an HRTF module that operates to perform HRTF processing on audio streams and other streams received from game audio, movie or video audio to create a virtual surround sound effect for the user. HRTF processing can be applied after audio processing including parameter equalization and other functions. The user can use GUI 7610 or activate and control the HRTF module and HRTF processing in any other suitable manner.

[0187] Figure 7M-7T An illustrative embodiment of a graphical user interface display for an audio processing system of a gaming system in accordance with aspects of the subject disclosure is depicted. Figure 7LA user of the audio processing system 7600 can interact with the graphical user interface to control various aspects of the audio processing system 7600. In an exemplary embodiment, the user can participate in the game process with other players including teammates through the game system. For example, the user can interact with a personal computer, laptop computer or other device that implements the game system and communicates data with a network such as the public Internet. The game system generates an audio stream of game audio that is presented to the user through an audio output system such as headphones. The player and the player's teammates talk together in an audio stream called game chat. The game chat audio stream is received by the user's computer and presented to the user through an audio output system such as headphones. The user generates an audio stream by speaking into a microphone, for example as part of a game chat. The user's voice is converted into data by the microphone and transmitted to the user's teammates through the network.

[0188] Figure 7M An illustrative embodiment of a mixer panel 7630 of a graphical user interface is depicted. The mixer panel 7630 includes an audio stream selector 7631, a series of audio stream controllers 7632, and a series of output selectors 7633. In addition to Figure 7M In addition to the features illustrated in FIG. , other embodiments may include other features.

[0189] Figure 7L The audio processing system of the game is configured to process three separate audio streams: game, chat, microphone. Each audio stream is processed independently of the other audio streams. In the mixer panel 7630, the user can assign different audio devices such as headphones, speakers, microphones, etc. to each corresponding audio stream and fine-tune the volume of the corresponding audio stream. In addition, the user also has a master volume. Processing each audio stream separately allows the user to fine-tune the game sound without affecting the voice audio of teammates on the chat audio stream.

[0190] Audio stream selector 7631 allows the user to select and control audio stream. Audio stream selector 7631 can be used to select display mixer panel 7630, game audio control panel, chat audio control panel, and microphone audio control panel. Audio stream selector 7631 can be used to select, for example, using a mouse or other accessories, or by touching a touch-sensitive screen to select a specific audio stream. In the illustrated embodiment, mixer panel 7630 enables volume control of three separate audio streams (including game audio stream, team chat audio stream, and user microphone audio stream) and the master audio volume of the gaming system. The user can select a specific audio stream, and, for example, using a mouse, the volume slider control for the selected audio stream can be dragged to a desired level. In this way, the volume of each separate audio stream can be adjusted according to user preferences or needs at a given time. The master volume slider allows the user to control the total game audio volume from all sources.

[0191] Output selector 7633 is embodied as a pop-up menu that can be selected by the user using a mouse or other accessory. The pop-up menu is populated by the system with the audio devices that the computer's operating system has recognized. Figure 7M-1 An example of selecting an audio output device on a pop-up menu is shown. Typically, the identified audio devices include a speaker system and a microphone system detected by the operating system.

[0192] Specifies one of the identified audio devices as the default audio device. The operating system has a default multimedia device. The default multimedia device is the stream for the game. The drop-down menu of the output selector 7633 can be used to control where the audio output is directed. To select the chat, the user can use the menu to select where the user wants to redirect the audio, such as headphones, speakers, audio drivers. On the output selector 7633, the list of the drop-down menu is populated from known windows audio devices. Windows has a default audio communication device. The audio processing system sets the chat as the Windows default audio communication device, and all audio from the chat audio stream is routed to the audio selected in the drop-down. In this example, the user does not select where the audio comes from, but where the audio is redirected. The game itself outputs the audio to the default Windows setting, which can be set by the Windows operating system control panel.

[0193] Figure 7M-2 Shown in the An example of selecting sound input and sound output options in the device of the operating system. Other devices can be selected by actuating the pop-up menu of the output selector 7633. Figure 7M In the example shown in , the Arctis 7Plus headset system is selected as the output device for the game audio stream, the chat audio stream, and the microphone audio stream. Using the output selector 7633, the user can route the audio streams to different output devices. Figure 7M-3 Shown is the mixing of different audio levels for the game audio stream, chat audio stream, and player microphone audio stream. Figure 7N-1 Show

[0194] Therefore, mixer panel 7630 can enable separate audio processing for each of the game audio stream, chat audio stream, and microphone audio stream. By interacting with a graphical user interface including mixer panel 7630, a user can independently control the volume and audio effects of each of these streams.

[0195] Figure 7NAn illustrative embodiment of a parametric equalizer 7635 of a graphical user interface is depicted. An equalizer is a tool through which a user can adjust and control the sound to improve the audio during a game session or other situation. An equalizer allows the user to hear more or less of sounds from a specific frequency range. Adjusting each frequency allows the user to hear more of what the user wants to hear and less of what the user does not want to hear.

[0196] The parametric equalizer 7635 provides an interactive but easy-to-use professional-grade 10-band parametric equalizer (EQ) for fully customizing the user's game audio stream. The game audio stream includes an audio stream of game sounds generated by the game system to accompany the game visuals. For example, the user can reduce the explosion sound generated by the game and increase the important sounds of footsteps and reloading generated by the game so that the user can better understand the activities of the opponent in the game. The parametric equalizer allows the user to boost or cut the sound on a custom frequency band according to a set of parameters, thereby giving precise control. In particular, for the selected frequency band, the user can select the center frequency, amplitude, bandwidth and filter type. Any tuning will be applied to the user sound and visible in real time on the graph. In some embodiments, the tuning will only affect the game audio stream. Other audio streams will not be affected.

[0197] The graphical user interface allows the user to interact with the display on the display screen of the user's device using a mouse or other accessory. In the parametric equalizer 7635, the center frequency can be shifted by the user to any value between 20 Hz and 20 kHz by dragging one of the available frequency band points on the graph or by selecting a frequency band (from 1 to 10) and then using the frequency slider. Once selected, the amplitude of the frequency band can be tuned up to +12 dB or down to -12 dB directly on the graph or using the gain slider.

[0198] exist Figure 7N In the graphical user interface illustrated in , the user can select one of the points corresponding to a frequency band or filter and adjust the gain applied by the corresponding filter by dragging the point vertically up or down on the display. Higher values ​​on the screen correspond to higher gain for that filter. The continuous line in the graphical user interface shows the actual frequency or curve between the points being applied. Selecting a point allows the user to move the line. Curve smoothing is applied between adjacent points to create a smooth frequency curve. Figure 7N-1 An example of adjusting the parametric equalizer 7635 for a chat audio stream to improve the intelligibility of a teammate's speech conveyed to a user via the chat audio stream is shown.

[0199] In the illustrated embodiment, the parametric equalizer 7635 includes a parameter selector 7636, an equalization selector 7637, and a filter type selector 7638. The parametric equalizer 7635 implements a series of filters for controlling audio processing. Using the features of the parametric equalizer 7635, a user can independently adjust gain, frequency, quality (Q) factor, and apply audio filters to up to 10 data points or frequency bands to customize game audio.

[0200] Parameter selector 7636 enables the user to select a parameter to be controlled on the graphical user interface. The user interface is divided into a series of frequency bands of the audible spectrum. Figure 7N In the example of , the frequency bands are centered at 20 Hz, 50 Hz, 100 Hz, 200 Hz, 500 Hz, 1 kHz, 2 kHz, 5 kHz, 10 kHz, and 20 kHz. These center frequencies are exemplary only. The center frequencies may be selected to emphasize certain areas of the audible spectrum, such as relatively low frequencies. The selection may be based on the specific audio content of the game being played or some aspect of the game being played. For example, in a game involving a player walking and shooting, relatively low frequency sounds carry a lot of information useful for the player to understand the unseen action of the game.

[0201] In the illustrated embodiment, the user can individually control the gain of the selected frequency band. Each corresponding frequency band corresponds to a corresponding filter in the filter series forming the parametric equalizer 7635. In the example shown, the frequency slider is actuated to select a frequency of 32.03 Hz, and the gain slider is used to adjust the gain at 32.03 Hz to a value of 2.1 dB. In addition, the Q factor slider can be actuated to adjust the Q factor of each corresponding filter. The Q factor slider adjusts the Q factor value of each individual filter. The Q factor is related to the width of the frequency modified by the filter. A narrow filter bandwidth corresponds to a smaller Q factor value and means that gain is applied to a smaller frequency range. A wider filter bandwidth corresponds to a larger Q value.

[0202] Once a frequency band is selected, the bandwidth of the frequency band can be widened or narrowed via the Q factor slider. Typically, the amplitude is inversely related to the Q factor. In a gaming environment, the Q factor can be useful for three types of filters, including peak filters, notch filters, and bandpass filters. For these filters, the higher the value of the Q factor, the narrower the curve or bandwidth affected by the filter. A peak filter with a low Q factor would be useful for controlling the timbre of a sound, for example, in situations where a user wants to add bass to give a car engine a more powerful sound or feel. A peak filter with a high Q factor would be useful for highlighting or de-emphasizing a certain frequency, for example, in situations where a user wants to highlight the sound of a bomb in a particular game such as CS: Go.

[0203] As noted, the Q factor is the parameter that relates the center frequency of a filter to its bandwidth. Figure 7N-2 The adjustment of the Q factor of a filter with a relatively narrow bandwidth is shown. The value of the Q factor is 10.00. Figure 7N-3 The adjustment of the Q factor of a filter with a relatively wide bandwidth is shown. The value of the Q factor is 0.5. Figure 7N-2 and Figure 7N-3 An example is also shown in which, when a mouse or other accessory is used to select a filter for adjustment, the current values ​​of the gain, frequency, and Q factor of the selected filter are displayed on the screen near the cursor to simplify the user's adjustment and improve the user's adjustment accuracy.

[0204] Filter type selector 7638 allows the user to select between different available filter types for parametric equalizer. Once selected, for each frequency band, the filter type can be selected in the filter type drop-down menu. In this example, the available filter types include bypass filter, low pass filter, high pass filter, peak filter, low shelf filter, high shelf filter. Other filter types can be specified, including combinations of filters. In a gaming context, a peak filter, low shelf filter, or high shelf filter with a low Q factor may be useful for enhancing the user's sense of entertainment. In addition, a peak filter and a notch filter with a high Q factor may be useful for giving a gamer a competitive advantage in a gaming context. In many applications, a bandpass filter is useless during a gaming session. However, a bandpass filter may be very useful for searching for the frequency of a particular sound. For example, a user may want to know what frequency is associated with footsteps in a gaming audio stream. A bandpass filter allows the user to listen to only a small frequency region or band.

[0205] Once the parametric equalizer has been used to establish a set of preferences for audio processing for game audio streams, chat audio streams, and microphone audio streams, the established values ​​can be stored by the user for subsequent use. The user can store data for a single audio stream, or the user can store data for multiple audio streams. The data can be stored on the user's computer, or can be transmitted over a network to be stored at a remote location such as in the cloud. The data can be retrieved for editing or other modifications. In some embodiments, the user can share a profile or configuration with another gamer. This can be done in any convenient manner. In addition, the profile or configuration can be specific to a specific game that the user is playing in multiple games. Therefore, the user can store a first configuration or profile for the Fortnite game, and store a second configuration or profile for the World of Warcraft game.

[0206] Fig.7OAn exemplary embodiment of a graphical user interface is shown, in which a user can apply a predetermined configuration or profile to a corresponding audio stream. The graphical user interface includes a configuration selector 7640. The configuration selector 7640 includes an audio stream selector 7641 and a configuration pop-up menu 7642. The user can use a mouse or another accessory to select an audio stream of interest in order to apply a profile or configuration. In an exemplary embodiment, the user can select a mixer, a game audio stream, a chat audio stream, or a microphone audio stream for the application of the configuration. Once the audio stream of interest is selected, the user can use a pop-up menu to select a predetermined profile or configuration for the selected audio stream. In one example, for the game audio stream, the user has selected a configuration titled "Fortnite footsteps", which may include data defining a set of audio settings for a parametric equalizer, which improves the clarity of footsteps heard by the user during the game. Adjusting the frequency, gain, and Q factor of a set of filters can greatly improve the user's ability to distinguish footsteps during the game process.

[0207] Thus, using the graphical user interface, a user can create, copy, edit, and delete configurations for each unique audio stream. This allows the user to change the audio stream effect with just one touch to select a predetermined configuration, without having to select each specific filter setting again. In an embodiment, a configuration can be set and applied to each individual audio stream to maximize personalization for each gamer.

[0208] In certain embodiments, when the user starts the game, the audio processing system will automatically retrieve and realize the user's configuration.For example, the user provides identification information, such as logging into a game application or website.Based on the identification information or other information provided by the user, the audio processing software can select the pre-stored configuration for the user of this game.In certain embodiments, the process of retrieving the pre-stored configuration of the user can be completed automatically based on user information or other information.In other embodiments, the user can be prompted to select a group of configuration or profile.In other embodiments, the user may have to actively locate and apply a pre-stored configuration set for the user of the game or a profile for the user of the game.

[0209] In other examples, a set of preset configurations can be made available to users of some games, or to some users of some games. For example, a manufacturer can design a set of profiles or configuration files that a user can access and select to apply to the user's audio processing system. For example, access to preset configuration files can be subscription-based. In addition, gamers can prepare libraries of profiles or configurations for different games and different preferences and store them in a common location. Members of a gamer group can retrieve, modify and improve stored profiles to enhance the enjoyment of the group.

[0210] Figure 7P An exemplary embodiment of a graphical user interface is shown in which a user can apply basic audio rendering presets so that audio effects can be quickly and easily applied to each individual audio stream. The graphical user interface includes a game audio rendering effect selector 7645. The game audio rendering effect selector 7645 in this example includes an audio stream selector 7646 and an audio adjustment panel 7647. Other examples may include other adjustments or features.

[0211] Audio stream selector 7646 allows the user to select the audio stream to which the audio rendering preset is to be applied. In this example, the user can select between a mixer, a game audio stream, a chat audio stream, and a microphone audio stream.

[0212] The audio adjustment panel 7647 includes a set of sliders that can be actuated by the user using a mouse or other accessories. Each slider is associated with an audio adjustment. Each slider is also associated with a switch selector that can be actuated by the user on the graphical user interface to enable or disable a specific audio adjustment. In an example embodiment, the available audio adjustments include bass boost, treble boost, voice clarity, smart volume, overall gain, and virtual surround.

[0213] Tuning the Bass Boost audio adjustment allows the user to boost or cut low frequencies. In one example, the bass frequencies are around 80Hz. This can have the effect of making deep low frequency sounds (such as explosions and engine sounds) in the game audio stream more or less audible. In an example, the Bass Boost audio adjustment can be tuned through a slider in a range from +12dB to -12dB.

[0214] The audio adjustment allows the user to boost or cut high frequencies. In the example, the treble frequency is around 8000Hz. This can have the effect of making high frequency sounds (such as footsteps, gunshots, or whistles) more or less audible in the game audio stream. This can give the user a real advantage in a shooter game, for example. In one example, the treble boost can be tuned via a slider in a range from -12dB up to +12dB.

[0215] The voice clarity audio adjustment recognizes and processes the dialogue or speech in the audio stream generated by the game. This means that tuning the voice clarity through the slider enables the user to boost or cut the voice in the game, movie, and incoming communication. In the example, the voice enhancement audio adjustment can be adjusted from -12dB to +12dB through the slider.

[0216] Some audio streams have a very large dynamic range, or the difference between the loudest sounds and the softest sounds. Smart Volume audio adjustment automatically makes small adjustments and helps maintain a steady volume in the user's voice. For example, if a user is constantly adjusting the volume on the user's computer while playing a game or listening to music, Smart Volume will solve this problem. In the example, the user can tune the impact of this audio effect on the sound from 0 to 1 with a slider.

[0217] The general gain audio adjustment only allows the user to adjust the overall volume. Such a feature is useful when the audio source volume is low. In an example, the general gain audio adjustment can be adjusted through a slider from 0dB up to +12dB. Moreover, the general gain audio adjustment does not allow exceeding the system limit on the audio volume or gain.

[0218] Virtual surround sound audio adjustment can be used in an arrangement that includes stereo equipment such as a headset, or internal speakers of a notebook computer. In one embodiment, a multi-channel audio stream such as 5.1-channel audio or 7.1-channel audio is available, and virtual surround sound audio adjustment is turned on to recreate a multi-channel three-dimensional audio listening experience on the user's stereo headphones or speakers. When activated, the virtual surround sound audio adjustment operation virtualizes the multi-channel game audio stream from the game engine, or any other media content, and mixes the audio stream to obtain a multi-channel listening experience on the user's stereo headphones or speakers. On the other hand, if a multi-channel audio stream such as a stereo content is not available, the virtual surround sound audio adjustment operation is turned on to expand the stereo sound to a wider sound field.

[0219] In another example, if the user's audio device is a multi-channel device such as a home theater system, and if a multi-channel audio stream such as 5.1-channel or 7.1-channel audio is available, turning on virtual surround sound audio adjustment will virtualize the multi-channel audio stream on the left and right speakers. In an embodiment, only the left and right speakers will output sound. In this case, it may be preferable not to activate virtual surround sound audio adjustment because a true surround sound effect is available. In this case, if a multi-channel audio stream such as stereo content is not available, turning on virtual surround sound audio adjustment operation to expand the stereo sound to a wider sound field. In this example, only the left and right speakers can output sound.

[0220] Figure 7Q An exemplary embodiment of a graphical user interface including a chat equalizer 7650 is shown. The chat equalizer 7650 in the example is embodied as a ten-band graphic equalizer for a team chat audio stream. The chat equalizer 7650 includes an audio stream selector 7651, a series of audio stream controllers 7652. In addition to Figure 7QIn addition to the features shown in , other embodiments may also include other features.

[0221] Audio stream selector 7651 allows the user to select an audio stream to apply audio equalization. In this example, the user can choose between a mixer, a game audio stream, a chat audio stream, and a microphone audio stream. Figure 7Q 7652. The chat equalizer allows a user to customize the audio of the team chat to improve the way a user hears teammates or others speaking on the chat audio stream. In a typical embodiment, a game is played on a gaming system where two or more players work together as teammates. Each player has access to a microphone and speakers, such as an integrated headset that the player can wear. The players can use the microphone and speakers to talk to each other, or more generally share voice and non-voice audible audio. The chat audio stream can include non-voice such as screaming or laughter. The audio stream controller 7652 allows the user to adjust the gain up and down across ten frequency bands.

[0222] In the illustrated embodiment, the chat equalizer 7650 includes a configuration selector 7653. A user can create and store one or more configurations or profiles. For example, after a user adjusts the audio stream controller 7652 to customize the sound processing for the chat audio stream according to user preferences, the user can save the data currently adjusted by the device. The data can be saved as a profile file or configuration file and can be saved locally or remotely. The saved profile can be shared between users or can be part of an available configuration library. Profiles can be retrieved and edited by the user. The configuration selector 7653 in this example includes a pop-up menu that, when actuated by the user, displays the available profiles or configurations. In some examples, a profile or configuration can be designated as a default profile and automatically loaded when a game starts. Profiles can be established for corresponding games according to user preferences.

[0223] Figure 7R An exemplary embodiment of a graphical user interface is shown in which a user can apply basic audio rendering presets so that audio effects can be quickly and easily applied to a chat audio stream. The graphical user interface includes a chat audio rendering effect selector 7655. The chat audio rendering effect selector 7655 in the example includes an audio stream selector 7656 and an audio adjustment panel 7657. Other examples may include other adjustments or features.

[0224] Audio stream selector 7656 allows the user to select the audio stream to which the audio rendering preset is to be applied. In this example, the user can select between the mixer, the game audio stream, the chat audio stream, and the microphone audio stream. In the example shown, the chat audio stream is selected.

[0225] The audio adjustment panel 7657 allows the user to select and adjust multiple aspects of the chat audio stream. In the example shown, the user accesses noise reduction audio adjustment, volume stabilizer audio adjustment, noise gate audio adjustment, automatic noise gate audio adjustment, and impact noise reduction audio adjustment.

[0226] The noise reduction audio adjustment operates to remove static noise from the user's voice by cleaning up spoken words and sentences. This audio adjustment is particularly useful for removing constant noise such as a personal computer fan. The user can tune the effect of the noise reduction audio adjustment via a slider from 0 for a quiet environment to 1 for a noisy environment.

[0227] Smart Volume Audio Leveling automatically makes small adjustments and helps maintain a steady volume, especially when the chat audio stream has a relatively high dynamic range. Smart Volume Audio Leveling eliminates the need for users to constantly adjust their computer volume during gaming sessions or when listening to music. Users can adjust the effect of Smart Volume Audio Leveling on their voices via a slider ranging from 0 to 1.

[0228] For a given volume threshold, the noise gate audio adjustment operates to cut off any sound thereafter. If the sound is above the predetermined noise gate threshold, the audio processing system lets all sound through. In other words, the sound is allowed to pass through the gate. On the other hand, if the sound is below the predetermined noise gate threshold, the sound does not pass through the gate. The predetermined noise gate threshold can be changed within the range of -60dB to 10dB via the threshold slider. This effect can be particularly useful in situations where the user perceives annoying or distracting noise far behind the dialogue level.

[0229] For a given volume threshold, the noise gate audio adjustment will cut off any sound behind it. The automatic noise gate audio adjustment automatically calculates a predetermined noise gate threshold based on the detected background noise. In an embodiment, the automatic noise gate audio adjustment requires the noise gate audio adjustment to be on, i.e. both audio adjustments must be activated. The automatic noise gate audio adjustment operates automatically. However, the user has a layer of control over the automatic noise gate audio adjustment via an offset slider that allows the user to apply a gain over the calculated threshold in a range from -20dB to +20dB. The automatic noise gate is useful if the user requires a noise gate, but the user is not sure which threshold to use.

[0230] The impact noise reduction audio adjustment operates to detect and suppress impact noise from a user chat audio stream or input device. The impact noise reduction audio adjustment is useful in reducing noise from sources such as keystrokes on a keyboard or noise from mouse clicks.

[0231] Figure 7SAn exemplary embodiment of a graphical user interface including a microphone equalizer 7660 is shown. The microphone equalizer 7660 in this example is embodied as a ten-band graphic equalizer for equalizing the user's microphone audio stream. The microphone equalizer 7660 includes an audio stream selector 7661 and a series of audio stream controllers 7662. In addition to Figure 7S In addition to the features illustrated in FIG. , other embodiments may include other features.

[0232] Audio stream selector 7661 allows the user to select an audio stream to which audio equalization is applied. In this example, the user can select between a mixer, a game audio stream, a chat audio stream, and a microphone audio stream. Figure 7S . The microphone equalizer allows the user to customize the audio of the microphone provided to the team chat user to improve the way teammates or others hear the user speaking on the chat audio stream. In a typical embodiment, the game is played by two or more players as teammates. Each player has access to a microphone and a speaker. The players can use the microphone and speaker to talk to each other in the chat audio, which can include non-speech such as screaming or laughing. The audio stream controller 7662 allows the user to adjust the gain up and down across ten frequency bands.

[0233] In the illustrated embodiment, the chat equalizer 7650 includes a configuration selector 7663. Similar to the profiles for the game audio stream and the chat audio stream, a user can create and store one or more configurations or profiles. For example, after the user adjusts the audio stream controller 7662 to customize the sound processing for the microphone audio stream according to the user's preferences, the user can save the data currently adjusted by the device. The data can be saved as a profile file or configuration file and can be saved locally or remotely. The saved profile can be shared between users or can be part of an available configuration library. The profile can be retrieved and edited by the user. The configuration selector 7663 in this example includes a pop-up menu that displays available profiles or configurations when actuated by the user. In some examples, a profile or configuration can be designated as a default profile and automatically loaded when the game starts. Profiles can be established for the corresponding game according to user preferences.

[0234] Figure 7T An exemplary embodiment of a graphical user interface is shown in which a user can apply basic audio rendering presets so that audio effects can be quickly and easily applied to a microphone audio stream generated by the user's own microphone. The microphone audio stream can be shared with teammates, for example, during a game session or at other times. In the example shown, the graphical user interface includes a microphone audio rendering effect selector 7655. In the example, the chat audio rendering effect selector 7665 includes an audio stream selector 7666 and an audio adjustment panel 7667. In the exemplary embodiment, the microphone audio rendering effect selector 7665 is configured to select a desired audio stream from the user's own microphone. Figure 7R The chat audio rendering effect selector 7655 illustrated in FIG. 76A operates similarly and includes features similar thereto. Other examples may include other adjustments or features.

[0235] Audio stream selector 7656 allows the user to select an audio stream to which an audio rendering preset is to be applied from the audio streams processed by the audio processing system. In this example, the user can select between a mixer, a game audio stream, a chat audio stream, and a microphone audio stream. In the example shown, the microphone audio stream is selected.

[0236] The audio adjustment panel 7657 allows the user to select and adjust multiple aspects of the microphone audio stream. In the example shown, the user accesses noise reduction audio adjustment, volume stabilizer audio adjustment, noise gate audio adjustment, automatic noise gate audio adjustment, and impact noise reduction audio adjustment.

[0237] The noise reduction audio adjustment operates to remove static noise from the sound stream created by the user's microphone by cleaning up spoken words and sentences. This audio adjustment is particularly useful for removing constant noise such as a personal computer fan or other persistent noise in the user's space that may be picked up by the user's microphone. The user can tune the effect of the noise reduction audio adjustment via a slider from 0 for a quiet environment to 1 for a noisy environment.

[0238] Smart Volume Audio Leveling automatically makes small adjustments and helps maintain a stable volume, especially when the microphone audio stream has a relatively high dynamic range. Smart Volume Audio Leveling eliminates the need for users to constantly adjust the computer volume during gaming or listening to music. Users can adjust the effect of Smart Volume Audio Leveling on the microphone audio using a slider ranging from 0 to 1.

[0239] For a given volume threshold, the noise gate audio adjustment operates to cut off any sound thereafter. If the sound is above the predetermined noise gate threshold, the audio processing system lets all sound through. In other words, the sound is allowed to pass through the gate. On the other hand, if the sound is below the predetermined noise gate threshold, the sound does not pass through the gate. The predetermined noise gate threshold can be changed within the range of -60dB to 10dB via the threshold slider. This effect can be particularly useful in situations where the user perceives annoying or distracting noise far behind the dialogue level.

[0240] For a given volume threshold, the noise gate audio adjustment will cut off any sound behind it. The automatic noise gate audio adjustment automatically calculates a predetermined noise gate threshold based on the detected background noise. In an embodiment, the automatic noise gate audio adjustment requires the noise gate audio adjustment to be on, i.e. both audio adjustments must be activated. The automatic noise gate audio adjustment operates automatically. However, the user has a layer of control over the automatic noise gate audio adjustment via an offset slider that allows the user to apply a gain over the calculated threshold in a range from -20dB to +20dB. The automatic noise gate is useful if the user requires a noise gate, but the user is not sure which threshold to use.

[0241] The impact noise reduction audio adjustment operates to detect and suppress impact noise from a user chat audio stream or input device. The impact noise reduction audio adjustment is useful in reducing noise from sources such as keystrokes on a keyboard or noise from mouse clicks.

[0242] Figure 7U An illustrative embodiment of a method 7670 utilized in the subject disclosure is depicted. The method 7670 may be performed by one or more software routines including data and instructions operating on a processing device such as computing device 206. The method 7670 may be performed on a gamer's standalone device such as a laptop or desktop computer. In other embodiments, the method 7670 may be performed by a device located remotely from the gamer (such as in a cloud network or at a game server) and accessible by the gamer's device.

[0243] Method 7670 can be used to detect a predetermined audio event in an audio stream, such as a game system output audio stream, such as the sound of a gunshot or footsteps or other sound, and if detected, modify the audio processing to affect the audibility of the audio event. For example, this can be done to make the audio event easier for the game player to hear or easier to distinguish from other sounds in the game system output stream. In another example, this can be done to no longer strengthen the audio event, such as removing distracting sounds from the audio stream.

[0244] The method begins at block 7672, where the gaming system initiates a game. This can be accomplished in any suitable manner, such as a player activating an application on a computing device or accessing a remote location such as a website over a network to initiate a game, or by a player joining an already initiated game. In an exemplary embodiment, a player can play a game as a member of a team with teammates against an opponent.

[0245] At box 7674, method 7670 includes receiving multiple game audio input streams from a video game. In the illustrated embodiment, multiple audio input streams include a game output audio stream, a gamer chat audio output stream, and a microphone audio output stream. In other embodiments, fewer, additional, or alternative audio streams may be received. In addition, in other embodiments, the received audio stream may originate from a source different from the video game, such as an application running on a user device or over a network. The received audio stream may include multiple audio streams and box 7674 may include a process of separating the received audio stream into constituent audio streams. In this example, the game output audio stream includes game sounds, such as explosion sounds, footstep sounds, gunshot sounds, door opening and closing sounds, music, and other sounds related to activities occurring in the video game. The gamer chat audio output stream includes spoken audio shared between a group of gamers (e.g., teammates in a team) participating in the game together. Gamer chats may include spoken words of voices in multiple languages, as well as non-voice sounds such as laughter, shouting, or screaming. The microphone audio output stream includes spoken audio of a local gamer to be shared with other gamers, such as a part of a gamer chat with a teammate. For example, microphone audio may include spoken words of speech in one or more languages ​​as well as non-speech sounds such as laughter and shouting.

[0246] At block 7676, a predetermined audio event is detected in the audio stream. The predetermined audio event may be any suitable sound or combination of sounds. In an example, the predetermined audio event may be the sound of footsteps in a game that the player is playing. In particular, the sound may be the sound of footsteps associated with the player's opponent in the game. In another example, the predetermined audio event may be the sound of a gunshot in a game that the player is playing. The gunshot may be associated with the player's opponent, or may be a random sound that occurs during the game. The detection of the audio event at block 7676 may be performed in any suitable manner. In combination Figure 7U Discuss example embodiments.

[0247] In these examples, the predetermined audio event is detected in a game output audio stream including a player chat audio output stream and a microphone audio output stream. In other examples, the predetermined audio event can be detected in a certain combination of a player chat audio stream or a microphone audio stream or a plurality of audio streams.

[0248] At box 7680, a predetermined audio profile is retrieved for processing the detected audio event. In an embodiment, the nature or identification of the audio profile is based on the nature or identification of the detected audio event. In one example, the sound of a gunshot is detected at box 7676, and the audio event is identified as a gunshot. Based on this identification, at box 7680, a predetermined audio profile is retrieved for processing the gunshot sound. In another example, the audio event detected at box 7676 is identified as a car door opening. At box 7680, a profile associated with the sound of the car door opening is retrieved. In addition, a more detailed or fine-grained identification of the audio event can be performed and used to select a specific profile at box 7680. For example, the steps of box 7676 can include distinguishing the sound of a single gunshot from the sound of multiple rounds of automatic weapons. In the first case, an audio profile for a single gunshot is retrieved. In the second case, an audio profile for multiple rounds of automatic weapons is retrieved.

[0249] The predetermined audio profile may be retrieved from any suitable location at block 7680. This includes local storage at the computing device on which the gaming system is running, or remote storage at a location, such as at a network location or in a network cloud accessible via a network location such as the public Internet. In some embodiments, the latency for retrieving the profile should be very short, such as in the range of 100 to 500 milliseconds, so that audio events can be processed in near real time with little audible delay for the listener.

[0250] The predetermined audio profiles may be selected from a group of previously established profiles provided by third party sources. For example, a gaming accessory manufacturer may supply and sell a set of audio filters for use with gaming accessories or for use with a particular gaming system. A set of recorded profiles may be available for the gamer to select and download over a network. In another example, the gamer may use trial and error to develop a custom profile to enhance, suppress, or otherwise modify specific sounds in the game audio stream. In one example, the gamer may use a graphical user interface to customize the parameters of a parametric equalizer and store the parameters for subsequent access and use. For example, FIG. 7M to FIG. 7T The graphical user interface illustrated in FIG. 1 may be used by a gamer to customize a profile for a user.

[0251] The predetermined audio profile can include any suitable data or information for processing the detected audio events according to the requirements or preferences of the listener. In some examples, the audio events can be made easier for the listener to hear. For example, in the case of playing games, the player can gain an advantage by hearing footsteps, particularly the footsteps of the opponent in the game. Then, for the sound of the detected footsteps, the predetermined audio profile can include the filter settings of the gain or volume in order to enhance the frequency of most of the energy of the footstep spectrum. Other frequencies can be suppressed by filtering or other suitable methods. And, in this example, the predetermined audio profile of the footstep sound can be included in the filtering in the low-pass filter with a specific roll-off frequency and a specific Q value or quality factor.

[0252] In some embodiments, the predetermined audio profile includes data and information for controlling a parametric equalizer. The parametric equalizer may be configured to process an audio stream according to settings established by a user, including settings such as filter frequency and gain and filter type and filter Q factor. The player may set the number of equalizer inputs according to user preferences or other factors. The predetermined audio profile may include at least one equalizer setting for a parametric equalizer, or a set of equalizer settings. The at least one equalizer setting of the profile replaces at least one equalizer input to modify the response of the parametric equalizer.

[0253] In box 7682, a predetermined audio profile is applied to a parametric equalizer. In addition, in box 7684, an audio stream is provided to a parametric equalizer. Under the control of the audio profile, the parametric equalizer operates to, for example, enhance the game system audio stream, to improve the ability of the player of the game system to hear the predetermined event in the game system audio stream. In one example, the spatial or temporal aspects of the input audio stream can be modified. For example, an audio processing system can identify the origin direction of the sound (e.g., footsteps) in the input audio stream. In this example, the player controls the avatar in the video game, and the direction or origin can be relative to the position of the avatar during the game. The audio processing system can modify aspects of the input audio stream, such as the audio spectrum of the game audio stream, to strengthen the origin direction of the footsteps. This can include, for example, including aspects of echo or reverberation, so that the position and direction of the footsteps are clearer. If the player wears headphones, and the audio stream provides a virtual surround sound experience for the player, this is particularly useful. The output of the parametric equalizer is a modified audio stream.

[0254] In embodiments where a parametric equalizer is not employed, any suitable device or method may be used to modify the input audio stream according to data, parameters or information of a predetermined audio profile. For example, the modification may be performed by digital signal processing or by analog signal processing.

[0255] At box 7886, the modified audio stream is provided to the audio device of the gamer or other user. For example, the gamer can wear a headset with a speaker and a microphone. The modified audio stream can be provided to the gamer's headset. In another example, the computing device used by the gamer has stereo speakers, and the modified audio is provided to those speakers. In some applications, the modified audio stream can be provided as a virtual surround sound audio stream, for example, by processing via a head-related transfer function.

[0256] Figure 7V Shows Figure 7U An exemplary embodiment of block 7676 of is for detecting a predetermined audio event in an input audio stream, such as an audio stream from a gaming system. At block 7690, the input audio stream is applied to an artificial intelligence engine, such as a neural network. The neural network has been trained to recognize or detect specific sounds or predetermined audio events using training data. For example, by being provided with training data corresponding to the sound of a gunshot, the neural network is able to recognize the sound of a gunshot in the input audio stream with a certain probability. Therefore, at block 7690, in response to receiving the input audio stream, the neural network returns a probability value in the range of 0 to 1. In the example, the larger the probability value, the greater the likelihood or confidence that a predetermined sound, such as a gunshot, is detected.

[0257] At block 7692, the probability value returned by the neural network is compared to a threshold. The threshold may be set at any level, such as 0.5, 0.8, or 0.95. In some examples, the threshold may be set dynamically based on factors such as the state of the game process. If the probability value does not exceed the threshold, then it is considered that no audio event has been detected. Control returns to Figure 7U 7674, and processing of the input audio continues. If the probability value exceeds the threshold, an audio event is considered to have been detected. Control returns to block 7680, and a predetermined audio profile is retrieved to process the detected audio event.

[0258] Although for the purpose of simplifying the explanation, the corresponding process is shown and described as Figure 7U and 7V The present invention is a series of blocks in the present invention, but it should be understood and appreciated that the subject matter claimed is not limited by the order of the blocks, and some blocks can be performed in a different order than that depicted and described herein, and / or performed concurrently with other blocks. In addition, the implementation of the method described herein may not require all the blocks shown.

[0259] Figure 7W and Figure 7X Depicted is an exemplary audio scene 7700 ( Figure 7W ) and audio mix 7702( Figure 7X). Audio scene 7700 illustrates an example of audio presented to a listener 7704. In this example, the listener 7704 uses a multi-channel audio arrangement for audio playback. In one example, the multi-channel audio arrangement includes Figure 7W 7704, the five speakers are labeled as front-L, center, front-R, surround-L, and surround-R. Further in this example, the audio scene 7700 includes several audio sources, including speech 7706, footsteps 7708, thunderstorm sounds 7710, vehicle sounds 7712, and gunshots 7714. In the exemplary audio scene 7700, the speech 7706 is located directly in front of the listener 7704, the footsteps 7708 are located to the right of the listener 7704, the thunderstorm sounds 7710 are located behind the listener 7704, the vehicle sounds are located to the left of the listener 7704, and the gunshots 7714 are located in front of the left of the listener 7704.

[0260] The audio scene 7700 can be used with, for example, a game console 206 ( Figure 2 ) is associated with a gaming system audio stream received during a video game played by a gaming system. The gaming system audio stream may be generated by the gaming system and may include a variety of sounds, including, for example, voices, footsteps, natural sounds such as thunderstorms, and artificial sounds such as vehicle sounds and gunfire. To help create an immersive experience for the listener, the gaming system audio stream may use surround sound capabilities and include, for example, Figure 7W 7704 so that sounds can appear from different directions relative to the listener 7704. In another example, as Figure 7W In addition to or in lieu of sounds of the type illustrated in , audio scene 7700 may also include music. Listener 7704 may listen to any type of audio content using a surround sound arrangement.

[0261] exist Figure 7X , audio mix 7702 shows the corresponding Figure 7W Example audio channels of audio scene 7700 of FIG. 7702. Audio mix 7702 shows the audio channels provided to Figure 7W The five audio channels of the loudspeaker are shown in FIG. 1 . These include front left (FL), front right (FR), center (C), surround left (SL), and surround right (SR). The vertical axis of each corresponding channel corresponds to the audio volume. The horizontal axis of each corresponding channel corresponds to time. The corresponding sounds, including gunshots, speech, and footsteps, occur simultaneously in time. Each audio channel receives a portion of each sound.

[0262] In the example, each respective audio channel illustrates the relative sound composition for a separate sound, including gunshots, spoken words, and footsteps. Figure 7X As shown on the left hand side of the audio scene 7700 ( Figure 7W ) that appears to originate from the left front of listener 7704 includes a relatively large audio volume or amplitude in the left front (FL) channel, a smaller amplitude in the center channel (C), and an even smaller amplitude in the left surround channel (SL). In addition, the right front (FR) channel and the right surround (SR) channel have essentially no audio volume or amplitude associated with the gunfire.

[0263] Similarly, for a speaker's spoken words, the front left and right channels have a medium audio volume or amplitude. Figure 7W The center channel has considerable audio volume or amplitude because the center channel appears to be located directly in front of the listener 7704 in the audio scene 7700. The two surround channels, left surround and right surround, have essentially no audio volume or amplitude.

[0264] Furthermore, in audio scene 7700, the footsteps appear to come from the right side of listener 7704. Figure 7X In the audio mix 7702 of , the right front channel and the right surround channel have substantially all audio volume or amplitude. In addition, in this example, because the gunshot and voice are relatively loud, these sounds have relatively large signal amplitudes relative to softer footsteps.

[0265] exist Figure 7X On the right hand side of the , a global mix of each of the five corresponding audio channels is shown. Each corresponding channel in the global mix represents the sum of all the different sounds heard from each source (including gunshots, voices, and footsteps). Therefore, for the time period shown, the left front channel is a combination of gunshot sounds and voice sounds. Similarly, for the time period shown, the right front channel is a combination of voice sounds and footsteps.

[0266] The global mix may correspond to the audio produced by the gaming system during the playing of the video game. Figure 7X The game audio stream is a global mix of audio signals comparable to the global mix. Positional information is provided by the five channels, including the mix of sound frequencies and amplitudes that drive the corresponding speakers. However, information about the source of any particular sound in this global mix is ​​not available from the information in the game audio stream. As a result, it is generally not possible to recover individual sounds from either the global mix or from the game audio stream.

[0267] It is desirable to improve the audio listening experience of some users. For example, for some gamers, it is beneficial to be able to better listen to some sounds or sound categories than are conventionally available in the game audio stream. Some individual gamers feel that better hearing footsteps can help gamers identify individuals around the gamer's position in the game environment. If the volume or sound of footsteps can be increased, or if footsteps can be made more prominent in the audio stream, gamers can better hear footsteps and be more competitive in the game. Similarly, suppressing some sounds may be helpful to gamers. For example, if a gamer believes that there is another person nearby in the game, the gamer may want to emphasize the footsteps in the game audio stream. If a gamer can hear footsteps, the gamer can locate potential enemies and be more aware of the actions in the game. Suppressing or reducing the volume or prominence of gunshots in the game can help gamers better hear other sounds in the audio stream. This can enable gamers to manipulate audio scenes such as audio scene 7700 to improve the gamer's competitiveness in the game or the fun of the game.

[0268] It is known to use equalization to attempt to separate individual sounds in an audio stream, such as a game audio stream. An equalizer circuit or a digital signal processor configured as an equalizer can receive a game audio stream and act globally on the channels of the audio stream of the audio scene 7700. The equalizer is operable to boost selected frequencies of interest and suppress other selected frequencies. However, equalization operates on all frequencies in a selected frequency band, and some individual sounds overlap in frequency. For example, footsteps and gunshots both have relatively large frequencies that overlap in frequency. Therefore, equalization of a game audio stream is generally not helpful in distinguishing or separating individual sounds in an audio stream. For music audio streams, it is known to use neural networks on a stereo audio stream to isolate individual instruments, such as a guitar.

[0269] According to some aspects of the present subject disclosure, a method and apparatus operate to extract a class of sound or multiple classes of sound within a multi-channel audio stream, such as the sound of footsteps. The extraction of such sounds is performed to, for example, enable manipulation of such sounds, such as increasing the gain of such sounds or reducing the gain of such sounds, or applying audio effects to such sounds. The effect may include equalization or compression of only the sounds of this category. The manipulation of the sounds of this category is completed without changing other sounds that are not in this category. Alternatively, manipulation may include reducing or removing all sounds that are not in the sound category of interest, leaving only the sounds in the category of interest. Sound categories may be defined in any suitable manner. In the example of creating a game audio stream of an audio scene 7700, sound categories may include gunshots, footsteps, vehicle noises, voices, and thunderstorm sounds. In the example of music, sound categories may include music produced by specific instruments such as drums or pianos.

[0270] Figure 7Y 77 is a block diagram of a system 7720 for extracting individual sounds from an audio stream according to some aspects of the subject disclosure. In an exemplary embodiment, the system 7720 includes a deep neural network 7722 that receives a multi-channel input audio stream 7724 and provides a multi-channel output audio stream 7726 corresponding to an individual sound or sound category of interest. Other embodiments of the system 7720 may include additional or alternative elements for extracting sound categories from the input audio stream 7724. For example, other types of machine learning modules or artificial intelligence modules may be used instead of deep neural networks. In some embodiments, the output audio stream 7726 includes multiple audio streams, where, for example, one audio stream includes sounds of a selected sound category such as footsteps or gunshots in the input audio stream 7724, and another audio stream includes the remaining sounds of the input audio stream 7724. A gamer can, for example, select which output audio stream to listen to during a specific time segment or time period in order to enhance his gaming experience or performance.

[0271] A deep neural network (DNN) 7722 is an artificial neural network having an input layer and an output layer and having multiple layers between the input layer and the output layer. The layers include artificial neurons, which include mathematical functions that receive one or more data or signal inputs and generate outputs according to a transfer function. The layers of artificial neurons are connected by synapses, and data or signals are transmitted between artificial neurons. Artificial neurons and synapses can have weights that change as DNN 7722 learns. The weights of artificial neurons or synapses will change the data or signals transmitted to subsequent artificial neurons. DNN 7722 can be constructed in any suitable form, including circuits and other hardware components, software, and a combination of the two.

[0272] System 7720 may be part of an audio processing system for processing one or more audio streams from an audio source. In one example, the one or more audio streams are output audio from a gaming system. Figure 7Y In the example, the input audio stream 7724 is marked as a game mix and is received from the game system. In an embodiment, the input audio stream 7724 may correspond to the game audio source 7614 ( Figure 7L ), and may be, for example, a gaming system, one or more applications running on a device such as computing device 206, and other sources such as network sources accessed by a user of the gaming system. In the illustrated embodiment, gaming audio source 7614 may be an application or online source. Input audio stream 7724 from gaming system may include audio streams that are associated with computing device 206 ( Figure 2) associated with the video displayed on the video display device. In the case of a war simulation game, for example, the audio input stream 7724 may include the sound of gunfire, footsteps of teammates or opponents, explosions, vehicle noises, speech including spoken words, and the like.

[0273] The DNN 7722 may be trained in any suitable manner to recognize and identify specific sounds or classes of sounds. Training involves selecting appropriate weights for each artificial neuron and each synapse. The training data may include a buffer of sound samples from the input audio stream 7724 or another source. The DNN 7722 learns to recognize sounds or classes of sounds from the training data. The training may be supervised, meaning that the training data is labeled to recognize sounds or classes of sounds, or unsupervised. In one embodiment, a single DNN 7722 is trained to recognize a single class of sounds. Thus, in Figure 7Y In the example of , DNN 7722 analyzes five channels of input audio stream 7724 to identify the sound of footsteps in input audio stream 7724. In other embodiments, a single DNN 7722 is trained to identify multiple classes of sounds. In yet another embodiment, DNN 7722 includes multiple DNN modules. Each corresponding module is trained to identify a class of sounds of interest. DNN 7722 may include appropriate input controls or interfaces such as an application programming interface (API) to receive control information defining, for example, what the class of sounds of interest is and other control information.

[0274] In one example, audio output from one or more games is obtained. This can be, for example, raw audio from a game without any preprocessing. The raw data can be used to simulate what a game engine can do and constitute training data for DNN 7722. In other embodiments, the training data can be increased by including audio from several or many different games or other sources. The training data should include intermediate mixes, especially dense mixes for the categories of interest. In order to operate on a music audio stream, the training data can include the final master of the sound track and the intermediate multi-track recording. The multi-track recording provides spatial information about where the sound is located in the stereo mix. The musical genre of the training data should match the musical genre of the music of interest.

[0275] In an embodiment, DNN 7722 receives input audio stream 7724 after being trained to extract a type of sound such as footsteps or gunshots. DNN 7722 converts input audio stream 7724 into Figure 7Y7722 is a game mix in a video game - separated into two components. The first component is the category of interest, such as footsteps. The second component is the rest of the input stream 7724, minus the sounds of the category of interest, or minus the sounds of footsteps. The sounds of footsteps have been extracted from the input audio stream, and the output audio stream 7726 includes a stream containing the sounds of footsteps and a stream containing the rest of the sounds from the input audio stream. Therefore, DNN 7722 operates a bit like a filter to select and filter out (or extract) the footsteps from the rest of the mix.

[0276] The DNN 7722 operates substantially in real time. In this example, substantially in real time means that the output audio stream 7726 is generated at the same rate as the input audio stream 7724 is received, with minimal delay between the input audio stream 7724 and the output audio stream 7726. In some embodiments, a buffer may be used to hold short segments of the sound of the input audio stream 7724 or the output audio stream 7726.

[0277] exist Figure 7Y In the example of , both the input audio stream 7724 and the output audio stream 7726 are multi-channel audio streams. That is, each corresponding audio stream includes multiple audio channels. In this example, five channel audio streams are used, including a left front channel, a right front channel, a center channel, a left surround channel, and a right surround channel. In other examples, the multi-channel audio stream may be a 5.1 audio stream, a 7.1 audio stream, a stereo audio stream, or the like. The channels of the input audio stream 7724 provide spatial information to the listener. In the example, the listener hears footsteps from the right side of the listener. Similarly, the channels of the output audio stream retain spatial information. In this way, the footsteps extracted by DNN 7722 retain spatial information for playback to the listener. A listener who only hears the footsteps hears them from the right hand side. The frequency and amplitude characteristics of the extracted right front channel and the extracted right surround channel of the output audio stream 7726 are the same as those of the audio stream 7724. Figure 7X The input audio stream described in is substantially similar. The rest of the output audio stream 7726 similarly retains spatial information. The frequency components and amplitudes of each of the channels of the multi-channel output audio stream substantially match the channels of the multi-channel input audio stream 7724.

[0278] Figure 7Z7730 is a block diagram of a system for extracting individual sounds from an audio stream according to some aspects of the subject disclosure. In an exemplary embodiment, the system 7730 includes a deep neural network 7722 that receives an input audio stream 7734 and provides an output audio stream 7736 corresponding to an individual sound or sound category of interest. The output audio stream 7736 may include multiple audio streams, including, for example, an audio stream with sounds of a sound category of interest, such as footsteps or spoken voice, and an audio stream with the remaining sounds of the input audio stream. The gamer can choose which output audio stream to hear or can modify the output audio stream to emphasize the features of the first audio stream and the second audio stream. In addition, the system 7730 includes multiple audio processing circuits 7732, which in the example include audio processing circuits 7732a, audio processing circuits 7732b, ... audio processing circuits 7732n and audio processing circuits 7738. Further, the system 7730 includes a mixer 7742. Other embodiments of the system 7730 may include additional or alternative elements for extracting various types of sounds from the input audio stream 7724. For example, other types of machine learning modules or artificial intelligence modules can be used instead of deep neural networks.

[0279] exist Figure 7Z In an embodiment of the invention, DNN 7722 is configured to extract several categories of sounds of interest. DNN 7722 receives an input audio stream 7734, which may include, for example, a few milliseconds of audio, such as a game mix from a gaming system. The game mix may include multiple categories of sounds of interest to the listener. Similar to Figure 7W For example, the first category (category 1) may include footsteps. The second category (category 2) may include thunderstorm sounds. The third category (category 3) may include vehicle sounds. The fourth category (category 4) may include gunshot sounds. The fifth category (category 5) may include voice or spoken conversations.

[0280] In the example, DNN 7722 receives a control signal at input 7740. In response to the control signal, the DNN extracts one or more of the sound categories of interest from the game mix of the input audio stream 7734. The extracted sounds from the categories of interest are provided as outputs from DNN 7722. This includes the sound of footsteps, the sound of gunfire, the sound of speech, etc. In addition, the rest of the mix that is not extracted is provided as being labeled "rest of the mix"; Figure 7Z middle.

[0281] In an embodiment, each of the outputs of the DNN 7722, including the extracted sound along with the rest of the mixed output, is time aligned or synchronized with the other outputs. In addition, in an embodiment, the DNN 7722 operates substantially in real time to extract the categories of interest from the input audio stream.

[0282] Multiple audio processing circuits 7732 can be individually controlled to customize the sound category provided to each corresponding audio processing circuit. For example, the audio processing circuit 7732 can provide gain to increase or decrease the level of a class of sounds. The audio processing circuit 7732 can provide equalization such as filtering or compression for each class of sounds. Therefore, each corresponding class of sounds can receive different audio processing.

[0283] In an embodiment, the type of audio processing provided by each of the audio processing circuits 7732 may be selected and controlled by the user. Other types of audio processing may also be provided. A user interface, such as a touch screen graphical user interface or voice input using voice recognition, may be provided to the user to enable the user to modify and enhance one or more categories of sounds associated with the gaming system. Thus, if the user wishes to hear footsteps and speech more clearly, suppress gunshots and vehicle sounds, the user may actuate the user interface accordingly, and the audio processing circuits 7732 will be adjusted to provide the necessary sound shaping and customization.

[0284] In an embodiment, each of the sound categories processed by DNN 7722 is a multi-channel audio stream. Figure 7W In an example of a multi-channel stream, spatial information is provided to the listener by providing channels having different frequencies and amplitudes, for example. The DNN 7722 operates on the multiple channels to extract specific categories of sounds from the input audio stream 7724. The spatial information is preserved in each of the extracted sound categories, such that a gunshot in the input audio stream 7734 originating from the front left of the listener continues to appear to originate from the front left after extraction by the DNN 7722 and audio processing by one of the audio processing circuits 7732.

[0285] The audio processing circuit 7732 may include any suitable processing operation, such as a digital signal processor. The audio processing circuit 7732 may include any combination of hardware and software. The audio processing circuit 7732 may all be substantially the same and may be modified based on, for example, control information received from a game player. In other embodiments, the audio processing circuit 7732 may be selected or configured to process specific sounds or sound categories. For example, footsteps and gunshots have relatively wide frequency bandwidths, but spoken speech from a gaming system may have relatively narrow bandwidths. Different components may be selected for each corresponding category of sound based on the corresponding bandwidth.

[0286] The mixer 7742 combines the output signals from the audio processing circuit 7732 and generates an output audio stream 7736. The output audio stream 7736 includes a modified mix of the game mix received by the input audio stream 7734. In an embodiment, the mixer 7742 operates under the control of the user, such as adjusting the desired audio characteristics through a user interface. For example, if the listener or user wants to hear the footsteps better, the DNN 7722 is trained to extract the footstep sounds from the input audio stream 7734. The DNN 7722 provides two streams as outputs, for example, a first audio stream including all footstep sounds, and a second audio stream including the rest of the game mix from the input audio stream 7734. In other embodiments, other combinations of output audio streams may be provided. In this example, the audio processing circuit 7732a will increase the gain of the footstep sounds in class 1, and the audio processing circuit 7738 will reduce the gain on the rest of the mix, which includes all sounds except the extracted footsteps. The two audio streams are then recombined in the mixer 7742 to produce a modified game mix. In the modified game mix, the sounds of footsteps are enhanced and the rest of the mix is ​​reduced.

[0287] Figure 8-9 An embodiment of a system with a corresponding communication flow diagram for associating incentives and game action results is shown. In this illustration, the user clicks the left button 119 of the game controller 115. The game controller 115 may include firmware (or circuitry) that creates the following Figure 8 The event described in Event 2 in the . Button is pressed and the event is created in Fig. 9 902 and 904. In step 904, the firmware of the game controller 115 may, for example, generate an event type "left button #3" and a unique GUID with a timestamp that is submitted to the AMS application. Return to Reference Figure 8 , the AMS application catalogs event 3 and, if an alternative stimulus has been predefined, remaps the event according to the alternative. The remapped event is then transmitted to the gaming application at event 4. Figure 8 Event 3 is depicted as Fig. 9 906 in FIG. 10. In this illustration, the AMS application replaces the left button #3 press actuation with a "keyboard 'F'" press that can be interpreted by the game application as a fire command. In this illustration, the AMS application continues to use the same GUID, but replaces that timestamp with another timestamp to identify when the replacement occurred.

[0288] Returning to event 4, the game application processes the event and sends the game action result back to the AMS application at event 5, which is processed by the AMS application at event 6. The AMS application then submits the result to the accessory at event 7. Events 4 and 5 are depicted as Fig. 9908 in FIG. 10 . In this step, the game application processes "F" as an action of the player firing a gun, and then determines an outcome from the action according to a logical game outcome generated by the game application. In this illustration, the firing action results in a hit. The game application submits an outcome type of "Hit" with a new timestamp to the AMS application, while utilizing the same GUID for tracking purposes. At step 910, the AMS application associates actuation "left button #3" (and / or alternate actuation keypad "F") with game outcome "F" and catalogs them in memory. The AMS application then submits the game action outcome "hit" to the accessory (e.g., game controller 115) at step 910, with the same GUID, and a new timestamp indicating when the outcome was received. Upon receiving the message from the AMS application, the accessory processes the "hit" at step 912 by activating a red LED on the accessory (e.g., left button 119 glows red or other LED of game controller 115 glows red) to indicate the hit. Other notifications may be used, such as another color of the LED to indicate a miss, a specific sound to indicate a hit or kill, vibration, or other suitable techniques for notifying the player of the game action outcome.

[0289] In reviewing the foregoing embodiments, it will be apparent to one of ordinary skill in the art that the embodiments of the subject disclosure may be modified, reduced, or enhanced without departing from the scope of the claims described below. For example, an AMS application may be executed from an accessory 115 or a computing device 206 to perform the embodiments described in the subject disclosure. The AMS application may also be operated from a remote server ("cloud service"). In another embodiment, the functionality of the AMS application may be distributed among devices. In yet another embodiment, the AMS application may be configured to track a player's performance and adjust thresholds as the player's performance improves or declines.

[0290] For example, as a player's performance improves through a particular game action, a threshold associated with the game action may be adapted to be less sensitive in detecting an overuse state. Similarly, if a player's performance declines due to overuse of a game action, the sensitivity of the threshold may be increased to quickly identify the overuse state of the game action. Additionally, when a player's performance significantly improves due to the use of an excluded game action, the AMS application may be adapted to add the game action to the exclusion list. The AMS application may also modify the exclusion list by removing the game action from the exclusion list in response to overuse of the game action causing the player's performance to decline.

[0291] Other embodiments may be applicable to the subject disclosure.

[0292] It should be understood that the devices described in the exemplary embodiments can communicate with each other via various wireless and / or wired methods. The method can be a link described as a coupling, connection, etc., which can include unidirectional and / or bidirectional communication over a wireless path and / or a wired path using one or more of various protocols or methods, wherein the coupling and / or connection can be direct (e.g., without an intermediate processing device) and / or indirect (e.g., an intermediate processing device such as a router).

[0293] Fig.10 An exemplary pictorial representation of a machine in the form of a computer system 1000 is depicted within which a set of instructions, when executed, may cause the machine to perform any one or more of the methodologies described above. One or more instances of the machine may operate, for example, as an accessory, a computing device, or a combination thereof. In some embodiments, the machine may be connected (e.g., using a network 1026) to other machines. In a networked deployment, the machine may operate in the capacity of a server or a client user machine in a server-client user network environment, or as a peer machine in a peer-to-peer (or distributed) network environment.

[0294] The machine may include a server computer, a client user computer, a personal computer (PC), a tablet computer, a smart phone, a laptop computer, a desktop computer, a control system, a network router, a switch or a bridge, or any machine capable of executing a set of instructions (sequential or otherwise) specifying actions to be taken by the machine. It will be understood that the communication devices disclosed in the subject matter broadly include any electronic device that provides voice, video or data communications. In addition, although a single machine is shown, the term "machine" should also be considered to include any collection of machines that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methods discussed herein.

[0295] The computer system 1000 may include a processor (or controller) 1002 (e.g., a central processing unit (CPU)), a graphics processing unit (GPU, or both), a main memory 1004, and a static memory 1006, which communicate with each other via a bus 1008. The computer system 1000 may also include a display unit 1010 (e.g., a liquid crystal display (LCD), a flat panel, or a solid-state display). The computer system 1000 may include an input device 1012 (e.g., a keyboard), a cursor control device 1014 (e.g., a mouse), a disk drive unit 1016, a signal generating device 1018 (e.g., a speaker or a remote control), and a network interface device 1020. In a distributed environment, the embodiments described in the subject disclosure may be adapted to utilize multiple display units 1010 controlled by two or more computer systems 1000. In such a configuration, the presentation described by the subject disclosure may be partially shown in a first display unit in the display units 1010, while the remaining portion is presented in a second display unit in the display units 1010.

[0296] The disk drive unit 1016 may include a tangible computer-readable storage medium 1022 on which is stored one or more sets of instructions (e.g., software or instructions 1024) embodying any one or more of the methods or functions described herein, including those methods shown above. The instructions 1024 may also reside completely or at least partially within the main memory 1004, the static memory 1006, and / or within the processor 1002 during execution thereof by the computer system 1000. The main memory 1004 and the processor 1002 may also constitute tangible computer-readable storage media.

[0297] Special hardware implementations including but not limited to application specific integrated circuits, programmable logic arrays, and other hardware devices can also be constructed to implement the methods described herein. Application specific integrated circuits and programmable logic arrays can use downloadable instructions to execute state machines and / or circuit configurations to implement the embodiments disclosed in this topic. Applications of the devices and systems that may include various embodiments widely include various electronic and computer systems. Some embodiments implement functions in two or more specific interconnected hardware modules or devices, and related control and data signals are transmitted between hardware modules or devices and through modules, or as part of application specific integrated circuits to implement functions. Therefore, the example system is suitable for software, firmware, and hardware implementations.

[0298] According to various embodiments disclosed in the subject matter, the operations or methods described herein are intended to be operated as software programs or instructions running on or executed by a computer processor or other computing device, which may include instructions in other forms of state machines implemented using logic components in a dedicated integrated circuit or field programmable gate array. In addition, software implementations (e.g., software programs, instructions, etc.) including but not limited to distributed processing or component / object distributed processing, parallel processing, or virtual machine processing may also be constructed to implement the methods described herein. It should be further noted that a computing device, such as a processor, controller, state machine, or other suitable device, used to execute instructions to perform an operation or method may perform such operations directly or indirectly with the aid of one or more intermediate devices directed by the computing device.

[0299] Although the tangible computer-readable storage medium 1022 is shown as a single medium in the example embodiment, the term "tangible computer-readable storage medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and / or associated caches and servers) storing the one or more instruction sets. The term "tangible computer-readable storage medium" should also be taken to include any non-transitory medium that is capable of storing or encoding an instruction set for execution by a machine and causing the machine to perform any one or more of the methodologies disclosed herein. The term "non-transitory" as in non-transitory computer-readable storage includes, but is not limited to, memory, drives, devices, and anything tangible but not a signal itself.

[0300] The term "tangible computer-readable storage medium" shall accordingly be deemed to include, but not be limited to, solid-state memories, such as memory cards or other packages containing one or more read-only (non-volatile) memories, random access memories or other rewritable (volatile) memories, magneto-optical or optical media such as disks or tapes, or other tangible media that can be used to store information. Thus, the present disclosure is deemed to include any one or more of the tangible computer-readable storage media, as listed herein and including art-recognized equivalents and successor media, in which the software implementations herein are stored.

[0301] Although this specification describes components and functions implemented in embodiments with reference to specific standards and protocols, the present disclosure is not limited to such standards and protocols. Each of the Internet and other packet-switched network transmission standards (e.g., TCP / IP, UDP / IP, HTML, HTTP) represents an example of the prior art. Such standards are superseded from time to time by faster or more efficient equivalents having substantially the same functionality. For device detection (e.g., RFID), short-range communications (e.g., Wi-Fi, ) and wireless standards for telecommunications (e.g., WiMAX, GSM, CDMA, LTE) may be used by the computer system 1000.

[0302] Fig.13 A system 1300 for identifying a set of possible trigger clips according to aspects of the present disclosure is depicted and Fig.14 A system 1400 for distributing a set of trigger clips to a set of players according to aspects of the present disclosure is depicted. Fig.13 and 14 Both, in one or more embodiments, a computer server 1402 may implement a video game (hereinafter video game server 1402) involving a number of different players 1304, 1306, 1308, 1314, 1316, and 1318 at different locations communicating with each other, and the video game server 1402 on a communication network (e.g., the Internet). In addition, some of the players 1304, 1306, 1314, and 1316 may comprise a team playing the video game, with another team of players 1308 and 1318 playing as opponents. Gaming devices 1404, 1408, 1412, and 1416 are associated with each player 1304, 1306, 1314, and 1316 of a team.

[0303] In one or more embodiments, video game server 1402 can detect current triggers during video games and record trigger clips. Triggers are events that players may find important and / or that will improve player performance in the future if reviewed. When the example of a trigger can include a player killing, hitting, being hit, dying, being shot, implementing weapon shooting, another player dying, ammunition changes, etc., the player may want to review the trigger to avoid such actions / events (e.g., dying, being hit, being shot, etc.) or continue such actions / events (e.g., killing, hitting, etc.) to improve future video game performance. Trigger clips are recorded parts of the video game that contain triggers. As described herein, a video game can have a sliding window of game content that continuously records the parts of the video game. Once video game server 1402 identifies a trigger, video game server 1402 can process the recorded sliding window / recorded part of the video game content that contains the trigger as a trigger clip. In addition, trigger clips can be provided and presented to the player for the player to review.

[0304] In one or more embodiments, a trigger clip is a recorded portion of the game content from the perspective or point of view of a player. In some embodiments, a trigger may involve one or more players. Thus, a distributed trigger clip may be requested or provided to all players involved in a trigger. That is, a distributed trigger clip is a set of trigger clips provided to each player involved in a trigger. Each trigger clip provided to each player is part of the game content from the perspective or point of view of the associated player. Fig.13 , a clip 1302 of a war scenario video game may include a first player 1304 on a first team. The trigger clip 1302 may be from the perspective of the first player 1304. The trigger may be a loss of life of a second player 1306 on the first team. Additionally, the trigger clip 1302 may include a first player 1308 on a second team, who is hiding behind a building 1310. The first player 1304 of the first team may request to intercept the trigger of the loss of life of the second player 1306 on the first team because the first player 1304 on the first team is unsure whether he / she killed the second player 1306 on the first team (i.e., a loss of life due to friendly fire), or whether the first player 1308 on the second team killed the second player 1306 on the first team (i.e., a loss of life due to enemy fire). The first player 1304 may want to review the trigger clip to review whether he / she committed a friendly fire, and if so, to avoid doing so in the future.

[0305] In one or more embodiments, the video game server 1402 can recognize that the first player 1304 and the second player 1306 on the first team both participated in the trigger. Therefore, the video game server 1402 can generate another trigger clip 1302a from the perspective of the second player 1306. Fig.14 , upon request by first player 1304, the video game server may provide distributed trigger clips, which include trigger clip 1302 and trigger clip 1302a. Trigger clip 1302 may be sent via a communication network (wired, wireless, or a combination thereof) to a gaming device 1404 associated with first player 1304 for presentation on display 1406 for review by first player 1304. Additionally, trigger clip 1302a may be sent (wired, wireless, or a combination thereof) to a gaming device 1408 associated with second player 1306 for presentation on display 1410 for review by first player 1306.

[0306] refer to Fig.13, the third player 1314 and the fourth player 1316 on the first team (the same team as the first player 1304 and the second player 1306) do not participate in the trigger because they are playing against the second player 1318 on the second team in a forest in a different scene / setting 1312 of the video game. Fig.14 , the video game server 1402 will recognize that neither the third player 1314 nor the fourth player 1316 will be provided with the distributed trigger clip because they were not involved in the trigger. Therefore, even though the video game server 1402 is communicatively coupled to the gaming devices 1412, 1416 and displays 1414, 1418 associated with the players 1314 and 1316 via a communication network (wired, wireless, or a combination thereof), the video game server 1402 will not provide the distributed trigger clip 1302, 1302a to the gaming devices 1412 and 1416.

[0307] In one or more embodiments, the video game server 1402 can detect or identify a trigger by receiving a user-generated input from the first player 1304, the user-generated input indicating that a clip is generated according to the event / trigger and the player involved in the event / trigger. The user-generated input can also include a timestamp. The user-generated input can be provided by inputting text input, gesture input (using a touch screen graphical user interface), and / or voice input (involving voice recognition technology) into the gaming device 1404. In addition, the user-generated input provided to the gaming device 1404 is then transmitted to the video game server 1402 for further processing. The video game server 1402 can then record the distributed trigger clips (trigger clip groups) for the trigger / event indicated in the user-generated input and the player's (or otherwise detected) timestamps included in the user-generated input (e.g., 20 seconds before and after the indicated or detected trigger).

[0308] In one or more embodiments, before detecting a trigger, the video game server 1402 may receive user-generated input indicating different events associated with different trigger types (as described herein). For example, the trigger types may include kill, hit, weapon fire, loss of life for the current player, loss of life for another player, and change of ammunition. The video game server 1402 may associate each event with a trigger type based on the user-generated input.

[0309] In one or more embodiments, the video game server 1402 may detect a trigger by implementing image recognition technology to identify or otherwise determine that an event within a portion of the game content is a trigger. In other embodiments, the video game server 1402 may access historical game content and identify previous triggers within the historical game content. That is, the video game server 1402 may analyze the historical game content to determine when an event was identified as a trigger based on its trigger type. Based on this determination, the video game server 1402 may detect a current trigger by determining that the current trigger / event is the same type as a previous trigger.

[0310] In one or more embodiments, detection of the current trigger can include accessing training game content, acquiring machine learning techniques to identify different triggers from different events in the training game content. That is, different events identified as different triggers in the training game content can be used to train the artificial intelligence engine on video game server 1402 to acquire machine learning techniques to identify triggers in portions of future game content. In addition, detection of the current trigger can include identifying the current trigger by using or in response to implementing machine learning techniques on the portion of the game content.

[0311] In one or more embodiments, the detection of a trigger may include detecting a specific sound or speech or word or statement of a player to identify or otherwise determine that an event within a portion of the game content is a trigger. This can be done in any suitable manner, including using artificial intelligence or machine learning techniques. In an example, a neural network is trained to estimate in real time whether there is, for example, a player's laughter or scream. In an example, real time means substantially no delay detectable by the player. In another example, real time means within a time delay of less than one second. Any suitable neural network can be used. The training data can be audio data from other game processes, which is marked to identify laughter or screams or other words or statements to be designated as triggers. The neural network is trained to provide an indication that a trigger has been detected or identified as an output. After training, a portion of the audio stream is provided to the neural network, and the neural network generates a value corresponding to the probability that a trigger has been detected. The processing system can compare the value with a predetermined threshold. If the probability exceeds a predetermined threshold, such as 50% or 85% or 95%, the feature can be identified as a significant trigger. The training data can include any kind of audio, including dialogue, laughter, screams and other sounds, and also sounds representing typical sounds.

[0312] including audio processing system 7600 ( Figure 7L) may include a chat audio stream and a microphone audio stream. The chat audio stream may include audio corresponding to conversations and other utterances from a player such as a teammate of a local user. The microphone audio stream may include audio corresponding to conversations and other utterances from a local user detected by a microphone adjacent to the local user. In a first use case, one or more neural networks may be trained to detect an occurrence in the chat audio stream or an occurrence in the microphone audio stream, and in response to the detected occurrence, cause subsequent actions within the game system. In other embodiments, under a second use case, a neural network may operate on a combined audio stream including both a chat audio stream and a microphone audio stream. Similarly, in a third use case, a single audio stream includes a combined audio stream including a chat audio stream, a microphone audio stream, and a game audio stream, the game audio stream including sounds generated by the game. Training data for any of these use cases is customized for a particular use case.

[0313] In a particular embodiment, the trigger is detected by automatically detecting laughter or screaming in the audio stream. The laughter or screaming can originate from any player. The motivation for this automatic trigger is the idea that if someone is laughing or screaming, then something funny must have happened in the game and it should be recorded in the trigger clip. Both audio and video are recorded in the clip.

[0314] In some embodiments, a first window of video of a first predetermined duration is recorded before the trigger occurs, and a second window of a second predetermined duration is recorded after the trigger occurs, which second predetermined duration may be different from the first predetermined duration. The nature of the detected trigger, such as laughter or screaming, can be used to select the duration of the trigger clip. For example, laughter detected as a trigger can result in recording a 30 second trigger clip with a duration of 30 seconds, and screaming detected as a trigger can result in recording a clip with a duration of 60 seconds. In some embodiments, any aspect of detecting a trigger, identifying a trigger, or the duration of a trigger clip associated with a particular trigger can be user selectable, such as through a GUI such as GUI 7610( Figure 7L )'s graphical user interface.

[0315] In some embodiments, one or more players have a webcam or other device that collects video of the player or the player's environment. The video stream from the webcam can be provided to the neural network to identify the video trigger in a manner similar to identifying the audio trigger in the audio stream. The neural network can be trained with appropriate data that is marked to identify the trigger of interest. In a further embodiment, both the audio stream and the video stream can be monitored in a multimodal method for identifying trigger events. For example, the video neural network will generate a video probability that a video trigger has been detected, and the audio neural network will generate an audio probability that an audio trigger has been detected. The video probability and the audio probability can be combined and evaluated for one or more probability thresholds in any suitable manner. For example, if the player laughs silently, the audio neural network may not detect the sound corresponding to the laughter, but the video neural network may detect the action or movement associated with the laughter, or the probability that the movement corresponds to the laughter. Any change in the emotion or other response captured in the audio or video can be designated as a trigger.

[0316] In some embodiments, triggers can be defined based on specific cultural aspects of the player. For example, studies can show that European players react to a game situation in a first way, American players react to the same game situation in a second way, South Asian players react to the same game situation in a third way, and East Asian players react to the same game situation in a fourth way. The responses of different cultures can include verbal words and statements or body movements or gestures, or any combination of these. Different models can be targeted to different geographic or linguistic or other characteristics of different players.

[0317] In some embodiments, processing the audio and video streams to identify triggers may occur at, for example, computing device 206 ( Figure 2 ) on a user device. The processing may be part of a game application running on the player device. In other embodiments, the processing of the audio and video streams may be done remotely, such as at a server that hosts the game and is accessible over a network such as the public Internet.

[0318] In some embodiments, the audio stream may be provided to a speech recognition module. The speech recognition module may detect and recognize words stated by the player. Any suitable speech recognition module may be used for this purpose. In this way, the user may be able to provide commands to the gaming system. In addition, triggers may be detected based on statements or commands of the recognized player.

[0319] In some embodiments, the data for user audio and video recognition can be stored in a user profile. The user profile can contain any suitable information useful to the audio or video associated with the user. The user profile can be accessed when the user initiates the game or during the game process or at any appropriate time. The user profile can be stored locally at the user's device. In other embodiments, the user profile can be remotely stored at a server or other storage location for access by the user's local device.

[0320] In some embodiments, the detected triggers may be characterized to determine specific characteristics of the triggers. The detected characteristics may be used to automatically select different characteristics of the recorded trigger clips. For example, a sound may be identified as a scream. If the volume of the scream exceeds a certain threshold, or if the duration of the scream exceeds a certain duration, or a combination of these characteristics, the scream may be considered a more significant trigger. The identification of more significant triggers may enable an audio processing system or other aspects to record trigger clips with specific characteristics (such as longer duration). Similarly, the identification of more significant triggers may enable an audio processing system or other aspects to distribute the recorded trigger clips to audiences different from those for conventional triggers. A neural network may be trained using appropriate training data to distinguish any interesting characteristics in an audio stream and generate a value corresponding to the probability that a trigger has been detected. If the probability exceeds a predetermined threshold, such as 85%, the characteristic may be identified as a significant trigger. Any suitable statistics in the audio stream and in the neural network may be used to identify one or more triggers.

[0321] In one or more embodiments, the video game server can determine the player group involved in the current trigger by using or in response to implementing image recognition technology on the trigger clip to identify the player group involved in the current trigger. In one or more embodiments, alternatively or additionally, the video game server can identify the player group involved in the current trigger by using or in response to analyzing one or more audio streams such as chat audio stream or microphone audio stream on the trigger clip, to identify the player group. In some embodiments, the video game server 1402 can determine the player group involved in the current trigger in the following manner: by or in response to analyzing the game action of each possible trigger clip in the group of trigger clips during the same time period to identify the player group. That is, when the video game server 1402 identifies the trigger in a part of the game content from the perspective of a player, the video game server can then generate a trigger clip (that is, record a part of the game content from the player's perspective) for the player. However, the video game server 1402 can also record other parts of the game content, and each part of the game content is from the perspective of different players. These partial groups of game content are possible trigger clips, to include distributed trigger clips. However, after determining only the players involved in the trigger, video game server 1402 discards the portions of the game content from the perspective of the players not involved in the trigger and generates a distributed trigger clip to include a group of portions of the game content from the perspective of each player involved in the trigger (i.e., trigger clips).

[0322] Fig.15A -D depicts an illustrative embodiment of the method utilized in the subject disclosure. Further, Fig.15A The method described in -D may be implemented by a video game server as described herein. Fig.15A , method 1500 may include the video game server associating each event in a set of events with each trigger type from a set of trigger types at 1502. In addition, method 1500 may include the video game server detecting a current trigger during the video game at 1504. In addition, method 1500 may include the video game server determining a player group involved in the current trigger at 1506. Moreover, method 1500 may include the video game server recording a trigger clip of the game content of each player in the player group at 1508, thereby generating a trigger clip group. Each trigger clip in the trigger clip group is a portion of the game content including the current trigger from the perspective of each player in the group of players. In addition, method 1500 may include the video game server providing the trigger clip group to a group of game devices via a communication network at 1510. Each game device in the group of game devices is associated with a player in the player group.

[0323] See also Fig. 15B, method 1511 may implement detection of a current trigger in different embodiments. Method 1511 may include the video game server receiving a first user-generated input at 1512. The first user-generated input indicates a current trigger and / or records the trigger clip group. In addition, method 1511 may include the video game server implementing image recognition technology on each portion of the game content associated with each trigger clip in the trigger clip group at 1514 to determine that an event occurred within each portion of the game content such that the event includes the current trigger. In addition, method 1511 may include the video game server accessing historical game content at 1516. In addition, method 1511 may include the video game server identifying a previous trigger within the historical game content at 1518. In addition, method 1511 may include the video game server determining at 1520 that the current trigger is a trigger of the same type as the previous trigger. In addition, method 1511 may include the video game server accessing training game content at 1522. Moreover, method 1511 may include the video game server acquiring machine learning technology at 1524 to identify different triggers from different events from the training game content. Additionally, method 1511 may include the video game server identifying a current trigger in response to implementing the machine learning technique on the portion of the game content at 1526. In some embodiments, the current trigger includes a trigger type, where the trigger type includes one of a kill, a hit, a weapon shot, a current player dies, another player dies, and an ammunition change.

[0324] refer to Fig. 15C , associating events with triggers can be accomplished by the video game server in different embodiments. The method 1531 can include the video game server determining an event group for the video game at 1530. Furthermore, the method 1531 can include the video game server associating each event in the event group for the video game with each trigger type from the trigger type group at 1532. Furthermore, the method 1531 can include the video game server receiving a second user-generated input at 1534. The second user-generated input provides an indication of each event associated with each trigger type.

[0325] refer to Fig.15D, determining the player group involved in the current trigger can be completed by the video game server in different embodiments. Method 1541 can include the video game server implementing image recognition technology or performing audio stream analysis for each trigger clip in the group of trigger clips at 1540. In response, method 1541 can include the video game server identifying the player group at 1542. In addition, method 1541 can include the video game server analyzing the game action during the same time period as each trigger clip of the group of trigger clips at 1544. In response, method 1541 can include the video game server identifying the player group at 1546. This analysis can include the video game server determining the position of each player in the video game from the game action and determining the player group involved in the current trigger by determining the position of each player in the game. In addition, those players within a specific threshold distance from the event / current trigger are determined to be involved in the current trigger.

[0326] In an embodiment, one or more aspects of the subject disclosure may include the following features: receiving a plurality of audio streams by a processing system including a processor, the plurality of audio streams including a game output audio stream of a game application, a gamer chat audio stream generated by a plurality of gamers participating in the game application, and a gamer microphone audio stream generated by one of the plurality of gamers; processing the game output audio stream by the processing system according to a first audio processing setting of the one gamer to form a processed game output audio stream; processing the gamer chat audio stream by the processing system according to a second audio processing setting of the one gamer to form a processed gamer chat audio stream; processing the gamer microphone audio stream by the processing system according to a third audio processing setting of the one gamer to form a processed gamer microphone audio stream; providing the processed game output audio stream and the processed gamer chat audio stream to an audio device of the one gamer by the processing system; and providing the processed player microphone audio stream to one or more of the plurality of gamers by the processing system.

[0327] Aspects may further include receiving, by a processing system, audio processing settings from the one player, the audio processing settings comprising the first audio processing setting, the second audio processing setting, and the third audio processing setting of the one player.

[0328] Aspects may further include providing a graphical user interface on the display device of the one player by the processing system for the one player to select audio processing settings. Aspects may further include receiving data defining audio processing settings from the graphical user interface by the processing system. Aspects may further include providing a parameter equalizer for a game output audio stream on a portion of the display device of the one player by the processing system, and receiving a player equalizer input of the one player at the parameter equalizer by the processing system, the player equalizer input being used to select a corresponding gain value for a corresponding frequency band of the parameter equalizer. Aspects may further include filtering the game output audio stream in a plurality of filters by the processing system, wherein the filtering is based on the player equalizer input. Aspects may also include receiving a player quality (Q) value input of the player by the processing system at the parameter equalizer, the player Q value input being used to select a Q value for the parameter equalizer, and adjusting the frequency response of at least one filter in a plurality of filters by the processing system according to the player Q value input.

[0329] Aspects may further include receiving a game audio stream generated by a game application by a processing system, establishing a game output virtual audio driver by the processing system, generating a game output audio stream by the game output virtual audio driver, establishing a game chat virtual audio driver by the processing system, generating a game chat audio stream by the game chat virtual audio driver, establishing a game microphone virtual audio driver by the processing system, and generating a game microphone audio stream by the game microphone virtual audio driver. Aspects may further include receiving a player profile of a player by the processing system, the player profile defining an audio processing setting, the audio processing setting including a first audio processing setting, a second audio processing setting, and a third audio processing setting for the player. Aspects may further include identifying a game application by the processing system, and receiving a game profile of the game application by the processing system based on identifying the game application, the game profile defining an audio processing setting, the audio processing setting including a first audio processing setting, a second audio processing setting, and a third audio processing setting for the player for the game application.

[0330] Aspects may further include providing, by the processing system, a graphical user interface on the one gamer's display device for the one gamer to use to select the audio processing settings, the graphical user interface including a mixer input actuated by the one gamer to change the audio volume of the processed game output audio stream and the audio volume of the processed player chat audio stream to the one gamer's audio device, and receiving, by the processing system, data defining the audio processing settings from the graphical user interface. Aspects may further include forming, by the processing system, virtual surround sound game audio, and providing, by the processing system, the virtual surround sound game audio to the one gamer's audio device.

[0331] In some embodiments, aspects may further include a non-transitory machine-readable medium including executable instructions that, when executed by a processing system including a processor, facilitate the execution of operations, the operations including receiving game output information including a game output audio stream from a game application; and receiving the game output audio stream from the game application at an audio processing module in data communication with the game application, and generating a plurality of game audio streams; and applying the corresponding audio customization to each of the plurality of game audio streams by the audio processing module according to a player input defining a corresponding audio customization for each corresponding game audio stream. Aspects may further include providing a graphical user interface on a display device of a player, and receiving a player input defining a corresponding audio customization for each corresponding game audio stream in response to a player actuation of the graphical user interface. Aspects may further include providing a parametric equalizer on the graphical user interface, and receiving a player equalizer input of a player at the parametric equalizer, the player equalizer input being used to select a corresponding gain value for a corresponding frequency band of the parametric equalizer. Aspects may further include establishing a plurality of filters, respective filters in the plurality of filters corresponding to respective frequency bands of the parametric equalizer, and filtering the game output audio stream in the plurality of filters, wherein the filtering is based on the gamer equalizer input. Aspects may further include identifying one or more audio output devices of a gamer, and receiving information from the graphical user interface that controls mapping of respective ones of the plurality of game audio streams to selected ones of the one or more audio output devices.

[0332] In some embodiments, aspects may further include a processing system including a processor and a memory storing executable instructions, wherein the executable instructions, when executed by the processing system, facilitate the execution of operations, the operations including receiving game output information including a game output audio stream from a game application, the game audio output stream including a plurality of game audio streams, delivering the plurality of game audio streams of the game application to one or more audio output devices of a player by a virtual audio device in data communication with the game application, applying the corresponding audio customization to each of the plurality of game audio streams by the virtual audio device in data communication with the game application according to player input defining a corresponding audio customization for each corresponding game audio stream. Aspects may further include receiving player input from a player from a graphical user interface, wherein the graphical user interface is in data communication with an audio processing module to control the corresponding audio customization. Aspects may further include receiving information from the graphical user interface that controls the mapping of a corresponding game audio stream in the plurality of game audio streams to a selected audio output device in the one or more audio output devices. Aspects may further include receiving information from an operating system of the device that controls the mapping of a corresponding game audio stream in the plurality of game audio streams to a selected audio output device in the one or more audio output devices.

[0333] In some embodiments, various aspects may include a method, comprising receiving, by a processing system comprising a processor and a memory, a live game output audio stream of a game application currently being played by multiple gamers including a first gamer, providing, by the processing system, a graphical user interface on a display device of the first gamer among the multiple gamers, the graphical user interface including a spectrum analyzer interface for the first gamer to use in selecting one or more live audio processing settings for the live game output audio stream, receiving, by the processing system, data defining the one or more live audio processing settings from the graphical user interface, and modifying, by the processing system, the live game output audio stream of the game application according to the one or more live audio processing settings.

[0334] Aspects may further include separating the live game output audio stream of the game application into a plurality of audio streams by the processing system, receiving a corresponding live audio setting for a corresponding audio stream in the plurality of audio streams by the processing system, and modifying each corresponding audio stream in the plurality of audio streams by the processing system according to the corresponding live audio setting. Aspects may further include providing a plurality of frequency bands by the processing system, each corresponding frequency band in the plurality of frequency bands having a user-selectable center frequency and gain, receiving a user input defining the user-selectable center frequency and gain of each corresponding frequency band by the processing system, wherein receiving the user input includes receiving data in response to an interaction of a user with a graphical element displayed on a display device of a first gamer, adjusting a filter setting of a corresponding filter in a plurality of filters corresponding to the plurality of frequency bands by the processing system, modifying the live game output audio stream of the game application by the processing system according to the corresponding filter, and providing the modified live game output audio to the audio device of the first gamer.

[0335] Aspects may further include receiving data of filter types defining multiple filters from a graphical user interface by a processing system, each filter in a plurality of filters corresponding to a frequency band in a plurality of frequency bands, establishing the multiple filters by the processing system, and modifying the live game output audio stream of the game application according to the multiple filters by the processing system. Aspects may further include receiving data of user selectable center frequency and gain defining each corresponding frequency band in a plurality of frequency bands by the processing system from a graphical user interface. Aspects may further include receiving data of user selectable Q factor defining each corresponding frequency band in a plurality of frequency bands by the processing system from the graphical user interface. Aspects may further include storing data defining one or more live audio processing settings in a retrievable configuration file by the processing system.

[0336] Aspects may further include retrieving a stored configuration file by a processing system, the stored configuration file includes data defining the stored live audio processing settings, and wherein receiving the data defining one or more live audio processing settings includes receiving editing information by the processing system to modify the stored configuration file, wherein the editing information is received in response to the first game player accessing the graphical user interface. Aspects may further include providing an audio rendering effect selector on the display device of the first game player by the processing system, the first game player actuates the audio rendering effect selector to activate and adjust one or more audio effects, and the live game output audio stream of the game application is modified by the processing system according to the one or more audio effects. Aspects may further include providing a user selectable control for one or more of bass enhancement adjustment, treble enhancement adjustment, voice clarity adjustment, intelligent volume adjustment, overall gain adjustment, and virtual surround sound adjustment. Aspects may further include separating the live game output audio stream of the game application into multiple audio streams by the processing system, and providing an audio stream selector by the processing system, the first game player actuates the audio stream selector to select an audio stream in the multiple audio streams for activation and adjustment of the one or more audio effects.

[0337] In some embodiments, aspects may include receiving multiple audio streams from a gaming system, providing a graphical user interface on a gamer's display device, including an audio mixing panel, the audio mixing panel configured to receive audio adjustments for each of the multiple audio streams, the audio mixing panel including multiple selectable views for user interaction by the gamer to modify audio processing of each of the multiple audio streams, the multiple selectable views including audio rendering effects and a parametric equalizer for each of the multiple audio streams, modifying each of the multiple audio streams by a plurality of audio filters, wherein the modification is based on user input received at the graphical user interface, generating a modified audio stream, and providing the modified audio stream to the gamer's audio device.

[0338] Aspects may further include providing a parametric equalizer view on a display device of a gamer, the parametric equalizer view including a plurality of frequency bands, and receiving a gamer actuation of a graphical user interface from a gamer, the gamer actuation defining a frequency and gain value associated with each frequency band in the plurality of frequency bands. Aspects may further include displaying a graphical display of gain values ​​and frequency bands on a display device of a gamer, displaying a user-selectable graphical element corresponding to a corresponding gain value for each corresponding frequency band, and detecting a mouse input of a gamer's selection of the selected graphical element and sliding the selected graphical element to the selected gain value. Aspects may further include receiving a selection of a gamer profile at a graphical user interface, wherein the gamer profile includes information defining one or more audio processing settings of a plurality of audio streams from a gaming system, and modifying each audio stream in the plurality of audio streams according to the one or more audio processing settings. Aspects may further include receiving a selection of an audio processing preset file at a graphical user interface, the audio processing preset file includes information defining one or more standardized audio processing settings of a plurality of audio streams from a gaming system, and modifying each audio stream in the plurality of audio streams according to the one or more audio processing settings.

[0339] In some embodiments, various aspects may include receiving a game audio stream, providing a graphical user interface on a player's display device, the graphical user interface including a mixer, the mixer including user settings that can be selected by the player interacting with the features of the graphical user interface, and modifying the game audio stream according to the user settings. Various aspects may further include receiving a game output audio stream, a chat audio output stream, and a player microphone audio output stream in the game audio stream, and providing a selectable screen for each of the game output audio stream, the chat audio output stream, and the player microphone audio output stream on the graphical user interface, receiving a user selection of settings for each of the game output audio stream, the chat audio output stream, and the player microphone audio output stream at the graphical user interface, and modifying each of the game output audio stream, the chat audio output stream, and the player microphone audio output stream according to the user selection of the player. Various aspects may further include receiving a user selection of a parametric equalizer from the player, providing a parametric equalizer display on the player's display device, and receiving an actuation of the parametric equalizer display to select a filter frequency value and a filter gain value from the player. Aspects may further include establishing a filter based on a filter frequency value and a filter gain value, and modifying the game audio stream based on the filter frequency value and the filter gain value.

[0340] In some embodiments, aspects may include receiving, by a processing system including a processor and a memory, a live game output audio stream of a game application currently being played by a plurality of gamers, retrieving, by the processing system, a stored configuration file, the stored configuration file including data defining stored live audio processing settings, modifying, by the processing system, the live game output audio stream of the game application according to the stored live audio processing settings to form modified game audio, and providing, by the processing system, the modified game audio to one or more of the plurality of gamers. Aspects may further include dividing, by the processing system, the live game output audio stream of the game application into a plurality of audio streams, identifying, by the processing system, a selected audio stream associated with the stored configuration file, and modifying, by the processing system, the selected audio stream according to the stored live audio processing settings.

[0341] Aspects may also include a processing system that divides a live game output audio stream into a game audio stream including audio generated by a game application, a chat audio stream including a teammate's voice, and a microphone audio stream including the voice of a player among a plurality of players, a processing system that modifies the game audio stream according to a game setting of a stored configuration file, a processing system that modifies the chat audio stream according to a chat setting of a stored configuration file, and a processing system that modifies the microphone audio stream according to a microphone setting of a stored configuration file. Aspects may further include a processing system that identifies a player associated with a game application, and a processing system that retrieves a stored configuration file based on identifying the player. Aspects may further include a processing system that identifies a game application, and a processing system that retrieves a stored configuration file based on identifying the game application and identifying the player. Aspects may further include a processing system that identifies a game application, and a processing system that retrieves a stored configuration file based on identifying the game application and identifying the player. Aspects may further include a processing system that identifies a game application, and a processing system that retrieves a stored configuration file based on identifying the game application.

[0342] Aspects may further include detecting the start of a game application by a processing system, retrieving a stored configuration file by the processing system, wherein the stored configuration file is associated with the game application, and applying the stored live audio processing settings to the live game output audio stream of the game application by the processing system. Aspects may also include detecting the start of a game application by a processing system, determining the identification of the game application by the processing system, retrieving a stored configuration file by the processing system, wherein the stored configuration file is associated with the game application, and wherein the stored configuration file includes a plurality of preset audio configurations for the game application, providing information about the plurality of preset audio configurations for the game application to a player in a plurality of gamers by the processing system, wherein the player accesses a graphical user interface, receiving a player selection of the selected preset audio configuration from a player in the plurality of gamers by the processing system, and applying the selected preset audio configuration to the live game output audio stream of the game application by the processing system. Aspects may further include receiving a player modification of the selected preset audio configuration by the processing system, forming an updated configuration file, and storing the updated configuration file for subsequent retrieval and access by a player in a plurality of gamers. Aspects may further include receiving a player modification at a graphical user interface.

[0343] In some embodiments, aspects may include launching a game application for a player of a plurality of gamers, retrieving a player profile associated with the player, receiving a game application audio output stream, applying an audio modification specified by the player profile to the game application audio output stream, forming a modified game audio stream, and providing the modified game audio stream to one or more gamers including the player. Aspects may further include providing information about the audio modification specified by the player profile to a graphical user interface, wherein the graphical user interface is accessible by a player of a plurality of gamers, receiving updates to the audio modification specified by the player profile from actuation of the graphical user interface by a player of a plurality of gamers, forming an updated profile, and storing the updated profile for subsequent retrieval and access by a player of a plurality of gamers. Aspects may further include providing a parametric equalizer on the graphical user interface, wherein the parametric equalizer is associated with one or more filters to modify the game application audio output stream according to an equalizer setting, the equalizer setting being retrieved in the player profile, and receiving modifications to the equalizer setting from a player of a plurality of gamers to customize the game application audio output stream for a player of a plurality of gamers.

[0344] Aspects may further include displaying a graphical representation of a filter adjustment and a frequency response associated with an equalizer setting on a graphical user interface, receiving an actuation of the graphical representation of the frequency response from a player of the plurality of players to select and modify a corresponding filter adjustment, updating the equalizer setting based on the actuation of the graphical representation of the frequency response, forming an updated equalizer setting, and storing the updated equalizer setting with an updated configuration file. Aspects may further include, in response to launching a gaming application for a player of the plurality of players, determining an identity of the player of the plurality of players, identifying a plurality of stored configuration files associated with the identity of the player, wherein respective stored configuration files are associated with respective gaming applications, determining the identity of the gaming application, and retrieving a player configuration file associated with the player based on the identity of the gaming application. Aspects may further include retrieving a standard configuration file that includes standard audio modifications available to all gamers, applying the standard audio modifications specified by the standard configuration file to a game application audio output stream to form a standard modified game audio stream, providing information about the standard audio modifications specified by the standard configuration file to a graphical user interface, wherein the graphical user interface is accessible by the gamer of the plurality of gamers, receiving customization of the standard audio modifications specified by the standard configuration file from actuation of the graphical user interface by the gamer of the plurality of gamers, forming a customized configuration file for the gamer of the plurality of gamers, and storing the customized configuration file for subsequent retrieval and access by the gamer of the plurality of gamers.

[0345] In some embodiments, aspects may include launching a gaming application for a player among a plurality of players, retrieving a standard configuration file that includes standard audio modifications to a gaming audio stream of the gaming application, receiving a gaming application audio output stream, applying the standard audio modifications to the gaming application audio output stream to form a modified gaming audio stream, providing the modified gaming audio stream to one or more players including the player, receiving customization of the standard audio modifications specified by the standard configuration file from the player among the plurality of players through actuation of a graphical user interface, forming a customized configuration file for the player among the plurality of players, and storing the customized configuration file for subsequent retrieval and access by the player among the plurality of players.

[0346] Aspects may further include providing information about standard audio modifications specified by a standard configuration file to a graphical user interface, wherein the graphical user interface includes a parametric equalizer, and wherein the standard audio modifications include equalizer settings for controlling one or more filters associated with the parametric equalizer. Aspects may further include identifying a gaming application and retrieving a gaming-specific standard configuration file including gaming-specific standard audio modifications, wherein the retrieval is based on identifying the gaming application. Aspects may further include separating a gaming application audio output stream of a gaming application into multiple audio streams, modifying the multiple audio streams in accordance with the standard audio modifications; subsequently, retrieving a custom configuration file, and modifying the multiple audio streams in accordance with the customization of the standard audio modifications specified by the standard configuration file.

[0347] In some embodiments, aspects may further include receiving, by a processing system including a processor, a game system output stream during a video game, the game system output stream including audio output captured by a microphone, video output captured by a camera, or a combination thereof, detecting, by the processing system, a current trigger in the game system output stream during the video game, recording, by the processing system, a trigger clip of game content, wherein the trigger clip includes a portion of the game content including the current trigger, and providing, by the processing system, the trigger clip to a group of game devices over a communication network, wherein each game device in the group of game devices is associated with a player of the video game. Aspects may further include providing, by the processing system, the game system output stream to a neural network, and receiving, by the processing system, an indication that the current trigger has been detected. Aspects may further include receiving, by the processing system, a value corresponding to a probability that the current trigger has been detected from the neural network. Aspects may further include comparing, by the processing system, the value corresponding to the probability with a predetermined threshold, and determining, by the processing system in response to the comparison, that the current trigger has been detected. Aspects may further include training the neural network on training data from one or more video games. Aspects may further include training the neural network on training data corresponding to voiced and non-voiced speech of players of one or more video games. Aspects may further include training the neural network on training data corresponding to dialogue, laughter, and screams of players of one or more video games.

[0348] Aspects may further include identifying, by the processing system, one of a laugh or a scream of a player of a video game in a gaming system output stream. Aspects may further include receiving, by the processing system, a plurality of audio streams including a chat audio stream, a gaming audio stream, and a microphone audio stream, and detecting, by the processing system, the current trigger in one of the chat audio stream and the microphone audio stream. Aspects may also include determining, by the processing system, a group of players involved in the current trigger, recording, by the processing system, a trigger clip of the gaming content for each player in the group of players, thereby generating a group of trigger clips, wherein each trigger clip in the group of trigger clips includes a portion of the gaming content including the current trigger from the perspective of each player in the group of players, and providing, by the processing system, the group of trigger clips to the gaming device group via a communications network, wherein each gaming device in the gaming device group is associated with a player in the group of players.

[0349] In some embodiments, various aspects may include receiving an input stream during a video game by a processing system including a processor, the input stream including an audio input captured by a microphone, a video input captured by a camera, or a combination thereof. A trigger model is applied to the input stream by the processing system, wherein the trigger model is generated by a neural network trained on a training audio input, a training video input, or a combination thereof, and the processing system detects a current trigger in the input stream during the video game according to the trigger model, wherein the current trigger includes an audio trigger, a video trigger, or a combination thereof, and a trigger clip of game content is recorded by the processing system, wherein the trigger clip includes a portion of the game content in a time period before and after the current trigger. Various aspects may further include training audio input, wherein the audio input includes voice, laughter, and screaming. Various aspects may further include training video input, wherein the video input includes facial expressions, hand gestures, body movements, or a combination thereof. Various aspects may further include selecting a time period before and after the current trigger from a set of time periods according to the type of the current trigger. Various aspects may further include identifying a characteristic of the current trigger that causes the trigger characteristic, wherein the time period before and after the current trigger is selected from a set of time periods according to the trigger characteristic. Aspects may further include volume, tone, or words as trigger characteristics.

[0350] Aspects may further include selecting a time period before and after a current trigger from a set of time periods based on game characteristics. Aspects may further include detecting a change in a player's mood from a video input of the player and from an audio input captured by the player. Aspects may further include selecting a trigger model from a set of trigger models by a processing system based on a video game, a style of video game, a location of a gaming device capturing audio input or video input, or a combination thereof. Aspects may further include capturing speech as the audio input by the microphone, converting the speech to text and analyzing the text, wherein the detecting the current trigger in the input stream during the video game is based on the analyzing the text.

[0351] In some embodiments, aspects include a gaming application that generates gaming output information including a gaming output audio stream, and a head-related transfer function (HRTF) module in data communication with the gaming application, the HRTF module operable to receive the gaming output audio stream from the gaming application, and to apply a preferred head-related transfer function to the gaming output audio stream to generate a surround sound audio stream, the preferred head-related transfer function being selected by a user of the device from a plurality of standardized head-related transfer functions based on a comparison of the performance of the standardized head-related transfer functions by the user. Aspects may further include a speaker system coupled to the head-related transfer function module, operative to receive the surround sound audio stream and generate surround sound audio for the user of the device. Aspects may further include a headset worn by a user of the device as the speaker system. Aspects may further include a right speaker and a left speaker built into a laptop computer system as the speaker system for providing audio to the user of the device.

[0352] Aspects may further include a graphical user interface module operative to provide instructions to a user to guide the user through a HRTF selection process to identify a preferred head-related transfer function, and to receive input from the user identifying the preferred head-related transfer function. Aspects may further include, on the graphical user interface module, providing one or more audio test files to a plurality of standardized head-related transfer functions, and receiving an indication of a preferred head-related transfer function from a user. Aspects may further include presenting to a user, on the graphical user interface module, a series of A / B tests, each A / B test in the series of A / B tests using a first head-related transfer function and a second head-related transfer function in the plurality of standardized head-related transfer functions, wherein the graphical user interface module is operative to receive from the user an indication of a best head-related transfer function for each A / B test in the series of A / B tests. Aspects may further include an audio processing module that performs data communication with a gaming application, the audio processing module being operative to receive a game output audio stream from the gaming application and generate a plurality of audio streams, the plurality of audio streams comprising a gaming audio stream and a chat audio stream, the audio processing module being further operative to apply a respective audio customization to each respective audio stream in accordance with a user input defining a respective audio customization for each respective audio stream in the plurality of audio streams, wherein the HRTF module is operative to receive the gaming audio stream from the audio processing module and to apply the preferred head-related transfer function to the gaming audio stream. Aspects may further include a graphical user interface module being operative to receive user input from a user of the device to control the HRTF module.

[0353] In some embodiments, aspects include providing an audible comparison of the performance of a plurality of standard head-related transfer functions to a user of a gaming system, receiving a selection of a best head-related transfer function from the user among the plurality of standard head-related transfer functions, receiving a game output audio stream from the gaming system, applying the best head-related transfer function to the game audio output stream to produce a surround sound audio stream, and providing the surround sound audio stream to an audio device of the user. Aspects may further include providing a first listening test of a first standard head-related transfer function, providing a second listening test of a second standard head-related transfer function, and receiving a selection of a best head-related transfer function from the user among the first standard head-related transfer function and the second standard head-related transfer function based on a user's preference for the first listening test or the second listening test.

[0354] Aspects may further include selecting a next head related transfer function from the plurality of standard head related transfer functions, selecting the best head related transfer function as a preferred head related transfer function, performing a listening test of the next head related transfer function and the preferred head related transfer function, receiving a user selection of a new preferred head related transfer function from the next head related transfer function and the preferred head related transfer function, and applying the new preferred head related transfer function to a game audio output stream to generate a surround sound audio stream. Aspects may further include receiving a game audio stream and a chat audio stream, receiving user input defining respective audio customizations for the game audio stream and the chat audio stream, applying respective audio customizations to the game audio stream and the chat audio stream based on the user input, generating a customized game audio stream and a customized chat audio stream, and applying the best head related transfer function to the customized game audio stream. Aspects may further include providing a graphical user interface operative to provide instructions to the user to guide the user through a head related transfer function selection process to identify the best head related transfer function, and receiving the user input selecting the best head related transfer function.

[0355] Aspects may further include presenting to a user on a graphical user interface a series of A / B tests, each A / B test in the series of A / B tests using a first head-related transfer function and a second head-related transfer function from a plurality of standardized head-related transfer functions, and receiving from the user via the graphical user interface an indication of a preferred head-related transfer function for each A / B test in the series of A / B tests. Aspects may further include providing the surround sound audio stream to headphones worn by the user.

[0356] In embodiments, aspects may include providing, by a processing system including a processor, a series of listening tests to a user of a gaming system, each listening test in the series of listening tests providing the user with a comparison of a first standard head related transfer function and a second standard head related transfer function in a plurality of standard head related transfer functions, receiving, by the processing system, an indication of a preferred head related transfer function for each listening test in the series of listening tests from the user, including receiving from the user an indication of a best head related transfer function in the plurality of standard head related transfer functions, receiving, by the processing system, a game output audio stream from the gaming system, applying, by the processing system, the best head related transfer function to the game audio output stream to produce a surround sound audio stream, and providing, by the processing system, the surround sound audio stream to an audio device of the user. Aspects may further include conducting, by the processing system, a tournament of listening tests for the user to select the best head related transfer function.

[0357] Aspects may further include receiving, by a processing system, a game audio stream and a chat audio stream, receiving, by the processing system, user input defining respective audio customizations for the game audio stream and the chat audio stream, applying, by the processing system, the respective audio customizations to the game audio stream and the chat audio stream based on the user input, generating a customized game audio stream and a customized chat audio stream, and applying an optimal head-related transfer function to the customized game audio stream. Aspects may further include providing, by the processing system, a graphical user interface operative to provide instructions to the user to guide the user through a head-related transfer function selection process to identify the optimal head-related transfer function, and receiving input from the user selecting the optimal head-related transfer function.

[0358] In embodiments, aspects may also include receiving, by a processing system including a processor, a game system audio stream during a video game, the game system audio stream including a plurality of audio streams, the plurality of audio streams including a chat audio stream, a game audio stream, and a microphone audio stream, detecting, by the processing system, a predetermined event in the game system audio stream during the video game, modifying, by the processing system, an audio spectrum of the game system audio stream to emphasize the predetermined event in the game system audio stream, generating an equalized audio stream, and transmitting, by the processing system, the equalized audio stream to an audio device of a player of the video game. Aspects may also include: in response to detecting the predetermined event in the game system audio stream, selecting, by the processing system, a predetermined audio profile, and applying, by the processing system, the predetermined audio profile to the game system audio stream to enhance audio reproduction of the predetermined event. Aspects may also include selecting, by the processing system, a profile for controlling a parametric equalizer, and processing, by the parametric equalizer, the game system audio stream according to the profile.

[0359] Aspects may further include detecting gunshot sounds in a game audio stream by a processing system, wherein the gunshot sounds are associated with another player of a video game. Aspects may further include detecting footstep sounds in a game audio stream by a processing system, wherein the footstep sounds are associated with another player of a video game. Aspects may further include identifying the origin direction of footstep sounds in a game audio stream by a processing system, wherein the origin direction is relative to the position of an avatar in a video game associated with a player of the video game, and modifying the audio spectrum of the game audio stream by the processing system to strengthen the origin direction of footstep sounds. Aspects may further include providing a visual identification of footstep sounds corresponding to footsteps on a graphical user interface accessible to a player of a video game by a processing system.

[0360] Aspects may further include providing, by the processing system, the equalized audio stream to a headset of a player of the video game. Aspects may further include providing, by the processing system, the gaming system audio stream to the neural network, and receiving, by the processing system, an indication from the neural network that a predetermined event has been detected. Aspects may further include receiving, by the processing system, from the neural network, a value corresponding to a probability that a predetermined event has been detected. Aspects may further include comparing, by the processing system, the value corresponding to the probability with a predetermined threshold, and determining, by the processing system in response to the comparison, that the predetermined event has been detected. Aspects may further include the predetermined event corresponding to a footstep sound in the gaming audio stream.

[0361] In embodiments, aspects may include receiving a gaming system audio stream from a gaming system, detecting a predetermined sound in the gaming system audio stream, selecting a predetermined audio profile in response to detecting a predetermined event in the gaming system audio stream, modifying the gaming system audio stream in accordance with the predetermined audio profile, producing the modified audio stream, and providing the modified audio stream to an audio device of a player of the gaming system. Aspects may also include providing the gaming system audio stream to a parametric equalizer, receiving an equalizer input for the parametric equalizer from a player of the gaming system, adjusting a response of the parametric equalizer in accordance with the equalizer input, applying the predetermined audio profile to the parametric equalizer, the predetermined audio profile including at least one equalizer setting for the parametric equalizer, the at least one equalizer setting replacing at least one equalizer input to modify a response of the parametric equalizer, and modifying the gaming system audio stream by the parametric equalizer to produce the modified audio stream.

[0362] Aspects may further include identifying a predetermined sound in the gaming system audio stream and selecting a predetermined audio profile based on the identification of the predetermined sound. Aspects may also include enhancing the gaming system audio stream to improve the ability of a player of the gaming system to hear predetermined events in the gaming system audio stream.

[0363] In embodiments, aspects may include receiving a game system audio stream during a video game, splitting the game system audio stream into a plurality of audio streams including a chat audio stream, a game audio stream, and a microphone audio stream, processing the game audio stream to detect a predetermined audio event in the game audio stream, providing a visual indication of the detection of the predetermined audio event to a first player of the video game on a display device, modifying the game audio stream to enhance audibility of the predetermined audio event to the first player of the video game, generating a modified audio stream, and providing the modified audio stream to an audio device of the first player of the video game. Aspects may further include identifying the predetermined audio event in the game audio stream, providing the game audio stream to a parametric equalizer, selecting a predefined profile for the parametric equalizer, the predefined profile including one or more profile settings to modify the performance of the parametric equalizer to enhance audio reproduction of the predetermined audio event for the first player of the video game, wherein the selection is based on the identifying of the predetermined audio event, and modifying the game audio stream by the parametric equalizer to generate the modified audio stream.

[0364] Aspects may further include sounds in a game audio stream associated with a second player of the video game that form a predetermined audio event. Aspects may further include identifying a direction of origin of the sounds associated with the second player of the video game, wherein the direction of origin is relative to a position in the video game of an avatar associated with a first player of the video game, and modifying, by the processing system, an audio spectrum of the game audio stream to emphasize the direction of origin of the sounds associated with the second player.

[0365] In some embodiments, aspects may include receiving an input audio stream from a gaming system, the input audio stream including game audio of a video game played by a game player, the input audio stream including a plurality of sound categories, providing the input audio stream to a neural network, extracting, by the neural network, sounds of a selected sound category of the plurality of sound categories, and providing a plurality of output audio streams, including providing a first audio stream including sounds of the selected sound category of the input audio stream and a second audio stream including remaining sounds of the input audio stream. Aspects may also include receiving sounds of the gaming audio having an apparent position, and wherein providing the first audio stream includes providing sounds of the selected sound category having the same apparent position as sounds of the gaming audio. Aspects may further include receiving a multi-channel audio signal from a gaming system.

[0366] Aspects may further include receiving surround sound audio from a gaming system, and wherein providing a first audio stream includes providing sounds of a selected sound category in the surround sound audio. Aspects may further include receiving footstep sounds with an apparent origin direction about a game player, extracting the footstep sounds from the input audio stream, and providing the first audio stream including the footstep sounds to the game player with the same apparent origin direction about the game player. Aspects may further include receiving control information from the game player, modifying one or more of a plurality of output audio streams, forming modified output audio, and providing the modified output audio to the game player. Aspects may further include modifying the gain of the sounds of the selected sound category according to the control information from the game player to form the modified output audio. Aspects may further include equalizing the sounds of the selected sound category according to the control information from the game player to form the modified output audio.

[0367] Aspects may further include modifying sounds of the selected sound category based on control information from a game player to form a modified sound of interest, modifying sounds of remaining sounds of the input audio stream based on control information from the game player to form a modified remaining sound, combining the modified sound of interest and the modified remaining sound to form a modified output audio to form a combined output audio, and providing the combined output audio to the game player. Aspects may further include providing one of the combined output audio and the input audio stream to the game player based on control information from the game player.

[0368] In some embodiments, aspects may include receiving an input audio stream from a gaming system, the input audio stream including game audio of a video game played by a game player, the input audio stream including multiple categories of sound, extracting sounds of a selected sound category from the multiple sound categories from the input audio stream, receiving audio processing control information from the game player, and providing multiple output audio streams to the game player based on the audio processing control information, including selectively providing one of a first audio stream including sounds of the selected sound category of the input audio stream and a second audio stream including remaining sounds of the input audio stream. Aspects may further include receiving gain control and equalization control information from the game player, modifying the gain or spectrum of sounds of the selected sound category based on the gain control and equalization control information from the game player, forming modified output sounds, and providing the modified output sounds to the game player.

[0369] Aspects may further include selecting a selected sound category in response to audio processing control information from a game player. Aspects may further include receiving a multi-channel audio stream from a gaming system as the input audio stream, the multi-channel audio stream including position information of sounds of game audio of a video game, and providing a plurality of output audio streams to the game player with the same position information as the sounds of the game audio of the video game. Aspects may further include selecting one of footsteps and gunfire sounds of the game audio of the video game as the selected sound category, wherein the selection is in response to audio processing control information from the game player. Aspects may further include providing the input audio stream to a neural network, and receiving sounds of a selected sound category from the plurality of sound categories from the neural network.

[0370] In some embodiments, aspects may include receiving, by a processing system including a processor, an input audio stream of a gaming system, the input audio stream including game audio of a video game played by a game player, the input audio stream including game sounds, the game sounds being organized into a plurality of categories of sounds, providing, by the processing system, the input audio stream to a neural network, the neural network being trained to recognize game sounds of at least one of the plurality of sound categories, receiving, by the processing system, extracted game sounds of at least one of the plurality of sound categories from the neural network, and providing, by the processing system, to the game player, a first output audio stream including the extracted game sounds and a second output audio stream including remaining sounds in the game audio of the video game. Aspects may also include receiving, by the processing system, audio processing control information from a game player, and modifying, by the processing system, the extracted game sounds based on the audio processing control information.

[0371] Aspects may also include modifying, by the processing system, the gain of the extracted game sounds, equalizing, by the processing system, or compressing, by the processing system, the extracted game sounds, or a combination thereof. Aspects may further include: enhancing, by the processing system, the extracted game sounds of the game audio of the video game; reducing, by the processing system, residual sounds in the game audio of the video game; or a combination thereof.

[0372] The illustrations of the embodiments described herein are intended to provide a general understanding of the structures of the various embodiments, and they are not intended to be used as a complete description of all elements and features of the devices and systems that may utilize the structures described herein. After reviewing the above description, many other embodiments will be apparent to those skilled in the art. Exemplary embodiments may include a combination of features and / or steps from a plurality of embodiments. Other embodiments may be used and may be derived therefrom so that structural and logical replacements and changes may be made without departing from the scope of the present disclosure. The accompanying drawings are also merely representative and may not be drawn to scale. Some of their proportions may be exaggerated, while other proportions may be minimized. Therefore, the specification and the accompanying drawings are considered to be illustrative rather than restrictive.

[0373] Although specific embodiments have been illustrated and described herein, it should be appreciated that any arrangement calculated to achieve the same purpose may replace the specific embodiments shown. The present disclosure is intended to cover any and all modifications or variations of various embodiments. Combinations of the above embodiments and other embodiments not specifically described herein may be used in the present subject disclosure. In one or more embodiments, the features actively listed may also be excluded from the embodiments without or without replacement of another component or step. The steps or functions described with respect to the exemplary process or method may be performed in any order. The steps or functions described with respect to the exemplary process or method may be performed alone or in combination with other steps or functions (from other embodiments or from other steps not yet described).

[0374] Less than all steps or functions described with respect to the exemplary process or method may also be performed in one or more exemplary embodiments. In addition, unless explicitly stated, the use of numerical terms to describe devices, components, steps or functions, such as first, second, third, etc., is not intended to describe order or function. Unless otherwise explicitly stated, the use of the terms first, second, third, etc. is generally to distinguish devices, components, steps or functions. In addition, one or more devices or components described with respect to the exemplary embodiments may contribute to one or more functions, where contributing (e.g., contributing to access or contributing to establishing a connection) may include less than each step required to perform the function, or may include all steps required to perform the function.

[0375] In one or more embodiments, a processor (which can include a controller or circuit) that performs various functions has been described. It should be understood that the processor can be a plurality of processors, which can include a distributed processor or a parallel processor in a single machine or a plurality of machines. The processor can be used to support a virtual processing environment. The virtual processing environment can support one or more virtual machines representing a computer, a server or other computing device. In such a virtual machine, components such as a microprocessor and a storage device can be virtualized or logically represented. The processor can include a state machine, an application specific integrated circuit and / or a programmable gate array including a field PGA. In one or more embodiments, when the processor executes an instruction to perform an "operation", this can include the processor directly performing the operation and / or promoting, guiding or cooperating with another device or component to perform the operation.

[0376] It should be understood that providing an abstract of the present disclosure will not be used to interpret or limit the scope or meaning of the claims. In addition, in the aforementioned detailed description, it can be seen that various features are combined together in a single embodiment for the purpose of streamlining the present disclosure. This method of disclosure should not be interpreted as reflecting the intention that the claimed embodiments require more features than the features explicitly recited in each claim. On the contrary, as reflected in the following claims, the creative subject matter lies in less than all the features of a single disclosed embodiment. Therefore, the following claims are hereby incorporated into the detailed description, with each claim independently serving as a separately claimed subject matter.

Claims

1. A method comprising: receiving, by a processing system including a processor, a plurality of audio streams, the plurality of audio streams comprising a game output audio stream of a gaming application, a gamer chat audio stream generated by a plurality of gamers participating in the gaming application, and a gamer microphone audio stream generated by one of the plurality of gamers; The processing system processes the game output audio stream according to the first audio processing setting of the one gamer to form a processed game output audio stream; The processing system processes the player chat audio stream according to the second audio processing setting of the one player to form a processed player chat audio stream; The processing system processes the player microphone audio stream according to the third audio processing setting of the one player to form a processed player microphone audio stream; The processing system provides the processed game output audio stream and the processed player chat audio stream to the audio device of the one player; as well as The processed player microphone audio stream is provided by the processing system to one or more players of the plurality of players.

2. The method according to claim 1, further comprising: Audio processing settings are received by the processing system from the one player, the audio processing settings including the first audio processing setting, the second audio processing setting, and the third audio processing setting of the one player.

3. The method according to claim 2, further comprising: providing, by the processing system, a graphical user interface on a display device of the one player for the one player to select the audio processing setting, Data defining audio processing settings is received by the processing system from the graphical user interface.

4. The method according to claim 3, wherein: Providing a graphical user interface comprises: providing, by the processing system, a parametric equalizer for the game output audio stream on a portion of the display device of the one gamer; receiving, by the processing system, a player equalizer input from the one player at the parametric equalizer to select a respective gain value for a respective frequency band of the parametric equalizer; and The game output audio stream is filtered by the processing system in a plurality of filters, wherein the filtering is performed based on the gamer equalizer input.

5. The method according to claim 1, further comprising: Receiving, by the processing system, a game audio stream generated by the game application; The processing system establishes a game output virtual audio driver; The game output virtual audio driver generates the game output audio stream; Establishing, by the processing system, a gamer chat virtual audio driver; The player chat audio stream is generated by the player chat virtual audio driver; Establishing, by the processing system, a virtual audio driver for a player's microphone; as well as The player microphone audio stream generated is generated by the player microphone virtual audio driver.

6. The method according to claim 1, further comprising: A player profile for the one gamer is received by the processing system, the player profile defining audio processing settings including the first audio processing settings, the second audio processing settings, and the third audio processing settings for the one gamer.

7. The method according to claim 1, further comprising: identifying, by the processing system, the gaming application; as well as The processing system receives a game profile of the game application based on identifying the game application, the game profile defining audio processing settings, the audio processing settings including the first audio processing setting, the second audio processing setting, and the third audio processing setting for the one gamer for the game application.

8. The method of claim 1, wherein said processing said game output audio stream comprises: forming virtual surround sound game audio by the processing system; as well as The virtual surround sound game audio is provided by the processing system to an audio device of the one gamer.

9. A non-transitory machine-readable medium comprising executable instructions that, when executed by a processing system comprising a processor, facilitate performance of operations comprising: receiving game output information including a game output audio stream from a game application; receiving the game output audio stream from the game application at an audio processing module that performs data communication with the game application, and generating a plurality of game audio streams; as well as Based on player input from the player defining a respective audio customization for each respective game audio stream, the respective audio customization is applied by the audio processing module to each respective game audio stream in the plurality of game audio streams.

10. The non-transitory machine-readable medium of claim 9, wherein: The operations also include: providing a graphical user interface on the gamer's display device; and In response to player actuation of the graphical user interface, the player input defining respective audio customizations for each respective game audio stream is received.

11. The non-transitory machine-readable medium of claim 10, wherein: The operations also include: The player input from the player is received from the graphical user interface, wherein the graphical user interface is in data communication with the audio processing module to control corresponding audio customization.

12. The non-transitory machine-readable medium of claim 10, wherein: The operations also include: providing a parametric equalizer on the graphical user interface; and A player equalizer input is received from the player at the parametric equalizer to select a respective gain value for a respective frequency band of the parametric equalizer.

13. The non-transitory machine-readable medium of claim 12, wherein: The operations also include: establishing a plurality of filters, respective filters of the plurality of filters corresponding to respective frequency bands of the parametric equalizer; and The game output audio stream is filtered in the plurality of filters, wherein the filtering is performed based on the gamer equalizer input.

14. The non-transitory machine-readable medium of claim 10, wherein: The operations also include: identifying one or more audio output devices of the player; and Information controlling mapping of a corresponding one of the plurality of game audio streams to a selected one of the one or more audio output devices is received from the graphical user interface.

15. The non-transitory machine-readable medium of claim 14, wherein the operations further comprise: Information controlling mapping of a corresponding one of the plurality of game audio streams to a selected one of the one or more audio output devices is received.

Images