Dynamic audio optimization

Abstract

Systems and methods for dynamic audio optimization associated with digital content are provided. One or more audio profiles including one or more audio settings for one or more audio output devices available for a user may be stored in a memory. An incoming audio stream associated with the user's current interactive session and real-world audio within an identified real-world space in which the user is located may be monitored. One or more audio deviations may be detected based on a comparison of the incoming audio stream to the real-world audio within the identified real-world space. An audio profile associated with the current interactive session may be automatically recalibrated to modify at least one of the audio settings. The audio output device may then process audio of the incoming audio stream according to the at least one modified audio setting.

Description

Background Art 1. Technical Field

[0001] The present invention relates generally to dynamic audio optimization. More particularly, the present invention relates to dynamic audio optimization related to virtual environments associated with digital content. 2. Background technology

[0003] Currently available digital content may include audiovisual data and other types of data presented in association with a virtual environment. Accordingly, playing audio associated with such digital content (e.g., interactive game titles and other content titles associated with a virtual environment) may involve the use of one or more audio output devices, including home theater devices, televisions, consoles, controllers, handheld devices, computing devices (including desktops, laptops, tablets and other mobile devices), other types of speakers and subwoofers, headphones (including headphones, earbuds or other head-worn or ear-hung speakers), and other peripheral devices or accessory devices that may include associated audio output components. Furthermore, each type of audio output device may have different capabilities and settings available, and each audio output device may produce different sounds when played in a room or space with different acoustic characteristics.

[0004] For example, playing an interactive game title may involve presenting a variety of different audio-visual effects associated with a virtual environment, but the player may use a specific combination of audio devices in a designated playback room. The audio-visual effects may include soundtracks, musical scores, background noise associated with a virtual (e.g., in-game) environment, sounds associated with virtual characters and objects, etc. During a game session of an interactive game title, multiple different types of audio may be presented simultaneously. For example, an in-game scene may be associated with the following: musical scores, ambient sounds associated with a specific location in a virtual environment, and one or more virtual characters and objects that are speaking or otherwise making noises or other audio effects. However, different users may have different combinations of audio output devices (each device has its own audio capabilities and settings, and produces different types of sounds in rooms or spaces of different shapes and configurations). Due to this difference, each user's audio experience of the same digital content may be different.

[0005] Additionally, different intermittent or repetitive constraints may further result in or require compliance with certain audio parameters. For example, a shared household (or multi-family housing) with young children, elderly parents, or dissatisfied neighbors within earshot may implement noise restrictions during certain hours of the day or night. In addition, audio associated with digital content (e.g., played during an immersive virtual reality session) may also overwhelm the user's ability to hear what is happening in the real-world environment. In addition, other users - especially those with hearing-related diseases - may have different preferences for audio settings than users without such diseases. Therefore, audio parameters may need to be adjusted based on many different factors, including playing different digital content titles, different audio output devices available, using different playback spaces, different user abilities and sensitivities, different existing conditions, etc., to optimize the user's audio experience.

[0006] Therefore, there is a need in the art for systems and methods that improve dynamic audio optimization associated with digital content. Summary of the invention

[0007] Embodiments of the present invention include systems and methods for dynamic audio optimization associated with digital content. One or more audio profiles including one or more audio settings for one or more audio output devices available to a user may be stored in a memory. An incoming audio stream associated with a current interactive session of the user and real-world audio within an identified real-world space in which the user is located may be monitored. One or more audio deviations may be detected based on a comparison of the incoming audio stream with the real-world audio within the identified real-world space. An audio profile associated with the current interactive session may be automatically recalibrated to modify at least one of the audio settings. The audio output device may then process the audio of the incoming audio stream according to the at least one modified audio setting. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 A network environment is shown in which a system for dynamic audio optimization associated with digital content may be implemented.

[0009] Figure 2 An exemplary unified data system (UDS) that may be used to provide data to a system for dynamic audio optimization associated with digital content is shown.

[0010] Figure 3 is a flow chart illustrating an exemplary method for dynamic audio optimization associated with digital content.

[0011] Figure 4 is a diagram illustrating an exemplary implementation of dynamic audio optimization associated with digital content.

[0012] Figure 5 is a block diagram of an exemplary electronic entertainment system that may be used in embodiments of the present invention. DETAILED DESCRIPTION

[0013] Embodiments of the present invention include systems and methods for dynamic audio optimization associated with digital content. One or more audio profiles including one or more audio settings for one or more audio output devices available to a user may be stored in a memory. An incoming audio stream associated with a current interactive session of the user and real-world audio within an identified real-world space in which the user is located may be monitored. One or more audio deviations may be detected based on a comparison of the incoming audio stream with the real-world audio within the identified real-world space. An audio profile associated with the current interactive session may be automatically recalibrated to modify at least one of the audio settings. The audio output device may then process the audio of the incoming audio stream according to the at least one modified audio setting.

[0014] Figure 1 A network environment 100 is shown in which a system for dynamic audio optimization associated with digital content can be implemented. The network environment 100 may include: one or more content source servers 110 that provide digital content (e.g., games, other applications, and services) for distribution; one or more content provider server application program interfaces (APIs) 120; a content distribution network server 130; an audio analysis server 140; and one or more user devices 150. The devices in the network environment 100 communicate with each other using one or more communication networks, which may include a local private network (e.g., an intranet) and / or may be part of a larger wide area network. The communication network may be a local area network (LAN) that is communicatively coupled to a wide area network (WAN) such as the Internet. The Internet is a wide network of interconnected computers and servers that allows the transmission and exchange of Internet Protocol (IP) data between users connected through a network service provider. Examples of network service providers are: a public switched telephone network, a cable service provider, a digital subscriber line (DSL) service provider, or a satellite service provider. The one or more communication networks allow communication between various components in the network environment 100.

[0015] The servers described herein may include any type of server as known in the art, including standard hardware computing components, such as network and media interfaces, non-transitory computer-readable storage devices (memory), and a processor for executing instructions or accessing information that may be stored in the memory. The functionality of multiple servers may be integrated into a single server. Any of the above servers (or integrated servers) may have certain client-side, cache, or proxy server features. These features may depend on the specific network arrangement of the server or certain configurations of the server.

[0016] The content source server 110 may maintain and provide a variety of digital content and digital services that are available for distribution over the communication network. The content source server 110 may be associated with any content provider that makes its content available for access over the communication network. The content source server 110 may thus host a variety of different content titles that may also be associated with object data (e.g., activity information, area information, character information, player information, other game media information, etc.) regarding digital or virtual objects that are displayed in a digital or virtual environment during an interactive session.

[0017] Such content may include not only digital videos and games, but also other types of digital applications and services. Such applications and services may include any of a variety of different digital content and functionality that may be provided to user device 150, including providing and supporting chat and other communication channels. Chat and communication services may include voice-based, text-based, and video-based messages. Thus, user device 150 may participate in a gaming session concurrently with one or more communication sessions, and the gaming session and the communication session may be hosted on one or more of content source servers 110.

[0018] Content from the content source servers 110 may be provided through a content provider server API 120 that allows various types of content source servers 110 to communicate with other servers (e.g., user devices 150) in the network environment 100. The content provider server API 120 may be specific to the particular operating language, system, platform, protocol, etc. of the content source server 110 providing the content, as well as the user device 150 and other devices of the network environment 100. In a network environment 100 that includes a variety of different types of content source servers 110, there may likewise be a corresponding number of content provider server APIs 120 that allow various formatting, conversion, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150, which may each use different operating systems, protocols, etc. to process such content. Thus, applications and services in different formats may be made available for compatibility with a variety of different user devices 150. In a network environment 100 that includes multiple different types of content source servers 110 , content distribution network servers 130 , translation filter servers 140 , user devices 150 , and databases 160 , there may also be a corresponding number of APIs managed by the content provider server API 120 .

[0019] The content provider server API 120 may also facilitate each of the user devices 150 to access content hosted or services provided by the content source server 110, either directly or via the content distribution network server 130. Additional information about the accessed content or services, such as metadata, may also be provided to the user devices 150 by the content provider server API 120. As described below, the additional information (e.g., object data, metadata) may be useful for providing details about the content or services provided to the user devices 150. In some embodiments, the services provided from the content source server 110 to the user devices 150 via the content provider server API 120 may include supporting services associated with other content or services, such as chat services, ratings, and profiles associated with a particular game, team, community, etc. In such cases, the content source servers 110 may also communicate with each other via the content provider server API 120.

[0020] The content distribution network server 130 may include a server that provides resources, files, etc. (including various content and service configurations) related to content from the content source server 110 to the user device 150. The content distribution network server 130 may also be called by the user device 150 requesting access to specific content or services. The content distribution network server 130 may include a global management server, a game server, a streaming media server, a server that hosts downloadable content, and other content distribution servers known in the art.

[0021] The audio analysis server 140 may include any data server known in the art that is capable of communicating with different content source servers 110, content provider server APIs 120, content distribution network servers 130, user devices 150, and databases 160. Such an audio analysis server 140 may be implemented on one or more cloud servers that execute instructions associated with interactive content (e.g., games, activities, videos, podcasts, user-generated content ("UGC"), publisher content, etc.). The audio analysis server 140 may also execute instructions, for example, for monitoring one or more incoming audio streams of one or more audio output devices in a real-world space (e.g., a room), and real-world audio output within the real-world space. Specifically, the audio analysis server 140 may analyze the audio output against the incoming audio stream based on one or more stored audio profiles. The audio analysis server 140 may calibrate (and continually recalibrate) the audio settings of the audio output device based on the analysis of the incoming and outgoing audio. Different sets of audio settings may be stored in an audio profile, which may be specific to a space (e.g., Jen's living room), specific to a digital content title (e.g., Fortnite), and / or specific to a set of conditions (e.g., after midnight, ambient sound levels).

[0022] User devices 150 may include a variety of different types of computing devices known in the art. User devices 150 may be computing devices including any number of different game consoles, mobile devices, laptops, and desktops. Such user devices 150 may also be configured to access data from other storage media, such as, but not limited to, memory cards or disk drives that may be appropriate in the case of downloaded services. Such user devices 150 may include standard hardware computing components, such as, but not limited to, network and media interfaces, non-transitory computer-readable storage devices (memory), and processors for executing instructions that may be stored in memory. These user devices 150 may also run using a variety of different operating systems (e.g., iOS, Android), applications, or computing languages ​​(e.g., C++, JavaScript). In this article about Figure 5 Exemplary client devices 150 are described in detail. Each user device 150 may be associated with a participant (eg, a player) or other type of user (eg, a spectator) associated with a collection of digital content streams.

[0023] In some embodiments, the user device 150 may include a group of different types of user devices located in a real-world space (e.g., a living room, a sitting room, a game room, or other architectural space) associated with a particular user or user group. The user device 150 may include, for example, any device (including a smart device) capable of outputting and / or capturing audio or sound and communicating with other user devices 150 in the communication network 100 and / or the real-world space. Therefore, the user device 150 may include a home theater device, a television, a sound bar, a device controller, a handheld device, a speaker and a subwoofer, a headset (including a headset, an earbud, or other head-mounted or ear-hook speaker), other peripheral devices or accessory devices that may include an associated audio output component, and a microphone (which may be separated from another user device or built into another user device). Some devices may include both a microphone and an audio output component (e.g., a speaker). Therefore, the user device 150 may also include a smart home device, an Internet of Things (IoT) device, a virtual assistant device, a baby and pet monitoring device, and the like.

[0024] In an exemplary embodiment, a subset of user devices 150 distributed in a particular real-world space (e.g., a gaming room) may include audio output devices, each of which may be located at a different location within the real-world space and include different audio capabilities (e.g., mono vs. stereo, 3D audio effects, noise cancellation, frequency range, volume range, treble / bass, etc.) and related settings. A real-world space may be configured in a variety of different ways, which may have different effects on the audio and sound waves propagating within and throughout the space. For example, the configuration of walls, the height of the ceiling, surface materials with different reflective, absorptive, or diffusive properties, the presence of objects or individuals, etc., may affect the way a user experiences audio in the real-world space. In addition, because user devices 150 may include mobile, handheld, or other portable devices, a particular set of audio output devices within the real-world space may not only change in terms of presence within the real-world space, but also in relative positions within the real-world space based on the movement or repositioning of the user. In addition, the specific location of the user within the real-world space may also result in a slightly different audio experience.

[0025] Although depicted separately, database 160 may be stored on any of the servers and devices shown in network environment 100, on the same server, on a different server, or on any of user devices 150. Such database 160 may store or link to various sources and services for audio analysis and modification. In addition, database 160 may store various different audio profiles (e.g., associated with a specific real-world space, content title, or other condition of the current session) and information about available audio settings, as well as audio modifications and customizations that may be specific to a specific user, user group or team, user category, game title, game style, sound type, condition, etc. The audio profile may be used to identify a specific speaking voice (e.g., of a user, a specific other user), which may be used as an audio parameter that may be prioritized and audio modified. One or more audio profiles may also be stored in database 160 for each user, real-world space, content title, or condition.

[0026] The user-specific audio profile may also include a set of audio parameters specified or adopted by the user according to preferences. For example, the user may prefer a high volume or a high level of bass. Such audio parameters - and the settings of the specific audio device associated with the user and causing such audio parameters - can be stored in the audio profile and updated when it is determined that the preference has changed. In another example, some users may have different hearing-related diseases, which makes certain sounds or frequencies unpleasant or difficult to distinguish. The audio profile optimized for such users can specify the audio settings applicable to modify the incoming audio stream to reduce the problematic sounds or frequencies output from the audio output device associated with the user. Different hearing preferences (e.g., frequency response range, speed range, volume range) may also be specified by the user and stored in the audio profile, as well as the relevant audio settings for the specific audio output device associated with the user and applicable to the incoming audio stream to be optimized according to the user's preferences. Preferences and related audio settings can be applied to stream combinations, separate streams and / or specific sounds / speech within the stream, and audio output device combinations or separate audio output devices.

[0027] One or more microphones may also be used to capture data about sounds and audio within the real-world space in which the user is located during the interactive session. The captured data about sounds and audio may include or indicate ambient noise, echo, and other audio deviations or distortions compared to the digital audio stream. Thus, the audio analysis server 140 may monitor and compare the incoming digital audio stream to the real-world audio during the current session to identify and measure different types of audio deviations or distortions, and to determine which audio settings may compensate for or offset the identified audio deviations or distortions associated with a particular real-world space. The results of the analysis may be stored in an audio profile specifically associated with the real-world space and its characteristics and properties, including acoustic properties.

[0028] Different audio profiles may also be specified based on specific content titles, content types (e.g., movies vs. games vs. virtual concerts), peer groups (e.g., friends, teammates, competitors, spectators), channels or platforms (e.g., in-game chat, external chat services such as Discord), and other characteristics of the session. In a given session, one or more user devices 150 in the real-world space may receive the same or different audio streams. The audio analysis server 140 may act as an intermediary device that modifies one or more of the audio settings before delivering the stream to (one or more) audio output devices for processing and playback. Alternatively, the audio analysis server 140 may operate in conjunction with one or more local applications to provide and apply audio-related insights and setting modifications to (one or more) user devices 150.

[0029] Real-world audio may not constantly or consistently exhibit the same audio distortion or deviation over time. Therefore, the audio analysis server 140 can constantly recalibrate the audio profile based on changing conditions. The recalibrated audio profile can specify different audio settings associated with offsetting or compensating for the audio distortion or deviation detected in real time (or near real time) during the current session. In some homes or other real-world spaces, the type of audio distortion or deviation - or other constraints - may recur. For example, traffic noise may be particularly loud during rush hour, children may need low volume or switch to the continuous bedtime of the headphone audio device, and families near airport flight paths may experience noise associated with aircraft. Such regular and irregular patterns can be identified by the audio analysis server 140 based on comparison and association with historical data about similar conditions and real-world sounds and audio. The audio analysis server 140 can monitor historical data about conditions and audio to detect new and emerging audio patterns that recur over time. Such insights can be used to update the audio profile about what audio settings may be optimal for achieving certain results in the real-world space considering the identified audio pattern.

[0030] In some embodiments, audio profiles can be specific to digital content titles. Such audio profiles can be defined by a predetermined set of acoustic properties, which should be produced by an optimized audio system. For example, the content of horror style can include extremely quiet scenes that create suspense, tension or fear, and pave the way for noisy and dramatic climax scenes. Optimizing audio settings to enhance horror experience can include different groups of audio settings to ensure that quiet scenes have their expected effects and experiences according to the acoustic properties of real-world environments, and to ensure that noisy and dramatic scenes have their expected effects and experiences. Alternatively, audio settings can be optimized to minimize the tension and fear that horror style content of certain user groups (e.g., children, people with heart disease or other sensitive people) brings. Audio profiles optimized for a specific type of experience related to a content title (or a scene in a content title) can be generated by content developers, service providers, users or other users. Such audio profiles can be generated and shared between users who share expectations for a specific type of audio experience. Thus, not only can audio profiles be optimized for different audio output devices, systems, and content titles (e.g., genre, style, etc.), but audio profiles can also be optimized by real-world audio constraints, subject matter, mood, health / sensitivity / accessibility, and other preferences.

[0031] In some embodiments, the audio analysis server 140 can generate an audio profile for the real-world space by transmitting a calibration signal (which can be the same or different for each device) to an audio output device in the real-world space. The calibration signal can include a predetermined multi-layer signal or a set of signals, including multiple different frequencies. The audio output device can include not only traditional entertainment devices (e.g., televisions, consoles, sound bars, speakers, subwoofers), but also any audio output device connected to a communication network to receive signals, including devices with built-in speakers (e.g., controllers), mobile devices, headphones, smart home devices (including virtual assistant devices), IoT devices, etc. The audio output device can play the calibration signal in the real-world space, and the output signal (including the frequency response) can then be captured by a microphone in the real-world space (which can be associated or not associated with the audio output device).

[0032] Based on the captured frequency response data, the audio analysis server 140 can identify certain properties (including acoustic properties) about the real-world space. Such properties may include data about the relative position of the audio output device and other acoustic data that can be used to evaluate and measure audio distortion and deviation relative to (one or more) digital calibration signals. Once the audio distortion and deviation are identified, the audio analysis server 140 can further determine which audio settings for each audio output device in the real-world space can compensate or offset the audio distortion and deviation. In addition, because the user may move in the real-world space, even if the user may move from one position to another in the real-world space, the audio settings can be recalibrated to dynamically provide a consistent audio experience to the user. To help make such a judgment, other types of sensors can be used to track the user's position in the real-world space, including cameras, infrared sensors, motion sensors, beacon devices, wearable devices, handheld devices, etc.

[0033] As described above, historical data about various conditions can be tracked relative to the resulting audio experience. Audio analysis server 140 can also use historical data to determine which audio settings can best compensate or offset any audio degradation in the real world space, which can also include audio settings optimized for different types of audio experiences. Therefore, audio analysis server 140 can also generate instructions for one or more adjustments to the audio settings of one or more of the audio output devices. Such instructions can be generated using one or more APIs from a content provider server API 120 that may be associated with a specific audio output device. When recalibrating an audio profile using a new audio setting or a new audio setting modification, new instructions can be generated based on the recalibrated audio profile. In some embodiments, audio analysis server 140 can generate a notification about the recalibrated audio profile to the user, and request approval to implement the new or modified audio settings in the recalibrated audio profile. Such notifications can also include recommendations or suggestions on how to obtain a specific audio experience. Such recommendations may include checking whether certain audio output devices are plugged in, turned on, switching between a home entertainment system and headphones, or vice versa, and determining specific audio settings to achieve the user's desired experience, which may be stored in a user profile along with explicit and implicit preferences, historical interactive session data, interactive habits, etc. Upon user approval, the new or modified audio settings in the recalibrated audio profile may be provided to the corresponding audio output devices in the real-world space for use in processing the incoming audio stream.

[0034] Figure 2An exemplary unified data system (UDS) 200 that can be used to provide data to a system for dynamic audio optimization associated with digital content is shown. Based on the data provided by the UDS 200, the translation filtering server 140 can be informed of current session conditions, such as the digital content title, what virtual (e.g., in-game) objects, entities, activities, and events the user has participated in, thereby supporting the analysis of audio optimization by the audio analysis server 140 and the coordination with the current virtual interaction and / or in-game activities. Each user interaction within the virtual environment can be associated with the type of virtual interaction, the location within the virtual environment, the time point within the virtual world timeline, and the metadata of other players, objects, entities, etc. involved. Therefore, metadata can be tracked for any of the various user interactions that may occur during the current interactive session, including associated virtual activities, entities, settings, results, actions, effects, locations, and character statistics. Such data can also be aggregated, applied to data models, and analyzed. Such UDS data models can be used to distribute context information to each part of the information in a unified manner across digital content and / or game titles.

[0035] For example, various content titles may depict one or more objects (e.g., those involved in in-game activities) with which a user may interact and / or UGC (e.g., screenshots, videos, narrations, aggregated applications, etc.) created by peers, publishers of media content titles, and / or third-party publishers. Such UGC may include metadata by which such UGC is searched. Such UGC may also include information about the media and / or peers. Such peer information may be obtained from data collected during peer interactions with objects of interactive content titles (e.g., video games, interactive books, etc.), and may be "bound" to the UGC and stored with it. Such binding enhances the UGC because the UGC may be deeply linked (e.g., directly launched) to the object, may provide information about the object and / or peers of the UGC, and / or may allow the user to interact with the UGC.

[0036] like Figure 2, an exemplary console 228 (e.g., user device 150) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user generated content (UGC) server 232, and object server 226) are shown. In one example, console 228 may be implemented on platform server 120, cloud server, or on any of server 218. In the exemplary example, content recorder 202 may be implemented on platform server 120, on cloud server, or on any of server 218. Such content recorder 202 receives content (e.g., media) from interactive content title 230 and records the content on content ring buffer 208. Such ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), the start time of each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and the end time of each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as a media file 212 (e.g., MP4, WebM, etc.). Such media files 212 may be uploaded to the streaming server 220 for storage and subsequent streaming or use, but the media files 212 may be stored on any server, cloud server, any console 228, or any user device 150. Such start time and end time of each segment may be stored by the console 228 as a content timestamp file 214. Such content timestamp files 214 may also include a streaming ID that matches the streaming ID of the media file 212, thereby associating the content timestamp file 214 to the media file 212. Such content timestamp files 214 may be uploaded and stored to the activity feed server 224 and / or the UGC server 232, but the content timestamp files 214 may be stored on any server, cloud server, any console 228, or any user device 150.

[0037] Concurrently with the content recorder 202 receiving and recording content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine when an object starts and ends. The object library 204 and the object recorder 206 may be implemented on the platform server 120, the cloud server, or on any of the servers 218. When the object recorder 206 detects the start of an object, the object recorder 206 receives object data (e.g., whether the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity results, activity type, etc.) from the object library 204 and records the activity data on the object ring buffer 210 (e.g., ActivityID1, START_TS; ActivityID2, START_TS; ActivityID3, START_TS). Such activity data recorded on the object ring buffer 210 may be stored in the object file 216. Such object files 216 may also include the activity start time, activity end time, activity ID, activity results, activity type (e.g., competition, exploration, mission, etc.), user or companion data related to the activity. For example, object files 216 may store data about items used during the activity. The object files 216 may be stored on object server 226, but object files 216 may be stored on any server, cloud server, any console 228, or any user device 150.

[0038] Such object data (e.g., object file 216) may be associated with content data (e.g., media file 212 and / or content timestamp file 214). In one example, UGC server 232 stores content timestamp file 214 and associates the content timestamp file with object file 216 based on a match between a streaming ID of content timestamp file 214 and a corresponding activity ID of object file 216. In another example, object server 226 may store object file 216 and may receive a query for object file 216 from UGC server 232. Such a query may be performed by searching for an activity ID of object file 216 that matches the streaming ID of content timestamp file 214 transmitted with the query. In another example, the query may be performed by matching the start time and end time of content timestamp file 214 with the start time and end time of the corresponding object file 216 transmitted with the query for the stored content timestamp file 214. Such object files 216 may also be associated with matching content timestamp files 214 by the UGC server 232, but the association may also be performed by any server, cloud server, any console 228, or any user device 150. In another example, the object files 216 and content timestamp files 214 may be associated by the console 228 during creation of each file 216, 214.

[0039] In an exemplary embodiment, media files 212 and activity files 216 can provide information about current session conditions to audio analysis server 140, which can also be used as another basis for evaluating incoming audio streams and applying audio modifications thereto. Therefore, audio analysis server 140 can use such media files 212 and activity files 216 to identify specific conditions of the current session, including players, characters, and objects at specific locations and events who are currently speaking or making noises. Based on such files 212 and 216, for example, audio analysis server 140 can identify relevant audio configuration files associated with content titles, virtual environments, virtual scenes, or in-game events (e.g., major battles, near record breaking), which can be used to dynamically modify audio settings during the current session and throughout the entire current session. Such session conditions can drive how the audio settings of different audio output devices can be modified, resulting in a customized determination of whether to apply and which audio settings to apply and to what extent.

[0040] Figure 3 is a flow chart illustrating an exemplary method 300 for dynamic audio optimization associated with digital content. Figure 3The method 300 may be embodied as executable instructions in a non-transitory computer-readable storage medium, including but not limited to a CD, a DVD, or a non-volatile memory such as a hard drive. The instructions of the storage medium may be executed by a processor (or multiple processors) to cause various hardware components of a computing device that hosts or otherwise accesses the storage medium to implement the method. Figure 3 The steps (and their order) identified in the description are exemplary and may include various alternatives, equivalents, or derivatives of the steps, including but not limited to the order of execution of the steps.

[0041] In step 310, the audio profile can be stored in a memory (e.g., database 160). Each audio profile can include a set of audio settings for one or more audio output devices. As described above, the audio profile can be specific to a real-world space (e.g., a room in which a user conducts an interactive session), specific to a content title (or style), and other audio conditions. In some embodiments, a new user can develop or further customize an audio profile by specifying their personal preferences and priorities associated with different sounds and audio streams across different audio output devices and content titles. In some embodiments, the audio analysis server 140 can query a new user to determine how to generate an audio profile for a user and / or a real-world space associated with a user's interactive session. Users can also choose to customize an existing audio profile (perhaps related to a specific audio setting, content title, or its style) based on their own personal preferences, which can be shared with other users for implementation of dynamic audio optimization.

[0042] In step 320, one or more incoming (digital) audio streams may be monitored by the audio analysis server 140, which may also monitor the real-world audio output in the real-world space—through a microphone and an audio detection device. The audio stream associated with the digital content may be presented to the user who uses the user device 150 to stream and play the digital content and the associated audio in the current session. In the case where the digital content is a multiplayer game, another audio stream associated with the voice chat function may also include sounds and speech associated with other users. In some cases, other chat services (e.g., Discord servers) may be used to provide another audio stream. In some embodiments, the incoming audio stream may include a calibration signal, the frequency response of which may be captured when playing in the real-world space. Calibration and recalibration may occur periodically or may be triggered by conditions that are detected as having changed during an interactive session. For example, the battery of the audio output device may be exhausted, the plug of the audio device may fall off or be knocked out of the socket, or the speaker (or speakers) of the audio output device may be offline for any reason. Different virtual interactive scenes may require different types of audio settings to optimize for specific experiences, moods, or effects.

[0043] In step 330, the audio stream and real-world audio output are analyzed by the audio analysis server 140 against the applicable audio profile. Specifically, the target audio parameters or device-specific audio settings specified by the audio profile are identified as being different from the real-world audio being monitored in the real-world space. This difference may be caused by audio distortion or deviation caused by the configuration and properties of the real-world space. In addition, certain audio parameters, requirements, or constraints can be identified by reference to historical data, which can be aggregated and analyzed to identify indicated audio-related user behavior patterns. For example, a user may repeatedly lower the system audio volume after certain times of the night, switch between speakers and headphones, or otherwise change audio settings in response to certain content titles or their style. Therefore, various audio-related factors can indicate the presence of audio distortion or deviation in the real-world space.

[0044] In step 340, one or more audio settings associated with at least one of the audio devices in the real-world space may be calibrated based on the analysis performed in step 330. More specifically, the audio settings may be adjusted in a manner that compensates for or counteracts the identified audio distortion or deviation. For example, when certain sounds from a particular audio output device may be determined to be absorbed or reflected by a surface to a certain extent, the audio analysis server 140 may adjust the audio settings of the particular audio output device and / or other devices to compensate for or counteract such absorption or reflection. Therefore, based on the analysis performed in step 330, the audio analysis server 140 may make one or more audio setting modifications to one or more audio devices. In some implementations, calibration and recalibration may occur continuously, thereby automatically adjusting the audio settings to fine-tune the audio experience of the user in the real-world space. In some embodiments, user feedback on preferences, likes, and dislikes for the current audio experience may be used to predict what the user may prefer in relation to new content titles, changes in the real-world space, or other situations where the current audio profile may not be fully applicable. A new audio profile may be created, or an old audio profile may be further improved to take into account new and different combinations of conditions.

[0045] In step 350, the (one or more) incoming audio streams may have been provided to one or more audio output devices for playback and presentation in real time or near real time according to the audio settings calibrated in step 340, but audio deviations may be detected. Therefore, the (one or more) audio streams that have been processed can be presented to the user of the user device 150 to create an audio experience consistent with the applicable audio profile, but the current conditions may change the way the audio is presented or experienced. For example, a sudden increase in background noise (e.g., from children, neighbors, birds, buildings, or other sources) may prevent the user from discerning certain sounds associated with the virtual environment and provided in the incoming audio stream. Although the user may want to pause the current session in response to certain external noise sources (e.g., children), the user may also want to block or otherwise offset other background noise sources. In this way, not only can audio deviations or distortions be detected and measured, but they can also be further analyzed to characterize the type of audio deviations or distortions, thereby triggering different types of adjustments or responses. In addition, over time, users can also continue to specify improvements and other changes to the audio profile, which allows audio modifications to be applied to future audio streams in a manner that better reflects user preferences and priorities.

[0046] In step 360, adjustments or notifications may be triggered based on the audio deviations detected in step 350. As discussed herein, adjustments may include adjustments to one or more audio settings of one or more audio output devices in the real-world space. In some embodiments, additional recommendations or suggestions for acoustic improvements may be sent to the user via a notification. For example, one or more audio devices may be detected as being offline, and the notification may suggest that the user check the status of the offline devices and bring the offline devices back online. Certain implementations may request user approval before implementing certain adjustments. For example, the user may not currently be wearing headphones with spatial audio capabilities, and the audio of an upcoming scene may include spatial audio effects. The notification may suggest that the user put on headphones before an upcoming scene to experience the spatial audio effects.

[0047] In step 370, one or more audio profiles may be updated based on the detected audio deviations and associated adjustments or notifications. Such updates may include information about the detected audio deviations, the extent to which the adjustments offset or compensated for the audio deviations, and the effect that the user appears to be responding to the dynamic audio optimization actions. Such data may be used to continually fine-tune the audio profiles to provide an audio experience that meets the user's expectations and preferences.

[0048] Figure 4 1 is a diagram illustrating an exemplary implementation of dynamic audio optimization associated with digital content. As shown, different audio sources 410A-C may be monitored during a current session of a user device and provided to the audio analysis server 140 for analysis and processing. Incoming audio stream 410A may be associated with a content source server (e.g., Figure 1 Incoming audio stream 410A may be provided to an audio output device and may also be provided or monitored by audio analysis server 140. Real-world audio 410B may include a digital audio stream associated with a content title processed and played by a real-world audio output device, as well as background noise in a real-world space. Real-world audio 410C may be captured by one or more microphones of a user device 150 located in the real-world space and provided to audio analysis server 140 for analysis and processing. Predetermined audio signal 410C may also be provided to audio analysis server 140 and provided to an audio output device in the real-world space for playback for calibration (and recalibration).

[0049] The audio analysis server 140 may retrieve one or more audio profiles from one of the databases 160, which may include a specific database storing space-specific audio profiles 420A (which may be shared by a group of users in the same household) and / or content-specific audio profiles 420B (which may be shared by a group of users playing the same interactive content title). Using the audio profiles associated with the users of the user devices 150, the audio analysis server 140 may analyze various audio parameters of the sound and audio within the real-world space. Different audio parameters may be identified and characterized by the audio analysis server 140 as being associated with audio deviations or distortions present in the incoming audio stream 410A, the predetermined audio signal 410C, or the real-world space. The audio analysis server 140 may also identify what audio setting modifications the audio profile specifies to compensate or offset any detected deviations or distortions, and / or predict what audio setting modifications may best compensate or offset the detected deviations or distortions. The identified or predicted audio setting modifications may then be transmitted by the audio analysis server 140 to one or more audio output devices 150 , which may then process their respective incoming audio streams 410A, or portions thereof, according to the corresponding modified audio settings.

[0050] Figure 5 is a block diagram of an exemplary electronic entertainment system that may be used in embodiments of the present invention. Figure 5 The entertainment system 500 includes a main memory 505, a central processing unit (CPU) 510, a vector unit 515, a graphics processing unit 520, an input / output (I / O) processor 525, an I / O processor memory 530, a controller interface 535, a memory card 540, a universal serial bus (USB) interface 545, and an IEEE interface 550. The entertainment system 500 also includes an operating system read-only memory (OS ROM) 555 connected to the I / O processor 525 via a bus 575, a sound processing unit 560, an optical disk control unit 570, and a hard disk drive 565.

[0051] The entertainment system 500 may be an electronic game console. Alternatively, the entertainment system 500 may be implemented as a general purpose computer, a set-top box, a handheld gaming device, a tablet computing device, or a mobile computing device or a mobile phone. The entertainment system may include more or fewer operating components depending on a particular form factor, purpose, or design.

[0052] Figure 5 The CPU 510, vector unit 515, graphics processing unit 520 and I / O processor 525 communicate via the system bus 585. In addition, Figure 5The CPU 510 communicates with the main memory 505 via a dedicated bus 580 , while the vector unit 515 and the graphics processing unit 520 may communicate via a dedicated bus 590 . Figure 5 The CPU 510 executes programs stored in the OS ROM 555 and the main memory 505 . Figure 5 The main memory 505 may contain pre-stored programs and programs transferred from a CD-ROM, DVD-ROM or other optical disk (not shown) via the I / O processor 525 using the optical disk control unit 570 . Figure 5 The I / O processor 525 may also allow for the introduction of content delivered via wireless or other communication networks (e.g., 4$, LTE, 3G, etc.). Figure 5 The I / O processor 525 mainly controls data exchange between various devices of the entertainment system 500 , including the CPU 510 , the vector unit 515 , the graphics processing unit 520 , and the controller interface 535 .

[0053] Figure 5 The graphics processing unit 520 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate images for display on a display device (not shown). For example, Figure 5 The vector unit 515 can transform the object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. In addition, the sound processing unit 560 executes instructions to generate sound signals, which are output to an audio device such as a speaker (not shown). Other devices can be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550 such as a wireless transceiver, which can also be embedded in the system 500 or as part of some other component such as a processor.

[0054] Figure 5 A user of the entertainment system 500 provides instructions to the CPU 510 via the controller interface 535. For example, the user may instruct the CPU 510 to store certain game information on the memory card 540 or other non-transitory computer-readable storage medium, or instruct a character in the game to perform a certain specified action.

[0055] The present invention may be implemented in applications that may be operated by a variety of end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., a Sony or Sony or Sony ), portable gaming devices (e.g., Sony or Sony ), or a different, lesser manufacturer's home entertainment system. It is fully contemplated that the present method described herein can operate on a variety of devices. The present invention can also be implemented in a cross-title neutral situation, where embodiments of the present system can be utilized across a variety of titles from a variety of publishers.

[0056] The present invention may be implemented in applications that may be operated using a variety of devices. Non-transitory computer-readable storage media refers to any one or more media that participate in providing instructions to a central processing unit (CPU) for execution. Such media may take many forms, including but not limited to non-volatile and volatile media, such as optical or magnetic disks and dynamic memory, respectively. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, and any other memory chip or cassette.

[0057] Various forms of transmission media may be involved in transmitting one or more sequences of one or more instructions to the CPU for execution. The bus transmits the data to the system RAM, from which the CPU retrieves and executes the instructions. The instructions received by the system RAM may optionally be stored on a fixed disk before or after execution by the CPU. Various forms of storage devices and the necessary network interfaces and network topologies may be similarly implemented to achieve the same purpose.

[0058] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. In view of the above teachings, many modifications and variations are possible. The described embodiments are selected in order to best explain the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and make various modifications suitable for the intended specific use. It is intended that the scope of the technology be defined by the claims.

Claims

1. A method for dynamic audio optimization associated with digital content, the method comprising: storing one or more audio profiles in a memory, wherein each of the audio profiles includes one or more audio settings for one or more audio output devices available to a user; monitoring an incoming audio stream associated with a current interactive session of the user and real-world audio within an identified real-world space in which the user is located; detecting one or more audio deviations based on a comparison of the incoming audio stream to the real-world audio within the identified real-world space; automatically recalibrating an audio profile associated with the current interactive session, wherein at least one of the audio settings of the audio profile is modified; as well as The at least one audio setting of the audio output device within the identified real-world space is modified based on the recalibrated audio profile, wherein the audio output device processes audio of the incoming audio stream according to the at least one modified audio setting.

2. The method of claim 1, wherein the audio profile is specific to a digital content title associated with the incoming audio stream, and wherein the audio settings in the audio profile are based on a predetermined set of acoustic properties associated with the digital content title.

3. The method of claim 1 , wherein the audio profile is specific to the identified real-world space and includes one or more acoustic properties of the identified real-world space, and wherein the audio settings in the audio profile are based on the acoustic properties of the identified real-world space.

4. The method of claim 3, further comprising generating the audio profile, wherein generating the audio profile comprises: providing a calibration signal associated with one or more predetermined frequencies for playback by one or more of the audio output devices in the real-world space; receiving frequency response data captured by one or more microphones in the real-world space while the calibration signal is played by the one or more audio output devices; identifying one or more properties of the real-world space based on the frequency response data, wherein the identified properties include the acoustic properties; as well as Instructions are generated for making one or more adjustments to the audio settings of the audio output device based on the identified properties of the real-world space, wherein the instructions for audio setting adjustments are executable to offset one or more of the audio deviations associated with the identified properties of the real-world space, and wherein modifying the at least one audio setting is based on the instructions. 5 . The method of claim 4 , wherein identifying the attribute of the real-world space comprises identifying a relative position of the audio output device within the real-world space. The method of claim 4 , wherein the calibration signal is a predetermined multi-layer signal comprising a plurality of different frequencies.

7. The method of claim 4, wherein the audio output device in the real-world space comprises one or more of a television, a game console device, a console controller, a speaker, a subwoofer, a computing device, a mobile device, a handheld electronic device, a smart device, an Internet of Things (IoT) device, and a head-mounted speaker device.

8. The method of claim 1, wherein automatically recalibrating the audio profile comprises monitoring one or more conditions of the real-world space, and wherein the at least one audio setting of the audio profile is modified based on the monitored conditions.

9. The method of claim 1 , wherein automatically recalibrating the audio profile is further based on historical data about the user or the real-world space, and wherein at least one audio setting of the audio profile is modified based on one or more patterns identified in the historical data.

10. The method of claim 1, further comprising generating a notification to the user regarding the recalibrated audio profile, wherein modifying the at least one audio setting of the audio output device within the identified real-world space is further based on approval by the user in response to the notification.

11. A system for dynamic audio optimization associated with digital content, the system comprising: a memory storing one or more audio profiles, wherein each of the audio profiles includes one or more audio settings for one or more audio output devices available to a user; a communication interface that communicates over a communication network, wherein the communication interface receives an incoming audio stream associated with a current interactive session of the user; one or more microphones that detect real-world audio within the identified real-world space in which the user is located; as well as a processor that executes instructions stored in the memory, wherein the processor executes the instructions to: detecting one or more audio deviations based on a comparison of the incoming audio stream to the real-world audio within the identified real-world space; automatically recalibrating an audio profile associated with the current interactive session, wherein at least one of the audio settings of the audio profile is modified; as well as The at least one audio setting of the audio output device within the identified real-world space is modified based on the recalibrated audio profile, wherein the audio output device processes audio of the incoming audio stream according to the at least one modified audio setting.

12. The system of claim 11, wherein the audio profile is specific to a digital content title associated with the incoming audio stream, and wherein the audio settings in the audio profile are based on a predetermined set of acoustic properties associated with the digital content title.

13. The system of claim 11, wherein the audio profile is specific to the identified real-world space and includes one or more acoustic properties of the identified real-world space, and wherein the audio settings in the audio profile are based on the acoustic properties of the identified real-world space.

14. The system of claim 13, wherein the processor executes further instructions to generate the audio profile, wherein generating the audio profile comprises: providing a calibration signal associated with one or more predetermined frequencies for playback by one or more of the audio output devices in the real-world space; receiving frequency response data captured by one or more microphones in the real-world space while the calibration signal is played by the one or more audio output devices; identifying one or more properties of the real-world space based on the frequency response data, wherein the identified properties include the acoustic properties; as well as Instructions are generated for making one or more adjustments to the audio settings of the audio output device based on the identified properties of the real-world space, wherein the instructions for audio setting adjustments are executable to offset one or more of the audio deviations associated with the identified properties of the real-world space, and wherein modifying the at least one audio setting is based on the instructions.

15. The system of claim 14, wherein the processor identifies the property of the real-world space by identifying a relative position of the audio output device within the real-world space.

16. The system of claim 14, wherein the calibration signal is a predetermined multi-layer signal comprising a plurality of different frequencies.

17. The system of claim 14, wherein the audio output device in the real-world space comprises one or more of a television, a game console device, a console controller, a speaker, a subwoofer, a computing device, a mobile device, a handheld electronic device, a smart device, an Internet of Things (IoT) device, and a head-mounted speaker device.

18. The system of claim 11, wherein the processor automatically recalibrates the audio profile by monitoring one or more conditions of the real-world space and modifies the at least one audio setting of the audio profile based on the monitored conditions.

19. The system of claim 11, wherein the processor further automatically recalibrates the audio profile based on historical data about the user or the real-world space, and modifies the at least one audio setting of the audio profile based on one or more patterns identified in the historical data.

20. The system of claim 11, wherein the processor executes further instructions to generate a notification to the user regarding the recalibrated audio profile, wherein modifying the at least one audio setting of the audio output device within the identified real-world space is further based on approval by the user in response to the notification.

21. A non-transitory computer readable storage medium having a program embodied thereon, the program being executable by a processor to perform a method for dynamic audio optimization associated with digital content, the method comprising: storing one or more audio profiles in a memory, wherein each of the audio profiles includes one or more audio settings for one or more audio output devices available to a user; monitoring an incoming audio stream associated with a current interactive session of the user and real-world audio within an identified real-world space in which the user is located; detecting one or more audio deviations based on a comparison of the incoming audio stream to the real-world audio within the identified real-world space; automatically recalibrating an audio profile associated with the current interactive session, wherein at least one of the audio settings of the audio profile is modified; as well as The at least one audio setting of the audio output device within the identified real-world space is modified based on the recalibrated audio profile, wherein the audio output device processes audio of the incoming audio stream according to the at least one modified audio setting.

Images