Gaze-based adjustment of virtual effect indicators

Abstract

Methods and systems for providing gaze-based generation of a virtual effect indicator correlated with a directional sound are disclosed.SOLUTION: Gaze data is tracked by a camera associated with the client device to identify a focal point within the three dimensional virtual environment at which one or both eyes of the player are focused. Once the focus point is indicated by the gaze data, if the focus point does not move toward the source location in the three dimensional virtual environment when receiving the directional sound, it indicates that the virtual effect indicator associated with the directional sound type of the indicated directional sound needs to be generated.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present techniques relate to detecting line of sight and generating virtual effect indicators correlated with directional sounds. More specifically, the present techniques may provide dynamic generation of virtual effect indicators during gameplay based on detected line of sight. [Background technology]

[0002] 2. Description of Related Art Currently available interactive media titles, including virtual reality media titles, still have widespread accessibility issues related to perceptual disabilities, such as blindness and low vision, hearing loss, hearing loss, deaf-blindness, and other sensory processing disorders. While advances in technology, driven by improving the gameplay experience for players with disabilities, for example, have resulted in more inclusive environments, there is still much room for improvement.

[0003] In many interactive media titles, directional sounds are used to communicate incoming characters or actions and are key to storyline development. For some players, missing such directional sounds due to perceptual impairments creates a less than ideal experience.

[0004] Therefore, there is a need to provide a service that uses gaze-based detection to provide virtual effect indicators based on specific cues that the player is not hearing various directional sounds. Summary of the Invention

[0005] Aspects of the present disclosure include systems and methods for detecting gaze and generating virtual effect indicators correlated with directional sounds. Information regarding multiple directional sound types associated with one or more interactive content titles may be stored in a memory. Each directional sound type may be associated with one or more virtual effect indicators in the respective interactive content title. Gameplay data transmitted over a communications network from client devices of players participating in a current activity of one of the interactive content titles within a current gameplay session may be monitored. The gameplay data may indicate directional sounds associated with source locations within a three-dimensional virtual environment of the interactive content title.

[0006] Gaze data may be tracked by a camera associated with the client device during a current gameplay session to identify a focal point within the three-dimensional virtual environment upon which one or both of the player's eyes are focused. If the focal point indicated by the gaze data does not move toward a source location within the three-dimensional virtual environment upon receiving gameplay data indicating a directional sound, a virtual effect indicator associated with a directional sound type of the indicated directional sound may be generated and presented within a display of the client device. The virtual effect indicator indicates the source location of the indicated directional sound.

[0007] The virtual effect indicator may include at least one of a visual Doppler effect centered on the source location of the directional sound on the display, a written cue indicating which direction to look toward the source location, a visual pointing to the source location of the directional sound or the direction of the directional sound, and a notification alerting the player to the source location of the directional sound. The gaze data may be mapped as the focal point moves to multiple positions within the three-dimensional virtual environment. The focal point indicated by the gaze data may be identified as not moving toward the source location within the three-dimensional virtual environment based on the mapped gaze data.

[0008] Identifying that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment may further include comparing one or more three-dimensional coordinates associated with the source location of the directional sound with one or more three-dimensional coordinates associated with the determined focal point. Alternatively or additionally, identifying that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment may include determining an orientation of the player within the three-dimensional virtual environment and determining a field of view of the player including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation. Other three-dimensional coordinates associated with the three-dimensional virtual environment may be excluded from the field of view of the player.

[0009] Various aspects of the present disclosure may include line-of-sight-based methods for generating virtual effect indicators correlated with directional sounds. Such methods may include storing, in a memory, information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within the respective interactive content title. Such methods may include monitoring gameplay data transmitted over a communications network from client devices of players participating in a current activity of one of the interactive content titles within a current gameplay session, the gameplay data indicating directional sounds associated with source locations within a three-dimensional virtual environment of the interactive content title.

[0010] Such methods may include tracking gaze data by a camera associated with the client device during a current gameplay session to identify a focal point within the three-dimensional virtual environment at which one or both of a player's eyes are focused. When gameplay data indicating a directional sound is received, such methods may include identifying that the focal point indicated by the gaze data does not move toward a source location within the three-dimensional virtual environment. Such methods may include generating and presenting within a display of the client device a virtual effect indicator associated with a directional sound type of the indicated directional sound, the virtual effect indicator indicating a source location of the indicated directional sound.

[0011] Additional aspects of the present disclosure may include a gaze-based generation system for virtual effect indicators correlated with directional sounds. Systems such as these may include a memory that stores information regarding multiple directional sound types associated with one or more interactive content titles, each directional sound type associated with one or more virtual effect indicators within the respective interactive content title. Systems such as these may include one or more processors that execute instructions stored in the memory. Execution of the instructions by the one or more processors may monitor gameplay data transmitted over a communications network from client devices of players participating in a current activity of one of the interactive content titles within a current gameplay session, the gameplay data indicating directional sounds associated with source locations within a three-dimensional virtual environment of the interactive content title. Execution of the instructions by the one or more processors may track gaze data by cameras associated with the client devices during the current gameplay session to identify a focal point within the three-dimensional virtual environment at which one or both of the players' eyes are focused.

[0012] Execution of the instructions by the one or more processors may identify that when gameplay data indicating the directional sound is received, the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment. One or more object-object associations may be dynamically generated based on the one or more object-object associations displayed with the media. Execution of the instructions by the one or more processors may generate and present within a display of the client device a virtual effect indicator associated with a directional sound type of the indicated directional sound, the virtual effect indicator indicating a source location of the indicated directional sound.

[0013] A further aspect of the present disclosure includes a non-transitory computer-readable medium or storage medium having embodied thereon a program executable by a processor to provide a method for dynamically generating and displaying play data on the medium. [Brief explanation of the drawings]

[0014] [Figure 1] 1 illustrates an exemplary network environment in which a system for detecting line of sight and generating a virtual effect indicator correlated with directional sound may be implemented. [Figure 2A] 1 illustrates an exemplary unified data system (UDS) that may be used to provide data to a system for detecting line of sight and generating a virtual effect indicator correlated with directional sound. [Figure 2B] 1 illustrates an exemplary table of various objects and associated events, according to an aspect of the present disclosure. [Figure 3] 10 illustrates an exemplary display with generated virtual effect indicators. [Figure 4] 1 illustrates a flowchart of an exemplary method for detecting line of sight and generating a virtual effect indicator correlated with directional sound, according to aspects of the present disclosure. [Figure 5] FIG. 1 is a block diagram of an exemplary electronic entertainment system that may be used with embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0015] Aspects of the present disclosure include systems and methods for detecting gaze and generating virtual effect indicators correlated with directional sounds. Information regarding multiple directional sound types associated with one or more interactive content titles may be stored in a memory. Each directional sound type may be associated with one or more virtual effect indicators in the respective interactive content title. Gameplay data transmitted over a communications network from client devices of players participating in a current activity of one of the interactive content titles within a current gameplay session may be monitored. The gameplay data may indicate directional sounds associated with source locations within a three-dimensional virtual environment of the interactive content title.

[0016] 1 illustrates a network environment in which a system for detecting gaze and generating a virtual effect indicator correlated with directional sound may be implemented. The network environment 100 may include one or more interactive content servers 110 providing streaming content (e.g., interactive videos, podcasts, etc.), one or more platform servers 120, one or more user devices 130, and one or more databases 140.

[0017] The interactive content server 110 can maintain, stream, and host interactive media available for streaming on user devices 130 over a communications network. Such an interactive content server 110 can be implemented in the cloud (e.g., one or more cloud servers). Each media can include one or more object data sets available for user participation (e.g., viewing or interacting with an activity). Data regarding the objects shown in the media can be stored in object files 216 ("object files") by the media streaming server 110, the platform server 120, and / or the user devices 130, as discussed in detail with respect to FIGS. 2A and 3.

[0018] The platform server 120 may be capable of communicating with different interactive content servers 110, databases 140, and user devices 130. Such platform servers 120 may be implemented on one or more cloud servers. The streaming server 110 may communicate with multiple platform servers 120, while the media streaming server 110 may be implemented on one or more platform servers 120. The platform server 120 may also execute instructions, such as receiving a user request from a user to stream streaming media (i.e., games, activities, videos, podcasts, user-generated content (“UGC”), publisher content, etc.). Furthermore, the platform server 120 may execute instructions, such as streaming a streaming media content title. Such streaming media may have at least one object set associated with at least a portion of the streaming media. Each set of object data may have data about an object displayed during at least a portion of the streaming media (e.g., activity information, zone information, actor information, mechanic information, game media information, etc.).

[0019] The streaming media and the associated set of at least one object data may be provided by an application programming interface (API) 160, which enables various types of media streaming servers 110 to communicate with different platform servers 120 and different user devices 130. The API 160 may be specific to the particular computer programming language, operating system, protocol, etc. of the media streaming server 110 providing the streaming media content title, the platform server 120 providing the media and the associated set of at least one object data, and the user device 130 receiving the object data. In a network environment 100 including multiple different types of media streaming servers 110 (or platform servers 120 or user devices 130), there may likewise be a corresponding number of APIs 160.

[0020] User devices 130 may include multiple different types of computing devices. For example, user devices 130 may include any number of different gaming consoles, mobile devices, laptops, and desktops. In another example, user devices 130 may be implemented in the cloud (e.g., one or more cloud servers). Such user devices 130 may also be configured to access data from other storage media, such as, but not limited to, memory cards or disk drives, which may be appropriate for downloaded services. Such devices 130 may include standard hardware computing components, such as, but not limited to, network interfaces, media interfaces, non-transitory computer-readable storage (memory), and processors for executing instructions that may be stored in the memory. These user devices 130 may also run a variety of different operating systems (e.g., iOS®, Android®), applications, or computing languages ​​(e.g., C++®, JavaScript®). An exemplary user device 130 is described in detail herein with respect to FIG. 5.

[0021] Database 140 can be stored on platform server 120, media streaming server 110, any of servers 218 (shown in FIG. 2A), on the same server, on different servers, on a single server, across different servers, or on user device 130. Such database 140 may store sets of streaming media and / or related object data. Such streaming media may depict one or more objects (e.g., activities) in which users can participate, and / or UGC (e.g., screenshots, videos, interpretations, mashups, etc.) created by peers, publishers of media content titles, and / or third-party publishers. Such UGC may include metadata for searching such UGC. Such UGC may also include information about the media and / or peers. Such peer information may be derived from data collected during peer interactions with objects in interactive content titles (e.g., video games, interactive books, etc.) and may be “bound” to and stored with the UGC. Such binding extends the UGC because it can deep link (e.g., directly initiate) to objects, provide information about the objects and / or peers of the UGC, and / or allow users to interact with the UGC. One or more user profiles can also be stored in database 140. Each user profile can include information about a user (e.g., activity and / or user progress within a media content title, user ID, user game character, etc.) and can be associated with media.

[0022] FIG. 2A illustrates an exemplary universal or unified data system (UDS) that can be used to provide data to a system for detecting gaze and generating virtual effect indicators correlated with directional sounds. Based on the data provided by the UDS, the platform server 120 can recognize in-game objects, entities, activities, and events in which the user has engaged, thus supporting analysis and adjustment of in-game activity. Each user interaction can be associated with metadata, such as the type of in-game interaction, its location within the in-game environment, its point in time within the in-game timeline, and other players, objects, and entities involved. Thus, metadata can be tracked for any of a variety of user interactions that may occur during a game session, including associated activities, entities, settings, outcomes, actions, effects, locations, character statistics, and the like. Such data can be further aggregated, applied to a data model, and subjected to analysis. Such a UDS data model can be used to assign contextual information to each piece of information in a uniform manner across the game.

[0023] 2A , an exemplary console 228 (e.g., user device 130) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user-generated content (UGC) server 232, and object server 226) are shown. In one example, the console 228 may be implemented on either the platform server 120, a cloud server, or a server 218. In an illustrative example, a content recorder 202 may be implemented on the platform server 120, a cloud server, or any of the servers 218. Such a content recorder 202 receives content (e.g., media) from an interactive content title 230 and records it in a content ring buffer 208. Such a ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), a start time for each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and an end time for each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as media files 212 (e.g., MP4, WebM, etc.). Such media files 212 may be uploaded to a streaming server 220 for storage and subsequent streaming or use, although the media files 212 may be stored on any server, cloud server, any console 228, or any user device 130. The start and end times for each such segment may be stored by the console 228 as a content timestamp file 214. Such a content timestamp file 214 may also include a stream ID that matches the stream ID of the media file 212, thereby associating the content timestamp file 214 with the media file 212.Such content timestamp files 214 may be uploaded and stored on the activity feed server 224 and / or UGC server 232, but the content timestamp files 214 may be stored on any server, cloud server, any console 228, or any user device 130.

[0024] At the same time that the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine the start and end times of the object. The object library 204 and the object recorder 206 may be implemented on either the platform server 120, the cloud server, or the server 218. When the object recorder 206 detects the start of an object, it receives object data from the object library 204 (e.g., if the object is an activity, the user interaction with the activity, the activity ID, the activity start time, the activity end time, the activity result, the activity type, etc.) and records this activity data (e.g., ActivityID1, START_TS; ActivityID2, START_TS; ActivityID3, START_TS) in the object ring buffer 210. Such activity data recorded in the object ring buffer 210 may be stored in the object file 216. Such object files 216 may also include activity start time, activity end time, activity ID, activity result, activity type (e.g., competition, quest, task, etc.), and user or peer data related to the activity. For example, object files 216 may store data regarding items used during the activity. Such object files 216 may be stored on an object server 226, although object files 216 may be stored on any server, cloud server, any console 228, or any user device 130.

[0025] Such object data (e.g., object files 216) can be associated with content data (e.g., media files 212 and / or content timestamp files 214). In one example, the UGC server 232 stores and associates the content timestamp files 214 with the object files 216 based on a match between the stream ID of the content timestamp files 214 and the corresponding activity ID of the object files 216. In another example, the object server 226 can store the object files 216 and receive queries for the object files 216 from the UGC server 232. Such queries can be performed by searching for the activity ID of the object files 216 that matches the stream ID of the content timestamp files 214 sent with the query. In yet another example, queries of stored content timestamp files 214 can be performed by matching the start and end times of the content timestamp files 214 with the start and end times of the corresponding object files 216 sent with the query. Such object files 216 may also be associated with matching content timestamp files 214 by the UGC server 232, although this association may be performed by any server, cloud server, any console 228, or any user device 130. In another example, the object files 216 and content timestamp files 214 may be associated by the console 228 during creation of the files 216, 214, respectively.

[0026] 2B, such object data (e.g., object files 216) may be associated with event information related to activity availability changes and may be associated with other objects having associated object information. Media-object bindings may form telemetry between objects displayed in at least a portion of the live streaming media and the live streaming media. For example, such object data may be zone data files 252, actor data files 254, mechanic data files 256, game media data files 258, and other gameplay-related data files.

[0027] Such object data (e.g., object files 216) may be categorized as in-progress, open-ended, or competitive. Such activity data files 216 may include optional properties such as, for example, a more detailed description of the activity, an image associated with the activity, if the activity is available to the player before starting the game, whether completion of the activity is required to complete the game, whether the activity can be repeated within the game, whether there are nested tasks or associated child activities, etc. Such activity data files 216 may include an activity availability change event that may indicate a list or array of activities currently available to the player. This may be used, for example, to determine which activities to display in a game plan.

[0028] Such zone data files 252 may represent relevant game world regions with a single coordinate system, and zones may have 2D maps associated with them that may be used to display locations within the zone. If applicable, each zone may include a zone ID and a short, localizable name for the zone. Such zone data files 252 may be associated with a view projection matrix (4x4) to convert from 3D world coordinates to 2D map locations. Such zone data files 252 may be associated with position change events that indicate updates to a player's current in-game location. Such position change events may be posted periodically or whenever a significant change occurs in the player's in-game location. The platform server 120 may store the latest values ​​in a "state." Such zone data files 252 may include the x, y, and z positions of the player's character within the zone, as well as a, b, and c vectors indicating the player's character's orientation or direction. Such zone data files 252 may be associated with activity start and / or activity end events, and in the case of activity end events, the outcome of completion, failure, or abandonment may be associated with the activity (e.g., activity ID).

[0029] Such actor data files 254 may be associated with entities with in-game behaviors, may be player controllers, or may be game controlled, and may change dynamically during gameplay. Such actor data files 254 may include the actor's actor ID, the actor's localizable name, the actor's image, and / or a brief description of the actor. Such actor data files 254 may be associated with an actor selection event that indicates a change in the player's selected actor(s). The selected actor(s) may represent the actors the player is controlling in the game and may be displayed in the player's profile and other spaces via the platform server 120. Multiple actors may be selected at a time, and each game may replace its list of actors upon save load.

[0030] Such mechanic data files 256 may be associated with items, skills, or effects (e.g., bows, arrows, stealth attacks, fire damage) that can be used by a player or game to affect gameplay, and may exclude items that do not affect gameplay (e.g., collectibles). Such mechanic data files 256 may include the mechanic's mechanic ID, the mechanic's short name, an image of the mechanic, and / or a brief description of the mechanic. Such mechanic data files 256 may be associated with a mechanic availability change event, which indicates that a mechanic available to a player has changed. Available may mean that the mechanic is available to the player in the game world, but the player may need to take some steps to acquire it in their inventory (e.g., purchase it from a shop, receive it from the world) before use. Each game may replace its list of mechanics when loading save data.

[0031] Such mechanic data files 256 may be associated with a mechanic inventory change event, indicating that a player's inventory has changed. Inventory may refer to mechanics that a player can use immediately, without having to take additional steps in the game before using the mechanic. Inventory information is used to estimate a player's readiness for various activities, which may be transferred to the platform server 120. A game may replace the list of mechanic inventory when loading save data. Cooldown mechanics may be considered part of the inventory. Mechanic counts with any non-zero value (e.g., ammo, recovery points, etc.) may be treated as "in inventory." Inventory mechanics may be considered a subset of available mechanics.

[0032] Such mechanic data files 256 may be associated with mechanic-use events indicating that a mechanic has been used by or on a player and may be used to display a mechanic use in a UGC context. Such mechanic data files 256 may include a list or sequence of mechanics used (e.g., flaming arrow, fire damage), or whether the mechanic was used by or on a player, including whether the initiator is a player. Such mechanic data files 256 may include an initiator actor ID, the initiator actor's current zone ID, and / or the initiator actor's current x, y, and z positions. Such mechanic data files 256 may be associated with mechanic-impact events indicating that a mechanic has affected gameplay (e.g., an arrow hits an enemy) and may be used to display a mechanic image in a UGC context. Mechanic use and mechanic image events may not be linked. Such a mechanics data file 256 may include an initiator action ID, the initiator actor's current zone ID, the initiator actor's current x, y, z position, the target actor ID, the target actor's current zone ID, the target actor's current x, y, z position, and a mitigation mechanic that may mitigate the initiator mechanic.

[0033] Such game media data files 258 may include the game media ID of the game media, the localizable name of the game media, the media format (e.g., image, audio, video, text, etc.), the category or type of media (cutscene, audio log, poster, developer commentary, etc.), the URL or server provisioning media file, and / or whether the game media is associated with a particular activity. Such game media data files 258 may be associated with a game media start event, which indicates that a particular piece of game media has just started in the game, and a game media end event, which indicates that a particular piece of game media has ended.

[0034] 3 shows an exemplary display with generated virtual effect indicators. A client device display 300 shows an exemplary third-person perspective of a player 302 within a game environment. From the player's perspective, there is a flying bat 304 on the left and a mountain 306 on the right. Hiding behind the mountain is a snake 308, which is currently difficult to see. However, during gameplay, the sound of the snake 308 is audible, with directional sound emanating from the right side. In some instances, the sound is played louder from the right speaker or right side of a headset or other headphones.

[0035] If the player is deaf, particularly if they are deaf in their right ear, they may not hear the sound of the snake 308. In such a case, the player may still be looking at the bats 304, for example, as they are moving around. A camera (not shown) may be used to detect where the player is looking on the display 300. If the player appears unable to look to the right while the sound of the snake 308 is playing, this may be an indication that the player is deaf in their right ear. In such a case, a virtual effect indicator 310 is generated to indicate where the sound is coming from.

[0036] As discussed in more detail below, the virtual effect indicator 310 may be generated only after a certain threshold of mismatch between the focal point and the source location, as determined by gaze data. For example, after five consecutive failed gaze attempts to target the source location, the virtual effect indicator 310 may be generated. Furthermore, a player may consistently look some distance away from the source location. In such cases, the player may have a certain percentage of hearing loss in one or both ears, resulting in an auditory offset. Therefore, a compensation angle may be determined based on the gaze pattern and how far away the player is looking from the source location. The compensation angle may then be applied to dynamically rebalance the sound between the two speakers or between the two sides of the headset.

[0037] FIG. 4 is a flowchart illustrating an example method 400 for detecting line of sight and generating a virtual effect indicator correlated with directional sound. The method 400 of FIG. 4 may be embodied as executable instructions in a non-transitory computer-readable storage medium, including, but not limited to, non-volatile memory such as a CD, DVD, or hard drive. The instructions in the storage medium may be executed by a processor(s) to cause various hardware components of a computing device that hosts or otherwise accesses the storage medium to perform the method. Execution of the instructions may be performed on a cloud server (e.g., the steps identified in FIG. 4 are performed in the cloud). The steps (and their order) identified in FIG. 4 are examples and may include various alternative, equivalent, or derivative forms thereof, including, but not limited to, their order of execution.

[0038] At step 410, information regarding a plurality of directional sound types associated with one or more interactive content titles is stored in memory. Each directional sound type is associated with one or more virtual effect indicators in the respective interactive content title. The directional sound types may differ in direction, sound type, and other sonic and location-based parameters. The directional sound types may later be recorded by the object recorder 206 in the activity file 216. Statistical analysis may be performed on directional sound types that players miss, and these may be actively included in the gameplay of players with known hearing impairments.

[0039] Step 420 monitors gameplay data transmitted over a communications network from client devices of players participating in a current activity of one of the interactive content titles within a current gameplay session. During gameplay, the collected gameplay data may indicate directional sounds associated with a source location within the three-dimensional virtual environment of the interactive content title. For example, the source location may be within a display hidden behind another object, or too far away to be seen but heard. Using directional audio techniques, the directional sounds may be experienced as emanating from a side that correlates to where the source location is on the display.

[0040] In step 430, gaze data is tracked by a camera associated with the client device during the current gameplay session to identify a focal point within the three-dimensional virtual environment at which one or both of the player's eyes are focused. The camera may be a virtual reality headset, a standalone camera placed in front of the player, or a location above the console or display. The focal point may be determined through various training and calibration modules that adjust the determined focal point until the player indicates that the determined focal point is accurate. Alternatively, calibration can be performed by internal measurements determined by a processor based on known factors.

[0041] In step 440, if gameplay data indicating directional sound is received, it is identified that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment. The identifying may include comparing one or more three-dimensional coordinates associated with the source location of the directional sound with one or more three-dimensional coordinates associated with the determined focal point. The identifying may include determining a player's orientation within the three-dimensional virtual environment and, based on the determined orientation, determining a field of view for the player that includes one or more three-dimensional coordinates of the three-dimensional virtual environment, with other three-dimensional coordinates associated with the three-dimensional virtual environment being excluded from the player's field of view. The orientation may be determined based on a virtual reality headset that detects the orientation of the player's head or body. Therefore, in virtual reality, the sound may originate from a position outside the field of view, requiring the player to reorient their body. In such cases, the virtual effect indicator may be an arrow or shading indicating that the player should reorient their body in a specific direction.

[0042] Further, the gaze data may be mapped as the focal point moves to multiple locations within the three-dimensional virtual environment. The focal point indicated by the gaze data may be identified as not moving toward a source location within the three-dimensional virtual environment based on the mapped gaze data. Further, a portion of the gaze data that shares the same set of timestamps as the identified trigger point may be identified as not matching a location on the display emitting the directional sound. A threshold may be set depending on the degree of mismatch between the focal point and the source location of the directional sound source. A virtual effect indicator may further be generated based on the threshold being met.

[0043] In step 450, a virtual effect indicator associated with the directional sound type of the indicated directional sound is generated and presented within the client device's display, the virtual effect indicator indicating the source location of the indicated directional sound. The virtual effect indicator may be displayed near the location where the source location is located or may describe the location where the source location is located. Different virtual effect indicators may include a visual Doppler effect centered on the display around the source location of the directional sound, a written cue indicating which direction to look toward the source location, a visual pointing to the source location or direction of the directional sound, or a notification alerting the player to the source location of the directional sound. The virtual effect indicator may be generated as an overlay or object within the game environment. The virtual effect indicator may (or may not) be recorded by the object recorder 206 and may (or may not) be stored as an activity in the activity file. The virtual effect indicator may be shown or removed from the media file 212 during recording by the content recorder.

[0044] Figure 5 is a block diagram of an exemplary electronic entertainment system 500. The entertainment system 500 of Figure 5 includes a main memory 505, a central processing unit (CPU) 510, a vector unit 515, a graphics processing unit 520, an input / output (I / O) processor 525, an I / O processor memory 530, a controller interface 535, a memory card 540, a universal serial bus (USB) interface 545, and an IEEE interface 550. The entertainment system 500 may further include an operating system read-only memory (OS ROM) 555, an audio processing unit 560, an optical disc control unit 570, and a hard disk drive 565, which are connected to the I / O processor 525 via a bus 575.

[0045] Entertainment system 500 may be an electronic game console. Alternatively, entertainment system 500 may be implemented as a general-purpose computer, a set-top box, a handheld gaming device, a tablet computing device, a mobile computing device, or a mobile phone. Entertainment systems may include more or fewer operating components depending on the particular form factor, purpose, or design.

[0046] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 of FIG. 5 communicate via a system bus 555. Additionally, the CPU 510 of FIG. 5 may communicate with main memory 505 via a dedicated bus 550, and the vector unit 515 and graphics processing unit 520 may communicate via a dedicated bus 590. The CPU 510 of FIG. 5 executes programs stored in the OS ROM 555 and the main memory 505. The main memory 505 of FIG. 5 may contain pre-stored programs and programs transferred via the I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disk (not shown) using the optical disk control unit 570. The I / O processor 525 of FIG. 5 may also enable the introduction of content transferred over wireless or other communication networks (e.g., 4G, LTE, 3G, etc.). The I / O processor 525 of FIG. 5 primarily controls the exchange of data between various devices of the entertainment system 500, including the CPU 510, the vector unit 515, the graphics processing unit 520, and the controller interface 535.

[0047] The graphics processing unit 520 of Figure 5 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate images for display on a display device (not shown). For example, the vector unit 515 of Figure 5 may convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Additionally, the audio processing unit 560 executes instructions to generate audio signals, which are output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550, such as a wireless transceiver; these interfaces may also be embedded in the system 500 or as part of some other component, such as a processor.

[0048] 5 provides instructions to CPU 510 via controller interface 535. For example, the user may instruct CPU 510 to store certain game information on memory card 540 or other non-transitory computer-readable storage medium, or may instruct a character in the game to perform some particular action.

[0049] The present invention may be implemented in an application that may be operable by a variety of end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., Sony PlayStation2® or Sony PlayStation3® or Sony PlayStation4®), a portable gaming device (e.g., Sony PSP® or Sony Vita®), or a home entertainment system from a different, but lesser manufacturer. It is fully contemplated that the methods described herein will operate on a variety of devices. The present invention may also be implemented in a cross-title neutral manner, such that embodiments of the inventive system may be utilized across a variety of titles from a variety of publishers.

[0050] The present invention may be implemented in applications that may be operable using a variety of devices. A non-transitory computer-readable storage medium refers to any medium or media that participates in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, including, but not limited to, non-volatile media such as optical disks, magnetic disks, and volatile media such as dynamic memory. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tape, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, and any other memory chip or cartridge.

[0051] Various forms of transmission media may be involved in carrying one or more sequences of one or more instructions to the CPU for execution. A bus carries the data to system RAM, from which the CPU retrieves and executes the instructions. The instructions received by the system RAM may optionally be stored on a fixed disk either before or after execution by the CPU. Various forms of storage may also be implemented, as well as network interfaces and network topologies necessary to implement them.

[0052] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were selected to best explain the principles of the technology, its practical application, and to enable those skilled in the art to utilize the technology in various embodiments and with various modifications suitable for the particular uses contemplated. It is intended that the scope of the technology be defined by the claims.

Claims

1. 1. A line-of-sight based method for generating a virtual effect indicator correlated with directional sound, comprising: monitoring a current gameplay session transmitted over a communications network, wherein a player's client device is playing an interactive content title associated with a three-dimensional virtual environment during the current gameplay session; determining that the player is associated with an orientation within the three-dimensional virtual environment; determining a player's field of view including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein one or more other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the player's field of view; identifying that gameplay data associated with the current gameplay session indicates directional sound associated with a source location within the three-dimensional virtual environment of the interactive content title; identifying that the source location corresponds to one or more of the other three-dimensional coordinates excluded from the player's field of view; generating and presenting within a display of the client device a virtual effect indicator, the virtual effect indicator visually indicating the source location of the indicated directional sound; The method comprising:

2. 2. The method of claim 1, wherein the virtual effect indicator includes at least one of a visual Doppler effect centered on the source location of the directional sound on the display, a written cue indicating which direction to look towards the source location, a visual pointing to the source location of the directional sound or the direction of the directional sound, and a notification alerting the player to the source location of the directional sound.

3. tracking player gaze data with a camera during the current gameplay session; identifying a location of focus indicated by tracked gaze data of the player within the three-dimensional virtual environment; mapping gaze data of the player as the focal point moves to different positions within the three-dimensional virtual environment; The method of claim 1 further comprising:

4. and further comprising identifying, based on the mapped gaze data, that the focal point indicated by the gaze data does not move towards the source position within the three-dimensional virtual environment. The method of claim 3.

5. 5. The method of claim 4, wherein identifying that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment further comprises comparing one or more three-dimensional coordinates associated with the source location of the directional sound with one or more three-dimensional coordinates associated with the determined focal point.

6. 5. The method of claim 4, wherein identifying that the focus indicated by the gaze data does not move toward the source location includes identifying that a portion of the gaze data that shares the same set of timestamps as an identified trigger point does not match a location on a display that is emitting the directional sound.

7. tracking a plurality of mismatches associated with the player, each mismatch being between a focal point and a source location of a directional sound; determining a correction angle based on the determined offset between the focal point and the source position, the determined offset indicating unilateral hearing loss, whereby the correction angle corrects for unilateral hearing loss; The method of claim 1 further comprising:

8. 1. A line-of-sight based generation system for a virtual effect indicator correlated with directional sound, comprising: a communications interface for receiving monitored data relating to a current gameplay session transmitted over a communications network, the communications interface being configured to receive monitored data relating to a current gameplay session, the player's client device playing an interactive content title associated with the three-dimensional virtual environment during the current gameplay session; a processor for executing instructions stored in a memory; Including, The processor: determining that the player is associated with an orientation within the three-dimensional virtual environment; determining a player's field of view including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein one or more other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the player's field of view; identifying that gameplay data associated with the current gameplay session indicates directional sound associated with a source location within the three-dimensional virtual environment of the interactive content title; identifying that the source location corresponds to one or more of the other three-dimensional coordinates excluded from the player's field of view; generating and presenting within a display of the client device a virtual effect indicator, the virtual effect indicator visually indicating the source location of the indicated directional sound; The system executes the instructions for:

9. 9. The system of claim 8, wherein the virtual effect indicator includes at least one of a visual Doppler effect centered on the source location of the directional sound on the display, a written cue indicating which direction to look towards the source location, a visual pointing to the source location of the directional sound or the direction of the directional sound, or a notification alerting the player to the source location of the directional sound.

10. The processor: tracking player gaze data with a camera during the current gameplay session; identifying a location of focus indicated by tracked gaze data of the player within the three-dimensional virtual environment; mapping gaze data of the player as the focal point moves to different positions within the three-dimensional virtual environment; The system of claim 8 , further comprising instructions for:

11. The processor: Identifying, based on the mapped gaze data, that the focal point indicated by the gaze data does not move towards the source position within the three-dimensional virtual environment. The system of claim 10 , further comprising instructions for:

12. 12. The system of claim 11, wherein identifying that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment further comprises comparing one or more three-dimensional coordinates associated with the source location of the directional sound with one or more three-dimensional coordinates associated with the determined focal point.

13. The processor: The system of claim 11 , further comprising: identifying that a portion of the gaze data that shares the same set of timestamps as an identified trigger point does not match a location on the display that is emitting the directional sound.

14. The processor: tracking a plurality of mismatches associated with the player, each mismatch being between a focal point and a source location of a directional sound; determining a correction angle based on the determined offset between the focal point and the source position, the determined offset indicating unilateral hearing loss, whereby the correction angle corrects for unilateral hearing loss; The system of claim 8 , further comprising instructions for:

15. A non-transitory computer-readable storage medium having instructions embodied thereon, the instructions being executable by a computing system to perform a method for line-of-sight based generation of a virtual effect indicator correlated with directional sound; The method comprises: monitoring a current gameplay session transmitted over a communications network, wherein a player's client device is playing an interactive content title associated with a three-dimensional virtual environment during the current gameplay session; determining that the player is associated with an orientation within the three-dimensional virtual environment; determining a player's field of view including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein one or more other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the player's field of view; identifying that gameplay data associated with the current gameplay session indicates directional sound associated with a source location within the three-dimensional virtual environment of the interactive content title; identifying that the source location corresponds to one or more of the other three-dimensional coordinates excluded from the player's field of view; generating and presenting within a display of the client device a virtual effect indicator, the virtual effect indicator visually indicating the source location of the indicated directional sound; The non-transitory computer-readable storage medium.

16. 16. The non-transitory computer-readable storage medium of claim 15, wherein the virtual effect indicator comprises at least one of a visual Doppler effect centered on the source location of the directional sound on the display, a written cue indicating which direction to look towards the source location, a visual pointing to the source location of the directional sound or the direction of the directional sound, or a notification alerting the player to the source location of the directional sound.

17. tracking player gaze data with a camera during the current gameplay session; identifying a location of focus indicated by tracked gaze data of the player within the three-dimensional virtual environment; mapping gaze data of the player as the focal point moves to different positions within the three-dimensional virtual environment; 20. The non-transitory computer-readable storage medium of claim 15, further comprising instructions for:

18. Identifying, based on the mapped gaze data, that the focal point indicated by the gaze data does not move towards the source position within the three-dimensional virtual environment.

20. The non-transitory computer-readable storage medium of claim 17, further comprising instructions for:

19. 20. The non-transitory computer-readable storage medium of claim 18, wherein identifying that the focal point indicated by the gaze data does not move toward the source location within the three-dimensional virtual environment further comprises comparing one or more three-dimensional coordinates associated with the source location of the directional sound with one or more three-dimensional coordinates associated with the determined focal point.

20. 20. The non-transitory computer-readable medium of claim 18, wherein identifying that the focal point indicated by the gaze data does not move toward the source location includes identifying that a portion of the gaze data that shares the same set of timestamps as an identified trigger point does not match a position on a display that is emitting the directional sound.

21. tracking a plurality of mismatches associated with the player, each mismatch being between a focal point and a source location of a directional sound; determining a correction angle based on the determined offset between the focal point and the source position, the determined offset indicating unilateral hearing loss, whereby the correction angle corrects for unilateral hearing loss; 20. The non-transitory computer-readable storage medium of claim 15, further comprising instructions for:

Images