Adjustment Based on the Line of Sight of the Virtual Effect Indicator

Abstract

A method and system are disclosed for providing generation based on the line of sight of a virtual effect indicator correlated with a directional sound. 【Solution means】Track line-of-sight data by a camera associated with a client device and identify a focus point on which one or both eyes of a player are focused within a three-dimensional virtual environment. When the focus point is indicated by the line-of-sight data, if the focus point does not move towards the source position within the three-dimensional virtual environment when receiving the directional sound, it indicates that a virtual effect indicator associated with the directional sound type of the indicated directional sound needs to be generated.

Description

Technical Field

[0001] This technique relates to detecting a line of sight and generating a virtual effect indicator correlated with a directional sound. More specifically, this technique may provide for the dynamic generation of virtual effect indicators during gameplay based on the detected line of sight.

Background Art

[0002] Description of Related Art Currently available interactive media titles, including virtual reality media titles, still widely suffer from accessibility problems related to perceptual disabilities such as blindness and low vision, hearing impairment, deafness, deaf-blindness, and other sensory processing disorders. For example, as a result of the advancement of techniques driven by improving the gameplay experience of disabled players, a more inclusive environment has been brought about, but there is still much room for improvement.

[0003] In many interactive media titles, directional sound is used to convey incoming characters or actions and is key to the story development. For some players, due to perceptual disabilities, missing such directional sound makes the experience less than ideal.

[0004] Therefore, it is necessary to provide a service that gives virtual effect indicators based on specific cues that the player is not hearing various directional sounds using line-of-sight-based detection.

Summary of the Invention

[0005] Aspects of the present disclosure include systems and methods for detecting a line of sight and generating a virtual effect indicator correlated with a directional sound. Information regarding a plurality of directional sound types associated with one or more interactive content titles may be stored in a memory. Each directional sound type may be associated with one or more virtual effect indicators in a respective interactive content title. Gameplay data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current gameplay session may be monitored. The gameplay data may indicate a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title.

[0006] During a current gameplay session, line-of-sight data may be tracked by a camera associated with the client device to identify a focus at which one or both eyes of the player are focused within the three-dimensional virtual environment. When the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment when receiving gameplay data indicating a directional sound, a virtual effect indicator associated with the directional sound type of the indicated directional sound may be generated and presented within a display of the client device. The virtual effect indicator indicates the source position of the indicated directional sound.

[0007] The virtual effect indicator may include at least one of a visual Doppler effect centered on the source position of the directional sound on the display, a write queue indicating which direction to look towards the source position, a visual indicating the source position or direction of the directional sound, and a notification warning the player of the source position of the directional sound. The line-of-sight data may be mapped as the focus moves to multiple positions within the three-dimensional virtual environment. The focus indicated by the line-of-sight data may be identified as not moving towards the source position within the three-dimensional virtual environment based on the mapped line-of-sight data.

[0008] Identifying that the focus indicated by the gaze data does not move towards the source position within the three-dimensional virtual environment may further include comparing one or more three-dimensional coordinates associated with the source position of the directional sound with one or more three-dimensional coordinates associated with the determined focus. Alternatively or additionally, identifying that the focus indicated by the gaze data does not move towards the source position within the three-dimensional virtual environment may include determining the orientation of the player within the three-dimensional virtual environment and determining the field of view of the player, including one or more three-dimensional coordinates of the three-dimensional virtual environment, based on the determined orientation. Other three-dimensional coordinates associated with the three-dimensional virtual environment may be excluded from the field of view of the player.

[0009] Various aspects of the present disclosure may include methods for generating gaze-based virtual effect indicators correlated with directional sound. Such methods may include storing in memory information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within each respective interactive content title. Such methods may include monitoring game play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current game play session, the game play data indicating a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title.

[0010] Such methods may include tracking gaze data by a camera associated with a client device during a current gaming session to identify, within a three-dimensional virtual environment, a focus on which one or both eyes of a player are focused. Such methods may include identifying that, when gaming play data indicating a directional sound is received, the focus indicated by the gaze data does not move toward the source position within the three-dimensional virtual environment. Such methods may include generating a virtual effect indicator associated with the directional sound type of the indicated directional sound and presenting the virtual effect indicator within a display of the client device, the virtual effect indicator indicating the source position of the indicated directional sound.

[0011] Additional aspects of the present disclosure may include a gaze-based generation system for virtual effect indicators correlated with directional sounds. Such systems may include a memory storing information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within a respective interactive content title. Such systems may include one or more processors executing instructions stored in the memory. Execution of the instructions by the one or more processors may monitor gaming play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current gaming session, the gaming play data indicating a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title. Execution of the instructions by the one or more processors may track gaze data by a camera associated with the client device during the current gaming session to identify, within the three-dimensional virtual environment, a focus on which one or both eyes of the player are focused.

[0012] Execution of instructions by one or more processors can identify that when game play data indicating directional sound is received, the focus indicated by the eye gaze data does not move towards the source position within the three-dimensional virtual environment. One or more object-object associations can dynamically generate play data based on one or more object-object associations displayed with the media. Execution of instructions by one or more processors can generate a virtual effect indicator associated with the indicated directional sound type of the directional sound and present it within the display of the client device, and the virtual effect indicator indicates the source position of the indicated directional sound.

[0013] Further aspects of the present disclosure include a non-transitory computer-readable medium or storage medium that embodies a program executable by a processor therein to provide a method for dynamically generating and displaying play data of the medium.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2A

Figure 2B

Figure 3

Figure 4

Figure 5

Modes for Carrying Out the Invention

[0015] Aspects of the present disclosure include systems and methods for detecting a line of sight and generating a virtual effect indicator correlated with a directional sound. Information regarding a plurality of directional sound types associated with one or more interactive content titles may be stored in a memory. Each directional sound type may be associated with one or more virtual effect indicators in a respective interactive content title. Gameplay data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current gameplay session may be monitored. The gameplay data may indicate a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title.

[0016] FIG. 1 shows a network environment in which a system for detecting a line of sight and generating a virtual effect indicator correlated with a directional sound may be implemented. The network environment 100 can include one or more interactive content servers 110 that provide streaming content (e.g., interactive video, podcast, etc.), one or more platform servers 120, one or more user devices 130, and one or more databases 140.

[0017] The Interactive Content Server 110 can maintain, stream, and host interactive media that is available for streaming on a user device 130 via a communication network. Such an Interactive Content Server 110 can be implemented in a cloud (e.g., one or more cloud servers). Each media can include one or more object data sets that are available for user participation (e.g., display of an activity or interaction with an activity). Data regarding objects shown in the media can be stored in an object file 216 ("object file") by the Media Streaming Server 110, the Platform Server 120, and / or the user device 130, as discussed in detail with respect to FIGS. 2A and 3.

[0018] The Platform Server 120 can be capable of communicating with different Interactive Content Servers 110, the Database 140, and the user device 130. Such a Platform Server 120 can be implemented in one or more cloud servers. The Streaming Server 110 can communicate with multiple Platform Servers 120, although the Media Streaming Server 110 can be implemented in one or more Platform Servers 120. The Platform Server 120 can also execute instructions such as receiving requests from users to stream streaming media (i.e., games, activities, videos, podcasts, user-generated content ("UGC"), publisher content, etc.). Further, the Platform Server 120 can execute instructions such as streaming a streaming media content title. Such streaming media can have at least one object set associated with at least a portion of the streaming media. Each set of object data can have data regarding objects displayed among at least a portion of the streaming media (e.g., activity information, zone information, actor information, mechanic information, game media information, etc.).

[0019] A set of streaming media and at least one associated object data can be provided by an application programming interface (API) 160, whereby various types of media streaming servers 110 can communicate with different platform servers 120 and different user devices 130. The API 160 can be specific to the particular computer programming language, operating system, protocol, etc. of the media streaming server 110 that provides the streaming media content title, the platform server 120 that provides the media and at least one associated set of object data, and the user device 130 that receives the object data. In a network environment 100 that includes multiple different types of media streaming servers 110 (or platform servers 120 or user devices 130), there can similarly be a corresponding number of API 160s.

[0020] The user device 130 may include a plurality of different types of computing devices. For example, the user device 130 may include any number of different gaming consoles, mobile devices, laptops, and desktops. In another example, the user device 130 may be implemented in the cloud (e.g., one or more cloud servers). Such a user device 130 may also be, but is not limited to, a memory card or a disk drive that may be appropriate for downloaded services, and may be configured to access data from other storage media. Such a device 130 may include standard hardware computing components such as, but not limited to, a network interface, a media interface, a non-transitory computer-readable storage device (memory), and a processor that can execute instructions that may be stored in the memory. These user devices 130 may also be executed using various different operating systems (e.g., iOS®, Android®), applications, or computing languages (e.g., C++®, Java®Script®). An exemplary user device 130 is described in detail herein with respect to FIG. 5.

[0021] The database 140 can be stored in the platform server 120, in the media streaming server 110, in any of the servers 218 (shown in FIG. 2A), in the same server, in different servers, in a single server, across different servers, or in any of the user devices 130. Such a database 140 can store a set of streaming media and / or related object data. Such streaming media can depict one or more objects (e.g., activities) that a user can participate in, and / or UGC (e.g., screenshots, videos, interpretations, mashups, etc.) created by publishers of media content titles and / or third-party publishers of peers. Such UGC can include metadata for searching such UGC. Such UGC can also include information about the media and / or peers. Such peer information can be derived from data collected during peer interactions with objects of interactive content titles (e.g., video games, interactive books, etc.), can be "bound" to the UGC, and can be stored with the UGC. Such binding can extend the UGC because it enables the UGC to deep-link (e.g., directly initiate) to an object, provide information about the object and / or peers of the UGC, and / or enable a user to interact with the UGC. One or more user profiles can also be stored in the database 140. Each user profile can include information about the user (e.g., activities and / or user progress within media content titles, user ID, user's game character, etc.) and can be associated with the media.

[0022] FIG. 2A shows an exemplary Universal or Unified Data System (UDS) that can be used to provide data to a system for detecting a line of sight and generating a virtual effect indicator correlated with a directional sound. Based on the data provided by the UDS, the platform server 120 can recognize in-game objects, entities, activities, and events that a user has engaged in, and thus support the analysis and adjustment of in-game activities. Each user interaction can be associated with metadata about the type of in-game interaction, the location within the in-game environment, the point in time within the in-game timeline, and other players, objects, entities, etc. involved. Thus, the metadata can track any of the various user interactions that may occur during a game session, including related activities, entities, settings, results, actions, effects, locations, character statistics, etc. Such data can be further aggregated, applied to a data model, and be subject to analysis. Such a UDS data model can be used to assign context information to each part of the information in a unified manner across the game.

[0023] As shown in FIG. 2A, an exemplary console 228 (e.g., user device 130) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user generated content (UGC) server 232, and object server 226) are shown. In one example, the console 228 may be implemented on either the platform server 120, a cloud server, or one of the servers 218. In an exemplary example, the content recorder 202 may be implemented on the platform server 120, a cloud server, or any of the servers 218. Such a content recorder 202 receives content (e.g., media) from the interactive content title 230 and records it in the content ring buffer 208. Such a ring buffer 208 may store a plurality of content segments (e.g., v1, v2, and v3), a start time for each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and an end time for each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as a media file 212 (e.g., MP4, WebM, etc.). Such a media file 212 can be uploaded to the streaming server 220 for storage and subsequent streaming or use, but the media file 212 can be stored on any server, cloud server, any console 228, or any user device 130. The console 228 may store the start time and end time for each such segment as a content timestamp file 214. Also, such a content timestamp file 214 may include a streaming ID that matches the streaming ID of the media file 212, thereby associating the content timestamp file 214 with the media file 212.Such content time stamp file 214 can be uploaded and stored in activity feed server 224 and / or UGC server 232, but content time stamp file 214 can be stored in any server, cloud server, any console 228, or any user device 130.

[0024] While the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine the start time and end time of the object. The object library 204 and the object recorder 206 can be implemented on any of the platform server 120, the cloud server, or the server 218. When the object recorder 206 detects the start of an object, the object recorder 206 receives object data (for example, if the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity result, activity type, etc.) from the object library 204, and records this activity data (for example, ActivityID1, START_TS;ActivityID2, START_TS;ActivityID3, START_TS) in the object ring buffer 210. Such activity data recorded in the object ring buffer 210 can be stored in the object file 216. Such an object file 216 can also include the activity start time, activity end time, activity ID, activity result, activity type (for example, confrontation match, quest, task, etc.), user or peer data related to the activity. For example, the object file 216 can store data related to items used during the activity. Such an object file 216 can be stored in the object server 226, but the object file 216 can be stored on any server, cloud server, any console 228, or any user device 130.

[0025] Such object data (e.g., object file 216) can be associated with content data (e.g., media file 212 and / or content timestamp file 214). In one example, UGC server 232 stores and associates content timestamp file 214 with object file 216 based on a matching between the streaming ID of content timestamp file 214 and the corresponding activity ID of object file 216. In another example, object server 226 can store object file 216 and can receive a query for object file 216 from UGC server 232. Such a query can be executed by searching for the activity ID of object file 216 that matches the streaming ID of content timestamp file 214 sent with the query. In yet another embodiment, a query of the stored content timestamp file 214 can be executed by matching the start time and end time of content timestamp file 214 with the corresponding start time and end time of object file 216 sent with the query. Also, such object file 216 can be associated by UGC server 232 with the matching content timestamp file 214, although this association can be performed by any server, cloud server, any console 228, or any user device 130. In another example, object file 216 and content timestamp file 214 can be associated by console 228 during the creation of each of files 216, 214.

[0026] As shown in exemplary Table 250 of FIG. 2B, such object data (e.g., object file 216) can be associated with event information regarding activity availability changes and can be associated with other objects having associated object information. A media-object bind can form telemetry between an object displayed in at least a portion of the live streaming media and the live streaming media. For example, such object data may be a zone data file 252, an actor data file 254, a mechanic data file 256, a game media data file 258, and other game play-related data files.

[0027] Such object data (e.g., object file 216) can be classified as in-progress, open-ended, or as a battle. Such activity data file 216 may include, for example, a more detailed description of the activity, an image related to the activity, whether completion of the activity is required to complete the game if the activity is available to the player before starting the game, whether the activity can be repeatedly played within the game, whether there are nested tasks or related child activities, and other optional properties. Such activity data file 216 may include an activity availability change event that can indicate a list or array of activities currently available to the player. For example, this can be used to determine the activities to display in the game plan.

[0028] Such a zone data file 252 may represent an area of the associated game world having a single coordinate system. The zone may have an associated 2D map and may be used to display positions on the zone. When the zone data file 252 is applicable, each zone may include a zone ID and a short localizable name for the zone. Such a zone data file 252 may be associated with a view projection matrix (4x4) for converting from 3D world coordinates to 2D map positions. Such a zone data file 252 may be associated with a position change event indicating an update to the player's current in-game position. Such position change events may be posted periodically or whenever the player's in-game position changes significantly. The platform server 120 may store the latest values in a "state". Such a zone data file 252 may include the x, y, z positions of the player's character in the zone and the a, b, c vectors indicating the orientation or direction of the player's character. Such a zone data file 252 may also be associated with an activity start event and / or an activity end event. In the case of an activity end event, a result of completion, failure, or abandonment may be associated with the activity (e.g., activity ID).

[0029] Such actor data file 254 may be associated with an entity along with its in-game behavior, may be a player controller, or may be game-controlled, and may change dynamically during gameplay. Such actor data file 254 may include an actor's actor ID, the actor's localizable name, the actor's image, and / or a brief description of the actor. Such actor data file 254 may be associated with an actor selection event indicating that the player's selected actor(s) has been changed. The selected actor(s) may represent the actor(s) the player is controlling in the game and may be displayed in the player's profile and other spaces via the platform server 120. Multiple actors may be selected at once, and each game may replace the list of actors when loading saved data.

[0030] Such mechanic data file 256 may be associated with items, skills, or effects (e.g., bows, arrows, stealth attacks, fire damage) that can be used by the player or the game to affect gameplay, and may exclude items that do not affect gameplay (e.g., collectibles). Such mechanic data file 256 may include a mechanic's mechanic ID, the mechanic's short name, the mechanic's image, and / or a brief description of the mechanic. Such mechanic data file 256 may be associated with a mechanic availability change event indicating that the mechanics available to the player have changed. Available may mean that the mechanic is available to the player in the game world, but the player may need to perform some steps to acquire it in their inventory before using it (e.g., purchase from a shop, receive from the world). Each game may replace the list of those mechanics when loading saved data.

[0031] Such a mechanical data file 256 can be associated with a mechanical inventory change event indicating that the player's inventory has been changed. The inventory can refer to a mechanic that the player can use immediately without the need for additional steps in the game before using the mechanic. Inventory information is used to estimate the player's preparation for various activities that can be transferred to the platform server 120. When the game loads the saved data, it can replace the list of the mechanical inventory. The cooldown mechanic can be considered part of the inventory. A mechanic count with any non-zero value (e.g., ammunition, recovery points, etc.) can be treated as "within the inventory". The inventory mechanic can be considered a subset of the available mechanics.

[0032] Such a mechanic data file 256 may be associated with a mechanic usage event indicating that the mechanic has been used by or against a player and can be used to be displayed as mechanic usage in the UGC context. Such a mechanic data file 256 may include a list or array of the mechanics used (e.g., fire arrow, fire damage), or whether the initiator is a player, such as whether the mechanic has been used by or against a player. Such a mechanic data file 256 may include the initiator actor ID, the current zone ID of the initiator actor, and / or the current x, y, z positions of the initiator actor. Such a mechanic data file 256 may be associated with a mechanic impact event indicating that the mechanic has affected the gameplay (e.g., an arrow hits an enemy) and can be used to display a mechanic image in the UGC context. The events of mechanic usage and mechanic image may not be linked. Such a mechanic data file 256 may include the initiator action ID, the current zone ID of the initiator actor, the current x, y, z positions of the initiator actor, the target actor ID, the current zone ID of the target actor, the current x, y, z of the target actor, and a mitigation mechanic that can mitigate the initiator mechanic.

[0033] Such a game media data file 258 may include a game media ID of the game media, a localizable name of the game media, a media format (e.g., image, audio, video, text, etc.), a category or type of the media (cutscene, audio log, poster, developer commentary, etc.), a URL or a server provisioning media file, and / or whether the game media is associated with a specific activity. Such a game media data file 258 may be associated with a game media start event indicating that a part of a specific game media has just started within the game, and a game media end event indicating that a part of a specific game media has ended.

[0034] FIG. 3 shows an exemplary display with a generated virtual effect indicator. On the display 300 of the client device, an exemplary third-person perspective of the player 302 within the game environment is shown. From the player's perspective, there is a bat 304 flying to the left and a mountain 306 on the right. Hidden behind the mountain is a snake 308, which is difficult to see at the moment. However, during gameplay, the sound of the snake 308 is audible, and the directional sound is emitted from the right side. In some examples, the sound is played louder from the right speaker, or the right side of the headset or other headphones.

[0035] If the player is hearing impaired, especially if there is a hearing impairment in the right ear, the sound of the snake 308 may not be heard. Even in such a case, the player may be looking at the bats 304 as they are moving around, for example. A camera (not shown) may be used to detect where the player is looking on the display 300. If it appears that the player cannot turn their line of sight to the right direction during the playback of the sound of the snake 308, it may be an indication that the player has a hearing impairment in their right ear. In such a case, a virtual effect indicator 310 is generated to indicate where the sound is being emitted from.

[0036] As discussed in more detail below, the virtual effect indicator 310 may be generated only after a particular threshold of the degree of mismatch between the focus determined by the gaze data and the source position. For example, the virtual effect indicator 310 is generated after the player is unable to direct their gaze at the source position five times in a row. Further, the player may consistently look at a location that is somewhat distant from where the source position is. In such a case, the player may have a certain degree of hearing loss in one or both ears, and an auditory offset occurs. Therefore, the correction angle may be determined based on the gaze pattern and how far the player is looking away from the source position. Next, the correction angle can be applied to dynamically readjust the balance of sound between the two speakers or on both sides of the headset.

[0037] FIG. 4 is a flowchart showing an exemplary method 400 for detecting a gaze and generating a virtual effect indicator correlated with a directional sound. The method 400 of FIG. 4 may be embodied as instructions executable within a non-transitory computer-readable storage medium including, but not limited to, non-volatile memory such as a CD, DVD, or hard drive. The instructions of the storage medium may be executed by a processor(s) to cause various hardware components of a computing device hosting or otherwise accessing the storage medium to implement the method. Execution of the instructions may be performed on a cloud server (e.g., the steps identified in FIG. 4 are executed in the cloud). The steps (and their order) identified in FIG. 4 are examples and may include various alternative, equivalent, or derivative forms thereof including, but not limited to, their execution order.

[0038] In step 410, information regarding a plurality of directional sound types associated with one or more interactive content titles is stored in a memory. Each directional sound type is associated with one or more virtual effect indicators in each interactive content title. The directional sound types may differ in terms of direction, type of sound, and parameters based on other sound waves and positions. The directional sound types may later be recorded in the activity file 216 by the object recorder 206. A statistical analysis regarding the directional sound types that the player has missed may be performed and can be actively included in the gameplay of a player with a known hearing impairment.

[0039] In step 420, game play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within the current game play session is monitored. During gameplay, the collected game play data may indicate a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title. For example, the source position may be within a display hidden behind another object or may be too far away to be seen but can be heard. The directional sound may be experienced as originating from the side correlated with the position where the source position is on the display using directional audio techniques.

[0040] In step 430, gaze data is tracked by a camera associated with the client device during the current game play session to identify a focus at which one or both eyes of the player are focused within the three-dimensional virtual environment. The camera may be a virtual reality headset, a stand-alone camera installed in front of the player, or a position on top of a console or display. The focus may be determined through various training and calibration modules that adjust the determined focus until the player indicates that the determined focus is accurate. Alternatively, the calibration can be performed by internal measurements determined by a processor based on known factors.

[0041] In step 440, when game play data indicating directional sound is received, it is identified that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment. Identifying may include comparing one or more three-dimensional coordinates associated with the source position of the directional sound with one or more three-dimensional coordinates associated with the determined focus. Identifying may include determining the player's orientation within the three-dimensional virtual environment and, based on the determined orientation, determining the player's field of view including one or more three-dimensional coordinates of the three-dimensional virtual environment, and other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the player's field of view. The orientation may be determined based on a virtual reality headset that detects the orientation of the player's head or body. Thus, in virtual reality, since sound may be emitted from a position outside the field of view, the player needs to change the orientation of the body. In such a case, the virtual effect indicator may be an arrow or shading indicating that the player needs to change the orientation in a specific direction.

[0042] Furthermore, the line-of-sight data may be mapped as the focus moves to multiple positions within the three-dimensional virtual environment. The focus indicated by the line-of-sight data may be identified as not moving towards the source position within the three-dimensional virtual environment based on the mapped line-of-sight data. Furthermore, a portion of the line-of-sight data sharing the same set of timestamps as the identified trigger point may be identified as not matching the position on the display where the directional sound is being emitted. The threshold may be set according to the degree of mismatch between the focus and the source position of the directional sound. The virtual effect indicator may be further generated based on the threshold being met.

[0043] In step 450, a virtual effect indicator associated with the indicated directional sound's directional sound type is generated and presented within the display of the client device, and the virtual effect indicator indicates the source position of the indicated directional sound. The virtual effect indicator may be displayed near the position where the source position is, or may describe the position where the source position is. Different virtual effect indicators may include a visual Doppler effect centered on the source position of the directional sound on the display, a write queue indicating which direction to look in towards the source position, a visual indicating the source position or the direction of the directional sound, or a notification warning the player of the source position of the directional sound. The virtual effect indicator may be generated as an overlay or object within the game environment. The virtual effect indicator may (or may not) be recorded by the object recorder 206 and may (or may not) be stored as an activity in the activity file. The virtual effect indicator may be shown or removed from the media file 212 during recording by the content recorder.

[0044] FIG. 5 is a block diagram of an exemplary electronic entertainment system 500. The entertainment system 500 of FIG. 5 includes a main memory 505, a central processing unit (CPU) 510, a vector unit 515, a graphics processing unit 520, an input / output (I / O) processor 525, an I / O processor memory 530, a controller interface 535, a memory card 540, a universal serial bus (USB) interface 545, and an IEEE interface 550. The entertainment system 500 may further include an operating system read-only memory (OS ROM) 555, an audio processing unit 560, an optical disk control unit 570, and a hard disk drive 565, which are connected to the I / O processor 525 via a bus 575.

[0045] Entertainment system 500 can be an electronic game console. Alternatively, entertainment system 500 may be implemented as a general-purpose computer, a set-top box, a handheld game device, a tablet computing device, a mobile computing device, or a cellular phone. The entertainment system may include a greater or lesser number of operating components depending on a particular form factor, purpose, or design.

[0046] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 of FIG. 5 communicate via system bus 555. Further, the CPU 510 of FIG. 5 communicates with main memory 505 via dedicated bus 550, and the vector unit 515 and graphics processing unit 520 may communicate via dedicated bus 590. The CPU 510 of FIG. 5 executes programs stored in OS ROM 555 and main memory 505. The main memory 505 of FIG. 5 may include pre-stored programs and programs transferred via I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disk (not shown) using optical disk control unit 570. The I / O processor 525 of FIG. 5 may also enable the introduction of content transferred through a wireless or other communication network (e.g., 4G, LTE, 3G, etc.). The I / O processor 525 of FIG. 5 mainly controls data exchange between various devices of entertainment system 500, including CPU 510, vector unit 515, graphics processing unit 520, and controller interface 535.

[0047] The graphics processing unit 520 in FIG. 5 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate an image for display on a display device (not shown). For example, the vector unit 515 in FIG. 5 can convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Further, the audio processing unit 560 executes instructions to generate an audio signal, and the audio signal is output to an audio device such as a speaker (not shown). Other devices can be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550 such as a wireless transceiver, and these interfaces can also be embedded in the system 500 or as part of some other components such as a processor.

[0048] The user of the entertainment system 500 in FIG. 5 provides instructions to the CPU 510 via the controller interface 535. For example, the user can instruct the CPU 510 to store specific game information in the memory card 540 or other non-transitory computer-readable storage media, or can instruct the characters in the game to perform some specific actions.

[0049] The present invention can be implemented in an application that can be operated by various end-user devices. For example, the end-user device can be a personal computer, a home entertainment system (e.g., Sony PlayStation2 (registered trademark) or Sony PlayStation3 (registered trademark) or Sony PlayStation4 (registered trademark)), a portable game device (e.g., Sony PSP (registered trademark) or Sony Vita (registered trademark)), or, although lower, a home entertainment system of a different manufacturer. The method described herein is fully intended to be operable on various devices. The present invention can also be implemented in a cross-title neutral state, and embodiments of the system of the present invention can be utilized across various titles from various publishers.

[0050] The present invention may be implemented in an application that can be operated using various devices. A non-transitory computer-readable storage medium refers to any medium or media involved in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms including, but not limited to, non-volatile media such as optical disks, magnetic disks, and volatile media such as dynamic memory. Common forms of non-transitory computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, RAM, PROM, EPROM, FLASHEPROM, as well as any other memory chip or cartridge.

[0051] Various forms of transmission media may be involved in conveying one or more sequences of one or more instructions to a CPU for execution. A bus transfers data to system RAM from which the CPU fetches and executes instructions. Instructions received by system RAM can optionally be stored on a fixed disk either before or after being executed by the CPU. Various forms of storage, as well as the network interfaces and network topologies necessary to implement them, may be implemented as well.

[0052] The detailed description of the foregoing technology has been presented for purposes of illustration and description. The above detailed description is not intended to be exhaustive or to limit the technology to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were chosen in order to best explain the principles of the technology, its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and with various modifications suitable for the particular purposes contemplated. The scope of the technology is intended to be defined by the claims.

Claims

1. A method for generating based on the line of sight of a virtual effect indicator correlated with a directional sound, comprising: storing in a memory information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within a respective interactive content title; monitoring game play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current game play session, the game play data indicating a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title; tracking line of sight data by a camera associated with the client device during the current game play session to identify, within the three-dimensional virtual environment, a focus at which one or both eyes of the player are focused; identifying that the focus indicated by the line of sight data does not move towards the source position within the three-dimensional virtual environment when the game play data indicating the directional sound is received; generating a virtual effect indicator associated with the directional sound type of the indicated directional sound and presenting it within a display of the client device, the virtual effect indicator indicating the source position of the indicated directional sound; the method as described above.

2. The method according to claim 1, wherein the virtual effect indicator includes at least one of a visual Doppler effect centered on the source position of the directional sound on the display, a writing queue indicating in which direction to look towards the source position, a visual indicating the source position or the direction of the directional sound, and a notification warning the player of the source position of the directional sound.

3. further comprising mapping the line of sight data as the focus moves to multiple positions within the three-dimensional virtual environment Identifying that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment is based on the mapped line-of-sight data, the method according to claim 1.

4. Identifying that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment further includes comparing one or more three-dimensional coordinates associated with the source position of the directional sound with one or more three-dimensional coordinates associated with the determined focus, the method according to claim 1.

5. Identifying that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment is determining the orientation of the player within the three-dimensional virtual environment, and determining the field of view of the player including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the field of view of the player, the determining, further including, the method according to claim 1.

6. further including setting a threshold for the degree of mismatch between the focus and the source position of the source of the directional sound, the virtual effect indicator is further generated based on the threshold being met, the method according to claim 1.

7. tracking a plurality of mismatches associated with the player, each mismatch being between the focus and the source position of the directional sound, the tracking, determining a correction angle based on the determined offset between the focus and the source position, the determined offset indicating monaural hearing loss, and the correction angle correcting the hearing on one side, the determining, further including, the method according to claim 1.

8. The identifying further includes identifying that a portion of the line-of-sight data sharing the same set of timestamps as the identified trigger point does not match the position on the display where the directional sound is emitted, the method according to claim 1.

9. A line-of-sight-based generation system for a virtual effect indicator correlated with directional sound, The memory configured to store in a memory information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within each interactive content title, the memory, A communication interface that receives monitored game play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current game play session, the game play data indicating a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title, the communication interface, A processor that executes instructions stored in the memory, Including, The processor, Tracking line-of-sight data by a camera associated with the client device during the current game play session to identify, within the three-dimensional virtual environment, the focus at which one or both eyes of the player are focused, When the game play data indicating the directional sound is received, identifying that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment, Generating a virtual effect indicator associated with the directional sound type of the indicated directional sound and presenting it within a display of the client device, the virtual effect indicator indicating the source position of the indicated directional sound, the presenting, The system for executing the instructions for.

10. The virtual effect indicator includes at least one of a visual Doppler effect centered on the source position of the directional sound on the display, a write queue indicating which direction to look towards the source position, a visual indicating the source position or the direction of the directional sound, or a notification warning the player of the source position of the directional sound. The system according to claim 9.

11. Executing further instructions for mapping the line-of-sight data as the focus moves to multiple positions within the three-dimensional virtual environment, Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment is based on the mapped line-of-sight data, the system according to claim 9.

12. Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment further includes comparing one or more three-dimensional coordinates associated with the source position of the directional sound with one or more three-dimensional coordinates associated with the determined focus, the system according to claim 9.

13. Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment is determining the orientation of the player within the three-dimensional virtual environment, and determining the player's field of view including one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the player's field of view, the determining, further including, the system according to claim 9.

14. The processor executes further instructions for setting a threshold for the degree of mismatch between the focus and the source position of the source of the directional sound, The virtual effect indicator is further generated based on the threshold being met, the system according to claim 9.

15. The processor is tracking a plurality of mismatches associated with the player, each mismatch being between the focus and the source position of the directional sound, the tracking, determining a correction angle based on the determined offset between the focus and the source position, the determined offset indicating monaural hearing loss, and the correction angle correcting unilateral hearing, the determining, executing further instructions for, the system according to claim 9.

16. A non-transitory computer-readable storage medium in which instructions are embodied, the instructions being executable by a computing system to perform a method of generating a virtual effect indicator based on the line of sight correlated with a directional sound, The method is Storing in a memory information regarding a plurality of directional sound types associated with one or more interactive content titles, each directional sound type being associated with one or more virtual effect indicators within each interactive content title, said storing, Monitoring game play data transmitted via a communication network from a client device of a player participating in a current activity of one of the interactive content titles within a current game play session, said game play data indicating a directional sound associated with a source position within a three-dimensional virtual environment of the interactive content title, said monitoring, Tracking line-of-sight data by a camera associated with the client device during the current game play session to identify, within the three-dimensional virtual environment, a focus at which one or both eyes of the player are focused, Identifying that the focus indicated by the line-of-sight data does not move towards the source position within the three-dimensional virtual environment when the game play data indicating the directional sound is received, Generating a virtual effect indicator associated with the directional sound type of the indicated directional sound and presenting it within a display of the client device, said virtual effect indicator indicating the source position of the indicated directional sound, said presenting, The non-transitory computer-readable storage medium comprising the above.

17. The virtual effect indicator includes at least one of a visual Doppler effect centered on the source position of the directional sound on the display, a write queue indicating in which direction to look towards the source position, a visual indicating the source position or the direction of the directional sound, or a notification warning the player of the source position of the directional sound. The non-transitory computer-readable medium according to claim 16.

18. Further including instructions executable to map the line-of-sight data as the focus moves to multiple positions within the three-dimensional virtual environment, Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment is based on the mapped line-of-sight data, the non-transitory computer-readable medium according to claim 16.

19. Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment further includes comparing one or more three-dimensional coordinates associated with the source position of the directional sound with one or more three-dimensional coordinates associated with the determined focus, the non-transitory computer-readable medium according to claim 17.

20. Identifying that the focus indicated by the line-of-sight data does not move toward the source position within the three-dimensional virtual environment is determining the orientation of the player within the three-dimensional virtual environment and determining a field of view of the player that includes one or more three-dimensional coordinates of the three-dimensional virtual environment based on the determined orientation, wherein other three-dimensional coordinates associated with the three-dimensional virtual environment are excluded from the field of view of the player, the determining further including, the non-transitory computer-readable medium according to claim 17.

Images