Gesture-based skill search

Abstract

To provide a method for gesture-based skill search.SOLUTION: A map regarding one or more virtual actions may be stored in memory. The virtual actions may be associated with a set of data associated with performing a corresponding action in a real-world environment. Data regarding an action by a user in the real-world environment may be captured. A current progress level of the user within a virtual environment is identified. The current identified progress level is associated with one or more available virtual actions. The captured data may correspond to an identified one of the available virtual actions based on a match between the captured data and the set of data associated with the identified virtual action as indicated by the map. A search of one or more databases for instructions corresponding to the identified virtual action is initiated.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates generally to searching for network devices, and more particularly to gesture-based searching for in-game skills. [Background technology]

[0002] 2. Description of Related Art Currently available digital game titles support a variety of different in-game scenarios in which players (through their respective avatars or characters) can engage in a variety of different types of activities and exercise various in-game skills. For example, racing game titles may test a player's driving skills, while first-person shooter game titles may test a player's aiming accuracy, and other fighting game titles may test a player's skills involving weapons such as swords, knives, axes, hammers, shovels, magical effects, etc. Similarly, various virtual terrains may require a player to control and navigate their avatar through swimming, climbing, jumping, and other actions within the game environment.

[0003] A player may typically control his or her avatar using one or more types of game controller. Currently, a variety of different controllers (and controller modifications) exist with various buttons, touchpads, sensors, microphones, etc., in various configurations and layouts. Different combinations of user inputs (e.g., series or simultaneous button presses, touchpad or other gestures, verbal commands, or other inputs) may be required to perform various types of in-game movements, manipulations, or other in-game actions. Because different game titles may include different activities, particular input combinations may result in different in-game movements. Furthermore, entering an incorrect input combination may result in an in-game action different from what the player intended.

[0004] This variety and complexity of in-game actions and controllers creates a barrier to entry when a player is introduced to a new or unfamiliar game title, controller, or game console system. Thus, a novice or inexperienced player may find gameplay difficult and frustrating without help or other types of guidance. However, searching for such help or guidance may be complicated by language barriers that may arise if the player is not familiar with the vocabulary or names of virtual elements in the in-game environment. Players may not even know how to conduct such searches. For example, simply watching another player perform an action (e.g., whether or not using an in-game object) does not necessarily convey to the player the name of the action or object, or how to perform such an action themselves.

[0005] Therefore, there is a need in the art for improved systems and methods for gesture-based discovery of in-game skills. Summary of the Invention

[0006] Embodiments of the present invention include systems and methods for gesture-based skill search. A memory stores a map of one or more virtual actions, each associated with a data set associated with performing a corresponding action in a real-world environment. A gesture-based search server captures data regarding actions by a user in the real-world environment. The gesture-based search server identifies a current level of progress of the user in the virtual environment, which is associated with one or more available virtual actions. The gesture-based search server determines that the captured data corresponds to an identified virtual action among the available virtual actions based on a match between the captured data and a data set associated with the identified virtual action, as indicated by the map. The gesture-based search server begins searching one or more databases for instructions corresponding to the identified virtual action. [Brief explanation of the drawings]

[0007] [Figure 1] 1 illustrates an exemplary network environment in which a system for gesture-based skill search may be implemented. [Figure 2] 1 illustrates an exemplary uniform data system (UDS) that may be used to provide data to a system for gesture-based skill retrieval. [Figure 3] 1 is a flowchart illustrating an example method for gesture-based skill search. [Figure 4] FIG. 1 is a block-level diagram of an example user device that may be used to perform gesture-based skill search. [Figure 5] FIG. 1 is a block diagram of an exemplary electronic entertainment system that may be used with embodiments of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0008] Embodiments of the present invention include systems and methods for gesture-based skill search. A memory stores a map of one or more virtual actions, each associated with a data set associated with performing a corresponding action in a real-world environment. A gesture-based search server captures data regarding actions by a user in the real-world environment. The gesture-based search server identifies a current level of progress of the user in the virtual environment, which is associated with one or more available virtual actions. The gesture-based search server determines that the captured data corresponds to an identified virtual action among the available virtual actions based on a match between the captured data and a data set associated with the identified virtual action, as indicated by the map. The gesture-based search server begins searching one or more databases for instructions corresponding to the identified virtual action.

[0009] FIG. 1 illustrates an exemplary network environment in which a system for gesture-based skill search may be implemented. The network environment 100 may include one or more content source servers 110 providing digital content for distribution (e.g., games, other applications, and services), one or more content provider server application program interfaces (APIs) 120, a content delivery network server 130, a gesture-based search server 140, and one or more user devices 150A-150N. The servers described herein may include any type of server known in the art, including standard hardware computing components such as network and media interfaces, non-transitory computer-readable storage (memory), and a processor for executing instructions or accessing information that may be stored in the memory. The functionality of multiple servers may be integrated into a single server. Any of the aforementioned servers (or integrated servers) may exhibit specific client-side, caching, or proxy server characteristics. These characteristics may depend on the particular network placement or configuration of the server.

[0010] Content source server 110 may maintain and provide a variety of digital content available for distribution. Content source server 110 may be associated with any content provider that makes its content accessible over a communications network. Such content may include digital video and games, as well as other types of digital applications and services. Such applications and services may include any of a variety of different digital content and functionality that may be provided to user device 150. An example of content source server 110 may be an internet website that provides downloadable and / or streaming content. Content provided by content source server 110 may include any type of multimedia content, such as movies, games, static / dynamic content, photos, social media content, social media websites, etc. User device 150 may include multiple different types of computing devices. In some embodiments, content data is transmitted from content source server 110 to a computing device, which then renders the content data into a format suitable for use by user device 150.

[0011] Content from content source servers 110 may be provided via content provider server API 120, which allows different types of content source servers 110 to communicate with other servers (e.g., user devices 150) in network environment 100. Content provider server API 120 may be specialized for the content source server 110 providing the content, as well as the particular language, operating system, protocol, etc. of the user device 150. In network environment 100 including multiple different types of content source servers 110, there may likewise be a corresponding number of content provider server APIs 120, which enable various formatting, conversion, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150, which may process such content using different operating systems, protocols, etc. Thus, different formats of applications and services may be made available to be compatible with a variety of different user devices 150.

[0012] The content source server 110 may also include online servers associated with social media applications, which may be programmed to provide one or more social media graphs to a user to identify the user's social contacts. These users may not be social contacts or socially associated with the user, but may have played one or more video games with the user. The friend list may include additional information about the user's friends, such as a description of the games each friend owns, an identification of the friend's online status (e.g., online, offline, inactive, etc.), the friend's most recent login and for how long, and the friend's last game played. The social network includes user data, including data such as the user's social graph, posts, photos, videos, and biographical information.

[0013] The content provider server API 120 may further facilitate each of the user devices 150 accessing content hosted or services provided by the content source server 110, either directly or via the content delivery network server 130. Additional information about the accessed content or services, such as metadata, may also be provided to the user device 150 by the content provider server API 120. As described below, the additional information (i.e., metadata) may be used to provide details about the content or services provided to the user device 150. In some embodiments, the services provided by the content source server 110 to the user device 150 via the content provider server API 120 may include services associated with other content or services, such as chat services, ratings, and profiles associated with particular games, teams, communities, etc. In such cases, the content source servers 110 may also communicate with each other via the content provider server API 120.

[0014] The content delivery network servers 130 may include servers that provide resources, files, etc. related to content from the content source servers 110 to the user devices 150, including various content and service configurations. The content delivery network servers 130 may also be invoked by user devices 150 requesting access to particular content or services. The content delivery network servers 130 may include general purpose management servers, game servers, streaming media servers, servers hosting downloadable content, and other content delivery servers known in the art.

[0015] Gesture-based search server 140 may include any data server known in the art that can receive data from user device 150. The content rendered by gesture-based search server 140 may be for essentially any type of computer application and may include one or more types of content, such as games, movies, audio, images, and multimedia, among others. In some embodiments, the content or portions of content are generated by gesture-based search server 140. In some embodiments, the content or portions of content are streamed from content source server 110 to the computing device over network 100. In some embodiments, the content or portions of content are streamed from cloud gaming infrastructure to the computing device over network 100. The infrastructure may direct the transmission of various types of content from content source server 110 to the computing device over network 100.

[0016] In an exemplary embodiment, gesture-based search server 140 may store or access a store containing search maps for one or more game titles. Each game title may be associated with a different search map, which may include information about in-game actions available in the respective game title (e.g., swinging a sword in a game environment), information about how the in-game actions correspond to real-world equivalent actions (e.g., swinging an object in a real-world environment), and associated input combination(s) associated with initiating actions in the game environment using one or more types of controllers. Some game titles may further restrict the types of actions available in various levels, activities, or other portions of the associated in-game environment. Such restrictions may be saved in an activity file (described in more detail in connection with FIG. 2 ) generated by the UDS system and may be used to filter searches by search users who are actively engaged in gameplay of a particular game title in a particular portion of a game environment or gameplay session.

[0017] A gesture-based skill search may be initiated using various triggers that can be customized for the user. For example, a verbal command, a key-based command, or a gesture command may indicate that the user wants to initiate such a gesture-based skill search. Such a command may trigger a camera and / or other sensors (e.g., of user device(s) 150) to capture images and / or other sensor data regarding the user's real-world environment. For example, a camera may capture an image of the user waving their arms in the real world, and an avatar swinging a sword is reproduced in the digital environment. The camera (discussed in more detail in connection with FIG. 4 ) may be configured to include multiple image capture devices, such as a stereoscopic pair of cameras, an infrared camera, a depth camera, or a combination thereof. In some embodiments, one or more microphones (discussed in more detail in connection with FIG. 4 ) may also be used to capture audio from the user's real-world environment. Meanwhile, other sensors (e.g., accelerometers, gyroscopes) associated with the user (e.g., via a controller, mobile device, wearable device, handheld device, etc.) may also capture data regarding real-world actions.

[0018] Such image, audio, and other sensor data may be provided to the gesture-based search server 140 for analysis. The gesture-based search server 140 may be able to use stored maps to match received sensor data regarding real-world actions with available in-game actions within the associated game title. In some implementations, the received sensor data may indicate multiple available in-game actions, and in such cases, different parameters may be used to filter the available in-game actions. As noted above, in some game titles, different in-game actions are available in different parts of the game environment or game session. Such data may be provided in an activity file and used to filter the available in-game actions and identify in-game actions that may be available in a specified part of the game environment or game session currently associated with the search user. Alternatively or additionally, a menu of multiple available in-game actions may be displayed in the graphical user interface for selection by the search user.

[0019] Once an in-game action is identified or selected, the graphical user interface may be updated by the gesture-based search server 140 to include a representation of the controller associated with the user and instructions on how to use the controller to perform the identified in-game action. Such instructions may include visual, audio, and / or textual instructions. In some embodiments, the instructions may be provided as an overlay in a pop-up window. Alternatively, the instructions may be provided to the user device 150 during a gameplay session while the user is still engaged in gameplay within the game environment.

[0020] Thus, gesture-based search server 140 may map data regarding real-world actions to one or more in-game actions, and then further map an identified one of the in-game actions to an input combination and associated command. Thus, a user actively playing a game may initiate a search performed by gesture-based search server 140 without leaving the in-game environment (e.g., to perform a traditional online search), pausing or ending their current gameplay session, or switching to another device or platform.

[0021] The user device 150 may be a server that provides internal services (e.g., to other servers) within the network environment 100. In such a case, the user device 150 may correspond to one of the content source servers 110 described herein. Alternatively, the user device 150 may be a client device, which may include any number of different game consoles, mobile devices, laptops, and desktops. Such user devices 150 may also be configured to access data on other storage media, such as, but not limited to, a memory card or disk drive, which may be suitable for downloaded services. Such user devices 150 may include standard hardware computing components, such as, but not limited to, a network interface, a media interface, non-transitory computer-readable storage (memory), and a processor that executes instructions that may be stored in the memory. These user devices 150 may also run a variety of different operating systems (e.g., iOS, Android®), applications, or computing languages ​​(e.g., C++, JavaScript®). An exemplary client device 150 is described in detail herein with respect to FIG. 5. Each user device 150 may be associated with a participant or other type of audience member of a collection of digital content streams.

[0022] In some embodiments, user device 150 is configured to execute games locally on the processing hardware of a computing device or game console. The games or content may be obtained in any form, such as a physical media form (e.g., digital disc, tape, card, thumb drive, solid-state chip, or solid-state card), or by downloading from the Internet over a communications network. In some embodiments, the computing device acts as a client in communication with a cloud gaming infrastructure over the communications network. The cloud gaming infrastructure may maintain and execute the video game being played by user device 150. Inputs received from user device 150, controllers, and cameras (and other sensors) may be transmitted to the cloud gaming infrastructure, which processes the inputs and reflects them in the game state of the running video game.

[0023] Game data from a video game, such as video data, audio data, and haptic feedback data, may be transmitted from content source server 110 to user device 150. The computer system may further process the game data before transmitting it to a suitable device, or may transmit the game data directly to a suitable device. For example, video and audio streams may be transmitted to user device 150.

[0024] User device 150 may also include or be associated with a controller configured to receive inertial sensor data from inertial sensors within user device 150. The inertial sensor data indicates movement of the controller and / or user device 150 according to the movement of a user with whom user device 150 is associated. The user's movements are based on a virtual reality scene displayed within user device 150. From the inertial sensor data, a path or trajectory of movement of user device 150 and a speed of movement may be determined. In some embodiments, the path or trajectory of the head-mounted display movement corresponds to one or more user movements within a set of user movements including lean forward, lean back, tilt left, tilt right, head left turn, head right turn, head up tilt, head down tilt, squat, and jump. However, in other embodiments, the path or trajectory of the head-mounted display movement may correspond to essentially any user movement within the motor capabilities of the human body.

[0025] In some embodiments, user device 150 can be used to manipulate, for example, grab, move, push, pull, etc., virtual objects within a virtual reality (VR) or augmented reality (AR) scene displayed on user device 150 or another computing device, e.g., a television, computer, etc. When a user wears user device 150 and moves their hand, the virtual hand in the game moves. Additionally, when a user wears user device 150 and moves their finger, the fingers of the virtual hand in the game move. The position and / or orientation of the fingers are determined from image data captured using the camera described above and in FIG. 5 to generate finger movements for the virtual hand. One or more examples of virtual objects include a virtual vehicle, a user's virtual hand, a virtual user, a virtual sword, a user's avatar, virtual fingers, virtual knuckles, a virtual tree, a virtual flower, etc.

[0026] The display of the user device 150 may generate a three-dimensional scene incorporating the avatar in a virtual space or environment with virtual interactive objects. The gesture-based search server 140 receives visual and / or distance information about the user captured by the user device 150. The gesture-based search server 140 extracts motion information describing the user's movements. The motion information may be encoded to describe the movement of the user's "skeleton" composed of cardinal points within the user's body. The gesture-based search server 140 may enable content augmentation by allowing user-related information, such as the user's avatar, identifier, and representative image, to be introduced into the VR space presented on the user's user device 150A-150N.

[0027] A user's avatar may be automatically converted from a single digital image into an animated 3D avatar. Specifically, a user may upload a digital or other image to user device 150 and, in exchange, receive one or more avatars featuring various facial expressions and / or animations. Avatars may be displayed and used by the user as any still or animated image (e.g., in GIF format). For example, avatars may be sent to other users via SMS, MMS, email message, chat (such as Facebook® Chat), instant messenger (such as Skype or Windows® Messenger), Twitter®, blogs, forums, or other electronic communication means intended to display the user's emotions.

[0028] In some embodiments, the avatar may be realistically based on the user, e.g., having the same facial features, clothing, and body type. In other embodiments, the avatar may be intentionally unrealistic, e.g., by mapping the user's movements to the facial features of a celebrity, movie star, politician, video game character, or another user. The avatar may be a blend of realistic and intentionally unrealistic elements, e.g., the user's face may be used for the avatar, but the avatar may be given different clothing or a different body type. Alternatively, the user's body may be used for the avatar, but the avatar may be given the face of a celebrity, movie star, politician, video game character, or another user.

[0029] In an embodiment, a user of a user device 150A, such as a Sony® Playstation 3® entertainment device (PS3), navigates within an instance of one of many virtual 3D environments available from an online server, each virtual 3D environment populated with a relatively small percentage of the total number of other users currently online. Within this environment, these users are represented by avatars, which enable social interaction with other users on other user devices 150B-150N and with the virtual environment itself, which may offer amenities such as pool tables, bowling alleys, arcade games, and streaming online content to a television screen.

[0030] User device 150A transmits information describing the user's movements and / or the movements of the corresponding avatar to other user devices 150B-150N or to gesture-based search server 140, which then forwards this information to the other user devices 150B-150N, thereby enabling the other user devices 150B-150N to generate scenes featuring precise movements by the avatar. User device 150A receives information describing the other user's movements and / or the movements of the corresponding avatar from other user devices 150B-150N or from an intermediary device such as gesture-based search server 140, which then forwards this information to first user device 150A upon receiving it from the other user devices 150B-150N, thereby enabling the first user device 150A to generate scenes featuring precise movements by a second avatar.

[0031] A user associated with user device 150A may be matched with other user devices 150B-150N in a virtual reality environment based on characteristics of each user's physical environment. In one embodiment, information regarding the characteristics of the physical space in which each user is located may be received by gesture-based search server 140. The information regarding the characteristics of each user's physical space may be used to match users located in different physical spaces that have similar characteristics. Users are matched to participate in a virtual multiplayer entertainment experience. This matching may be based on the characteristics of the physical space in which each user is located. For example, matching may be performed based on the similarity of the characteristics of the physical spaces in which each user is located.

[0032] The user device 150 or computing device generates a representation of the captured user's movements and causes an avatar to perform the generated representation of the user's movements. The user device 150 also generates a representation of the received second user's movements and causes the second avatar to perform the generated representation of the second user's movements. The user device 150 also updates the virtual space or environment, as well as any virtual interactive objects, as needed.

[0033] Any relevant events in the virtual environment can be communicated between user devices 150 using a common coordinate system so that events in the virtual environment are referenced with respect to a physical reference position. To establish a common coordinate system, only a single instance of position data needs to be exchanged between user devices 150, and indeed this may indeed be a one-way transfer of data (one user device 150A tells another user device 150B the position and orientation of the first user device 150A relative to the second user device 150B). However, it may be more reliable if user devices 150 exchange position data multiple times, for example, frequently during a session of virtual interaction in a shared coordinate system. Subsequent relevant movements of user devices 150 in the real environment can be reproduced in the virtual environment. It will thus be understood that the actual and virtual positions of the user devices substantially coincide.

[0034] Subsequent movement away from the common reference position may be tracked by the gesture-based search server 140 (which acts as a motion tracking structure) using the accelerometers and gyroscopes (motion sensors) of the user device 150, and cumulative changes in displacement and orientation relative to the common reference position may be maintained. The relationships between acceleration, velocity, and displacement (both lateral and rotational) are well known and will not be described in detail here. While accelerometers and gyroscopes may be referred to separately or collectively as motion sensors, it will be understood that accelerometers and gyroscopes may also be considered collectively as accelerometers, with the primary difference being the lateral or rotational nature of the detected acceleration.

[0035] FIG. 2 illustrates an exemplary uniform data system (UDS) 200 that can be used to provide data to a system for gesture-based skill search. Based on the data provided by the UDS, the gesture-based search server 140 can recognize in-game objects, entities, activities, and events in which a user has engaged, thereby supporting analysis of and filtering related to in-game activity. Each user interaction can be associated with metadata, such as the type of in-game interaction, its location in the in-game environment, its point in the in-game timeline, and other players, objects, and entities involved. Thus, metadata can be tracked for any of the various user interactions that may occur during a game session, including associated activities, entities, settings, results, actions, effects, locations, and character statistics. Such data can then be aggregated, applied to a data model, and subjected to analysis. Using such a UDS data model, contextual information can be assigned to each piece of information in a uniform manner across games.

[0036] 2, an exemplary console 228 (e.g., user device 150) and exemplary servers 218 (e.g., streaming server 220, activity feed server 224, user-generated content (UGC) server 232, and object server 226) are shown. In an embodiment, console 228 may be implemented in or in association with content source server 110, content delivery network server 130, cloud server, or any combination of server 218 and the servers of FIG. 1. In an exemplary embodiment, content recorder 202 may receive content (e.g., media) from interactive content title 230 and record it in content ring buffer 208. Such ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), the start time of each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and the end time of each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored as media files 212 (e.g., MP4, WebM, etc.) by the console 228. Such media files 212 may be uploaded to the streaming server 220 for storage and subsequent streaming or use, although the media files 212 may also be stored on any server, cloud server, any console 228, or any user device 150. Such start and end times for each segment may be stored by the console 228 as a content timestamp file 214. Such content timestamp file 214 may also include a stream ID, which matches the stream ID of the media file 212, thereby associating the content timestamp file 214 with the media file 212.Such content timestamp files 214 may be uploaded and stored on the activity feed server 224 and / or the UGC server 232, but the content timestamp files 214 may also be stored on any server, cloud server, any console 228, or any user device 150.

[0037] While the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine when the object starts and ends. The object library 204 and the object recorder 206 may be implemented on either the platform server 120, the cloud server, or the server 218. When the object recorder 206 detects the start of an object, the object recorder 206 receives object data from the object library 204 (e.g., if the object is an activity, the user interaction with the activity, the activity ID, the activity start time, the activity end time, the activity result, the activity type, etc.) and records the activity data (e.g., ActivityID1,START_TS, ActivityID2,START_TS, ActivityID3,START_TS) in the object ring buffer 210. Such activity data recorded in the object ring buffer 210 may be stored in the object file 216. Such object files 216 may also include activity start times, activity end times, activity IDs, activity results, activity types (e.g., competitive matches, quests, tasks, etc.), and user or peer data related to the activity. For example, object files 216 may store data regarding items used during the activity. Such object files 216 may be stored on object servers 226, although object files 216 may also be stored on any server, cloud server, any console 228, or any user device 150.

[0038] Such object data (e.g., object files 216) may be associated with content data (e.g., media files 212 and / or content timestamp files 214). In one embodiment, the UGC server 232 stores the content timestamp files 214 and associates them with the object files 216 based on matching the stream ID of the content timestamp files 214 with the corresponding activity ID of the object files 216. In another embodiment, the object server 226 may store the object files 216 and receive queries for the object files 216 from the UGC server 232. Such queries may be performed by searching for the activity ID of the object files 216 that matches the stream ID of the content timestamp files 214 sent with the query. In yet another embodiment, queries of the stored content timestamp files 214 may be performed by matching the start and end times of the content timestamp files 214 with the start and end times of the corresponding object files 216 sent with the query. Such object files 216 may also be associated with matching content timestamp files 214 by the UGC server 232, although the association may be performed by any server, cloud server, any console 228, or any user device 150. In another embodiment, the object files 216 and content timestamp files 214 may be associated by the console 228 during the creation of each file 216, 214.

[0039] The activity file generated by the UDS 200 may be provided or accessed by the gesture-based search server 140, which analyzes, filters, and matches data about real-world actions with in-game actions in one or more maps. For example, the activity file may include gameplay data about a particular gameplay session of a particular game title by a particular peer player performing one or more in-game actions when the search user initiates a search. The gesture-based search server 140 may use data from the activity file to identify the game title, the player, the in-game action being performed, and the temporal proximity of the in-game action to the search trigger (e.g., for real-time or near-real-time searches). The identified parameters or conditions when conducting a search may be used by the gesture-based search server 140 to filter relevant search results.

[0040] 3 is a flowchart illustrating an example method for gesture-based skill discovery. The method 300 of FIG. 3 may be embodied as executable instructions in a non-transitory computer-readable storage medium, including, but not limited to, non-volatile memory such as a CD, DVD, or hard drive. The instructions in the storage medium may be executed by a processor(s) to cause various hardware components of a computing device that hosts or accesses the storage medium to perform the method. The steps (and order of steps) identified in FIG. 3 are exemplary and may include various alternatives, equivalents, or derivations of the steps, including, but not limited to, the order of execution of the steps.

[0041] At step 310, a map for one or more virtual actions is stored in memory. Each virtual action (e.g., an action performed in a virtual environment) is associated with a set of data associated with the execution of the corresponding action in the real-world environment and with instruction data regarding input combinations that result in the performance of the virtual action. Different variations of the map may be associated with a particular game title, a collection of game titles, a game genre, a game developer, a game console, a game controller, a collection of controller variations, a game environment, an in-game activity, etc.

[0042] At step 320, data regarding movements by the user within the real-world environment is captured. Movements by the user within the real-world environment may be represented by real-world environment data captured by user device 150 (and associated sensors). The data may be image data captured by a camera of user device 150. In another embodiment, the data may be captured by an inertial sensor on user device 150. The real-world environment data, along with one or more images and image data of the user interacting with the real-world environment captured by user device 150, may be transferred to gesture-based search server 140.

[0043] At step 330, a user's current level of progress within the virtual environment may be determined based on one or more activity files associated with the user's current gameplay. The current determined level of progress is associated with one or more available virtual actions. The user may be matched with one or more other users based on the user's determined level of progress and the progress levels of the other users. Activity files associated with the other users may also be obtained and used to analyze the real-world action data captured at step 320.

[0044] At step 340, it is determined that the captured data corresponds to an identified virtual action of the available virtual actions based on a match between the captured data and a data set associated with the identified virtual action as indicated by the map. In an embodiment, a difference between the captured data and a data set associated with the identified virtual action is indicated by the map. If the difference is within a predetermined threshold, it may be determined that the captured data corresponds to the identified available virtual action. In an embodiment, the available virtual actions are filtered based on other users who correctly perform the virtual action, and one or more available virtual actions are based on the filtered users.

[0045] At step 350, one or more databases are initiated to search for instructions corresponding to the identified virtual action. In some embodiments, the instructions may be stored in the map itself. The map may also have links to or point to other databases where instructions are held. Thus, instructions for the identified in-game action may be obtained and provided to the user device 150.

[0046] FIG. 4 is a block-level diagram of an exemplary user device that may be used to perform gesture-based skill search. It should be understood that the user device 400 may include or exclude more or fewer components than those shown in FIG. 4 , depending on the available configuration and functionality. The user device 400 may include a processor 402 for executing program instructions. A memory 406 is provided for data storage purposes and may include both volatile and non-volatile memory. A display 416 is included to provide a visual interface viewable by the user. The display 416 may be defined by one single display or in the form of separate display screens for each eye. If two display screens are provided, it is possible to provide separate video content for the left eye and video content for the right eye. For example, separate video content display for each eye may improve immersive control of three-dimensional content in a virtual reality scene.

[0047] The motion detection module 418 may include any of various types of motion sensing hardware, such as a magnetometer 420, an accelerometer 424, and a gyroscope 426. The user device 400 may be equipped with an inertial sensor 422 configured to generate inertial sensor data indicative of the movement of the user device 400 according to the movement of the user with whom the user device 400 is associated. The magnetometer 420 measures the strength and direction of magnetic fields near the user device 400. In some embodiments, three magnetometers 420 are used within the user device 400 to ensure an absolute reference for the world space yaw angle. In some embodiments, the magnetometers 420 are designed to measure within a range of ±80 microstellar, which is the Earth's magnetic field. The magnetometers are affected by metal and provide a monotonic yaw measurement due to the actual yaw. The magnetic field may be distorted by metal in the environment, which causes distortion in the yaw measurement. If necessary, this distortion can be calibrated using information from other sensors, such as a gyroscope or camera. In some embodiments, the accelerometer 424 is used in conjunction with the magnetometer 420 to obtain the tilt and azimuth angles of the user device 400 .

[0048] In some embodiments, the present invention may also include one or more external inertial sensors positioned on the user's body. The present invention may include operations for comparing the external inertial sensor data with inertial sensor data received from inertial sensor 422 within user device 400 to identify specific movements made by the user.

[0049] Accelerometers 424 are devices that measure acceleration and gravity-induced reaction forces. Single-axis and multi-axis (e.g., six-axis) models can detect the magnitude and direction of acceleration in various directions. Accelerometers 424 are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 424 are used to provide the gravity direction, which provides an absolute reference for two angles: world-space pitch and world-space roll.

[0050] Gyroscope 426 is a device for measuring or maintaining orientation based on the principles of angular momentum. In one embodiment, three gyroscopes 426 provide information about movement across each axis (x, y, and z) based on inertial sensing. Gyroscopes 426 are useful for detecting high-speed rotations. However, without an absolute reference, gyroscopes 426 may drift over time. For this reason, gyroscopes 426 must be periodically reset, which may be done using other available information, such as visual tracking of objects, position / orientation determination based on accelerometers, magnetometers, etc.

[0051] A camera 404 is provided for capturing images and image streams of the real-world environment to which the user device 400 is exposed. Multiple cameras 404 may (optionally) be included in the user device 400, including a rear-facing camera 404 (a camera that faces away from the user when the user is viewing the display of the user device 400) and a front-facing camera 404 (a camera that faces towards the user when the user is viewing the display of the user device 400). In some embodiments, the user device 400 may include a camera 404 for sensing depth information of objects in the real-world environment to which the user device 400 is exposed.

[0052] The user device 400 includes a speaker 412 for providing audio output and may also include a microphone 414 for capturing audio from the real-world environment, including sounds from the surrounding environment, utterances by the user, etc.

[0053] A WiFi module 410 may be included to enable connection of the user device 400 to the Internet via wireless network technology. The user device 400 may also include a Bluetooth module 408 to enable wireless connection to other devices.

[0054] It should be understood that the components of user device 400 shown in Figure 4 are examples of components that may be included in user device 400 and do not represent all possible components that may be included in user device 400. For example, in various embodiments, user device 400 may include some or none of the components shown in Figure 4. In some embodiments, user device 400 may include additional components not shown in Figure 4.

[0055] In an exemplary embodiment, a searching user may view an in-game action 430 being performed by an avatar (e.g., an avatar associated with a peer player) in the virtual environment. The in-game action 430 shown in FIG. 4 may be a jump or leap. After triggering a gesture-based search, the user may perform a corresponding gesture 440 in the real-world environment. The user may be holding, wearing, or within detection range of one or more sensors of the user device 400. The user device 400 may provide sensor data related to the user gesture 440 to the gesture-based search server 140 for analysis.

[0056] The gesture-based search server 140 may determine that the user gesture 440 characterized by the sensor data is associated with one or more available in-game actions. The available in-game actions may be further filtered based on one or more activity files associated with the conditions that triggered the search. For example, the search may have been triggered during a particular gameplay session or in close temporal proximity to a particular gameplay session. Thus, a filtered subset of in-game actions may be identified. The gesture-based search server 140 may further determine that the specified or selected in-game action is mapped to an instruction set corresponding to the input combination that produces the specified or selected in-game action. The gesture-based search server 140 may then generate a combo display 440 based on the mapped instruction set and provide the generated combo display 440 to the user device 150 for presentation and display.

[0057] 5 is a block diagram of an exemplary electronic entertainment system that may be used with embodiments of the present invention. Entertainment system 500 may be an electronic game console. Alternatively, entertainment system 500 may be implemented as a general-purpose computer, a set-top box, a handheld gaming device, a tablet computing device, a mobile computing device, or a mobile phone. Entertainment systems may include more or fewer operating components depending on their particular form factor, purpose, or design.

[0058] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 of FIG. 5 communicate via a system bus 585. Additionally, the CPU 510 of FIG. 5 may communicate with main memory 505 via a dedicated bus 580, while the vector unit 515 and graphics processing unit 520 may communicate via a dedicated bus 590. The CPU 510 of FIG. 5 executes programs stored in the OS ROM 555 and the main memory 505. The main memory 505 of FIG. 5 may include pre-stored programs and programs transferred via the I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disk (not shown) using the optical disk control unit 570. The I / O processor 525 of FIG. 5 may also enable the introduction of content transferred via wireless or other communication networks (e.g., 4G, LTE, 3G, etc.). The I / O processor 525 of FIG. 5 primarily controls the exchange of data between various devices of the entertainment system 500, including the CPU 510, the vector unit 515, the graphics processing unit 520, and the controller interface 535.

[0059] The graphics processing unit 520 of Figure 5 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate images for display on a display device (not shown). For example, the vector unit 515 of Figure 5 may convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Additionally, the audio processing unit 560 executes instructions to generate audio signals, which are output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550, such as a wireless transceiver, which may be embedded within the system 500 or as part of some other component, such as a processor.

[0060] 5 provides instructions to CPU 510 via controller interface 535. For example, the user may instruct CPU 510 to store certain game information on memory card 540 or other non-transitory computer-readable storage medium, or may instruct a character in a game to perform some specific action.

[0061] The present invention may be implemented in an application that may be operable by a variety of end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., Sony PlayStation2® or Sony PlayStation3® or Sony PlayStation4®), a portable gaming device (e.g., Sony PSP® or Sony Vita®), or a home entertainment system from a different, but subordinate, manufacturer. It is fully contemplated that the methods of the present invention described herein will be operable on a variety of devices. The present invention may also be implemented in a cross-title neutral manner, such that embodiments of the system can be utilized across a variety of titles from a variety of publishers.

[0062] The present invention can be implemented in applications that can operate using a variety of devices. A non-transitory computer-readable storage medium refers to any medium or media that participates in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, including, but not limited to, non-volatile and volatile media, such as optical or magnetic disks and dynamic memory, respectively. Common forms of non-transitory computer-readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic media, a CD-ROM disk, a digital video disk (DVD), any other optical media, a RAM, a PROM, an EPROM, a FLASHEPROM, and any other memory chip or cartridge.

[0063] Various forms of transmission media may be involved in carrying one or more sequences of one or more instructions to a CPU for execution. A bus carries the data to system RAM, from which the CPU retrieves and executes the instructions. Optionally, the instructions received by the system RAM may be stored on a fixed disk either before or after execution by the CPU. Various forms of storage may be implemented, as well as network interfaces and network topologies necessary to implement the storage.

[0064] The above detailed description of the present technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the present technology to the precise form disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were selected to best explain the principles of the present technology and its practical application, and to enable others skilled in the art to utilize the present technology in various embodiments and with various modifications suitable for the particular use contemplated. It is intended that the scope of the present technology be defined by the claims.

Claims

1. 1. A method for gesture-based skill retrieval, comprising: capturing data regarding real-world actions including one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, the real-world actions being a representation of a virtual action in the virtual environment; identifying a subset of one or more available virtual actions based on a current level of progress of the user within the virtual environment, the subset of virtual actions being determined to be available at the current level of progress within the virtual environment; determining, based on matching the captured data regarding the real-world action with a data set associated with the identified virtual action, that an identified virtual action of the available virtual actions in the subset includes a representation of the one or more behaviors described by the captured data regarding the real-world action; obtaining data regarding an input combination associated with performing the identified virtual action in the virtual environment; generating a display of instructions corresponding to the identified virtual action based on the captured data, the displayed instructions relating to how the user may use a user device to cause an associated avatar to perform the identified virtual action representing the one or more performances of the real-world action in accordance with the input combination associated with the captured data; A method comprising:

2. 2. The method of claim 1, further comprising: storing a map in a memory relating to a plurality of virtual actions, the stored map further storing data relating to the instructions corresponding to the identified virtual actions, and wherein generating the representation of the instructions is based on the stored map.

3. The method of claim 1 , wherein the data regarding the real-world actions by the user in the real-world environment is captured by at least one of a camera, an accelerometer, a gyroscope, and an inertial sensor.

4. 2. The method of claim 1, wherein determining that the identified virtual action of the available virtual actions in the subset includes the representation of the one or more movements described by the captured data is further based on a difference between the captured data and the data set associated with the identified virtual action being within a predetermined threshold.

5. 2. The method of claim 1, wherein determining that the identified virtual action of the available virtual actions in the subset includes the representation of the one or more behaviors described by the captured data comprises filtering the available virtual actions in the subset based on one or more conditions associated with a search trigger input or the current level of progress.

6. The method of claim 5 , further comprising identifying the condition based on one or more activity files associated with the search trigger input or the current level of progress.

7. 3. The method of claim 2, wherein the stored map is specific to at least one of a game title, a collection of game titles, a game genre, a game developer, a game console, a game controller, a collection of controller variants, a game environment, and an in-game activity.

8. The method of claim 2 , wherein the map is stored with a plurality of different maps, the method further comprising identifying the map based on the virtual environment of the user.

9. 1. A system for gesture-based skill retrieval, comprising: Memory, a processor for executing instructions stored in said memory; wherein the processor: instructions to initiate capture of data regarding real-world actions, including one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, the real-world actions being a reproduction of a virtual action in the virtual environment; instructions for identifying a subset of one or more available virtual actions based on a current level of progress of the user within the virtual environment, the subset of virtual actions being determined to be available at the current level of progress within the virtual environment; instructions for determining, based on matching the captured data regarding the real-world action with a data set associated with the identified virtual action, that an identified virtual action of the available virtual actions in the subset includes a representation of the one or more behaviors described by the captured data regarding the real-world action; instructions for obtaining data regarding an input combination associated with performing the identified virtual action in the virtual environment; instructions for generating a display of instructions corresponding to the identified virtual action based on the captured data, the displayed instructions relating to how the user may use a user device to cause an associated avatar to perform the identified virtual action representing the one or more performances of the real-world action in accordance with the input combination associated with the captured data; Run the system.

10. 10. The system of claim 9, wherein the memory stores a map relating to a plurality of virtual actions, the stored map further storing data relating to the instructions corresponding to the identified virtual actions, and the processor generates the representation of the instructions based on the stored map.

11. The system of claim 9 , wherein the real-world action comprises a movement that visually corresponds to the identified virtual action being performed in the virtual environment.

12. 10. The system of claim 9, further comprising a communications interface, wherein the data regarding the real-world actions by the user in the real-world environment is captured by at least one of a camera, an accelerometer, a gyroscope, and an inertial sensor in communication with the processor via the communications interface.

13. 10. The system of claim 9, wherein the processor determines that the identified virtual action of the available virtual actions in the subset includes the representation of the one or more movements described by the captured data based further on a difference between the captured data and the data set associated with the identified virtual action being within a predetermined threshold.

14. 10. The system of claim 9, wherein the processor determines that the identified virtual action of the available virtual actions in the subset includes the representation of the one or more behaviors described by the captured data by filtering the available virtual actions in the subset based on one or more conditions associated with a search trigger input or the current level of progress.

15. The system of claim 14 , wherein the processor further executes instructions to identify the condition based on one or more activity files associated with the search trigger input or the current level of progress.

16. 11. The system of claim 10, wherein the stored maps are specific to at least one of a game title, a collection of game titles, a game genre, a game developer, a game console, a game controller, a controller variation, a game environment, and an in-game activity.

17. The system of claim 10 , wherein the map is stored with a plurality of different maps, the system further comprising identifying the map based on the virtual environment of the user.

18. A non-transitory computer-readable storage medium having embodied thereon a program executable by a processor to perform a method for gesture-based skill retrieval, the method comprising: capturing data regarding real-world actions including one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, the real-world actions being a representation of a virtual action in the virtual environment; identifying a subset of one or more available virtual actions based on a current level of progress of the user within the virtual environment, the subset of the one or more virtual actions being determined to be available at the current level of progress within the virtual environment; determining, based on matching the captured data regarding the real-world action with a data set associated with the identified virtual action, that an identified virtual action of the available virtual actions in the subset includes a representation of the one or more behaviors described by the captured data regarding the real-world action; obtaining data regarding an input combination associated with performing the identified virtual action in the virtual environment; generating a display of instructions corresponding to the identified virtual action based on the captured data, the displayed instructions relating to how the user may use a user device to cause an associated avatar to perform the identified virtual action representing the one or more performances of the real-world action in accordance with the input combination associated with the captured data; A non-transitory computer readable storage medium comprising:

19. The method of claim 1 , further comprising identifying a search trigger for initiating a gesture-based search for the input combination, wherein the data regarding the real-world action is captured based on the identified search trigger.

20. The method of claim 1 , further comprising generating a menu of the subset of virtual actions in a graphical user interface, wherein identifying the identified virtual action is further based on a selection by the user from the menu.

Images