Gesture-based Skill Search

Abstract

To provide a method for gesture-based skill search.SOLUTION: A map regarding one or more virtual actions may be stored in memory. The virtual actions may be associated with a set of data associated with performing a corresponding action in a real-world environment. Data regarding an action by a user in the real-world environment may be captured. A current progress level of the user within a virtual environment is identified. The current identified progress level is associated with one or more available virtual actions. The captured data may correspond to an identified one of the available virtual actions based on a match between the captured data and the set of data associated with the identified virtual action as indicated by the map. A search of one or more databases for instructions corresponding to the identified virtual action is initiated.SELECTED DRAWING: None

Description

Technical Field

[0001] The present invention generally relates to the search of network devices. More specifically, the present invention relates to gesture-based search of in-game skills.

Background Art

[0002] Description of Related Art Currently available digital game titles correspond to various different in-game scenarios where players can engage in various different types of activities (through their respective avatars or characters) to exercise various in-game skills. For example, in a racing game title, a player's driving skills can be tested, while in a first-person shooting game title, a player's aiming accuracy can be tested, and in other combat game titles, a player's skills with weapons such as swords, knives, axes, hammers, shovels, magical effects, etc. can be tested. Similarly, in various virtual terrains, a player may be required to control and navigate their avatar through swimming, mountain climbing, jumping, and other actions within the game environment.

[0003] Players can usually use one or more types of game controllers to control their avatars. Currently, there are various different controllers (and controller variants) with various buttons, touch pads, sensors, microphones, etc. in various configurations and layouts. To perform various types of in-game movements, operations, or other in-game actions, various combinations of user inputs (e.g., a series or simultaneous button presses, touch pad gestures or other gestures, verbal commands, or other inputs) may be required. Since different game titles may include different activities, certain combinations of inputs may result in different in-game movements. Additionally, incorrect input of a combination of inputs may result in in-game actions different from what the player intended.

[0004] Such diversity and complexity of in-game actions and controllers pose an entry barrier when a player is introduced to a new or unfamiliar game title, controller, or game console system. Thus, novice or inexperienced players may find gameplay difficult and frustrating without help or other types of guidance. However, searching for such help or guidance can become complicated due to a language barrier that may arise when the player is not familiar with the vocabulary or names of the virtual elements in the in-game environment. The player may not even know how to perform such a search. For example, just watching another player perform an action (such as whether to use an in-game object) may not necessarily convey to the player the name of the action or object, or how to perform such an action themselves.

[0005] Accordingly, there is a need in the art for improved systems and methods for gesture-based searching of in-game skills. SUMMARY OF THE INVENTION

[0006] Embodiments of the present invention include a system and method for gesture-based skill searching. A memory stores a map for one or more virtual actions, each virtual action being associated with a data set related to performing a corresponding action in the real-world environment. A gesture-based search server captures data regarding an action by a user in the real-world environment. The gesture-based search server identifies a current progress level of the user within the virtual environment, which is associated with one or more available virtual actions. The gesture-based search server determines, based on a matching of the captured data with the data set associated with the identified virtual action as indicated by the map, whether the captured data corresponds to the identified virtual action among the available virtual actions. The gesture-based search server begins searching for instructions corresponding to the identified virtual action in one or more databases. BRIEF DESCRIPTION OF THE DRAWINGS

[0007]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

DETAILED DESCRIPTION OF THE INVENTION

[0008] Embodiments of the present invention include a system and method for gesture-based skill search. The memory stores a map regarding one or more virtual actions, and each virtual action is associated with a data set related to performing a corresponding action in the real-world environment. The gesture-based search server captures data regarding an action by a user in the real-world environment. The gesture-based search server identifies the current progress level of the user within the virtual environment, which is associated with one or more available virtual actions. The gesture-based search server determines, based on a matching of the captured data with the data set associated with the identified virtual action as indicated by the map, that the captured data corresponds to the identified virtual action among the available virtual actions. The gesture-based search server begins to search for instructions corresponding to the identified virtual action in one or more databases.

[0009] FIG. 1 shows an exemplary network environment in which a system for gesture-based skill search can be implemented. The network environment 100 can include one or more content source servers 110 that provide digital content for distribution (such as games, other applications, and services), one or more content provider server application program interfaces (APIs) 120, a content delivery network server 130, a gesture-based search server 140, and one or more user devices 150A - 150N. The servers described herein can include any type of server well-known in the art that includes standard hardware computing components such as a network interface and a media interface, a non-transitory computer-readable storage (memory), and a processor for executing instructions that can be stored in the memory or accessing information that can be stored in the memory. The functions of multiple servers can be integrated into a single server. Any of the aforementioned servers (or the integrated server) can exhibit specific client-side characteristics, cache characteristics, or proxy server characteristics. These characteristics can depend on the specific network arrangement of the server or the specific configuration of the server.

[0010] The content source server 110 can maintain and provide various digital contents that can be distributed. The content source server 110 can be associated with any content provider, and the content provider makes its own content accessible via a communication network. Such contents can include not only digital videos and games but also other types of digital applications and services. Such applications and services can include any various different digital contents and functions that can be provided to the user device 150. Examples of the content source server 110 can include Internet websites that provide downloadable content and / or streaming content. The content provided by the content source server 110 can include any type of multimedia content such as movies, games, static / dynamic content, photos, social media content, social media websites, etc. The user device 150 can include a plurality of different types of computing devices. In some embodiments, the content data is transmitted from the content source server 110 to the computing device, and then the content data is rendered by the computing device into a format suitable for use by the user device 150.

[0011] Content from the content source server 110 can be provided via the content provider server API 120, thereby enabling various types of content source servers 110 to communicate with other servers (such as user devices 150) within the network environment 100. The content provider server API 120 can be specialized for the content source server 110 that provides the content, as well as for the specific language, operating system, protocol, etc. of the user device 150. In the network environment 100 including a plurality of different types of content source servers 110, there may similarly be a corresponding number of content provider server APIs 120, thereby enabling various formatting, conversion, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150, and different user devices 150 can process such content using different operating systems, protocols, etc. Thus, applications and services in different formats can be made available to be compatible with various different user devices 150.

[0012] The content source server 110 may also include an online server associated with a social media application, and the social media application can be programmed to provide one or more social media graphs to the user to identify the user's social contacts. These users may not be social contacts or may not be socially associated with the user, but may have played one or more video games with the user. The friend list may include additional information about the user's friends, such as a description of the games owned by each friend, identification of the friend's online status (e.g., online, offline, inactive, etc.), the friend's most recent login and its duration, the game the friend last played, etc. The social network includes user data, which includes data such as the user's social graph, posts, photos, videos, historical information, etc.

[0013] The content provider server API 120 may further facilitate access by each of the user devices 150, directly or via the content delivery network server 130, to the content hosted by the content source server 110 or the services provided thereby. Additional information regarding the accessed content or services, such as metadata, may also be provided by the content provider server API 120 to the user devices 150. As described below, the additional information (i.e., metadata) may be used to provide details regarding the content or services provided to the user devices 150. In some embodiments, the services provided from the content source server 110 to the user devices 150 via the content provider server API 120 may include services associated with other content or services, such as chat services, ratings, and profiles associated with specific games, teams, communities, etc. In such cases, the content source server 110 may also communicate with each other via the content provider server API 120.

[0014] The content delivery network server 130 may include servers that provide resources, files, etc. related to the content from the content source server 110, including various content configurations and service configurations, to the user devices 150. The content delivery network server 130 may also be invoked by the user devices 150 that request access to specific content or services. The content delivery network server 130 may include general-purpose management servers, game servers, streaming media servers, servers that host downloadable content, and other content delivery servers known in the art.

[0015] The gesture-based search server 140 may include any data server known in the art that can receive data from the user device 150. The content rendered by the gesture-based search server 140 may be essentially for any type of computer application, and may include one or more types of content such as games, movies, audio, images, multimedia, etc. In some embodiments, the content or a portion of the content is generated by the gesture-based search server 140. In some embodiments, the content or a portion of the content is streamed from the content source server 110 to the computing device via the network 100. In some embodiments, the content or a portion of the content is streamed from the cloud game infrastructure to the computing device via the network 100. The infrastructure may be instructed to send various types of content from the content source server 110 to the computing device via the network 100.

[0016] In an exemplary embodiment, the gesture-based search server 140 may store or access a store that includes search maps for one or more game titles. Each game title may be associated with a different search map, and the search maps may include information about in-game actions available in each game title (e.g., swinging a sword within a game environment), information about how the in-game actions correspond to real-world equivalent actions (e.g., swinging an object within a real-world environment), and a combination (s) of related inputs associated with initiating an action within the game environment using one or more types of controllers. Some game titles may further restrict the types of actions available at various levels, activities, or other portions of the associated in-game environment. Such restrictions may be stored in an activity file generated by the UDS system (described in more detail in connection with FIG. 2), and may further be used to filter searches by search users who are actively involved in playing a particular game title in a particular portion of the game environment or game play session.

[0017] Gesture-based skill search can be initiated using various triggers that can be customized according to the user. For example, a verbal command, a key-based command, or a gesture command can indicate that the user wants to initiate such a gesture-based skill search. Such commands can trigger cameras and / or other sensors (e.g., of user device(s) 150) to capture images and / or other sensor data regarding the real-world environment where the user is located. For example, a camera can capture an image of a user waving an arm in the real world, and an avatar swinging a sword is reproduced in the digital environment. The camera (discussed in more detail in relation to FIG. 4) can be configured to include multiple image capture devices such as a stereo pair of cameras, an infrared camera, a depth camera, or a combination thereof. In some embodiments, one or more microphones (further described in more detail in relation to FIG. 4) can also be used to capture audio from the user's real-world environment. On the other hand, other sensors (e.g., accelerometers, gyroscopes) associated with the user (e.g., via a controller, a mobile device, a wearable device, a handheld device, etc.) can also capture data regarding real-world actions.

[0018] Such image, audio, and other sensor data can be provided to the gesture-based search server 140 for analysis. The gesture-based search server 140 can use the stored map to match received sensor data regarding real-world actions with in-game actions available within the relevant game title. In some embodiments, the received sensor data can indicate multiple available in-game actions, and in such cases, different parameters can be used to filter the available in-game actions. As described above, in some game titles, various in-game actions are possible at various parts of the game environment or game session. Such data is provided in an activity file and can be used to filter available in-game actions and identify in-game actions that may be available in the identified part of the game environment or game session currently associated with the searching user. Alternatively or additionally, a menu regarding multiple available in-game actions can be displayed on the graphical user interface for selection by the searching user.

[0019] When an in-game action is identified or selected, the graphical user interface can be updated by the gesture-based search server 140 to include a display of the controller associated with the user and instructions regarding how to use the controller to execute the identified in-game action. Such instructions can include visual instructions, audio instructions, and / or text-based instructions. In some embodiments, the instructions can be provided as an overlay within a pop-up window. Alternatively, they can be provided to the user device 150 during the game play session while the user is still engaged in game play within the game environment.

[0020] Therefore, the gesture-based search server 140 can map data regarding real-world actions to one or more in-game actions, and then further map a specified one of the in-game actions to input combinations and related instructions. Accordingly, a user who is actively playing a game can initiate a search performed by the gesture-based search server 140 without leaving the in-game environment (e.g., to perform a conventional online search), without pausing or ending the current game play session, or without switching to another device or platform.

[0021] The user device 150 can be a server that provides internal services within the network environment 100 (e.g., to other servers). In such a case, the user device 150 can correspond to one of the content source servers 110 described herein. Alternatively, the user device 150 can be a client device that can include any number of different game consoles, mobile devices, laptops, and desktops. Such a user device 150 can also be configured to access data on other storage media, such as a memory card or disk drive, which may be suitable, for example, in the case of downloaded services. Such a user device 150 can include standard hardware computing components, such as, but not limited to, a network interface, a media interface, a non-transitory computer-readable storage (memory), and a processor that can execute instructions stored in the memory. These user devices 150 can also be executed using a variety of different operating systems (e.g., iOS, Android (registered trademark)), applications, or computing languages (e.g., C++, JavaScript (registered trademark)). Exemplary client devices 150 are described in detail herein with respect to FIG. 5. Each user device 150 can be associated with a participant in a collection of digital content streams or other types of viewers.

[0022] In some embodiments, the user device 150 is configured to execute games locally on the processing hardware of a computing device or a game console. The game or content can be obtained in any form, such as in the form of physical media (e.g., digital disks, tapes, cards, thumb drives, solid state chips, or solid state cards, etc.), or by downloading from the Internet via a communication network. In some embodiments, the computing device functions as a client in communication with a cloud game infrastructure via a communication network. The cloud game infrastructure can maintain and execute the video game being played by the user device 150. Inputs received from the user device 150, the controller, and the camera (and other sensors) can be sent to the cloud game infrastructure, and the cloud game infrastructure processes the inputs and reflects them in the game state of the running video game.

[0023] Game data from a video game, such as video data, audio data, and tactile feedback data, can be sent from the content source server 110 to the user device 150. The computer system may further process the game data before sending it to a suitable device, or send the game data directly to a suitable device. For example, video and audio streams can be sent to the user device 150.

[0024] The user device 150 may also include, or be associated with, a controller configured to receive inertial sensor data from an inertial sensor within the user device 150. The inertial sensor data indicates the movement of the controller and / or the user device 150 in accordance with the movement of the user associated with the user device 150. The movement of the user is based on a virtual reality scene displayed within the user device 150. From the inertial sensor data, the path or trajectory of the movement of the user device 150 and the speed of the movement can be determined. In some embodiments, the path or trajectory of the movement of the head-mounted display corresponds to one or more of the movements of the user within a set of movements of the user including forward tilt, backward tilt, left tilt, right tilt, head left rotation, head right rotation, head upward tilt, head downward tilt, squat, and jump. However, in other embodiments, the path or trajectory of the movement of the head-mounted display can essentially correspond to any movement of the user within the movement capabilities of the human body.

[0025] In some embodiments, within a virtual reality (VR) or augmented reality (AR) scene displayed on the user device 150 or another computing device, such as a television, computer, etc., the user device 150 can be used to manipulate virtual objects, such as grab, move, push, pull, etc. When the user moves their hand while wearing the user device 150, the virtual hand in the game moves. Further, when the user moves their finger while wearing the user device 150, the fingers of the virtual hand in the game move. The position and / or orientation of the finger is determined from the image data captured using the cameras described above and in FIG. 5, and the movement of the fingers of the virtual hand is generated. Examples of one or more virtual objects include a virtual vehicle, the user's virtual hand, a virtual user, a virtual sword, the user's avatar, virtual fingers, virtual finger joints, a virtual tree, virtual flowers, etc.

[0026] The display of the user device 150 can generate a three-dimensional scene incorporating an avatar in a virtual space or virtual environment having virtual interactive objects. The gesture-based search server 140 receives visual information and / or distance information regarding the user captured by the user device 150. The gesture-based search server 140 extracts motion information describing the user's movement. The motion information can be encoded to describe the movement of the user's "skeleton" composed of key points within the user's body. The gesture-based search server 140 can enable content enhancement by enabling the introduction of user-related information such as the user's avatar, identifier, representative image, etc. into the VR space presented to the user's user devices 150A to 150N.

[0027] The user's avatar can be automatically converted from a single digital image into an animated 3D avatar. Specifically, the user can upload a digital image or other image to the user device 150 and in return receive one or more avatars featuring various facial expressions and / or animations. The avatar is displayed to and used by the user as any still image or animated image (e.g., in GIF format). For example, the avatar can be sent to other users via SMS, MMS, email messages, chat (such as Facebook (registered trademark) chat), instant messengers (such as Skype or Windows (registered trademark) Messenger), Twitter (registered trademark), blogs, forums, or other electronic communication means for the purpose of displaying the user's sentiment.

[0028] In some embodiments, the avatar may be based on the user in a realistic manner, for example, having the same facial features, clothing, and body type. In other embodiments, the avatar may be intentionally made unrealistic, for example, by mapping the user's movements to the facial features of a celebrity, movie star, politician, video game character, or another user. The avatar can be a blend of realistic and intentionally unrealistic elements. For example, the user's face can be used for the avatar, but different clothing or a different body type can be given to the avatar. Alternatively, the user's body can be used for the avatar, but the face of a celebrity, movie star, politician, video game character, or another user can be given to the avatar.

[0029] In an embodiment, a user of a user device 150A, such as a Sony (registered trademark) Playstation 3 (registered trademark) entertainment device (PS3), navigates within an instance of one of a number of available virtual 3D environments from an online server, and each virtual 3D environment has a relatively small percentage of the total number of other users currently online. Within this environment, these users are represented by avatars, which enables social interaction with other users on other user devices 150B - 150N and social interaction with the virtual environment itself, and the virtual environment can provide facilities such as a billiard table, bowling alley, arcade games, and streaming online content on a TV screen.

[0030] User device 150A transmits information describing the user's movement and / or the corresponding avatar's movement to other user devices 150B - 150N or to the gesture-based search server 140. The gesture-based search server 140 then sends this information to other user devices 150B - 150N, enabling other user devices 150B - 150N to generate a scene characterized by the accurate movement of the avatar. User device 150A receives information describing the movement of other users and / or the corresponding avatar's movement from other user devices 150B - 150N or from an intermediary device such as the gesture-based search server 140. When the gesture-based search server 140 receives this information from other user devices 150B - 150N, it sends this information to the first user device 150A, enabling the first user device 150A to generate a scene characterized by the accurate movement of the second avatar.

[0031] The user associated with user device 150A can be matched with other user devices 150B - 150N within the virtual reality environment based on the characteristics of each user's physical environment. In one embodiment, information regarding the characteristics of the physical space where each user is located can be received by the gesture-based search server 140. Using the information regarding the characteristics of each user's physical space, users located in different physical spaces with similar characteristics can be matched. Users are matched to participate in a virtual multiplayer entertainment experience. This matching can be based on the characteristics of the physical space where each user is located. For example, the matching can be performed based on the degree of similarity of the characteristics of the physical space where each user is located.

[0032] The user device 150 or computing device generates a representation of the captured user movement and causes the generated representation of the user movement to be executed on an avatar. The user device 150 also generates a representation of the received second user movement and causes the generated representation of the second user movement to be executed on a second avatar. The user device 150 also updates the virtual space or virtual environment, and any virtual interactive objects, as needed.

[0033] Any relevant events within the virtual environment can be communicated between user devices 150 using a common coordinate system so that events within the virtual environment are referenced with respect to a physical reference position. To establish the common coordinate system, only a single instance of position data needs to be exchanged between user devices 150, and indeed this can be in fact a one-way transfer of data (one user device 150A communicates to another user device 150B the position and orientation of the first user device 150A with respect to the second user device 150B). However, it may be more reliable if the user devices 150 exchange the position data multiple times, for example frequently during a session of virtual interaction in a shared coordinate system. Subsequent relevant movements of the user devices 150 in the real environment can be reproduced in the virtual environment. Thus, it will be understood that the actual position and the virtual position of the user device substantially coincide.

[0034] Subsequent movements away from the common reference position are tracked by the gesture-based search server 140 (functioning as a movement tracking structure) using the accelerometers and gyroscopes (movement sensors) of the user device 150, and cumulative changes in displacement and orientation with respect to the common reference position can be maintained. The relationships between acceleration, velocity, and displacement (both lateral and rotational) are well known and will not be described in detail here. The accelerometers and gyroscopes may be mentioned separately or may be mentioned together as movement sensors, but it will be understood that the accelerometers and gyroscopes can also be considered together as accelerometers, the main difference being the lateral or rotational nature of the detected acceleration.

[0035] FIG. 2 shows an exemplary Uniform Data System (UDS) 200 that can be used to provide data to a system for gesture-based skill search. Based on the data provided by the UDS, the gesture-based search server 140 can recognize in-game objects, entities, activities, and events that a user has engaged in, and thus can handle the analysis of in-game activities and filtering related to in-game activities. Each user interaction can be associated with metadata such as the type of in-game interaction, the location in the in-game environment, the point in time on the in-game timeline, and other players, objects, entities, etc. involved. Thus, metadata can be tracked for any of the various user interactions that can occur during a game session, including related activities, entities, settings, results, actions, effects, locations, and character statistics. Such data can be further aggregated, applied to a data model, and made available for analysis. Using such a UDS data model, context information can be assigned to each part of the information in a unified manner across the game.

[0036] As shown in FIG. 2, an exemplary console 228 (e.g., user device 150) and an exemplary server 218 (e.g., streaming server 220, activity feed server 224, user-generated content (UGC) server 232, and object server 226) are shown. In one embodiment, the console 228 can be implemented on, or in association with, the content source server 110, the content delivery network server 130, the cloud server, or can be implemented as any combination of the server 218 and the servers in FIG. 1. In an exemplary embodiment, the content recorder 202 can receive content (e.g., media) from the interactive content title 230 and record it in the content ring buffer 208. Such a ring buffer 208 can store a plurality of content segments (e.g., v1, v2, and v3), the start time of each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and the end time of each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments can be stored by the console 228 as a media file 212 (e.g., MP4, WebM, etc.). Such a media file 212 can be uploaded to the streaming server 220 for storage and subsequent streaming or use, but the media file 212 can also be stored on any server, cloud server, any console 228, or any user device 150. Such start times and end times of each segment can be stored by the console 228 as a content timestamp file 214. Such a content timestamp file 214 can also include a streaming ID, which matches the streaming ID of the media file 212, whereby the content timestamp file 214 is associated with the media file 212.Such a content time stamp file 214 can be uploaded and stored in the activity feed server 224 and / or the UGC server 232, but the content time stamp file 214 may be stored in any server, cloud server, any console 228, or any user device 150.

[0037] While the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to identify the start and end times of the object. The object library 204 and the object recorder 206 can be implemented on any of the platform server 120, the cloud server, or the server 218. When the object recorder 206 detects the start of an object, the object recorder 206 receives object data (e.g., when the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity result, activity type, etc.) from the object library 204 and records activity data (e.g., ActivityID1,START_TS, ActivityID2,START_TS, ActivityID3,START_TS) in the object ring buffer 210. Such activity data recorded in the object ring buffer 210 can be stored in the object file 216. Such an object file 216 can also include the activity start time, activity end time, activity ID, activity result, activity type (e.g., confrontation match, quest, task, etc.), user data or peer data related to the activity. For example, the object file 216 can store data related to items used during the activity. Such an object file 216 can be stored in the object server 226, but the object file 216 can be stored on any server, cloud server, any console 228, or any user device 150.

[0038] Such object data (e.g., object file 216) can be associated with content data (e.g., media file 212 and / or content timestamp file 214). In one embodiment, UGC server 232 stores content timestamp file 214 and associates content timestamp file 214 with object file 216 based on a matching of the streaming ID of content timestamp file 214 and the corresponding activity ID of object file 216. In another embodiment, object server 226 can store object file 216 and can receive a query regarding object file 216 from UGC server 232. Such a query can be executed by searching for an activity ID of object file 216 that matches the streaming ID of content timestamp file 214 sent with the query. In yet another embodiment, a query of stored content timestamp file 214 can be executed by matching the start time and end time of content timestamp file 214 with the corresponding start time and end time of object file 216 sent with the query. Such object file 216 can also be associated by UGC server 232 with the matched content timestamp file 214, although the association can be performed by any server, cloud server, any console 228, or any user device 150. In another embodiment, object file 216 and content timestamp file 214 can be associated by console 228 during the creation of each file 216, 214.

[0039] The activity file generated by UDS200 can be provided or accessed by the gesture-based search server 140 and is used to analyze, filter data related to real-world actions, and match them with in-game actions on one or more maps. For example, the activity file may include gameplay data related to a specific gameplay session of a specific game title by a specific peer player who executes one or more in-game actions when the search user starts a search. The gesture-based search server 140 can use the data from the activity file to identify the game title, player, in-game action being executed, and the temporal proximity of the in-game action to the search trigger (e.g., for searching in real-time or near real-time). The identified parameters or conditions during the search can be used by the gesture-based search server 140 to filter relevant search results.

[0040] Figure 3 is a flowchart showing an exemplary method for gesture-based skill search. The method 300 of Figure 3 can be embodied as executable instructions in a non-transitory computer-readable storage medium including, but not limited to, non-volatile memory such as a CD, DVD, or hard drive. The instructions of the storage medium can be executed by one or more processors to cause the various hardware components of a computing device hosting or accessing the storage medium to implement the method. The steps (and order of steps) specified in Figure 3 are exemplary and can include various alternatives, equivalents, or derivatives of the steps, including, but not limited to, the order of execution of the steps.

[0041] In step 310, a map regarding one or more virtual actions is stored in the memory. Each virtual action (for example, an action executed within a virtual environment) is associated with a data set related to the execution of the corresponding action in the real-world environment and with instruction data regarding the input combinations that result in the performance of the virtual action. Various variations of the map can be associated with a specific game title, a set of game titles, a game genre, a game developer, a game console, a game controller, a set of modified controller forms, a game environment, in-game activities, etc.

[0042] In step 320, data regarding the movement of a user within the real-world environment is captured. The movement of the user within the real-world environment can be indicated by real-world environment data captured by the user device 150 (and related sensors). The data can be image data captured by the camera of the user device 150. In another embodiment, the data can be captured by inertial sensors on the user device 150. Along with one or more images and the image data of the user interacting with the real-world environment captured by the user device 150, the real-world environment data can be transferred to the gesture-based search server 140.

[0043] In step 330, the current progress level of the user within the virtual environment can be determined based on one or more activity files associated with the current gameplay by the user. The currently determined progress level is associated with one or more available virtual actions. The user can be matched with one or more of other users based on the user's determined progress level and the progress levels of other users. Activity files associated with other users can also be obtained and used to analyze the real-world movement data captured in step 320.

[0044] In step 340, it is determined, based on a matching of the captured data with a data set associated with a specified virtual action among the available virtual actions, as indicated by a map, that the captured data corresponds to the specified virtual action. In an embodiment, the difference between the captured data and the data set associated with the specified virtual action is indicated by the map. If the difference is within a predetermined threshold, it may be determined that the captured data corresponds to the specified available virtual action. In an embodiment, the available virtual actions are filtered based on other users who correctly execute the virtual actions, and one or more available virtual actions are based on the filtered users.

[0045] In step 350, a search for instructions corresponding to the specified virtual action is initiated in one or more databases. In some embodiments, the instructions may be stored in the map itself. The map may also have a link to another database in which the instructions are held, or may refer to another database. Thus, the instructions for the specified in-game action can be obtained and provided to the user device 150.

[0046] FIG. 4 is a block level diagram of an exemplary user device that can be used to perform gesture-based skill search. It should be understood that depending on the effective configuration and functionality, more or fewer components than those shown in FIG. 4 may be included in or excluded from the user device 400. The user device 400 may include a processor 402 for executing program instructions. A memory 406 is provided for data storage purposes, and the memory 406 may include both volatile and non-volatile memory. A display 416 is included that provides a visual interface that can be viewed by the user. The display 416 can be defined as a single display or in the form of separate display screens for each eye. If two display screens are provided, it is possible to provide the left-eye video content and the right-eye video content separately. For example, by separately displaying video content for each eye, the sense of presence control of the three-dimensional content of the virtual reality scene can be improved.

[0047] The motion detection module 418 may include any of various types of motion detection hardware such as a magnetometer 420, an accelerometer 424, and a gyroscope 426. The user device 400 may be equipped with an inertial sensor 422, which is configured to generate inertial sensor data indicative of the motion of the user device 400 in accordance with the motion of the user associated with the user device 400. The magnetometer 420 measures the strength and direction of the magnetic field near the user device 400. In some embodiments, three magnetometers 420 are used within the user device 400 to ensure an absolute reference for the world space yaw angle. In some embodiments, the magnetometer 420 is designed to measure in the range of ±80 microtesla, which is the Earth's magnetic field. The magnetometer is affected by metal and provides a monotonic yaw measurement due to the actual yaw. The magnetic field may be distorted by metal in the environment, which causes distortion in the yaw measurement. If necessary, this distortion can be calibrated using information from other sensors such as a gyroscope or a camera. In some embodiments, the accelerometer 424 is used together with the magnetometer 420 to obtain the tilt angle and azimuth angle of the user device 400.

[0048] In some embodiments, the present invention may also include one or more external inertial sensors disposed on the user's body. The present invention may include an operation of comparing external inertial sensor data with inertial sensor data received from the inertial sensor 422 within the user device 400 to identify specific movements made by the user.

[0049] The accelerometer 424 is a device that measures acceleration and gravity-induced reaction forces. Uniaxial models and multi-axis (e.g., 6-axis) models can detect the magnitude and direction of acceleration in various directions. The accelerometer 424 is used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 424 are used to provide the direction of gravity, thereby providing an absolute reference for two angles (world space pitch and world space roll).

[0050] The gyroscope 426 is a device for measuring or maintaining orientation based on the principle of angular momentum. In one embodiment, three gyroscopes 426 provide information regarding movement across their respective axes (x, y, and z) based on inertial sensing. The gyroscope 426 helps in detecting high-speed rotation. However, the gyroscope 426 can drift over time without the presence of an absolute reference. For this reason, it is necessary to periodically reset the gyroscope 426, and the reset can be performed using other available information such as visual tracking of an object, position / orientation identification based on an accelerometer, magnetometer, etc.

[0051] A camera 404 is provided to capture images and image streams of the real-world environment exposed by the user device 400. A plurality of cameras 404, including a rear camera 404 (a camera that faces away from the user when the user is looking at the display of the user device 400) and a front camera 404 (a camera that faces the user side when the user is looking at the display of the user device 400), can optionally be included in the user device 400. In some embodiments, a camera 404 for sensing depth information of objects within the real-world environment exposed by the user device 400 can be included in the user device 400.

[0052] The user device 400 includes a speaker 412 for providing audio output. A microphone 414 may also be included to capture audio from the real-world environment, including sounds from the surrounding environment, speech by the user, etc.

[0053] A Wi-Fi (registered trademark) module 410 may be included to enable the user device 400 to connect to the Internet via wireless network technology. The user device 400 may also include a Bluetooth (registered trademark) module 408 to enable wireless connection to other devices.

[0054] The components of the user device 400 shown in FIG. 4 are examples of components that may be included in the user device 400, and it should be understood that they do not represent all possible components that can be included in the user device 400. For example, in various embodiments, the user device 400 may or may not include some of the components shown in FIG. 4. In some embodiments, the user device 400 may include additional components not shown in FIG. 4.

[0055] In an exemplary embodiment, a searching user can view in-game actions 430 being performed by an avatar (e.g., an avatar associated with a peer player) within a virtual environment. The in-game actions 430 shown in FIG. 4 can be jumps or leaps. After triggering a gesture-based search, the user can perform a corresponding gesture 440 in the real-world environment. The user can be holding, wearing, or within the detection range of one or more sensors of the user device 400. The user device 400 can provide sensor data regarding the user gesture 440 to the gesture-based search server 140 for analysis.

[0056] The gesture-based search server 140 can identify that a user gesture 440 characterized by sensor data is associated with one or more available in-game actions. Based on one or more activity files associated with the conditions that triggered the search, the available in-game actions can be further filtered. For example, the search may have been triggered during a particular game play session or in temporal proximity to a particular game play session. Thus, a subset of the filtered in-game actions can be identified. The gesture-based search server 140 can further identify that the identified or selected in-game action is mapped to a set of instructions corresponding to the input combination that causes the identified or selected in-game action. Thereafter, the gesture-based search server 140 can generate a combo display 440 based on the mapped set of instructions and provide the generated combo display 440 to the user device 150 for presentation and display.

[0057] FIG. 5 is a block diagram of an exemplary electronic entertainment system that can be used in an embodiment of the present invention. The entertainment system 500 can be an electronic game console. Alternatively, the entertainment system 500 can be implemented as a general-purpose computer, a set-top box, a handheld game device, a tablet computing device, a mobile computing device, or a cellular phone. The entertainment system can include more or fewer operating components depending on the particular form factor, purpose, or design.

[0058] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 in FIG. 5 communicate via the system bus 585. Further, the CPU 510 in FIG. 5 communicates with the main memory 505 via the dedicated bus 580, while the vector unit 515 and the graphics processing unit 520 may communicate via the dedicated bus 590. The CPU 510 in FIG. 5 executes programs stored in the OS ROM 555 and the main memory 505. The main memory 505 in FIG. 5 may include pre-stored programs and programs transferred via the I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disc (not shown) using the optical disc control unit 570. The I / O processor 525 in FIG. 5 may also enable the introduction of content transferred via a wireless or other communication network (such as 4G, LTE, and 3G, etc.). The I / O processor 525 in FIG. 5 mainly controls data exchange between various devices of the entertainment system 500 including the CPU 510, vector unit 515, graphics processing unit 520, and the controller interface 535.

[0059] The graphics processing unit 520 in FIG. 5 executes the graphics instructions received from the CPU 510 and the vector unit 515 to generate an image to be displayed on a display device (not shown). For example, the vector unit 515 in FIG. 5 may convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Further, the audio processing unit 560 executes instructions to generate an audio signal, and the audio signal is output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 500 via the USB interface 545 and the IEEE 1394 interface 550 such as a wireless transceiver, and these may be embedded within the system 500 or as part of some other component such as a processor.

[0060] The user of the entertainment system 500 of FIG. 5 provides commands to the CPU 510 via the controller interface 535. For example, the user may command the CPU 510 to store specific game information in the memory card 540 or other non-transitory computer-readable storage medium, or may command a character in the game to perform some specific action.

[0061] The present invention may be implemented in an application that may be operable by various end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., Sony PlayStation2 (registered trademark) or Sony PlayStation3 (registered trademark) or Sony PlayStation4 (registered trademark)), a portable game device (e.g., Sony PSP (registered trademark) or Sony Vita (registered trademark)), or a home entertainment system of a different manufacturer at a lower level. The methods of the present invention described herein are fully intended to be operable on various devices. The present invention may also be implemented with cross-title neutrality, and embodiments of the system may be utilized across various titles from various publishers.

[0062] The present invention may be implemented in an application that may be operable using various devices. A non-transitory computer-readable storage medium refers to any medium or media involved in providing instructions to a central processing unit (CPU) for execution. Such media may take many forms including, but not limited to, non-volatile media such as optical or magnetic disks and volatile media such as dynamic memory. Common forms of non-transitory computer-readable media include, for example, floppy (registered trademark) disks, flexible disks, hard disks, magnetic tape, any other magnetic medium, CD-ROM disks, digital video disks (DVDs), any other optical medium, RAM, PROM, EPROM, FLASH EPROM, and any other memory chip or cartridge.

[0063] Various forms of transmission media can be involved in communicating one or more sequences of one or more instructions to the CPU for execution. A bus conveys data to system RAM, and the CPU fetches and executes instructions from the system RAM. Optionally, the instructions received by the system RAM can be stored on a fixed disk either before or after execution by the CPU. Various forms of storage can be implemented, as well as the network interface and network topology necessary to implement the storage.

[0064] The foregoing detailed description of the technology has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations are possible in light of the above teachings. The described embodiments were chosen in order to best explain the principles of the technology and its practical application, and to enable others skilled in the art to utilize the technology in various embodiments and with various modifications suitable for the particular use contemplated. The scope of the technology is intended to be defined by the claims.

Claims

1. A method for gesture-based skill search, comprising: capturing data regarding a real-world action that includes one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, wherein the real-world action is a reproduction of a virtual action within the virtual environment; identifying a subset of one or more available virtual actions based on a current progress level of the user within the virtual environment, wherein the subset of virtual actions is determined to be available at the current progress level within the virtual environment; determining, based on a matching of the captured data regarding the real-world action and a data set associated with the identified virtual actions, that a particular virtual action among the available virtual actions within the subset includes an expression of the one or more actions described by the captured data regarding the real-world action; obtaining data regarding an input combination associated with executing the identified virtual action in the virtual environment; generating a display of instructions corresponding to the identified virtual action based on the obtained data, wherein the displayed instructions are related to a method of using the user's user device to cause an associated avatar to execute the particular virtual action that represents the one or more actions of the real-world action according to the input combination associated with the obtained data; A method comprising the above steps.

2. The method according to claim 1, further comprising storing a map in a memory regarding a plurality of virtual actions, wherein the stored map further stores data regarding the instructions corresponding to the identified virtual action, and wherein generating the display of the instructions is based on the stored map.

3. The method according to claim 1, wherein the data regarding the real-world action by the user in the real-world environment is captured by at least one of a camera, an accelerometer, a gyroscope, and an inertial sensor.

4. Determining that the identified virtual action among the available virtual actions within the subset includes the representation of the one or more operations described by the captured data is further based on that the difference between the captured data and the data set associated with the identified virtual action is within a predetermined threshold, according to the method of claim 1.

5. Determining that the identified virtual action among the available virtual actions within the subset includes the representation of the one or more operations described by the captured data includes filtering the available virtual actions within the subset based on one or more conditions associated with a search trigger input or the current progress level, according to the method of claim 1.

6. The method of claim 5 further includes specifying the conditions based on one or more activity files associated with the search trigger input or the current progress level.

7. The stored map is specific to at least one of a game title, a set of game titles, a game genre, a game developer, a game console, a game controller, a set of controller variants, a game environment, and in-game activities, according to the method of claim 2.

8. The map is stored together with a plurality of different maps, and the method of claim 2 further includes identifying the map based on the virtual environment of the user.

9. A system for gesture-based skill search, comprising: a memory; a processor configured to execute instructions stored in the memory; The processor is configured to: initiate capturing data regarding a real-world action including one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, wherein the real-world action is a reproduction of a virtual action within the virtual environment; identify a subset of one or more available virtual actions based on the current progress level of the user within the virtual environment, wherein the subset of virtual actions is determined to be available at the current progress level within the virtual environment; Instructions for determining that a specified virtual action among the available virtual actions within the subset includes an expression of the one or more actions described by the captured data regarding the real-world action, based on matching the captured data regarding the real-world action with a data set associated with the specified virtual action; Instructions for obtaining data regarding an input combination associated with executing the specified virtual action in the virtual environment; Instructions for generating a display of instructions corresponding to the specified virtual action based on the obtained data, wherein the displayed instructions are for causing an avatar associated with the specified virtual action representing the one or more actions of the real-world action to be executed according to the input combination associated with the obtained data, and relate to a method of using the user's user device, the generating instructions; A system for executing.

10. The memory stores a map regarding a plurality of virtual actions, the stored map further stores data regarding the instructions corresponding to the specified virtual action, and the processor generates the display of the instructions based on the stored map, the system according to claim 9.

11. The real-world action includes an action that visually corresponds to the specified virtual action executed within the virtual environment, the system according to claim 9.

12. Further comprising a communication interface, and the data regarding the real-world action by the user in the real-world environment is captured by at least one of a camera, an accelerometer, a gyroscope, and an inertial sensor that communicates with the processor via the communication interface, the system according to claim 9.

13. The processor further determines that the specified virtual action among the available virtual actions within the subset includes the expression of the one or more actions described by the captured data, based further on that a difference between the captured data and the data set associated with the specified virtual action is within a predetermined threshold, the system according to claim 9.

14. The system according to claim 9, wherein the processor discriminates that the specified virtual action among the available virtual actions in the subset includes the representation of the one or more operations described by the captured data by filtering the available virtual actions in the subset based on a search trigger input or one or more conditions associated with the current progress level.

15. The system according to claim 14, wherein the processor further executes an instruction to specify the condition based on the search trigger input or one or more activity files associated with the current progress level.

16. The system according to claim 10, wherein the stored map is specific to at least one of a game title, a set of game titles, a game genre, a game developer, a game console, a game controller, a controller variant, a game environment, and in-game activities.

17. The system according to claim 10, wherein the map is stored together with a plurality of different maps, and the system further includes specifying the map based on the virtual environment of the user.

18. A non-transitory computer-readable storage medium in which a program executable by a processor for executing a method of gesture-based skill search is embodied, the method comprising: capturing data regarding real-world actions including one or more actions by a user in a real-world environment during an interactive session associated with a virtual environment, the real-world actions being a reproduction of virtual actions within the virtual environment; identifying a subset of one or more available virtual actions based on the current progress level of the user within the virtual environment, the subset of the one or more virtual actions being determined to be available at the current progress level within the virtual environment; Determining that a specified virtual action among the available virtual actions within the subset includes an expression of the one or more operations described by the captured data regarding the real-world action, based on a matching of the captured data regarding the real-world action and a data set associated with the specified virtual action; Obtaining data regarding an input combination associated with executing the specified virtual action in the virtual environment; Generating a display of instructions corresponding to the specified virtual action based on the obtained data, wherein the displayed instructions relate to a manner of use of the user's user device for causing the associated avatar to execute the specified virtual action representing the one or more operations of the real-world action, in accordance with the input combination associated with the obtained data; A non-transitory computer-readable storage medium including the above.

19. The method according to claim 1, further comprising identifying a search trigger for initiating a gesture-based search for the input combination, wherein the data regarding the real-world action is captured based on the identified search trigger.

20. The method according to claim 1, further comprising generating a menu of the subset of virtual actions within a graphical user interface, wherein the identifying of the specified virtual action is further based on a selection by the user from the menu.

Images