Sharing of operational data

Abstract

To provide a system and method for capturing, analyzing, and sharing data regarding physical movement of a user.SOLUTION: A method for analyzing and sharing movement data includes: receiving data captured during a movement by a user from a user device associated with the user; rendering a corresponding movement by a virtual character in a virtual environment based on the data; and capturing a video of the corresponding movement in a virtual reality environment, to be associated with the data. The captured video and the data are provided to a recipient designated by the user. Data regarding the recipient reproducing the captured video is compared to the data associated with the captured video to verify whether the recipient is performing the movement.SELECTED DRAWING: Figure 3

Description

Technical Field

[0001] The present invention generally relates to the sharing of motion data. More specifically, the present invention relates to capturing, analyzing, and sharing data related to a user's physical motion.

Background Art

[0002] Currently available user devices include cameras that enable the capture of videos and the sharing of such videos via various online social networks (e.g., TikTok (registered trademark)). However, a user's physical motion may be characterized by more parameters than can be captured by just a video. This is because a video only shows how the end result of the motion looks. For example, there is no analysis or other type of data that can help viewers understand the mechanism, rhythm, or force behind the motion, or the method of performing such motion accurately.

[0003] In various computing systems such as video game systems and other virtual reality systems, it is possible to reproduce some elements of user motion within a virtual environment or other digital environment. Such systems may rely on controllers and other sensors to capture motion data and convert such motion data into instructions for controlling an avatar or character within the virtual environment. For example, certain gesture controls may be converted into various actions by an avatar when navigating and interacting with the virtual environment. The avatar or character may be viewed by other users or players within the virtual environment, but the motion data underlying the motion is not shared with others. Similar to a video captured by a camera, viewers do not receive analysis or instructions of the motion.

[0004] Therefore, there is a need in the art for improved systems and methods for capturing, analyzing, and sharing data related to a user's physical motion. [Overview of the project]

[0005] Embodiments of the present invention include a system and method for sharing motion data. A user device associated with a user communicates with a content management server. The content management server receives data captured from the user device during actions performed by the user in a real-world environment. Based on the data, the content management server renders the corresponding actions by a virtual character in a virtual environment. The content management server captures a video of the corresponding actions in the virtual reality environment associated with the data. The content management server provides the captured video and data to a recipient designated by the user. The content management server verifies that the recipient is performing the action by comparing data about the recipient during playback of the captured video with data associated with the captured video. [Brief explanation of the drawing]

[0006] [Figure 1] This illustrates an exemplary network environment in which a system for analyzing and sharing operational data may be implemented. [Figure 2] This document presents an example of a uniform data system (UDS) that can be used to analyze operational data related to virtual environments. [Figure 3] This flowchart shows an exemplary method for analyzing and sharing motion data. [Figure 4] This is an exemplary block-level diagram of a user device that can be used to capture operational data. [Figure 5] This is a block diagram of an exemplary electronic entertainment system that may be used in embodiments of the present invention. [Modes for carrying out the invention]

[0007] Embodiments of the present invention include a system and method for sharing motion data. A user device associated with a user communicates with a content management server. The content management server receives data captured from the user device during actions performed by the user in a real-world environment. Based on the data, the content management server renders the corresponding actions by a virtual character in a virtual environment. The content management server captures a video of the corresponding actions in the virtual reality environment associated with the data. The content management server provides the captured video and data to a recipient designated by the user. The content management server verifies that the recipient is performing the action by comparing data about the recipient during playback of the captured video with data associated with the captured video.

[0008] Figure 1 shows an exemplary network environment in which a system for analyzing and sharing operational data may be implemented. The network environment 100 may include one or more content source servers 110 that provide digital content for distribution (e.g., games, other applications and services), one or more content provider server application program interfaces (APIs) 120, a content distribution network server 130, a context management server 140, and one or more user devices 150A to 150N.

[0009] The servers described herein may include any type of server known in the art, including standard hardware computing components such as network interfaces, media interfaces, non-temporary computer-readable storage (memory), and processors for executing instructions that can be stored in memory or for accessing information that can be stored in memory. The functions of multiple servers may be integrated into a single server. Any of the aforementioned servers (or integrated servers) may exhibit the characteristics of a particular client-side, cache, or proxy server. These characteristics may depend on the particular network placement or configuration of the server. Each server and device may communicate with each other via a local or wide-area communication network, including embodiments of the cloud and the internet.

[0010] The content source server 110 may maintain and provide a variety of digital content available for distribution. The content source server 110 may be associated with any content provider that makes its content available for access over a communication network. Such content may include not only digital video and games, but also other types of digital applications and services. Such applications and services may include any variety of different digital content and functions that can be provided to the user device 150. An example of the content source server 110 may be an internet website that provides downloadable and / or streaming content. The content provided by the content source server 110 may include any type of multimedia content, such as movies, games, static / dynamic content, photographs, social media content, social media websites, virtual reality, augmented reality, and mixed reality. The user device 150 may include several different types of computing devices or computing systems known in the art. In some embodiments, content data is transmitted from the content source server 110 to the user device 150, where the content data is rendered by the user device 150 (or associated peripherals) in a format suitable for use by the user device 150.

[0011] Digital content from the content source server 110 may be provided by the content provider server API 120, which enables various types of content source servers 110 to communicate with other servers (e.g., user devices 150) in the network environment 100. The content provider server API 120 may be specific to the content source server 110 providing the content, and the user device 150, such as its language, operating system, protocol, etc. In a network environment 100 containing multiple different types of content source servers 110, there may be a corresponding number of content provider server APIs 120, which enable various formats, conversions, and other cross-device and cross-platform communication processes for providing content and other services to different user devices 150, and different user devices 150 may process such content using different operating systems, protocols, etc. Thus, applications and services may be made available in different formats to ensure compatibility with various different user devices 150.

[0012] The Content Provider Server API 120 may further facilitate access for each user device 150 to content hosted by the Content Source Server 110 or services provided by the Content Source Server 110, either directly or via the Content Distribution Network Server 130. Additional information, such as metadata about the accessed content or services, may also be provided to the user device 150 by the Content Provider Server API 120. As described below, additional information (i.e., metadata) may be available to provide details about the content or services provided to the user device 150. In some embodiments, services provided from the Content Source Server 110 to the user device 150 via the Content Provider Server API 120 may include support services associated with other content or services, such as chat services, ratings, and profiles associated with a particular game, team, community, etc. In such cases, the Content Source Servers 110 may communicate with each other via the Content Provider Server API 120.

[0013] The content distribution network server 130 may include servers that provide resources, files, etc., related to content from the content source server 110, including various content and service configurations with user devices 150. The content distribution network server 130 can also be invoked by user devices 150 requesting access to specific content or services. The content distribution network server 130 may include universe management servers, game servers, streaming media servers, servers hosting downloadable content, and other content distribution servers known in the art.

[0014] The content management server 140 may include any data server known in the art that can receive data from the user device 150. The content rendered by the content management server 140 may be for essentially any type of computer application and may include one or more types of content, in particular, such as games, movies, audio, images, and multimedia. In some embodiments, the content or a portion thereof is generated by the content management server 140. In some embodiments, the content or a portion thereof is streamed from the content source server 110 to the user device 150 via one or more communication networks. In some embodiments, the content or a portion thereof is streamed from a cloud gaming infrastructure to the user device 150 via a communication network(s). The infrastructure may manage to send various types of content from the content source server 110 to the user device 150 via a communication network(s).

[0015] In exemplary embodiments, the content management server 140 may specifically manage data exchange related to user actions in the physical space of the real world. Such data may be captured by various user devices 150 (and / or associated sensors) associated with the user. The action data may be further supplemented and augmented with analytical data, as well as data about the virtual or other digital environment in which the player or character associated with the user is located when the user performs actions in the real world.

[0016] In various embodiments, the content management server 140 may also allow social media applications to access one or more social media graphs of the user in order to identify the user's social contacts. In some embodiments, augmented behavioral data may be shared with other users who may not have social contact with the user and may not be socially associated with the user on one or more existing social networks, but who may have played one or more video games with the user. The friend list may include additional information about the user's friends, such as a description of the games each friend owns, identification of the friend's online status (e.g., online, offline, inactive, etc.), the friend's last login and the period since, and the last game the friend played. The social networks include user data, such as the user's social graph, posts, photos, videos, and history information.

[0017] In exemplary embodiments, the user may provide input commands to the user device 150 regarding the capture of real-world actions. Such commands may be given verbally, by gesture, or via other hardware or software-based user interfaces (e.g., buttons). In response to such commands, the user device 150's camera (e.g., camera 404 in Figure 4) and other sensors (e.g., motion detection system 418 in Figure 4) may be activated to capture images, videos, and other types of data about the real-world environment in which the user is located. In some embodiments, sensors may be distributed in the real-world environment to capture different viewpoints and aspects of user actions. For example, some sensors may be worn (or held) on the user's body, while others may be stationary and positioned at different distances from the user, etc. The various sensors associated with the user device 150 may be provided to the content management server 140 for analysis and augmentation.

[0018] The content management server 140 may work to analyze images, videos, sensor data, etc., captured by sensors to generate metadata and various metrics that characterize actions performed by the user. The analysis and resulting metrics may break down an action into a series of sub-movements and characterize the overall action and its sub-movements. Each action (and sub-movement) may be characterized by metadata relating to the sequence and order of the sub-movements, where each action (and sub-movement) occurs within a virtual environment, interacting with its virtual elements and other parameters that the user may input. For example, the user may specify that such an action be associated with a custom audiovisual effect in the digital environment (e.g., associated a specified sub-movement with a specific visual effect or sound in the virtual environment). The user may also specify that the same action (or sub-movement) be associated with different audiovisual effects if performed in different locations or under different conditions in the virtual environment.

[0019] Furthermore, metrics characterizing the movements may be generated. For example, metrics relating to the specific mechanisms, rhythm, tempo, force, and other characteristics of each movement (and sub-movements) may be generated based on an analysis of a combination of sensor data from various sensors used to capture the movement data. Computer vision may be used to analyze images and videos of the movements, and the resulting metrics may be combined with other types of sensor data to generate additional metrics characterizing the movements. For example, by analyzing images and videos of the movements, various positions, angles, shapes, etc., of different parts of the body relative to each other and to the ground (or other features of the physical space) may be identified during each movement (and sub-movement). Furthermore, elements of timing, rhythm, and tempo may be evaluated for each interrelated movement and sub-movement. Other types of sensor data may be used to generate metrics relating to the forces on which each movement or sub-movement is performed. The initial movement data captured by the sensors may be combined with metadata and metrics generated within the movement profile (or other types of shareable content). Users may attach custom names, parameters, and other settings before sharing with other users (associated with their user account and user device).

[0020] The camera of the user device 150 (described in more detail with respect to Figure 4) can be configured to include multiple image capture devices, such as a stereo camera pair, an infrared camera, a depth camera, or a combination thereof. In some embodiments, one or more microphones (described in more detail with respect to Figure 4) can be used to capture sound from the user and / or from the environment in which the user device 150 is located.

[0021] In some embodiments, the user device 150 is configured to run the game locally on the processing hardware of the computing device. The game or content can be obtained in any format, such as physical media format (e.g., digital disc, tape, card, thumb drive, solid-state chip, solid-state card, etc.), or by downloading from the Internet (or other communication network). In some embodiments, the user device 150 acts as a client in communication with a cloud gaming infrastructure via one or more communication networks. The cloud gaming infrastructure may maintain and run the video game being played by the user device 150. The computing device may be configured to send inputs received from the user device 150, controller, and camera to the cloud gaming infrastructure, which processes this input and reflects it in the game state of the running video game.

[0022] Game data from video games, such as video data, audio data, and haptic feedback data, can be transmitted from the content source server 110 to the user device 150. The computer system may further process the game data before transmitting it to the appropriate device, or it may transmit the game data directly to the appropriate device. For example, a video stream and an audio stream may be transmitted to the user device 150.

[0023] In some embodiments, the user device 150 may be associated with a server that provides internal services in the network environment 100 (e.g., to other servers). In such a case, the user device 150 may correspond to one of the content source servers 110 described herein. Alternatively, the user device 150 may be a client device that can include any number of different game consoles, mobile devices, laptops, and desktops. Such a user device 150 may be configured to access data from other storage media, such as, but not limited to, a memory card or disk drive that may be appropriate in the case of downloaded services. Such a user device 150 may include standard hardware computing components such as, but not limited to, a network interface, a media interface, a non-transitory computer-readable storage (memory), and a processor for executing instructions that may be stored in the memory. These user devices 150 may also be executed using various different operating systems (e.g., iOS, Android (registered trademark)), applications, or computing languages (e.g., C++, JavaScript (registered trademark)). Exemplary client devices 150 are described in detail herein with respect to FIG. 4. Each user device 150 may be associated with a participant in a collection of digital content streams or other types of viewers.

[0024] The user device 150 may also include a module configured to receive inertial sensor data from an inertial sensor within the user device 150. The inertial sensor data indicates the movement of the user device 150 in accordance with the movements of the user associated with the user device 150. The user's movements are based on a virtual reality scene displayed within the user device 150. The path of the movement of the user device 150 can be determined from the inertial sensor data and the speed of the movement of the user device 150. In some embodiments, the path of the movement of the head-mounted display corresponds to one or more user movements within a set of user movements including tilting forward, tilting backward, tilting left, tilting right, turning the head to the left, turning the head to the right, looking up, looking down, crouching, and jumping. However, in other embodiments, the path of the movement of the head-mounted display may basically correspond to any user movement within the movement capabilities of the human body.

[0025] In some embodiments, the user device 150 can be used to manipulate virtual objects within, for example, a virtual reality (VR) or augmented reality (AR) scene, such as grasping, moving, pushing, pulling, etc., and this manipulation is displayed on the user device 150 or another computing device, such as a television, computer, etc. When the user wears the user device 150 and moves their hand, the virtual hand in the game moves. Further, when the user wears the user device 150 and moves their finger, the fingers of the virtual hand in the game move. The position and / or orientation of the finger is determined from the image data captured using the cameras described above and in FIG. 4 to generate the movement of the fingers of the virtual hand.

[0026] A user's avatar may be automatically converted from a single digital image into an animated 3D avatar. More specifically, a user may upload a digital image or other image to a user device 150 and, in return, receive one or more avatars featuring different facial expressions and / or animations. The avatars are displayed to the user and used as still or animated images (e.g., in GIF format). For example, an avatar may be sent to other users via SMS, MMS, email messages, chat (e.g., Facebook® Messenger), instant messenger (e.g., Skype or Windows® Messenger), Twitter®, TikTok, blogs, forums, or other electronic means of communication intended for sharing with one or more other individuals.

[0027] In some embodiments, the avatar may be realistically based on the user (e.g., having similar facial features, clothing, and body shape). In other embodiments, the avatar may be deliberately non-realistic, for example, by mapping the user's actions to the facial features of a celebrity, movie star, politician, video game character, or another user. The avatar may be a fusion of realistic and deliberately non-realistic elements. For example, the user's face may be used for the avatar, but the avatar may be given different clothing and body shape. Alternatively, the user's body may be used for the avatar, but the avatar may be given the face of a celebrity, movie star, politician, video game character, or another user.

[0028] The display of user device 150 may generate a three-dimensional scene incorporating an avatar into a virtual space or virtual environment with virtual interactive objects. Content management server 140 receives visual and / or distance information about the user captured by user device 150. Content management server 140 extracts motion information describing the user's movements. The motion information can be encoded to describe the movement of the user's "skeleton," which consists of major points within the user's body. Content management server 140 can enable content expansion by allowing user-related information, such as the user's avatar, identifier, and representative image, to be introduced into the VR space presented to the user's user devices 150A-150N.

[0029] The second user of the second user device 150B may be a recipient designated by the user. User device 150A transmits information describing the user's actions and / or the corresponding avatar's actions to the second user device 150B or the content management server 140, which then transmits this information to the second user device 150B, so that user device 150B can generate a scene featuring the exact actions of the avatar. User device 150A receives information describing the actions of the second user and / or the corresponding second avatar's actions from the second user device 150B or from an intermediate device such as the content management server 140. The content management server 140 transmits this information to device 150A to the first user, as it was received from the second user device 150B, so that the first user device 150A can generate a scene featuring the exact actions of the second avatar.

[0030] In some embodiments, the action profile may be applied to data relating to the recipient. For example, the recipient of the action profile may play a video of the action or view images via a corresponding user device 150, which may be presented in combination with augmentative metadata and metrics. When the recipient is ready to attempt or otherwise attempt to perform the action, data relating to the recipient may be captured and compared with the action profile. Similar analysis and metrics may be generated for the recipient and compared with the corresponding analysis and metrics for the action profile. If the analysis and metrics match within one or more predetermined thresholds, the recipient may be verified as having performed the action correctly. In some embodiments, such verification may result in specific notifications or audiovisual effects in a virtual environment.

[0031] In some embodiments of the present invention, if the actions of a receiver do not yield analysis or metrics that fall within a predetermined threshold(s), the discrepancies may be identified and used to provide guidance to the receiver. For example, if a particular action or sub-action is performed too slowly or with too little force, the receiver may be provided with a side-by-side view of their action profile and the original action shown by video, along with a notification to move faster or with more force, respectively. Similarly, if the receiver's body shape or angle differs from that indicated by the action profile, the receiver may be provided with a notification featuring an image of the shape or angle specified by the action profile superimposed on the receiver's image. Thus, the receiver can learn precisely how their actions differ from those characterized by the action profile.

[0032] The user device 150 or computing device generates a representation of the user's actions when captured and causes the avatar to perform the generated representation of the user's actions. The user device 150 also generates a representation of a second user's actions when received and causes the second avatar to perform the generated representation of the second user's actions. The user device 150 also updates the virtual space or environment, as well as any virtual interactive objects, as necessary.

[0033] Figure 2 shows an exemplary Unified Data System (UDS) that may be used to analyze behavioral data related to a virtual environment. Based on the data provided by the UDS, the content management server 140 can recognize in-game objects, entities, activities, and events in which a user has interacted, and thus support the analysis and coordination of in-game activities. Each user interaction may be associated with metadata such as the type of in-game interaction, its location within the in-game environment, its time in the in-game timeline, and other players, objects, and entities involved. Thus, the metadata can track any of the various user interactions that may occur during a game session, such as related activities, entities, settings, results, actions, effects, locations, and character statuses. Such data may be further aggregated, applied to a data model, and subject to analysis. Using such a UDS data model, contextual information may be assigned to each part of the information in a unified manner across the game.

[0034] As shown in Figure 2, an exemplary console 228 (e.g., user device 150) and an exemplary server 218 (e.g., streaming server 220, activity feed server 224, user-generated content (UGC) server 232, and object server 226) are shown. In one example, the console 228 may be implemented on either the platform server 120, the cloud server, or server 218. In one exemplary example, the content recorder 202 may be implemented on the platform server 120, the cloud server, or any server 218. Such a content recorder 202 receives content (e.g., media) from the interactive content title 230 and records it in the content ring buffer 208. Such a ring buffer 208 may store multiple content segments (e.g., v1, v2, and v3), start times for each segment (e.g., V1_START_TS, V2_START_TS, V3_START_TS), and end times for each segment (e.g., V1_END_TS, V2_END_TS, V3_END_TS). Such segments may be stored by the console 228 as media files 212 (e.g., MP4, WebM, etc.). Such media files 212 may be uploaded to the streaming server 220 for storage and subsequent streaming or use, but media files 212 may be stored on any server, cloud server, any console 228, or any user device 150. Such start and end times for each segment may be stored by the console 228 as a content timestamp file 214. Such a content timestamp file 214 may also include a streaming ID that matches the streaming ID of the media file 212, thereby associating the content timestamp file 214 with the media file 212.Such content timestamp files 214 may be uploaded to and stored on the activity feed server 224 and / or the UGC server 232, but the content timestamp files 214 may be stored on any server, cloud server, any console 228, or any user device 150.

[0035] As the content recorder 202 receives and records content from the interactive content title 230, the object library 204 receives data from the interactive content title 230, and the object recorder 206 tracks the data to determine when the object starts and ends. The object library 204 and object recorder 206 may be implemented on the platform server 120, the cloud server, or any server 218. When the object recorder 206 detects the start of an object, it receives object data from the object library 204 (for example, if the object is an activity, user interaction with the activity, activity ID, activity start time, activity end time, activity result, activity type, etc.) and records this activity data (for example, ActivityID1,START_TS;ActivityID2,START_TS;ActivityID3,START_TS) in the object ring buffer 210. Such activity data recorded in the object ring buffer 210 may be stored in the object file 216. Such an object file 216 may also include data on the activity start time, activity end time, activity ID, activity result, activity type (e.g., competition, quest, task, etc.), and user or peer data related to the activity. For example, the object file 216 may store data on items used during the activity. Such an object file 216 may be stored on the object server 226, but the object file 216 may be stored on any server, cloud server, any console 228, or any user device 150.

[0036] Such object data (e.g., object file 216) may be associated with content data (e.g., media file 212 and / or content timestamp file 214). In one example, UGC server 232 stores the content timestamp file 214 and associates the content timestamp file 214 with object file 216 based on a match between the streaming ID of the content timestamp file 214 and the corresponding activity ID of the object file 216. In another example, object server 226 may store object file 216 and receive queries about object file 216 from UGC server 232. Such queries may be performed by searching for an activity ID of object file 216 that matches the streaming ID of the content timestamp file 214 sent with the query. In yet another example, a query of the stored content timestamp file 214 may be performed by matching the start and end times of the content timestamp file 214 with the start and end times of the corresponding object file 216 sent with the query. Such object files 216 may also be associated with a matched content timestamp file 214 by the UGC server 232, but this association may be performed by any server, cloud server, any console 228, or any user device 150. In another example, the object files 216 and content timestamp file 214 may be associated by the console 228 during the creation of each file 216, 214.

[0037] Figure 3 is a flowchart illustrating an exemplary method for analyzing and sharing operational data. Method 300 in Figure 3 may be embodied as executable instructions in a non-temporary computer-readable storage medium, including but not limited to non-volatile memory such as a CD, DVD, or hard drive. Instructions in the storage medium may be executed by a processor(s) to cause various hardware components of a computing device hosting or otherwise accessing the storage medium to implement the method. The steps (and their order) shown in Figure 3 are exemplary and may include, but not limited to, various alternatives, equivalents, or derivatives thereof, including their execution order.

[0038] In step 310, data is received from the user device associated with the user. The data is captured during the user's actions in the real-world environment. The user's actions in the real-world environment may be represented by real-world environment data captured by the user device 150. The data may be image data captured by the camera of the user device 150. In another embodiment, the data may be captured by the inertial sensor of the user device 150. The real-world environment data may be transferred to the content management server 140 along with one or more images and image data of the user interacting in the real-world environment captured by the user device 150.

[0039] In step 320, the corresponding actions of a virtual character in the virtual environment are rendered based on the data. In step 330, a video of the corresponding actions in the virtual reality environment is captured. The captured video is associated with the data. In one embodiment, the video of the corresponding actions in the virtual reality environment is captured when the data reaches a predetermined threshold.

[0040] In step 340, the captured video and data are provided to recipients designated by the user. The captured video and data may be provided to recipients via one or more social networks associated with the user and recipients.

[0041] In step 350, the receiver's actions are verified by comparing data about the receiver during playback of the captured video with data associated with the captured video. User input selections via user device 150 may be received to perform different actions within the virtual environment. Updates to the rendering of the virtual reality environment may be generated to reflect the performance of different actions. Signature actions associated with the user may be created based on data meeting predetermined thresholds.

[0042] Figure 4 is a block diagram of an exemplary electronic entertainment system 400. The entertainment system 400 in Figure 4 includes main memory 405, a central processing unit (CPU) 410, a vector unit 415, a graphics processing unit 420, an input / output (I / O) processor 425, an I / O processor memory 430, a controller interface 435, a memory card 440, a Universal Serial Bus (USB) interface 445, and an IEEE interface 450. The entertainment system 400 further includes operating system read-only memory (OS ROM) 455, an audio processing unit 460, an optical disc control unit 470, and a hard disk drive 465, which are connected to the I / O processor 425 via bus 475.

[0043] Figure 4 is a block-level diagram of an exemplary user device 400 that may be used to capture operational data. Naturally, depending on the effective configuration and functionality, more or fewer components than those shown in Figure 4 may be included in or removed from the user device 400. The user device 400 includes a processor 402 for executing program instructions. Memory 406 is provided for data storage purposes and may include both volatile and non-volatile memory. A display 416 is included that provides a visual interface that the user can see. The display 416 can be defined as a single display or as separate display screens for each eye. If two display screens are provided, it is possible to provide video content for the left eye and the right eye separately. For example, displaying video content separately for each eye can improve immersive control of three-dimensional content in a virtual reality scene.

[0044] The motion detection module 418 may include any of various types of motion detection hardware, such as a magnetometer 420, an accelerometer 424, and a gyroscope 426. The user device 400 may include an inertial sensor 422 configured to generate inertial sensor data indicating the movement of the user device 400 according to the actions of the user to which the user device 400 is associated. The magnetometer 420 measures the strength and direction of the magnetic field near the user device 400. In some embodiments, three magnetometers 420 are used within the user device 400 to ensure an absolute reference for the yaw angle in world space. In various embodiments, the magnetometers 420 are designed to span the Earth's magnetic field of ±80 microteslas. The magnetometer is affected by metals and provides a monotonic yaw measurement relative to the actual yaw. The magnetic field may be distorted by metals in the environment, which distorts the yaw measurement. If necessary, this distortion can be calibrated using information from other sensors, such as a gyroscope or camera. In some embodiments, an accelerometer 424 is used together with a magnetometer 420 to obtain the tilt angle and azimuth angle of the user device 400.

[0045] In some embodiments, the present invention may also include one or more external inertial sensors placed on the user's body. The present invention may include operations for comparing external inertial sensor data with inertial sensor data received from an inertial sensor 422 in the user device 400 in order to determine a particular action performed by the user.

[0046] The accelerometer 424 is a device that measures acceleration and the reaction force caused by gravity. Single-axis and multi-axis (e.g., 6-axis) models can detect the magnitude and direction of acceleration in various directions. The accelerometer 424 is used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 424 are used to provide the direction of gravity, which gives an absolute reference for two angles (world-space pitch and world-space roll).

[0047] A gyroscope 426 is a device that measures or maintains orientation based on the principle of angular momentum. In one embodiment, three gyroscopes 426 provide information about motion around their respective axes (x, y, and z) based on inertial sensing. The gyroscopes 426 help detect high-speed rotation. However, the gyroscopes 426 can drift over time if an absolute reference is not present. This necessitates periodic resetting of the gyroscopes 426, which can be done using other available information, such as visual tracking of an object or position / orientation determination based on an accelerometer, magnetometer, etc.

[0048] A camera 404 is provided to capture images and image streams of the real-world environment to which the user device 400 is exposed. The user device 400 may include multiple cameras 404, including a rear-facing camera 404 (facing away from the user when the user is looking at the display of the user device 400) and a front-facing camera 404 (facing towards the user when the user is looking at the display of the user device 400). In some embodiments, the camera 404 may be included in the user device 400 to sense depth information of objects in the real-world environment to which the user device 400 is exposed.

[0049] The user device 400 includes a speaker 412 for providing audio output. It may also include a microphone 414 for capturing audio from the real world environment, including sounds from the surrounding environment and speech made by the user.

[0050] A Wi-Fi module 410 may be included to enable the user device 400 to connect to the internet via wireless network technology. The user device 400 may also include a Bluetooth® module 408 to enable wireless connectivity to other devices.

[0051] Naturally, the components of the user device 400 shown in Figure 4 are examples of components that may be included in the user device 400, and do not represent all possible components that may be included in the user device 400. For example, in various embodiments, the user device 400 may include some of the components shown in Figure 4, or it may not. In some embodiments, the user device 400 may include additional components not shown in Figure 4.

[0052] Figure 5 is a block diagram of an exemplary electronic entertainment system that may be used in embodiments of the present invention. The entertainment system 500 may be an electronic game console. Alternatively, the entertainment system 500 may be implemented as a general-purpose computer, set-top box, handheld game device, tablet computing device, mobile computing device, or mobile phone. The entertainment system may include more or fewer operating components depending on the particular form factor, purpose, or design.

[0053] The CPU 510, vector unit 515, graphics processing unit 520, and I / O processor 525 in Figure 5 communicate via the system bus 585. Furthermore, the CPU 510 in Figure 5 may communicate with the main memory 505 via a dedicated bus 580, and the vector unit 515 and graphics processing unit 520 may communicate via a dedicated bus 590. The CPU 510 in Figure 5 executes programs stored in the OS ROM 555 and the main memory 505. The main memory 505 in Figure 5 may include pre-stored programs and programs transferred via the I / O processor 525 from a CD-ROM, DVD-ROM, or other optical disc (not shown) using an optical disc control unit 570. The I / O processor 525 in Figure 5 may also enable the introduction of content transferred via wireless or other communication networks (e.g., 4G, LTE, 3G, etc.). The I / O processor 525 in Figure 5 primarily controls data exchange between various devices of the entertainment system 500, including the CPU 510, vector unit 515, graphics processing unit 520, and controller interface 535.

[0054] The graphics processing unit 520 in Figure 5 executes graphics instructions received from the CPU 510 and the vector unit 515 to generate an image for display on a display device (not shown). For example, the vector unit 515 in Figure 5 may convert an object from three-dimensional coordinates to two-dimensional coordinates and send the two-dimensional coordinates to the graphics processing unit 520. Furthermore, the audio processing unit 560 executes instructions to generate an audio signal to be output to an audio device such as a speaker (not shown). Other devices may be connected to the entertainment system 500 via a USB interface 545 and an IEEE 1394 interface 550 such as a wireless transceiver, and these may be embedded in the system 500 or as part of some other component such as a processor.

[0055] The user of the entertainment system 500 in Figure 5 provides commands to the CPU 510 via the controller interface 535. For example, the user may instruct the CPU 510 to store specific game information on a memory card 540 or other non-temporary computer-readable storage medium, or to instruct a character in the game to perform a specific action.

[0056] The present invention may be implemented in applications that can be operated by various end-user devices. For example, the end-user device may be a personal computer, a home entertainment system (e.g., Sony PlayStation 2®, Sony PlayStation 3®, or Sony PlayStation 4®), a portable game device (e.g., Sony PSP® or Sony Vita®), or a lower-end but different manufacturer's home entertainment system. The method described herein is fully intended to be operable on a variety of devices. The present invention may also be implemented in cross-title neutrality, and embodiments of the system may be used across various titles from various publishers.

[0057] The present invention may be implemented in applications that may be operable using various devices. Non-temporary computer-readable storage media refer to any media(s) involved in providing instructions to a central processing unit (CPU) for execution. Such media can take many forms, including but not limited to non-volatile media such as optical disks or magnetic disks, and volatile media such as dynamic memory. Common forms of non-temporary computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, any other magnetic media, CD-ROM disks, digital video disks (DVDs), any other optical media, and RAM, PROM, EPROM, FLASHEPROM, and any other memory chips or cartridges.

[0058] Various forms of transmission media may be involved in conveying one or more sequences of one or more instructions to the CPU for execution. A bus transmits data to system RAM, and the CPU retrieves and executes the instructions from system RAM. Instructions received by system RAM can optionally be stored on a fixed disk either before or after execution by the CPU. Various forms of storage may be implemented along with the network interfaces and network topologies necessary to implement the storage.

[0059] The above detailed description of the Art is presented for illustrative and explanatory purposes only. The above detailed description is not intended to be comprehensive or to limit the Art to the exact form disclosed. Many modifications and variations are possible in light of the above teachings. The embodiments described have been selected to best illustrate the principles of the Art and its practical application, and to enable those skilled in the art to utilize the Art in various embodiments and in various modifications suitable for the specific intended use. The scope of the Art is intended to be defined by the claims.

Claims

1. A method for sharing operational data, Receiving sensor data from one or more sensors associated with a user during an interactive session, wherein the sensor data is captured in a real-world environment during the user's actions, and the receiving of such data is... Decomposing the aforementioned operation into a series of sub-operations (each sub-operation having a variable-length segment of the sensor data corresponding to a discrete portion of the aforementioned operation performed by the user), and analyzing the sensor data to generate metadata characterizing the series of sub-operations, Receiving user input that one or more of the subordinate operations are associated with a specified audiovisual effect within the virtual environment of the interactive session, Based on the aforementioned sensor data, the corresponding actions of the virtual characters in the virtual environment are rendered. Capturing a video of the corresponding operation within the virtual environment, wherein the captured video is associated with the sensor data, Providing to the recipient device of the user-specified recipient an operational profile that includes the specified audiovisual effects associated with the specified suboperation, characterized by the metadata generated from the captured video and the sensor data, The verification that the receiver is performing the specified suboperation is performed by comparing data relating to the receiver during playback of the captured video with the sensor data associated with the captured video in the operation profile, wherein the specified audiovisual effect occurs in the virtual environment based on the verification. Includes, method.

2. The operation profile further includes custom parameters specified by the user, The method according to claim 1, wherein the custom parameter includes one or more virtual elements of the virtual environment.

3. The process involves generating the receiver's video within the virtual environment while the receiver is performing the aforementioned operation, Based on the verification, the virtual environment is updated with the virtual element associated with the custom parameter. The method according to claim 2, further comprising:

4. The sensor data includes image data or video data of the user's actions in the real-world environment. The method according to claim 1, further comprising applying computer vision to the image data or video data to generate metadata or metrics.

5. The method according to claim 1, wherein verification that the receiver is performing the specified suboperation is based on one or more matches of the data relating to the receiver to the operation profile which is within a predetermined threshold.

6. To analyze the data relating to the recipient in order to identify one or more differences in the aforementioned operating profile, To generate a notification to the recipient regarding the identified differences. The method according to claim 1, further comprising:

7. Overlaying one or more images from the aforementioned operating profile onto one or more images of the recipient, To provide the superimposed image to the receiver device. The method according to claim 1, further comprising:

8. The method according to claim 1, wherein the operation profile further specifies one or more conditions of the virtual environment.

9. The method of claim 8, wherein verification that the receiver is performing the specified subordinate operation is based on one or more matches of the specified conditions in the virtual environment.

10. A system for sharing operational data, A sensor interface that receives sensor data from one or more sensors associated with a user during an interactive session, wherein the sensor data is captured during the user's actions in a real-world environment, and the sensor interface A processor that executes instructions stored in memory, wherein the processor executes the instructions, Decomposing the aforementioned operation into a series of sub-operations (each sub-operation having a variable-length segment of the sensor data corresponding to a discrete portion of the aforementioned operation performed by the user), and analyzing the sensor data to generate metadata characterizing the series of sub-operations, Receiving user input that one or more of the subordinate operations are associated with a specified audiovisual effect within the virtual environment of the interactive session, Based on the aforementioned sensor data, the corresponding actions of the virtual characters in the virtual environment are rendered. Capturing a video of the corresponding operation within the virtual environment, wherein the captured video is associated with the sensor data, The processor performs the following: A communication interface that provides a user-designated recipient device to an audiovisual effect associated with a designated suboperation characterized by the captured video and metadata generated from the sensor data, wherein the recipient is verified to be performing the designated suboperation based on comparing data about the recipient during playback of the captured video with the sensor data associated with the captured video in the operation profile, and the designated audiovisual effect occurs in the virtual environment based on the verification, and A system that includes this.

11. The operation profile further includes custom parameters specified by the user, The system according to claim 10, wherein the custom parameter includes one or more virtual elements of the virtual environment.

12. The aforementioned processor, The process involves generating the receiver's video within the virtual environment while the receiver is performing the aforementioned operation, Based on the verification, the virtual environment is updated with the virtual element associated with the custom parameter. The system according to claim 11, which executes further instructions to perform the following:

13. The sensor data includes image data or video data of the user's actions in the real-world environment. The system according to claim 10, wherein the processor executes further instructions for applying computer vision to the image data or video data to generate metadata or metrics.

14. The system according to claim 10, wherein the receiver is verified to be performing the specified suboperation based on one or more matches of the data relating to the receiver to the operation profile which is within a predetermined threshold.

15. The aforementioned processor, To analyze the data relating to the recipient in order to identify one or more differences in the aforementioned operating profile, To generate a notification to the recipient regarding the identified differences. The system according to claim 10, which executes further instructions to perform the following:

16. The processor executes further instructions to superimpose one or more images from the operation profile onto one or more images of the receiver. The system according to claim 10, wherein the communication interface provides the superimposed image to the receiving device.

17. The system according to claim 10, wherein the operation profile further specifies one or more conditions of the virtual environment.

18. The system according to claim 17, wherein the receiver is verified to be performing the specified operation based on one or more matches of the specified conditions in the virtual environment.

19. A non-temporary computer-readable storage medium that embodies a program executable by a processor for performing a method for sharing operational data, the method being: Receiving sensor data from one or more sensors associated with a user during an interactive session, wherein the sensor data is captured in a real-world environment during the user's actions, and the receiving of such data is... Decomposing the aforementioned operation into a series of sub-operations (each sub-operation having a variable-length segment of the sensor data corresponding to a discrete portion of the aforementioned operation performed by the user), and analyzing the sensor data to generate metadata characterizing the series of sub-operations, Receiving user input that one or more of the subordinate operations are associated with a specified audiovisual effect within the virtual environment of the interactive session, Based on the aforementioned sensor data, the corresponding actions of the virtual characters in the virtual environment are rendered. Capturing a video of the corresponding operation within the virtual environment, wherein the captured video is associated with the sensor data, Providing a user-specified recipient device with an action profile including the specified audiovisual effect associated with the specified sub-action, characterized by the captured video and the metadata generated from the sensor data, The verification that the receiver is performing the specified suboperation is performed by comparing data relating to the receiver during playback of the captured video with the sensor data associated with the captured video in the operation profile, wherein the specified audiovisual effect occurs in the virtual environment based on the verification. Non-temporary computer-readable storage media, including [specific type of storage medium].

Images