System and method for guiding a user in playing a game

Abstract

A method for processing a self-coaching interface is described. The method includes identifying, by a user, a gameplay event during gameplay. The gameplay event is flagged as falling below a skill threshold. The method also includes generating a recording of a time window of the gameplay event and processing game telemetry for the recording of the gameplay event. The game telemetry is used to identify a progression of an interaction action prior to the gameplay event of the time window. The method includes generating overlay content in the self-coaching interface. The overlay content is applied to one or more image frames of the recording when viewed via the self-coaching interface. The overlay content is surfaced in the one or more image frames during playback of the recording. The overlay content provides a prompt for improving a skill of the user above the skill threshold.

Description

Technical Field

[0001] This disclosure relates to systems and methods for instructing users to play games. Background Technology

[0002] Typically, video games are designed to challenge players to figure out how to overcome obstacles. In some games, this might be the challenge of defeating another player in a multiplayer setting, while in others it might be the challenge of defeating an entity that the computer struggles to control. Often, players attempt these challenges multiple times, gradually learning several complexities of the game to improve their skills. Additionally, some of the most challenging situations may not occur frequently, making it harder for players to remember lessons learned from previous experiences. Some players may feel frustrated by the amount of time spent improving their skills at a particular point in the game.

[0003] It is under these circumstances that the embodiments of the present invention came into being. Summary of the Invention

[0004] The embodiments disclosed herein provide systems and methods for guiding users to play games.

[0005] In one implementation, a mentoring system is provided to reduce frustration for players of various game genres when learning new games. Mentoring accelerates player skill development.

[0006] In one implementation, a series of gameplay events, such as the death of a virtual user and virtual user misses, are recorded and collected into a grouped sequence over time. The systems and methods described herein provide a user interface (UI) to review previously failed video and controller input captures in specific situations to examine and learn which alternative strategies should be used. The systems and methods highlight failures that occurred during gameplay.

[0007] In one implementation, the systems and methods described herein use game hooks or game telemetry to identify points during gameplay where guidance can be applied, such as gameplay events. For example, guidance may be applied after a virtual user dies or after a virtual user is attacked. A record of the last X time units of gameplay prior to a gameplay event is stored in a memory device, where X is a positive real number. This record is stored along with tags describing the problem (such as a virtual user dying or a virtual user being attacked). The systems and methods generate a dedicated UI to replay the record with overlays such as input selections received from the user. Additionally, a timeline scribble bar is provided along the dedicated UI to allow the user to scroll backward and forward during the last X time units. Furthermore, the dedicated UI includes prompts for the user regarding mechanisms or situations to improve the user's skill level.

[0008] In one embodiment, a method for processing a self-guided interface is described. The method includes: identifying gameplay events by a user during gameplay; marking the gameplay events as falling below a skill threshold; and generating a record of a time window of the gameplay events, and processing game telemetry for the record of the gameplay events. The game telemetry is used to identify the progress of interactive actions prior to the gameplay events within the time window. The method also includes: generating overlay content in the self-guided interface; applying the overlay content to one or more image frames of the record when viewed via the self-guided interface; displaying the overlay content in the one or more image frames during playback of the record; and providing a prompt to raise the user's skill above the skill threshold.

[0009] In one implementation, a method is described to facilitate self-guidance for players during gameplay. The method includes: recording a series of gameplay events as game event data. The series of gameplay events is associated with a user identity (ID). The method further includes: categorizing the series of gameplay events based on multiple predetermined gameplay events; and receiving one or more inputs to initiate a guidance session regarding one of the multiple predetermined gameplay events. In response to the one or more inputs, the method includes: obtaining a portion of the game event data. Recording the portion of the game event data within a predetermined time period in which one of the series of gameplay events occurs. One of the series of gameplay events is categorized as one of the multiple predetermined events. The method includes: displaying the portion of the game event data using a timeline to facilitate the player's self-guidance.

[0010] In one implementation, when the user accesses the game with the series of gameplay events after authenticating their identity, the series of gameplay events is associated with the user identity.

[0011] In one embodiment, the method further includes: receiving input indicating time along the timeline; and translating the portion of the game event data based on the input indicating the time.

[0012] In one implementation, the method further includes: recording additional series of gameplay events as additional game event data. The additional series of gameplay events is associated with the user ID. The method includes: classifying the additional series of gameplay events based on the plurality of predetermined gameplay events. In response to the one or more inputs, the method includes: obtaining a portion of the additional game event data. Recording the portion of the additional game event data within a predetermined time period in which one of the additional series of gameplay events occurs. One of the additional series of gameplay events is classified as one of the plurality of predetermined events. The method includes: displaying the portion of the additional game event data using the timeline to facilitate the player's self-guidance.

[0013] In one implementation, the method includes: recording additional series of gameplay events as additional game event data. The additional series of gameplay events is associated with another user ID. The method includes: categorizing the additional series of gameplay events based on a plurality of predetermined gameplay events. In response to the one or more inputs, the method includes: obtaining a portion of the additional game event data. Recording the portion of the additional game event data within a predetermined time period in which one of the additional series of gameplay events occurs. One of the additional series of gameplay events is categorized as one of the plurality of predetermined events. The method includes: displaying the portion of the additional game event data using the timeline to facilitate the player's self-guidance.

[0014] Some advantages of the system and method described in this paper include providing a user interface that enables users to understand video games and quickly acquire the skills to play them. Additionally, video games are improved because the guidance session is associated with and accessed to allow users to self-guide. Furthermore, unnecessary network traffic is reduced. Compared to playing video games without a guidance session, users can play video games by making fewer mistakes when they apply the experience learned during the guidance session. These mistakes unnecessarily increase network traffic between the client device used by the user and the server system in which the video game is executed.

[0015] Other aspects of this disclosure will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which exemplify the principles of the embodiments described herein. Attached Figure Description

[0016] Various embodiments of this disclosure are best understood by reference to the following description taken in conjunction with the accompanying drawings, in which:

[0017] Figure 1 This is a diagram illustrating the implementation scheme of a system for playing games.

[0018] Figure 2 This is a diagram illustrating the implementation scheme of a system for demonstrating a virtual scene in a game.

[0019] Figure 3 It is used to illustrate another virtual scene in the game. Figure 1 A diagram illustrating the implementation scheme of the system.

[0020] Figure 4 This is yet another virtual scenario used to illustrate the death of a virtual user. Figure 1 A diagram illustrating the implementation scheme of the system.

[0021] Figure 5A This is a diagram illustrating an implementation scheme for a system that uses a game logger to record game event data for user ID1.

[0022] Figure 5B This is an example of an implementation scheme for a game logger that records game event data for user IDs 2 to N.

[0023] Figure 6 This is a diagram illustrating an implementation scheme for a system that uses a guidance procedure to generate guidance sessions for training users.

[0024] Figure 7A It is used to instantiate one or more virtual objects in Figure 1 A diagram illustrating the implementation of a guided scenario overlaid on one or more virtual objects in a virtual scene.

[0025] Figure 7B Is Figure 1 A diagram of the implementation scheme for a guiding scenario, rendered and displayed on the system's display device.

[0026] Figure 8A It is used to illustrate in Figure 1 Simultaneously display on the display device Figure 7A A diagram showing the implementation scheme for one guidance scenario and another guidance scenario.

[0027] Figure 8B This is used to illustrate the simultaneous rendering and display of the guidance scenario in Figure 7 associated with user ID1 and another guidance scenario associated with user IDN. Figure 1 A diagram illustrating the implementation scheme of the display device.

[0028] Figure 8C This diagram illustrates an implementation scheme for a system in which the processor generates audio data that is output to the user via headphones, serving as an alternative to and supplement to rendering a guided scene on a display screen.

[0029] Figure 9 This is used as an example to illustrate the use of an inference training engine to generate... Figure 6 A diagram illustrating the implementation scheme of the system for guiding sessions.

[0030] Figure 10 This is a flowchart illustrating an implementation scheme for a method used to train a model.

[0031] Figure 11A This is a diagram illustrating an implementation scheme of a system for communication between a processor system and multiple devices via routers, modems, and computer networks.

[0032] Figure 11B This is a diagram illustrating an implementation scheme for a processor system that can be implemented within a game console.

[0033] Figure 11C This is a diagram illustrating an implementation scheme for a system that allows communication between a smartphone and a processor system via a computer network without the need for a router and modem between the computer network and the smartphone.

[0034] Figure 12A This is a diagram of the implementation scheme for headphones.

[0035] Figure 12B This is a diagram illustrating an implementation scheme for a display device used to display images of a guiding scene or virtual scene on a display screen.

[0036] Figure 13 This is a flowchart that conceptually illustrates various implementations of this disclosure, executed to stream cloud video games to client devices.

[0037] Figure 14 It is a block diagram of an implementation scheme for a game console that is compatible with a display device for interfacing with a client device and is capable of communicating with a game hosting system via a computer network.

[0038] Figure 15 This is a diagram illustrating an implementation of an HMD (Head-Mounted Display) component.

[0039] Figure 16 An example implementation of an Information Service Provider (INSP) architecture. Detailed Implementation

[0040] Systems and methods for guiding users to play games are described. It should be noted that various embodiments of this disclosure are practiced without some or all of these specific details. In other instances, well-known processing operations are not described in detail to avoid unnecessarily obscuring the various embodiments of this disclosure.

[0041] Figure 1 This is a diagram illustrating an embodiment of a system 100 for playing a game. System 100 includes a display device 1 that displays a virtual scene 110 of the game. The display device 1 has a camera 101 that captures one or more images of the real-world environment in front of the camera 101. For example, the camera 101 captures one or more images within its field of view. As an example, each virtual scene is an image frame. In one embodiment, the terms image and image frame are used interchangeably herein. Examples of display devices as used herein include smart TVs, televisions, plasma display devices, liquid crystal display (LCD) devices, and light-emitting diode (LED) display devices. User 1 is playing a game, such as a video game, using a game controller 1. Examples of video games include single-player games or multiplayer games. Examples of game controller 1 include PlayStation. TM controller, Xbox TM controllers and Nintendo TM Controller. As an example, a game controller includes multiple joysticks and multiple buttons.

[0042] Display device 1 displays multiple images of the game. User 1 logs into user account 1 assigned to user 1 by the server system to access the game from the server system. For example, the user identity (ID) and password assigned to the user by the server system are authenticated by the server system to determine whether user access to the game is permitted.

[0043] When accessing the game, User 1 plays the game using Game Controller 1. During gameplay, a virtual scene 110 is generated, and User ID 1 assigned to User 1 is displayed in the virtual scene 110.

[0044] In the game, User 1 uses Game Controller 1 to construct a virtual ramp 112, on which Virtual User 102 can climb to gain a height advantage over another Virtual User 104. Virtual User 102, Virtual Ramp 112, and Virtual User 104 exist within a virtual scene 110. Furthermore, in the game, the movement of Virtual User 102 is controlled by User 1 using Game Controller 1. User 1 uses Game Controller 1 to control a virtual gun 108 held by Virtual User 102. For example, User 1 uses Game Controller 1 to control Virtual Gun 108 to fire at Virtual User 104. Virtual Gun 108 is part of the virtual scene 110. In multiplayer games, Virtual User 104 is controlled by another user, such as User 2 or User 3.

[0045] During gameplay, in virtual scene 110, virtual bullet 106 points at virtual user 102, while virtual user 102 fires at virtual user 104, and virtual user 104 fires at virtual user 102. Virtual bullet 106 points at virtual user 102 from one side of virtual user 102. Virtual scene 110 does not include the virtual user firing virtual bullet 106 at virtual user 102. Virtual scene 110 includes virtual tree 105 and virtual wall 107. It should be noted that each of virtual tree 105, virtual wall 107, virtual ramp 112, virtual user 102, virtual gun 108, virtual bullet 108, and virtual user 104 is an example of a virtual object.

[0046] In one implementation, the server system authenticates the user ID and password to allow the user to access multiple games with gameplay options.

[0047] In one implementation, instead of display device 1, a head-mounted display (HMD) is used. The HMD is worn by user 1 on his / her head.

[0048] Figure 2 This is a diagram illustrating an implementation scheme of a system 100 with a virtual scene 202 for illustrative purposes in a game. The virtual scene 202 is displayed on display device 1 and follows virtual scene 110 during gameplay. For example, both virtual scenes 110 and 202 are displayed during the same game session. User 1 cannot save virtual user 102 from being hit by virtual bullets 106. Figure 1 The virtual bullet 108 hits the virtual user 102. After the virtual user 102 is hit, the virtual user 102 dies in the game and is detected by the virtual drone 204 of the virtual scene 202. Additionally, before the virtual user 102 dies in the game, user 1 manages to control the game controller 1 to further control the virtual user 102 and the virtual gun 108. Figure 1 The system then kills virtual user 104. When virtual user 104 is killed, it is detected by another virtual drone 206 in virtual scene 202. Additionally, user ID 1 is displayed in virtual scene 202.

[0049] Figure 3 This is a diagram illustrating an implementation scheme of system 100 for another virtual scene 302 in the game. Virtual scene 302 is located within virtual scene 202 ( Figure 2 After display or in virtual scene 110 ( Figure 1The virtual scene 302 is displayed before the actual scene. In the virtual scene 302, virtual user 102 is controlled by user 1 via game controller 1. Virtual scene 302 is displayed on display device 1. Virtual user 102 is controlled to use a virtual gun 108 to shoot at virtual user 104. Additionally, virtual scene 302 includes a virtual wall 304 constructed by virtual user 102. For example, user 1 uses game controller 1 to control virtual user 102 to construct the virtual wall 304 to protect virtual user 102 from virtual bullets fired by virtual user 104.

[0050] In virtual scene 302, virtual bullet 306 hits virtual user 102 from behind. User 1 cannot see from virtual scene 302 which virtual user fired virtual bullet 306 at virtual user 102. Virtual scene 302 does not include the virtual user who fired virtual bullet 306 at virtual user 102.

[0051] It should be noted that virtual scene 302 is displayed within it. Figure 1 Virtual scene 110 and Figure 2 The virtual scene 202 is displayed on display device 1 during the same or different game sessions. For example, after virtual user 102 dies in virtual scene 202, the game program is executed by the processor of the server system to provide a waiting period during which user 1 waits for virtual user 102 to respawn, revive, or be resurrected. As used herein, examples of processors include microprocessors or microcontrollers or central processing units (CPUs) or application-specific integrated circuits (ASICs) or programmable logic devices (PLDs). The waiting period occurs before the game displays virtual scene 302. In this example, user 1 does not log out of user account 1 between the display of virtual scene 202 and the display of virtual scene 302, and therefore virtual scenes 110, 202, and 302 are displayed during the same game session. As another example, after virtual user 102 dies in virtual scene 202, user 1 logs out of user account 1. User 1 then logs back into user account 1. After user 1 logs back into user account 1, virtual scene 302 is displayed. In this example, virtual scene 302 is displayed during a different game session than the game session in which virtual scenes 110 and 202 are displayed.

[0052] In one implementation, a game session ends when virtual user 102, controlled by user 1, dies or wins the game session. Virtual user 102 wins the game session when it kills all other virtual users or survives for more than a pre-stored time period during the game session. Once virtual user 102 dies, it can respawn and another game session begins, which ends when virtual user 102 dies or wins another game session.

[0053] Figure 4 This is a diagram illustrating an implementation scheme of system 100 in another virtual scenario 402 where virtual user 102 dies. Virtual scenario 402 is in virtual scenario 302 ( Figure 3 The virtual scenes 302 and 402 are displayed after the virtual scenes 402 and 404 are displayed during the same game session. In virtual scene 402, virtual users 102 and 104 are killed. Virtual user 102 is killed by virtual bullet 306. Figure 3 (Kill). Virtual scene 402 includes virtual drones 204 and 206 for detecting virtual users 102 and 104.

[0054] Figure 5A This is a diagram illustrating an embodiment of a system 500 that records game event data 510 by a game recorder 502. System 500 includes a processor system 505. An example of processor system 505 is a server system comprising one or more servers in one or more data centers. Another example of processor system 505 includes one or more processors in a server system. Yet another example of processor system 505 includes one or more virtual machines. An example of game recorder 502 is a digital video recorder that records both video and audio data of game event data 510. Processor system 505 includes game recorder 502, game processor 506, and guidance processor 508. Game processor 506 is coupled to game recorder 502 and guidance processor 508. Guidance processor 508 is coupled to game recorder 502.

[0055] Game processor 506 executes game program 528 to facilitate gameplay for user 1 and other users. As an example, game program 528 includes a game engine and a rendering engine. The game engine is executed to determine the positioning and orientation of various virtual objects in the virtual scene. The virtual scene includes a background as an example of virtual objects. The rendering engine is executed to determine graphics parameters such as the color, intensity, shadows, and lighting of the virtual objects in the virtual scene. Game program 528 is stored in one or more memory devices of processor system 502 and said one or more memory devices are coupled to game processor 506. When game program 528 is executed, game event data 510 is generated and virtual scenes 110, 220, 302, and 402 are displayed on display device 1. Figures 1 to 4 ).

[0056] The generated game event data 510 is for user ID 1. For example, the virtual scene 110 of game event data 510 ( Figure 1 ), 202 Figure 2 ), 302 Figure 3 ) and 402 Figure 4This is generated when game program 528 is executed. Game program 528 is executed when processor system 505 authenticates user ID 1 and other information of user 1, such as password or phone number or a combination thereof.

[0057] As an example, game event data 510 includes events that occur in virtual scene 202 (…). Figure 2 In the game event data 510, multiple game events 1 to m occur when virtual user 102 dies, where m is a positive integer. For illustration, the game event data 510 includes virtual scene 110 ( Figure 1 The data for virtual scene 402 is 110. For further illustration, game event 512 is data for virtual scene 110, and game event 514 is data for virtual scene 202. As another example, game event data 510 includes data that causes virtual scene 402 to be displayed. Figure 4 In the game event data 510, multiple game events 1 to n occur for the death x of virtual user 102, where n and x are positive integers. For illustration, the game event data 510 includes virtual scene 302 ( Figure 3 The data for virtual scene 302 and 402 are listed below. For further illustration, game event 516 is data for virtual scene 302, and game event 518 is data for virtual scene 402. As another example, game event data 510 includes multiple game events 1 to 0 that cause virtual user 102's health level to decrease below a predetermined threshold, where 0 is a positive integer. For illustration, game event 520 is data for a virtual scene where virtual user 102 has full health, and game event 522 is data for a virtual scene where virtual user 102's health decreases below a predetermined threshold. As yet another example, game event data 510 includes multiple game events 1 to p that cause virtual user's health level to decrease below a predetermined threshold, where p is a positive integer. For illustration, game event 524 is data for a virtual scene where virtual user 102 has full health, and game event 526 is data for a virtual scene where virtual user 102's health decreases below a predetermined threshold. It should be noted that a series of game events from game event 520 to game event 522 occur during the same or different game sessions, followed by a series of game events from game event 524 to game event 526.

[0058] During the execution of game program 528, game recorder 502 records game event data 510. For example, the processor of game recorder 502 stores or writes the game event data 510 to one or more storage devices of game recorder 502. Examples of storage devices include read-only memory, random access memory, and combinations thereof. For illustration, the storage device is a flash memory device or a redundant array of independent disks (RAID).

[0059] During or after the execution of game program 528, recorded game event data 510 is provided from game recorder 502 to guidance processor 508. For example, guidance processor 508 periodically requests game recorder 502 to obtain game event data 510 from game recorder 502. In response to receiving a request from guidance processor 508, game recorder 502 accesses (e.g., reads) game event data 510 from one or more memory devices of game recorder 502 and sends game event data 510 to guidance processor 508. As another example, game recorder 502 periodically sends game event data to guidance processor 508. To illustrate, even without receiving any request from guidance processor 508, game recorder 502 accesses game event data 510 from one or more memory devices of game recorder 502 and sends game event data 510 to guidance processor 508.

[0060] During or after the execution of game program 528, instructor program 530 is executed by instructor processor 508. For example, instructor program 530 is executed in virtual scene 202 ( Figure 2 After display and in the virtual scene 302 ( Figure 3 ) before display or in virtual scene 402 ( Figure 4 ) will be executed after being displayed.

[0061] It should be noted that when instruction 530 is executed by processor system 505, instruction 530 performs the functions described herein as being performed by instruction 530. It should also be noted that in one embodiment, multiple game recorders are used instead of game recorder 502. Similarly, in one embodiment, multiple game processors are used instead of game processor 506. In one embodiment, multiple instruction processors are used instead of instruction processor 508.

[0062] In one implementation, the game recorder 502, the game processor 506, and the instruction processor 508 are coupled to each other via a bus.

[0063] In this implementation, the instruction program 530 is part of and integrated into the game program 528. For example, the computer program code of the instruction program 520 is interspersed with the computer program code of the game program 528.

[0064] In one implementation, game events are sometimes referred to herein as gameplay events.

[0065] In one implementation, whenever virtual user 102 dies, the guidance program 530 identifies the death from game event data 510. For example, the guidance program 530 determines that virtual scene 202 includes a virtual drone 204 located above virtual user 102 to determine that virtual user 102 has died. The guidance program 520 tags (such as assigns) keywords (such as death or extinction) to the virtual scene 202 in which virtual user 102 died, and stores the keywords as metadata in one or more memory devices of the processor system 505. Whenever virtual user 102 dies, the guidance program 530 determines that user 1's skill level is below a preset threshold stored in one or more memory devices of the processor system 505. For example, whenever virtual user 102 dies in the game, the game program 528 reduces the skill level corresponding to user ID 1 to below the preset threshold.

[0066] Figure 5B This is an example of a game logger 502 used to record game event data 504 for user IDs 2 to N. User IDs 2 to N are assigned to other users 2 to N who are playing the game. For example, user ID 2 is assigned to user 2, and user ID N is assigned to user N.

[0067] Game event data 504 includes a series of game events 1 to q for user IDX and another series of game events 1 to r for user IDN records. For example, game event 550 includes the execution of game program 528 ( Figure 5A Data accessed during gameplay of a virtual environment. Game event q includes the death q of virtual user 102. In processor system 505 ( Figure 5A After authenticating user IDN, access to game program 528 is granted. As another example, game event 552 includes data accessed during the execution of game program 528 related to a virtual scene. Game event 552 includes the death of a virtual user controlled by user N via game controller N. As another example, game event 554 includes data accessed during the execution of game program 528 related to a virtual scene, and game event 556 includes data accessed during the execution of game program 528 related to a virtual scene. Game event 556 includes the death of a virtual user controlled by user N via game controller N, where r is a positive integer. During or after the execution of game program 528, game event data 504 will be recorded to be compared with game event data 510 (…). Figure 5A The game recorder 502 provides the data to the guidance processor 508 in the same way. Figure 5A ).

[0068] Figure 6This diagram illustrates an implementation of a system 600 that uses a mentor 530 to generate a mentoring session 608 for training user 1. System 600 includes game event data 510 for user ID 1. Additionally, system 600 includes a mentor 530, game event data 504 for user IDs 2 through N, a client device 602, selected game event data 604, and metadata 606. Examples of client devices, as used herein, include combinations of desktop computers, laptops, tablets, smartphones, handheld remote controls and smart TVs, combinations of handheld controllers (HHC) and game consoles, and combinations of handheld controllers and head-mounted displays (HMDs). A game controller is an example of an HHC. Display device 1 is an example of a display device for client device 602.

[0069] Game event data 510 for user ID 1 includes video data 612 and audio data 614. As an example, the video data of the game event data includes the positioning, orientation, and graphical parameters of multiple virtual objects displayed in the virtual scene. Additionally, as an example, the audio data of the game event data consists of phonemes, words, lyrics, or phrases spoken by one or more virtual objects in the virtual scene. Similarly, game event data 504 for user IDs 2 through N includes video and audio data for each of user IDs 2 through N.

[0070] Examples of metadata 606 include game telemetry, such as one or more game states used to generate virtual objects in the game. For example, metadata 606 includes game states, based on which virtual objects not displayed, output, or spoken in a guided scene are displayed. As another example, metadata 606 is related to virtual scenes displayed during the execution of game program 528 (such as virtual scenes 110, 202, 302, or 402). Figures 1 to 4 Metadata 606 is information associated with the positioning and orientation of one or more virtual objects in virtual scene 110. For example, metadata 606 is the positioning of a virtual rectangle displayed around the positioning of a virtual bullet 106 shown in virtual scene 110. The virtual rectangle is not displayed in virtual scene 110. As another example, metadata 606 is information associated with the positioning of virtual objects not displayed in virtual scene 110, 202, 302, or 402. Figures 1 to 4 The positioning and orientation of virtual users in ) . The virtual user not shown fires 106 virtual bullets ( Figure 1 ) or 306 Figure 3The instruction procedure 530 generates a virtual user 102 that fires at a location next to (e.g., to the right) the location of the virtual bullet 106, so that the virtual user is displayed when the virtual bullet 106 is fired toward the virtual user 102. As yet another example, metadata 606 is the username of the virtual user who fires the virtual bullet but is not displayed in virtual scenes 110, 202, 302, or 402. As yet another example, metadata 606 is not used to generate a camera view from the perspective of a virtual user who is never displayed in virtual scenes 110, 202, 302, or 402. For example, metadata 606 is not used to generate a motion replay of virtual scenes 110, 202, 302, or 402 from the perspective of a virtual user who is never displayed in virtual scenes 110, 202, 302, or 402.

[0071] Examples of guided sessions 608 include the display of a guided scenario where user 1 cannot play a game but can learn based on games he / she has previously played, or games previously played by users 2 through N, or a combination thereof. For example, the guided program 530 is powered by a processor system 505 (…). Figure 5A The process executes to generate a guidance session 608 for user ID1. An example of a guidance scenario is a virtual scene generated and rendered when the guidance program 530 is executed. When guidance session 608 is generated, one or more guidance scenarios are displayed on client device 602. The guidance scenarios do not allow user ID to play the game when displayed to user ID. For example, during the display of a guidance scenario to user ID1, processor system 505 does not execute game program 528 for user ID1. Figure 5A For example, for user ID1, processor system 505 does not render the virtual scene generated by executing game program 528 during the display of one or more guided scenes to user ID1. Instead, processor system 505 executes a tutorial program 530 for rendering the one or more guided scenes for user ID1.

[0072] In operation 610, the guidance program 530 receives an input signal 616 from the client device 602 to initiate a guidance session 608. For example, during gameplay, user 1 selects a button on client device 1 to generate an input signal 616 for initiating the guidance session 608. For illustration, in virtual user 102 within virtual scene 202 (… Figure 2 ) or 402 Figure 4 After dying in the game, User 1 selects Game Controller 1 ( Figure 1 The button is selected. Upon receiving a selection, the game controller 1 generates an input signal 616 to initiate a guidance session 608. The input signal 616 is sent from the client device 602 to the guidance program 530.

[0073] The guidance procedure 530 receives an input signal 616 and identifies from the input signal 616 a guidance session 608 to be initiated. Upon identifying a guidance session 608 to be initiated, the guidance procedure 530 generates an output signal 618, which includes a request 620 for one or more user IDs 622 for which a guidance procedure 520 is to be initiated. For example, the output signal 618 includes querying the client device 602 to obtain information about whether the guidance procedure 530 is to be initiated based on gameplay by a combination of two or more of user ID1, user ID2, user IDN, or user ID (i.e., two or more of ID1, ID2, and IDN). For illustration, the output signal 618 includes querying to obtain a user ID1, user ID2, user IDN, or a combination of two or more of them for accessing gameplay of the game.

[0074] Client device 602 receives a request 620 for one or more user IDs 622 from instruction 530. For example, client device 602 displays the request 602 for one or more user IDs 622 on its display screen. For illustration, client device 602 displays a list of user IDs 1 to N on its display screen. As another example, client device 602 outputs sound via an earpiece worn by user 1 to query user 1 for one or more user IDs 622. The earpiece is part of or coupled to client device 602. As yet another example, client device 602 outputs sound via one or more speakers to query user 1 for one or more user IDs 622.

[0075] Client device 602 receives one or more user IDs 622 from user 1 and generates an input signal 623 including one or more user IDs 622. For example, user 1 selects one or more buttons on game controller 1 to provide one or more user IDs 622. For illustration, user 1 selects one or more buttons on game controller 1 to select one or more of user IDs 1 to N displayed on the screen of client device 602. Client device 602 sends the input signal 623 to instruction 530.

[0076] Upon receiving one or more user IDs 622 within input signal 623, the instructor 530 generates output signal 624, which includes a request 626 for identifying the game sequence type, for which an instructor session 608 is to be generated for one or more user IDs 622. For example, request 626 is a query about whether the game sequence type is type 1 or type 2. Examples of type 1 game sequences include multiple game events that cause a virtual user to die, and examples of type 2 game sequences include multiple game events that cause a virtual user's health level to decrease. For illustration, type 1 includes game events 512 to 514 and game events 516 and 518, and type 2 includes game events 520 to 522 and game events 524 to 526. Figure 5A ).

[0077] Client device 602 receives an output signal 624, including a request 626 for identifying the game sequence type, via its user interface or another type of interface, and outputs the request to user 1. For example, client device 602 displays or renders the request 626 via its display screen. For illustration, client device 602 displays a signal including virtual user 102 (…). Figures 1 to 4 The list of game sequence types whose death and the virtual user 102's health level decrease below a predetermined threshold. As another example, client device 602 outputs sound via one or more speakers of client device 602, and said sound includes a request 626 for a game sequence type. As yet another example, client device 602 outputs sound to user 1 via headphones to obtain a response to request 626.

[0078] In response to request 626, user 1 provides a game sequence type identifier 628 via the user interface or another type of interface of client device 602. For example, user 1 selects one or more buttons on game controller 1 to provide the game sequence type identifier 628. For illustration, user 1 selects one or more buttons to select a game sequence type where virtual user 102 dies or virtual user 102's health level decreases. Upon receiving a selection of game sequence type identifier 628, client device 602 generates an input signal 630 including game sequence type identifier 628 and sends input signal 630 to instruction program 530.

[0079] The guidance program 530 receives input signal 630 and identifies game sequence type identifier 628 from input signal 630. Upon identifying game sequence identifier 628, the guidance program 530 generates output signal 634, which includes a request 632 for identifying a game event of the sequence type identified by game sequence identifier 628. For example, to determine whether game event 514 or 518 (representing the death of virtual user 102) is pending. Figure 5A A guidance session 608 is generated, and the guidance program 530 generates a request 632 to identify game events 514 or 518. The guidance program 530 sends an output signal 634 to the client device 602.

[0080] Upon receiving output signal 634, client device 602 identifies request 632 from output signal 634 and provides request 632 to user 1 via client device 602's user interface or another interface. For example, client device 602 displays a list of game events 514 and 518 on its display screen. For illustration, client device 602 displays virtual scene 202 and virtual scene 402 on its display screen to allow user 1 to select one of virtual scenes 202 and 402.

[0081] User 1 responds to request 632 to identify a game event for which a guidance session 608 is to be generated. The game event is identified to provide a game event identifier 636. For example, User 1 selects virtual scene 202 to identify game event 514 or selects virtual scene 402 to identify game event 518 via one or more buttons on game controller 1. In response to receiving a selection of virtual scene 202 or 402, client device 602 generates an input signal 638 with game event identifier 636 and sends input signal 638 to guidance program 530.

[0082] Upon receiving input signal 638, the guidance program 530 identifies game event identifier 636 and selects game event data 604. For the game event data, a guidance session 608 is generated for game user ID 622. For example, the guidance program 530 identifies game event identifier 636, which is based on the virtual scene 202 displayed when game program 528 is executed for user ID 1. Figure 2 Recorded game events 514 ( Figure 5A Examples of selected game event data 604 include game event 514 or game event 518. Figure 5A ).

[0083] When identifying a game event identifier 636 for which a guidance session 608 is to be generated, the guidance program 530 is executed by the processor system 505 to store a portion of the game event data 510 recorded within a predetermined time interval from the time the game event identified by the game event identifier 636 was recorded, in order to output the selected game event data 604. For example, when identifying a game event 514 for which a guidance session 608 is to be generated, the guidance program 530 stores one or more game events recorded within a predetermined time interval prior to the time the game event 514 was recorded. The guidance program 530 also stores game event 514. One or more game events cause or result in the occurrence of game event 514. For illustration, the guidance program 530 deletes one or more game events recorded outside the predetermined time interval prior to the time the game event 514 was recorded from one or more memory devices of the processor system 505. Game event 514 and the game events within the predetermined time interval are stored by the guidance program 530 as the selected game event data 604. Additionally, in this example, the processor system 505 retains one or more game events recorded within a predetermined time interval prior to the recording of game event 514 in one or more memory devices of the processor system 505, to retain selected game event data 604. Deletion of one or more game events results in the retention of one or more game events, and the retention of one or more game events results in game event 514. The retained one or more game events are the selected game event data 604.

[0084] To further explain, during a game session, a series of virtual scenarios are generated and recorded while the game program 528 is executed. The series includes a first series, a second series, and a third series. The first series includes a first set of successive virtual scenarios in which virtual user 102 collects items, such as virtual weapons, and virtual objects, such as bricks or metal bars, to protect virtual user 102 from virtual bullets. The second series includes a second set of successive virtual scenarios in which virtual user 102 battles other virtual users. The third series includes a third set of successive virtual scenarios, such as virtual scenarios 110 and 202 (…). Figure 1 and Figure 2 Game events 512 and 514 are recorded for the virtual scene. Figure 5A The second series continues from the first series, and the third series continues from the second series. Processor system 505 deletes the virtual scenes of the first and second series and retains the virtual scenes of the third series. The third series is the selected game event data 604.

[0085] As another example, the instruction 530 stores selected game event data 604, which includes a portion of game event data 510, within one or more memory devices of the processor system 505. The one or more memory devices of the processor system 505 in which the selected game event data 604 is stored are different from, for example, not identical to, the one or more memory devices in which the game event data 510 is recorded. In one embodiment, instead of storing it in different memory devices, the selected game event data 604 is stored in a memory device at a different memory address than the game event data 510 at the memory address where it is recorded.

[0086] As yet another example, when identifying game event 514 for which a guidance session 608 is to be generated, the instructor 530 stores one or more game events recorded within a predetermined time interval before the time of recording game event 514, one or more game events recorded within a predetermined time interval after the time of recording game event 514, and game event 514 itself. The one or more game events recorded within the predetermined time interval before the time of recording game event 514, game event 514, and the one or more game events recorded within the predetermined time interval after the time of recording game event 514 constitute selected game event data 604. For example, when identifying game event 514 for which a guidance session 608 is to be generated, the instructor 530 stores one or more game events recorded within a predetermined time interval after the time of recording game event 514. Game event 514 is followed by one or more game events recorded within the predetermined time interval after the time of recording game event 514. For illustration, the instructor 530 deletes one or more game events recorded outside the predetermined time interval after the time of recording game event 514 from one or more memory devices of the processor system 505. One or more game events occurring within a preset time interval after the recording time of game event 514 result in the deletion of one or more game events. Additionally, in this example, the processor system 505 retains one or more game events recorded within a preset time interval after the recording time of game event 514 in one or more memory devices of the processor system 505.

[0087] To further explain, during a game session, a series of virtual scenes are generated when game program 528 is executed. The series includes a first series, a second series, and a third series. The first series includes a first set of successive virtual scenes in which, after virtual user 102 dies in game event 514 as indicated in virtual scene 202, virtual user 102 collects props, such as virtual weapons, and virtual objects such as bricks or metal bars to protect virtual user 102 from virtual bullets. The second series includes a second set of successive virtual scenes in which virtual user 102 dances with other virtual users. The third series includes a third set of successive virtual scenes in which virtual user 102 continues to dance with other virtual users. The second series succeeds the first series, and the third series succeeds the second series. Processor system 505 deletes the second and third series of virtual scenes and retains the first series of virtual scenes. The first series is the selected game event data 604.

[0088] Based on the selected game event data 604, the instructor 530 processes metadata 606 to generate and render virtual objects, such as overlays, which can improve user 1's gaming skills by allowing user 1 to see one or more reasons why a game event, identified by game event identifier 636, occurred. For example, the instructor 530 processes metadata 606 to identify the location and orientation of the virtual user shooting virtual user 102 and determines an overlay that includes the virtual user shooting virtual user 102. Once the overlay is displayed, user 1 can see the reason for virtual user 102's death. The reason for death is the virtual user shooting virtual user 102. The overlay is an example of a virtual object. Additionally, the instructor 530 generates a virtual box surrounding the virtual user to highlight the virtual user. As another example, the instructor 530 identifies the location of virtual bullet 106 from metadata 606 and determines that virtual user 102 will not be killed by virtual bullet 106 if virtual user 102 has already jumped or is protected by a virtual wall. The guidance program 530 generates instructions for displaying to user 1 to control virtual user 102 to jump and fire simultaneously or to build a wall around virtual user 102. The instructions enable user 1 to protect virtual user 102 from being hit by virtual bullets 106. User 1 can consult the instructions. When faced with a situation where virtual user 102 is about to be killed by another virtual bullet, if user 1 follows the instructions, virtual user 102 can be protected from the other virtual bullet.

[0089] The guidance procedure 530 generates an overlay of virtual objects based on metadata 606 to be overlaid on the selected game event data 604 to generate a guidance session 608. For example, the guidance procedure 530 overlays virtual frames around virtual objects stored in a virtual scene as the selected event data 604. As another example, the guidance procedure 530 overlays virtual objects onto a virtual scene 202. The virtual objects overlaid on the virtual scene 202 fire virtual bullets 106. Figure 1 As yet another example, instruction 530 adds the username or user ID of the virtual object that fired bullet 106 as an overlay item.

[0090] Instruction program 530 provides instruction session 608 to client device 602. For example, processor system 505 executes instruction program 530 to generate one or more instruction scenarios in which one or more virtual objects generated based on metadata 606 are overlaid on selected game event data 604. For illustration, one or more instruction scenarios include an overlay of one or more virtual objects on virtual scene 202. One or more instruction scenarios are displayed on the display screen of client device 602.

[0091] The guidance session 608 continues until input 640 is received from client device 602. For example, input 640 is generated when user 1 selects one or more buttons on game controller 1 to end guidance session 608. Input 640 is sent to instructor 530 within input signal 642. Instructor 530 identifies input 640 from instructor 530 to end guidance session 608 and ends guidance session 608. For example, instructor 530 ends guidance session 530 for user ID1 in response to receiving input 640. For illustration, game program 528 continues for user ID1 after guidance session 608 ends for user ID1.

[0092] In one implementation, the guidance procedure 530 does not request the client device 602 to identify the game sequence type and the game event that identifies the game sequence type. For example, the guidance procedure 530 does not send signals 624 and 634 to the client device 602. User 1 does not need to identify the game sequence type and the game event that identifies the game sequence type. Instead, the guidance procedure 530 determines to receive an input signal 616 for initiating a guidance session 608 immediately after the game event 514 or 518 in which the virtual user 102 dies. For example, the guidance procedure 530 determines to receive the input signal 616 at the end of game event 514 or 518 and before the occurrence of the game event following game event 514 or 518. Upon determining this, the guidance procedure 530 determines to generate a guidance session 608 for game event 514 or 518.

[0093] In one implementation, the guidance session 608 occurs during gameplay. For example, both the guidance program 530 and the game program 528 are executed simultaneously by the processor system 505. Before user 1 plays the game, the processor system 505 receives an input signal from the game controller 1 of client device 602 indicating that the guidance program 530 and the game program 528 should be executed simultaneously. Upon receiving the input signal, the processor system 505 executes the guidance program 530. When the guidance program 530 is executed, one or more virtual objects are generated by the guidance program 530 of the processor system 506 based on metadata 606 and rendered for display along with virtual objects in the virtual scenes generated by executing the game program 528. For example, during the guidance session 608, one or more virtual objects generated based on metadata 606 are displayed in virtual scenes 110, 202, 302, and 402.

[0094] In one implementation, the guidance scenario is referred to herein as a self-guided interface, such as a self-guided user interface.

[0095] Figure 7A It is used to instantiate one or more virtual objects and virtual scenes 110 ( Figure 1 A diagram illustrating the implementation of scenario 700, used together with guidance. One or more virtual objects are based on metadata 606 ( Figure 6 The guidance scenario 700 includes a virtual user 102, a virtual gun 108, a virtual bullet 106, and a virtual ramp 112. The guidance scenario 700 is generated during the execution of the guidance program 530. Figure 6 It is generated at time ) and displayed on display device 1.

[0096] Guidance scenario 700 includes virtual opinion 702, virtual box 704, another virtual box 706, gun ID 708, virtual box 710, bullet ID 712, gun ID 714, virtual box 716, virtual gun 718, virtual user 720, virtual box 722, virtual user ID 724, virtual game controller 726, and virtual button ID 728. Guidance scenario 700 does not include virtual tree 105 and virtual wall 107 to avoid confusion between guidance scenario 700 and virtual objects not needed by the user. Virtual opinion 702, virtual box 704, virtual box 706, gun ID 708, virtual box 710, bullet ID 712, gun ID 714, virtual box 716, virtual gun 718, virtual user 720, virtual box 722, virtual user ID 724, virtual game controller 726, and virtual button ID 728 are examples of one or more virtual objects generated based on metadata 606, such as overlays. It should be noted that virtual user 720 is not in virtual scenario 110. Figure 1 A) in.

[0097] Virtual suggestion 702 is a guidance message for user 1: "Jump now! Or build a wall now!" to avoid being hit by virtual bullet 106. Virtual box 704 expands around virtual user 102 to highlight virtual user 102. Similarly, virtual box 706 expands around virtual gun 108. Gun ID 708 identifies the type of virtual gun 108 to highlight virtual gun 108. For example, gun ID 708 identifies whether virtual gun 108 is a shotgun, pistol, semi-automatic gun, double-barreled gun, or AK-47. Virtual box 710 expands around virtual bullet 106 to highlight virtual bullet 106. Bullet ID 712 identifies the type of virtual bullet 106. Examples of virtual bullet types include lead bullets, semi-jacketed bullets, and fully jacketed bullets.

[0098] Virtual frame 716 extends around virtual gun 718 held by virtual user 720. Gun ID 714 identifies the type of virtual gun 718. Virtual frame 722 extends around virtual user 720 to highlight virtual user 720. Virtual user ID 724 is a user ID assigned to another user who controls virtual user 720, who fires virtual user 102 with virtual bullet 106. User ID 724 is controlled by processor system 505. Figure 5A The virtual game controller 726 is an image of the game controller 1 used or held by user 1 who plays the game and controls virtual user 102. The virtual button ID 728 is in the virtual scene 110 that causes virtual user 102 to die. Figure 1 The button selected by user 1 during the event. By providing metadata 606 in the virtual scene 700, the guidance procedure 530 facilitates user 1's self-guidance to improve user 1's skill level.

[0099] The guidance scene 700 is displayed along or simultaneously with the timeline scribble bar 730, which is a time bar, time scale, time axis, or timeline. The guidance program 530 generates and renders the timeline scribble bar 730 as part of the guidance session 608. User 1 uses game controller 1 to select one of many segments of the timeline scribble bar 732, such as segments 732, 734, and 736, to access and view the guidance scene 700 from the processor system 505. For example, when segment 732 is selected, the guidance scene 700 is displayed on display device 1.

[0100] The timeline scrubber 730 includes segments for displaying selected game event data 604 on the display screen 1. As an example, the timeline scrubber 730 includes segments, each of which can be selected by the user 1 via the game controller 1 to generate an input signal. When an input signal is received, a replay of the virtual scene that caused the game event, the virtual scene of the game event, and a virtual scene occurring after the game event are replayed by the guidance program 530, wherein overlay content is superimposed on one or more of the virtual scenes. As another example, the timeline scrubber 730 includes a set of segments 738 that the user 1 can select via the game controller 1 to display a guidance scene in which one or more virtual objects generated based on metadata 606 are superimposed on a portion of a virtual scene recorded during a predetermined time period prior to the time of recording the virtual scene (such as virtual scene 202) where the virtual user 102 dies. The predetermined time period extends from -P time units to 0 time units, where the time unit can be seconds or minutes. Furthermore, the timeline scrubber 730 includes a segment 736 of a guided scene that user 1 can select via game controller 1 to view, in which one or more virtual objects generated based on metadata 606 are overlaid on a portion of a virtual scene (such as virtual scene 202) recorded when virtual user 102 dies. When user 1 selects segment 736 of the timeline scrubber 730 via game controller 1, the guided scene, including one or more virtual objects generated based on metadata 606, is overlaid on one or more virtual objects in virtual scene 202, in which the virtual user 102 who dies is rendered by the guide program 530 and displayed on display device 1. The death of virtual user 102 is recorded at unit 0. The timeline scrubber 730 includes a set of segments 740 that user 1 can select via game controller 1 to display a guided scene, in which one or more virtual objects generated based on metadata 606 are overlaid on a portion of a virtual scene recorded during a preset time period following the time when the virtual scene (such as virtual scene 202) where virtual user 102 dies. The preset time period extends from 0 time units to M time units, where the time unit can be seconds or minutes. A time window exists between time units – P and M.

[0101] In one implementation, guidance scenario 700 does not include one or more of virtual frames 704, virtual frames 706, and gun ID 708. For example, guidance procedure 530 determines that one or more of virtual frames 704, virtual frames 706, and gun ID 708 are not needed to improve user 1's skill level, and therefore does not generate one or more of virtual frames 704, virtual frames 706, and gun ID 708.

[0102] In one implementation, instead of expanding the bounding box around a virtual object in the guidance scene, the guidance procedure 530 highlights the virtual object by rendering it with a color, intensity, or shadow that is significantly different from that of other virtual objects in the guidance scene. For example, instead of expanding the bounding box 710 around virtual bullet 106 and the bounding box 722 around virtual user 720, the guidance procedure 530 renders virtual bullet 106 and virtual user 720 with an intensity significantly greater than that of virtual user 102 and virtual gun 108. Furthermore, highlighting virtual bullet 106 and virtual user 720 compared to virtual user 102 and virtual gun 108 facilitates self-guidance for user 1.

[0103] In one implementation, the guidance scenario 700 includes a virtual tree 105 and a virtual wall 107.

[0104] In one implementation, the guidance procedure 530 changes the time unit—the time period between P and M—based on the number of game events that lead to game-ending events (such as death or a decrease in health points). For example, the guidance procedure 520 determines that the first time length of a combat sequence in which virtual user 102 participates that leads to virtual user 102's first death is greater than the second time length of a second combat sequence in which virtual user 102 participates that leads to virtual user 102's second death. The first combat sequence involves fighting one or more virtual users, and the second combat sequence involves fighting one or more virtual users. The guidance procedure 520 stores selected game event data 604 that leads to the first death for the first time length and stores selected game event data 604 that leads to the second death for the second time length. In this way, the guidance procedure 520 captures and stores the entire first and second combat sequences.

[0105] Figure 7B This is a diagram illustrating an implementation of a guidance scene 750 rendered and displayed on display device 1. The guidance scene 750 is rendered in response to an input signal instructing selection of a segment 752 on the timeline scrub bar 730. The segment 752 is selected by user 1 using one or more buttons on game controller 1.

[0106] Fragment 752 follows fragment 734, and fragment 734 follows fragment 732. For example, fragment 752 includes data from recording virtual scene 110 (…). Figure 1 Time and Recording of Virtual Scenes 202 Figure 2 One or more virtual objects of a virtual scene recorded between time intervals. Each virtual scene corresponding to a segment of the timeline scrubber 730 is recorded after being displayed on the display screen 1.

[0107] The guidance scenario 750 includes virtual opinion 754, virtual frame 756, virtual bullet 758, and bullet ID 760. Virtual opinion 754 is generated by guidance program 530 (…). Figure 6 This is generated to guide user 1 in protecting virtual user 102 from being hit by virtual bullets 758. For example, virtual advice 754 includes "Jump now." Virtual boxes 756 surround or encircle the virtual user 720 when firing virtual bullets 106. Figure 7A The virtual bullet 758 is then extended by the subsequent firing. A virtual frame 756 highlights the virtual bullet 758. The bullet ID 760 identifies the type of the virtual bullet 758. The guidance scene 750 does not include the virtual tree 105 and the virtual wall 107 to avoid confusing the guidance scene 750 with virtual objects that are not needed by the user 1.

[0108] It should be noted that in instructional scenario 750, examples and... Figure 7A The guidance scenario 700 compares the progress of virtual objects with virtual objects. For example, in guidance scenario 750, a virtual frame 756 is shown surrounding a virtual bullet 758 fired from a virtual gun 718. Virtual frame 756 is generated after a virtual frame 710 surrounding a virtual bullet 106 is generated. Virtual bullet 758 is fired after virtual bullet 106 is fired. Additionally, in guidance scenario 750, virtual opinion 754 instructs user 1 to control virtual user 102 to jump without the option to construct a virtual wall. Guidance procedure 530 determines that it is too late for user 1 to construct a virtual wall when virtual bullet 758 is fired.

[0109] In one implementation, when user 1 selects segment 753 following segment 752 via game controller 1, processor system 505 processes metadata 606 to display the location of the virtual bullet 758 closer to virtual user 102 compared to the location of the virtual bullet 758 illustrated in guidance scene 750. Segment 753 corresponds to a guidance scene including one or more virtual objects from a first virtual scene, which is temporally sequential from the time the game recorder 502 records the second virtual scene. The location of the virtual bullet 758 in the first and second virtual scenes defines the movement of the virtual bullet 758. Guidance scene 750 includes replayable virtual objects from the second virtual scene.

[0110] In one implementation, the guidance scenario 750 includes a virtual tree 105 and a virtual wall 107.

[0111] In one implementation, virtual suggestions 702 and 754 are examples of prompts for user 1 to raise their skill level above a preset threshold. For example, during a next game session, when user 1 controls virtual user 102 or constructs a virtual wall on at least one side of virtual user 102, the chance of virtual user 102 being hit by virtual bullets from the side or back of virtual user 102 is reduced. Virtual user 102 can survive longer during the game session, and game program 528 raises user 1's skill level above the preset threshold. Each suggestion provides a reason for virtual user 102's death. For example, virtual user 102 died in a previous game session because virtual user 102 did not construct a virtual wall or jump during the previous game session.

[0112] Figure 8A This diagram illustrates an implementation scheme in which guidance scene 700 and another guidance scene 800 are simultaneously displayed on display device 1. In addition to guidance scene 700, guidance scene 800 is also composed of guidance program 530 (…). Figure 6 ) is rendered on display device 1. For example, guide session 608 ( Figure 6 This includes guidance scenarios 700 and 800. As another example, input signals are received by the guidance program 530 from the client device 602. The input signals indicate the input to the virtual scenario 110 upon which the guidance scenario 700 is based. Figure 1 Game event identifier 636 (records game event data) Figure 6 The selection of ) and the virtual scene 302 on which the generated guidance scene 800 is based ( Figure 3 The selection of another game event identifier for the recorded game event data. In this example, when a selection of segment 732 on the timeline scrubber 730 is received from the client device 602, both guide scenes 700 and 800 are generated and rendered by the instructor 530. Additionally, in this example, the timeline scrubber 730 is generated by the instructor 530 and rendered along guide scenes 700 and 800.

[0113] Guidance Scenario 800 is based on virtual scenario 302 ( Figure 3 The virtual scene 302 is recorded, and one or more virtual objects from the virtual scene 302 are included in the guidance scene 800. For example, a virtual user 102, a virtual bullet 306, and a virtual gun 108 are included in the guidance scene 800. As another example, the guidance scene 800 is generated based on the virtual scene 302, which is displayed and recorded at a time corresponding to segment 732. For example, the virtual scene 302 is recorded in multiple time segments prior to the time of the game event q of the virtual user 102. The virtual scene 302 is recorded in the same manner as the virtual scene 110 from which the guidance scene 700 is generated.

[0114] Guidance scenario 800 includes user ID 802, virtual frame 804, virtual user 806, virtual gun 808, virtual frame 810, gun ID 812, virtual bullet ID 816, and virtual frame 818. User ID 802, virtual frame 804, virtual user 806, virtual gun 808, virtual frame 810, gun ID 812, bullet ID 816, and virtual frame 818 are based on metadata 606 (…). Figure 6 Examples of one or more overlay items of the generated virtual object.

[0115] Virtual frame 804 extends around virtual user 806 to highlight virtual user 806 as the cause of virtual user 102's death x. Virtual user 806 is controlled by another user via a game controller. Virtual frame 810 extends around virtual gun 808 held by virtual user 806 to highlight virtual gun 808. Gun ID 812 identifies the type of virtual gun 808. User ID 802 is assigned to the other user controlling virtual user 806. User ID 802 is determined by processor system 505 ( Figure 5A )distribute.

[0116] A virtual frame 818 extends around the virtual bullet 306 to highlight the virtual bullet 306 that the virtual user 806 points to the virtual user 102. A bullet ID 816 identifies the virtual bullet 306. Additionally, the guidance scenario 800 includes virtual comments 702 to facilitate guidance for user 1.

[0117] In one implementation, the virtual feedback displayed in guidance scenario 800 differs from virtual feedback 702. For example, the virtual feedback displayed in guidance scenario 800 is "Jump!" or "Build a wall" instead of "Jump now! or start building a wall now!".

[0118] In one implementation, the guidance procedure 530 does not highlight one or more virtual objects in the guidance scenario 800. For example, the guidance procedure 530 determines that it is not necessary to highlight virtual user 102 and virtual gun 108 to improve user 1's skill level, and determines not to generate virtual boxes 704 and 706.

[0119] Figure 8B This is a diagram illustrating an implementation scheme of a display device 1 that simultaneously renders and displays a guidance scenario 700 associated with user ID 1 and another guidance scenario 850 associated with user ID N. Guidance scenario 850 is composed of guidance program 530 ( Figure 5A Generate guidance scenario 850 for user ID N. For example, when input signal 623 ( Figure 6When a user ID1, identifying user 1, is included, and another user ID N, identifying user N assigned to user 1, is included, the instructor 530 accesses game event data 504 recorded for user ID N from one or more memory devices of the processor system 505, and stores a portion of the game event data 504 as part of selected game event data 604 in one or more memory devices of the processor system 505. For illustration, the one or more memory devices in which the game event data 504 is recorded are different from the one or more memory devices in which the selected game event data 604 is recorded. One or more virtual objects generated for user ID N based on metadata 606 are overlaid by the instructor 530 on a portion of the selected game event data 604 for user ID N to generate an instructional scene 850 for user ID N. Game event data 504 is generated when other users with user ID N play the game.

[0120] The guidance scenario 850 includes a virtual tree 852, a virtual user 854, a virtual gun 856, a gun ID 858, a user ID 860, a virtual frame 862, another virtual frame 864, a virtual user 866, a virtual gun 868, a virtual frame 870, another virtual frame 872, a gun ID 874, virtual bullets 876, a virtual frame 878, a gun ID 880, a virtual user 882, a virtual frame 884, a virtual gun 886, a virtual frame 888, a virtual wall 890, another virtual wall 892, virtual health 894, a user ID 897, and a virtual ramp 899. The guidance scenario 850 also includes a virtual controller 896 and a virtual button 898. User ID 860, Virtual Box 862, Virtual Box 864, Virtual User 854, Virtual Gun 856, Gun ID 858, Virtual Box 872, Gun ID 874, Virtual Box 888, Virtual Box 884, Virtual Box 870, Virtual Controller 896, and Virtual Button 898 are examples of one or more virtual objects generated based on metadata 606 for overlay.

[0121] Guidance Scenario 850 targets user ID N based on the execution of game program 528 ( Figure 5AThe virtual scene generated when virtual user 882 dies is generated. For example, the virtual scene generated according to the guidance scene 850 includes one or more virtual objects displayed in the guidance scene 850. The virtual scene generated according to the guidance scene 850 does not include one or more virtual objects generated based on metadata 606 for display in the guidance scene 850. For example, the virtual scene generated according to the guidance scene 850 includes virtual bullet 876, virtual gun 886, virtual user 882, virtual ramp 899, virtual walls 890 and 892, virtual tree 852, user ID 897, virtual user 866, and virtual gun 868. A virtual scene for user ID N is generated after another user accesses the game program 528. The other user accesses the game program 528 when user ID N is authenticated by the processor system 505.

[0122] Virtual frame 862 surrounds virtual user 854, and virtual frame 864 surrounds virtual gun 856. Gun ID 858 identifies the type of virtual gun 856. User ID 860 is assigned to the user who controls virtual user 854 via a game controller or another type of controller. Virtual frame 878 extends around virtual bullet 876 fired from virtual gun 856 towards virtual user 856.

[0123] Virtual health 894 represents the health of virtual user 882. Virtual frame 884 extends around virtual user 882, and virtual frame 888 extends around virtual gun 886. Virtual user 882 stands on virtual ramp 899 and is surrounded on both sides by virtual walls 890 and 892. Gun ID 880 identifies the type of virtual gun 886. Additionally, gun ID 874 identifies the type of virtual gun 868. Virtual frame 872 surrounds virtual gun 868, and virtual frame 870 surrounds virtual user 866. User ID 897 identifies the user who controls virtual user 866 via a game controller or another type of controller.

[0124] The virtual controller 896 is an image of a controller (such as a keyboard) used by another user to control the virtual user 882. Virtual buttons 898 identify buttons on the controller represented by the virtual controller 896. During gameplay, another user selects said buttons at times corresponding to segment 732.

[0125] The timeline scribble bar 730 is displayed along with or simultaneously with the display of the guidance scenes 700 and 850. For example, guidance program 530 ( Figure 6 The timeline scrubber 730 and guide scenes 700 and 850 are displayed simultaneously for displaying the timeline scrubber 730 and guide scenes 700 and 850. When from client device 602 ( Figure 6 ) game controller 1 ( Figure 1Upon receiving a selection for fragment 732, the guidance procedure 530 renders a guidance scene 850, which is generated based on a virtual scene recorded at the time corresponding to fragment 732. Fragment 736 indicates the time 0 during which virtual user 882 dies while another user controlling virtual user 882 is playing a game.

[0126] Virtual user 882 is firing at virtual user 866. During the gunfight, virtual user 854 is firing at virtual user 882. Virtual wall 892 protects virtual user 882 from virtual bullet 876. User 1 can learn from tutorial scenario 850 to construct virtual walls, such as virtual wall 890, on one side of virtual user 102 to protect virtual user 102 from virtual bullet 106 fired by virtual user 720.

[0127] In one implementation, a guidance engine is used instead of guidance procedure 530. As an example, an engine is a combination of software and hardware used to perform the functions described herein as being performed by an engine. For illustration, an engine is a PLD or ASIC programmed to perform the functions described herein as being performed by an engine.

[0128] In one implementation, any virtual box described herein surrounds a virtual object to highlight it.

[0129] Figure 8C This diagram illustrates an embodiment of a system 801 in which the processor system 505 generates audio data 805 as sound output to the user 1 via a headset 803, serving as an alternative to and supplement to rendering a guidance scene on the display screen 1. The headset 803 is worn by the user 1 and placed close to the user 1's ear. During the guidance session 608, the guidance program 530 generates audio data 805 as sound output to the user 1. For example, during the guidance session 608, as an alternative to and supplement to displaying virtual opinions 702, the guidance program 530 sends audio data 805 to the headset 803. The audio data 805 includes messages such as "jump and shoot simultaneously" as sound output or in lieu of the virtual opinions 702 displayed on the display screen 1. The headset 803 outputs the audio data 805 as a message to the user 1.

[0130] In one implementation, virtual objects 702, 704, 706, 708, 710, 712, 714, 716, 718, 720, 722, 724, 726, 728 ( Figure 7A ), 756, 758 ( Figure 7B ), 802, 804, 806, 808, 810, 812, 816, 818 ( Figure 8A), 854, 856, 858, 860, 864, 878, 880, 884, 888, 870, 872, 874 and 897 are examples of overlay content generated by Instruction 530 based on metadata 606.

[0131] Figure 9 This is used as an example to illustrate the use of the inference training engine 902 to generate guided sessions 608. Figure 6 A diagram illustrating an implementation scheme of system 900. System 900 includes an artificial intelligence (AI) processor 904, game event data 510 for user ID 1, and game event data 504 for user IDs 2 to N. The AI ​​processor 904 is a processor system 505 ( Figure 5A ) processor.

[0132] The inference training engine 902 includes a feature extractor 906, a feature classifier 908, and a model 910 to be trained. An example of each of the feature extractor 906, feature classifier 908, and model 910 is an ASIC. Another example of each of the feature extractor 906, feature classifier 908, and model 910 is a PLD. An example of model 910 is a network of circuit elements. Each circuit element has one or more inputs and one or more outputs. The inputs of the circuit elements are coupled to one or more outputs of one or more circuit elements. For illustration, model 910 is a neural network or artificial intelligence model. The feature classifier 908 is coupled to the feature extractor 906 and model 910.

[0133] The inference training engine 902 accesses (e.g., reads) game event data 510 and game event data 504 from one or more memory devices of the game recorder 502. The feature extractor 906 extracts features from the game event data 510 and 504. Once the features are extracted, the feature classifier 908 classifies the extracted features. The classified features are used to train the model 910 to determine game events, such as death or a decrease in health level, to initiate a guidance session 608 for said game event, and to identify one or more virtual objects in the virtual scene to be associated with one or more overlays of virtual objects generated based on metadata 606.

[0134] Figure 10 This is a flowchart illustrating an implementation scheme of method 1000 for training model 910. In operation 1002 of method 1000, feature extractor 906 extracts features from game event data 510 and game event data 504 ( Figure 9 ) Identifying features. For example, feature extractor 906 identifies virtual user 102 in virtual scene 202. Figure 2The feature extractor determines the death of the virtual user 102 by determining that the skeleton of the virtual user 102 is located in a horizontal plane rather than a vertical plane, or by determining that the virtual scene 202 includes a virtual drone 204 located above the skeleton of the virtual user 102 for detecting the virtual user 102, or by determining that the virtual user 102 in the virtual scene 202 utters words such as "I'm going to die" or "I'm dead," or a combination thereof. As another example, the feature extractor 906 determines that the virtual user 102's virtual health value level in the virtual scene 202 has been reduced to zero to determine that the health value level has been reduced below a predetermined threshold.

[0135] Operation 1004 of method 1000 occurs after operation 1002. In operation 1004, feature classifier 908 classifies the features extracted from game event data 510 and 504. For example, feature classifier 908 determines that virtual user 102 is dead because the skeleton of virtual user 102 is located in a horizontal plane. As another example, feature classifier 908 determines that virtual user 102 is dead because virtual scene 202 includes virtual drone 204 located above the skeleton of virtual user 102. As yet another example, feature classifier 908 determines that virtual user 102 is dead because virtual scene 202 includes the words "I" and "dead" or "I" and "dead" in the same sentence spoken by virtual user 102 in virtual scene 202. As yet another example, feature classifier 908 determines that virtual user 102's health is low because the health level of virtual user 102 in virtual scene 202 has decreased below a predetermined threshold.

[0136] Operation 1006 of method 1000 occurs after operation 1004. In operation 1006, model 910 is trained based on the number of sequence-type game events to determine the probability of a sequence-type game event occurring during the next game session. For example, model 910 determines the probability that virtual user 102 will die during the next game session or the probability that virtual user 102's health level will decrease below a threshold during the next game session. To illustrate, when it is determined that virtual user 102 has died in at least 6 of the past 10 game sessions, model 910 determines that the probability of virtual user 102 dying in the next game session is high, for example, above a preset threshold. On the other hand, when it is determined that virtual user 102 has survived in at least 6 of the past 10 game sessions, model 910 determines that the probability of virtual user 102 dying in the next game session is low, for example, below a preset threshold.

[0137] Operation 1008 of method 1000 follows operation 1006. Operation 1008 is executed by processor system 505. In operation 1008, when it is determined that the probability of a sequence-type game event occurring during the next game session is low, processor system 505 does not recommend initiating a guidance session 608 for the next game session. On the other hand, in operation 1008, when it is determined that the probability of a sequence-type game event occurring during the next game session is high, processor system 505 recommends initiating a guidance session 608 for the next game session. For example, processor system 505 generates a message and renders the message for display on display device 1. The message queries user 1 whether user 1 wishes to initiate guidance session 608. As another example, processor system 505 generates audio data including a message instructing user 1 whether user 1 wishes to initiate guidance session 608, and sends the audio data to headset 910 (…). Figure 9 Audio data, as sound, is output to user 1 via headphones 910.

[0138] When viewing messages displayed on screen 1 or listening to messages output as sound by headset 910, user 1 selects one or more buttons on game controller 1 to instruct the initiation of guidance session 608 in order to generate an input signal. Upon receiving the input signal instructing the initiation of guidance session 608, processor system 505 initiates guidance session 608. For example, processor system 505 renders guidance scene 700 (… Figure 7A ) or guidance scenario 750 ( Figure 7B ) or guidance scenarios 700 and 800 ( Figure 8A Both or guide scenarios 700 and 850 ( Figure 8B Both are used for display on display screen 1. On the other hand, when an input signal indicating that guidance session 608 should not be initiated is received, the processor system 505 does not initiate guidance session 608.

[0139] Figure 11A This is a diagram illustrating an embodiment of a system 1100 for communication between a processor system 505 and multiple devices, including a display device 1106 and a handheld controller 1108, via a router and modem 1104 and a computer network 1102. Examples of display devices 1106 include display device 1 (…). Figure 1LCD display devices, LED display devices, plasma display devices, and HMDs are examples of handheld controllers 1108. Examples of handheld controllers 1108 include touchscreens, keypads, mice, game controllers 1, and keyboards. Each of the display devices 1106 and handheld controllers 1108 is used by user 1. Additionally, each of users 2 through N uses a handheld controller with the same structure and function as handheld controller 1108 and a display device with the same structure and function as display device 1106. Display device 1106 and handheld controller 1108 are client devices 602. Figure 6 Examples of ).

[0140] System 1100 also includes a router and modem 1104, a computer network 1102, and a processor system 505. System 1100 also includes a headset 1110 and a display device 1106. Display device 1106 includes a display screen 1112, such as an LCD display, an LED display, or a plasma display. Display device 1 ( Figure 1 Example 1106 is a display device. Example 1102 of computer network includes the Internet or intranet or a combination thereof. Example 1104 of router and modem includes a gateway device. Another example of router and modem 1104 includes router devices and modem devices.

[0141] Display 1112 is coupled to router and modem 1104 via a wired connection. Examples of wired connections, as used herein, include a transport cable that transmits data serially or in parallel or via a Universal Serial Bus (USB) protocol.

[0142] The handheld controller 1108 includes controls 1118, a digital signal processor system (DSPS) 1120, and a communication device 1122. Controls 1118 are coupled to the DSPS 1120, which is coupled to the communication device 1122. Examples of controls 1118 include buttons and joysticks. Examples of communication devices 1122 include those enabling the use of wireless protocols such as Wi-Fi. TM or Bluetooth TM A communication circuit for communication between communication device 1122 and router and modem 1104.

[0143] Communication device 1122 is coupled to headset 1110 via a wired or wireless connection. Examples of wireless connections, as used herein, include applications using wireless protocols such as Wi-Fi. TM or Bluetooth TM The protocol is connected. Additionally, communication device 1122 is wirelessly coupled to router and modem 1104. Examples of wireless connections include Wi-Fi.TM Connection and Bluetooth TM Connection. Router and modem 1104 are coupled to computer network 1102, which is coupled to processor system 505.

[0144] During gameplay, processor system 505 executes game program 528 to generate image frame data from one or more game states, and applies network communication protocols, such as Transmission Control Protocol (TCP / IP) based on Internet Protocol, to the image frame data to generate one or more packets, which are then sent to router and modem 1104 via computer network 1102. The modem in router and modem 1104 applies the network communication protocol to the one or more packets received from computer network 1102 to obtain or extract image frame data, and provides the image frame data to the router in router and modem 1104. The router routes the image frame data to display screen 1112 via a wired connection between the router and display screen 1112 for displaying one or more images of the game based on the image frame data received within one or more packets.

[0145] During the display of one or more game images, game recorder 502 records game event data 510 and 504, which are used to generate a guidance session 608 or a training model 910, or a combination thereof. Additionally, during the execution of game program 528, user 1 selects or moves controls 1118 of handheld controller 1108 to generate input signals, such as input signals 616, 623, 630, 638, or 642, or input signals indicating the selection of segments of the timeline scribble bar 730, or any other input signals processed herein by DSPS 1120. DSPS 1120 processes (such as measuring, sampling, filtering, or amplifying, or a combination thereof) the input signals to output processed input signals having the same information as the information within the input signals. For example, DSPS 1120 identifies a button on game controller 1 selected by user 1. As another example, DSPS 1120 identifies a joystick on game controller 1 or a button on game controller 1 selected by user 1. The processed input signal is transmitted from DSPS 1120 to communication device 1122. Communication device 1122 applies a wireless protocol to the processed input signal to generate one or more wireless packets and transmits the wireless packets to router and modem 1104. The wireless packets include the same information as that included in the processed input signal.

[0146] The router in router and modem 1104 receives wireless packets from communication device 1122 and applies a wireless protocol to obtain or extract processed input signals from the wireless packets. The router in router and modem 604 provides the processed input signals to the modem in router and modem 604. The modem applies a network communication protocol to the processed input signals to generate one or more network packets. For example, the modem determines that the processed input signal will be sent to processor system 502 executing game program 506 and embeds the network address of processor system 505 within one or more network packets. The modem sends one or more network packets to processor system 505 via computer network 1102.

[0147] The processor system 505 applies network communication protocols to one or more network packets received from the router and modem 1104 to obtain or extract information within the processed input signal, and refers to the above. Figure 6 The information is processed in an interpretable manner to initiate guidance session 608 or to recommend initiating guidance session 608. Figure 6 The processor system 505 generates output signals, such as output signals 618, 624, or 634. Figure 6 The processor system 505 transmits one or more network packets to the router and modem 1104 via the computer network 1102, or any other output signal described herein, or any other output signal described herein that includes image frame data of the guiding scene, or any other output signal described herein that includes audio data, and applies a network communication protocol to the output signal to generate one or more network packets.

[0148] The modem in router and modem 1104 applies a network communication protocol to one or more network packets received via computer network 1102 to obtain or extract an output signal. The router in router and modem 1104 applies a wireless protocol to the output signal to generate one or more wireless packets and transmits the wireless packets to communication device 1122 of handheld controller 1108. Communication device 1122 of handheld controller 1108 applies a wireless protocol to one or more wireless packets received from router and modem 1104 to obtain or extract an output signal and transmits the output signal to headset 1110 to output audio data as sound to user 1. For example, communication device 1122 applies a wired protocol, such as Universal Serial Bus (USB) protocol, to generate one or more packets with audio data and transmits one or more packets to headset 1110 via a wired connection. As another example, communication device 1122 applies a wireless protocol to audio data to generate one or more wireless packets and transmits one or more wireless packets to headset 1110 via a wireless connection.

[0149] Additionally, the router in router and modem 1104 sends an output signal to display screen 1112. Upon receiving the output signal, display screen 1112 displays one or more image frames based on the image frame data of the output signal.

[0150] In one implementation, the communication device 1122 communicates with the router and modem 1104 via a wired connection (such as a cable).

[0151] In one embodiment, display screen 1112 is coupled to router and modem 1104 via a communication device, such as a communication device applying a wireless communication protocol. The communication device is part of display device 1106. For example, display screen 1112 is coupled to the communication device. Router 1104 applies a wireless protocol to image frame data received via computer network 1102 to generate one or more wireless packets, and transmits said one or more wireless packets to the communication device of display device 1106. The communication device applies a wireless protocol to the one or more wireless packets to extract or obtain image frame data, and transmits the image frame data to display screen 1112 for displaying one or more images of the instructional scenario described herein.

[0152] In one implementation, the display device 1106 and the handheld controller 1108 are integrated into a mobile device such as a smartphone, tablet, or laptop.

[0153] Figure 11BThis diagram illustrates an implementation of system 1120, where processor system 505 can be implemented within game console 1122. System 1120 includes a display device 1106, a handheld controller 1108, a headset 1110, game console 1122, a router and modem 1104, a computer network 1102, and a server system 1124. An example of game console 1124 is a video game console or a computer or a combination of a central processing unit (CPU) and a graphics processing unit (GPU). For illustration, game console 1124 is the Sony PlayStation. TM Or Microsoft Xbox TM The game console 1124 includes a processor system 505 and communication devices 1126, such as Wi-Fi. TM Communication devices or Bluetooth TM Communication devices. As an example, a processor system as used herein includes one or more CPUs and one or more GPUs, with the one or more CPUs coupled to one or more GPUs.

[0154] Example of server system 1124 includes one or more servers within one or more data centers. For illustration, the server could be a game console or a blade server. As another example, server system 1124 includes one or more virtual machines. Communication device 1126 is connected via a wireless connection such as Wi-Fi. TM Connection or Bluetooth TM A communication device is connected to display device 1106. Additionally, communication device 1126 is wirelessly coupled to communication device 1122 of handheld controller 1108. Communication device 1126 is coupled to processor system 505. Processor system 505 is wired to router and modem 1104. Router and modem 1104 is coupled to server system 1124 via computer network 1102.

[0155] Processor system 505 executes the game in place of or in conjunction with server system 1124 for displaying a virtual scene on display screen 1112 of display device 1106. For example, in response to receiving login information provided by user 1 via game controller 1, processor system 505 sends a request to server system 1124 via computer network 1102 to determine whether the login information is valid. When processor system 505 receives an indication from server system 1124 via computer network 1102 that the login information received from game controller 1 is valid, processor system 505 executes game program 528 for user 1 to play the game via game controller 1 and game console 1122. On the other hand, when processor system 505 receives an indication from server system 1124 via computer network 1102 that the login information received from game controller 1 is invalid, processor system 505 does not execute game program 528 for user 1 to play via game controller 1 and game console 1122.

[0156] Communication device 1126 receives input signals from handheld controller 1108, such as input signals 616, 623, 630, 638, or 642. Figure 6 The processor system 505 provides an input signal, or a wireless packet indicating the selection of a segment of the timeline rubbing bar 730, or any other input signal described herein, and applies a wireless protocol to the wireless packet to extract the input signal from the wireless packet, and provides the input signal to the processor system 505. The processor system 505 generates an output signal, such as output signal 618, 624, or 634, based on the input signal in the manner described above. Figure 6 This could be any other output signal described herein, or any other output signal described herein that includes image frame data of the guidance scene, or any other output signal described herein that includes audio data. For example, processor system 505 generates data for guidance session 608. Processor system 505 provides the output signal to communication device 1126. Communication device 1126 applies a wireless protocol to the output signal to generate one or more wireless packets and sends the wireless packets to communication device 1122 of handheld controller 1108, or display device 1106, or both display device 1106 and handheld controller 1108.

[0157] In one implementation, some of the functions described herein as being performed by processor system 505 are performed by processor system 505 of game console 1122, while the remaining functions described herein as being performed by processor system 505 are performed by server system 1124 instead.

[0158] In one embodiment, processor system 505 is coupled to display device 1106 via a wired connection. Output signals are sent from processor system to display device 1106 via wired connection for displaying one or more images of the instructional scene or virtual scene described herein on display device 1106.

[0159] In one implementation, the game is stored on a game console 1122 and is an offline game. In this implementation, the game is either a single-player game or a multiplayer game, and the game console 1122 does not communicate with the server system 1124 to access any part of the game from the server system 1124.

[0160] Figure 11C This diagram illustrates an implementation of system 1140 for communication between smartphone 1142 and processor system 505 via computer network 1102 without the need for a router and modem 1104 between computer network 1102 and smartphone 1142. System 1140 includes smartphone 1142, headset 1110, cellular network 1144, computer network 1102, and processor system 505. Smartphone 1142 is client device 602 ( Figure 6 Examples of ).

[0161] Smartphone 1142 is coupled to cellular network 1144 via a cellular wireless connection such as a fourth-generation cellular (4G) connection or a fifth-generation cellular (5G) connection. Cellular network 1144 is coupled to computer network 1102, which is coupled to processor system 505.

[0162] Smartphone 1142 generates one or more packets by applying cellular communication protocols such as 4G or 5G protocols to input signals, such as input signals 616, 623, 630, 638, or 642, or input signals indicating the selection of segments of timeline scribbles 730, or any other input signals described herein, and sends one or more packets to cellular network 1144. Cellular network 1144 receives one or more packets and applies cellular communication protocols to obtain or extract input signals, and applies network communication protocols to input signals to generate one or more network packets. One or more network packets generated by cellular network 1144 are sent to processor system 505 via computer network 1102. Processor system 505 processes one or more network packets received from cellular network 1102 in the manner described above to generate output signals, such as output signals 618, 624, or 634, or any other output signals described herein, or any other output signals described herein including image frame data guiding the scene, or output signals described herein including audio data, and sends one or more network packets including the output signals to cellular network 1144 via computer network 1102.

[0163] Cellular network 1144 applies network communication protocols to one or more network packets received from processor system 505 to extract or obtain output signals, and applies cellular communication protocols to the output signals to generate one or more packets. Cellular network 1102 transmits one or more packets, including the output signals, to smartphone 1142.

[0164] Figure 12A This is a diagram illustrating an embodiment of the headset 1110. The headset 1110 includes a communication device 1202, a digital-to-analog (D / A) converter 1204, an audio amplifier 1206, and a speaker 1208. An example of the communication device 1202 is a communication circuit that applies a wired or wireless protocol.

[0165] Communication device 1202 is coupled to communication device 1122 of handheld controller 1108. Figure 11A ) or smartphone 1142 ( Figure 11C A digital-to-analog converter 1204 is coupled to a communication device 1202, and an audio amplifier 1206 is coupled to a digital-to-analog converter 1204. Additionally, a speaker 1208 is coupled to an audio amplifier 1206.

[0166] Communication device 1202 from communication device 1122 ( Figure 11A ) or from smartphone 1142 ( Figure 11C ) Receives audio data 805 ( Figure 8CThe communication device 1202 sends the audio data 805 to a digital-to-analog converter 1204. The digital-to-analog converter 1204 converts the audio data 805 from digital format to analog format to output an analog audio signal. The digital-to-analog converter 1204 sends the analog audio signal output based on the audio data 805 to an audio amplifier 1206. The audio amplifier 1206 amplifies (e.g., increases) the amplitude or magnitude of the analog audio signal to output an amplified audio signal, which is an electrical signal. A loudspeaker 1208 converts the electrical energy of the amplified audio signal into sound energy to output a signal to be heard by user 1. Figure 1 The sound heard.

[0167] In one implementation, multiple speakers are used instead of speaker 1208.

[0168] Figure 12B This is a diagram illustrating an embodiment of a display device 1250 for displaying images of the instructional or virtual scene described herein on a display screen 1112. The display device 1250 is a component of display device 1 (…). Figure 1 Examples of display devices 1250 include communication devices 1252 and display screens 1112. Examples of display devices 1250 include LCD display devices, LED display devices, and plasma display devices. Examples of display screens 1112 include LCD displays, LED displays, and plasma displays. For illustration, display device 1250 is a display device for a smartphone 1142, a game controller 1, a tablet computer, or a computer. Examples of computers include desktop computers and laptop computers. Examples of communication devices 1252 include communication circuitry that applies wired or wireless protocols for data communication. Communication device 1252 is coupled to display screen 1112.

[0169] As a means of generating other forms of data such as audio data 805 ( Figure 8C As a substitute or supplement to the above, processor system 505 generates image frame data of the guided scene or virtual scene described herein. In the same manner as processor system 505 generates one or more packets with audio data 505, processor system 505 or communication device 1126 ( Figure 11B One or more packets are generated by applying protocols, such as network communication protocols, wired protocols, or wireless protocols, to the image frame data, and one or more packets are sent to the display device 1250. For example, see reference... Figure 11A The processor system 505 applies network communication protocols to the image frame data to generate one or more network packets, and sends one or more network packets via computer network 1102 to router and modem 1104. Figure 11ARouter and modem 1104 processes one or more network packets with image frame data in the same manner as router and modem 1104 processes one or more network packets with audio data 805 to obtain image frame data from one or more network packets, applies a wireless protocol to the image frame data to generate one or more wireless packets, and transmits one or more wireless packets to communication device 1252. As another example, see reference... Figure 11B The communication device 1126 of the game console 1122 applies a wireless protocol to the image frame data to generate one or more wireless packets, and sends the wireless packets to the communication device 1252 of the display device 1250. As another example, see [reference needed]. Figure 11C Cellular network 1144 receives one or more network packets containing image frame data from processor system 505 via computer network 1102, applies network communication protocols to extract image frame data from the one or more network packets, and applies cellular communication protocols to the image frame data to generate one or more network packets. Cellular network 1144 then sends the one or more packets containing image frame data to smartphone 1142.

[0170] Return to reference Figure 12B The communication device 1152 receives one or more packets containing image frame data, and applies protocols such as cellular communication protocols, wired protocols, or wireless protocols to extract or obtain the image frame data from the one or more packets, and sends the image frame data to the display screen 1112. Upon receiving the image frame data, the display screen 1112 displays the guidance scene or virtual scene described herein.

[0171] Figure 13 This is a flowchart conceptually illustrating various operations performed according to an implementation of this disclosure for streaming a cloud video game to a client device. Examples of client devices include game controllers, smartphones, game consoles, and computers. Game server 1302 executes the video game and generates raw (uncompressed) video 1304 and audio 1306. Game event data 510 or game event data 504, or a combination thereof, is an example of a record of a combination of video 1004 and audio 1306. Game server 1302 is an example of processor system 505. Video 1304 and audio 1306 are captured and encoded for streaming purposes, as indicated by reference numeral 1308 in the illustrated figures. Encoding provides compression of the video and audio streams to reduce bandwidth usage and optimize the gaming experience. Examples of encoding formats include H.265 / MPEG-H, H.264 / MPEG-4, H.263 / MPEG-4, H.262 / MPEG-2, WMV, VP6 / 7 / 8 / 9, etc.

[0172] The encoded audio 1310 and encoded video 1312 are further divided into network packets, as indicated by reference numeral 1314, for transmission over computer network 1320, which is computer network 1102. Figure 11A Examples are provided. In some implementations, the network packet coding process also employs a data encryption process, thereby providing enhanced data security. In the illustrated implementation, audio packets 1316 and video packets 1318 are generated for transmission over computer network 1320.

[0173] The game server 1302 also generates haptic feedback data 1322, which is also packetized into network packets for network transmission. In the illustrated implementation, haptic feedback packets 1324 are generated for transmission over computer network 1320.

[0174] The aforementioned operations of generating raw video, audio, and haptic feedback data are performed on the game server 1302 in the data center, and the streaming engine of the data center performs the following operations: encoding the video and audio, and packetizing the encoded audio / video and haptic feedback data for transmission. As indicated by reference numeral 1320, the audio, video, and haptic feedback packets are transmitted over a computer network. As indicated by reference numeral 1326, audio packet 1316, video packet 1318, and haptic feedback packet 1324 are received by client device 602 (…). Figure 6 The data is fragmented, for example, parsed, to extract encoded audio 1328, encoded video 1330, and haptic feedback data 1322 from network packets at client device 602. If the data has been encrypted, it is also decrypted. The encoded audio 1328 and encoded video 1330 are then decoded by the client device, as indicated by reference numeral 1334, to generate raw audio and video data on the client side for rendering on display device 1340 of client device 602. The haptic feedback data 1322 is processed by the processor of client device 602 to produce haptic feedback effects at controller device 1342 or other interface devices (e.g., HMD, etc.) where haptic effects can be presented. Controller device 1342 is an example of game controller 1. An example of a haptic effect is vibration or rumble from controller device 1342.

[0175] It should be understood that video games respond to user input and therefore execute a program flow similar to the above-described program flow for transmitting and processing user input, but in the opposite direction from the client device to the server. As shown, controller device 1342 or another input device, such as a body part of user 1, or a combination thereof, generates input data 1348. Control input signals 616, 623, 630, 638, and 642 ( Figure 6Any of the following is an example of input data 1348. Input data 1348 is packetized at the client device for transmission to the data center via computer network 1320. Input data packet 1346 is depacked and reassembled by game server 1302 to define input data 1348 on the data center side. Input data 1348 is fed to game server 1302, which processes input data 1348 to generate the game state.

[0176] During the transmission of audio packets 1316, video packets 1318, and haptic feedback packets 1324 via computer network 1320, in some embodiments, the data transmitted through computer network 1320 is monitored to ensure quality of service. For example, as indicated by reference numeral 1350, the network condition of computer network 1320 is monitored, including both upstream and downstream network bandwidth, and the game streaming is adjusted in response to changes in available bandwidth. That is, as indicated by reference numeral 1352, the encoding and decoding of network packets are controlled based on the current network condition.

[0177] Figure 14 This is a block diagram of an implementation of a game console 1400 compatible with a display device for interfacing with a client device and capable of communicating with a game hosting system (such as processor system 505 (Figure 5)) via computer network 1320. Game console 1122 (Figure 11) is an example of game console 1400. Game console 1400 is located within data center A or at the location of user 1. In some implementations, game console 1400 is used to execute a game displayed on HMD 1405. Game console 1400 is equipped with various peripheral devices that can be connected to game console 1400. Game console 1400 has a unit processor 1428, a dynamic random access memory (XDRAM) unit 1426, a reality synthesizer graphics processor unit 1430 with a dedicated video random access memory (VRAM) unit 1432, and an input / output (I / O) bridge 1434. Game console 1400 also has Blu-ray discs for reading from disk 1440a. The game console 1400 includes a disc-only memory (BD-ROM) optical disc reader 1440 and a removable insertable hard disk drive (HDD) 1436 accessible via an I / O bridge 1434. Optionally, the game console 1400 also includes a compact flash memory card reader, similarly accessible via the I / O bridge 1434. A memory card reader 1438 for memory cards, etc. An I / O bridge 1434 also connects to a USB 2.0 port 1424, a Gigabit Ethernet port 1422, and an IEEE 802.11b / g wireless network (Wi-Fi). TMPort 1420 and Bluetooth connectivity Wireless link port 1418.

[0178] During operation, the I / O bridge 1434 handles all wireless, USB, and Ethernet data, including data from the game controller and from the HMD 1405. For example, when user 1 is playing, by executing game code such as game program 528 ( Figure 11A When a game is generated as part of a process, I / O bridge 1434 receives input data or signals described herein from game controller 1342 or 1403 and / or from HMD 1405 via a Bluetooth link, and directs the input data to unit processor 1428, which updates the current state of the game accordingly. For example, a camera within HMD 1405 captures the pose of user 1 to generate an image representing the pose. Game controller 1342 is an example of game controller 1, and the game controller is an example of HHC.

[0179] In addition to game controllers 1342 and 1403 and HMD 1405, wireless, USB, and Ethernet ports provide connectivity for other peripheral devices such as, for example, remote control 1404, keyboard 1406, mouse 1408, and portable entertainment devices 1410 (such as, for example, Sony PlayStation). Entertainment equipment, etc.), cameras (such as, for example, Camera 1412, etc.), microphone / earphone 1414, and microphone 1415. Portable entertainment device 1410 is an example of a game controller. In some implementations, such peripheral devices are wirelessly connected to game console 1400, for example, portable entertainment device 1410 via Wi-Fi. TM Communication is achieved via a dedicated connection, while the microphone headset 1414 communicates via a Bluetooth link. The microphone headset 1414 is a type of over-ear headphone 803 (…). Figure 8C Examples of ).

[0180] The availability of these interfaces means that the Game Console 1400 is also potentially compatible with other peripherals such as digital video recorders (DVRs), set-top boxes, digital cameras, portable media players, Internet Protocol (IP) phones, mobile phones, printers, and scanners.

[0181] In addition, a conventional memory card reader 1416 is connected to the game console 1400 via a USB port 1424, enabling it to read memory cards 1448 of the type used by the game console 1400. Game controllers 1342 and 1403 and HMD 1405 are operable to communicate wirelessly with the game console 1400 via a Bluetooth link 1418, or to connect to the USB port 1424, thereby also receiving power from the batteries of game controllers 1342 and 1403 and HMD 1405. In some embodiments, each of game controllers 1342 and 1403 and HMD 1405 includes memory, a processor, a memory card reader, persistent memory such as flash memory, light emitters such as light-emitting spherical segments, light-emitting diodes (LEDs), or infrared light, a microphone and speaker for ultrasonic communication, a soundproof enclosure, a digital camera, an internal clock, a spherical segment with a recognizable shape such as facing the game console 1400, and a wireless device using protocols such as Bluetooth, Wi-Fi, etc.

[0182] Game controller 1342 is designed for use by user 1's two hands, and game controller 1403 is a one-handed controller with an accessory. HMD 1405 is designed to fit above user 1's head and / or in front of their eyes. In addition to one or more analog joysticks and conventional control buttons, each game controller 1342 and 1403 is susceptible to three-dimensional position determination. Similarly, HMD 1405 is susceptible to three-dimensional position determination. Therefore, in some embodiments, as a supplement to or alternative to conventional button or joystick commands, user 1's gestures and movements of game controllers 1342 and 1403 and HMD 1405 are translated into game input. Optionally, other wirelessly enabled peripherals such as PlayStation... TM Portable devices, etc., are used as controllers. (Regarding PlayStation...) TM For portable devices, additional game or control information, such as control commands or the number of lives, is displayed on the device's screen. In some implementations, other alternative or supplementary control devices are used, such as, for example, a dance mat (not shown), a light gun (not shown), a steering wheel and pedals (not shown), a custom controller, etc. Examples of custom controllers include a single or several large buttons for a quick quiz game (also not shown).

[0183] The remote control 1404 can also be operated to wirelessly communicate with the game console 1400 via Bluetooth link 1418. The remote control 1404 includes components suitable for operating Blu-ray... TM The BD-ROM reader 1440 is suitable for navigation disk content control.

[0184] In addition to regular pre-recorded and recordable CDs and so-called Super Audio CDs, Blu-ray... TM The Blu-ray BD-ROM reader 1440 is operable to read CD-ROMs compatible with the game console 1400. In addition to regular pre-recorded and recordable DVDs, Blu-ray... TM The BD-ROM reader 1440 is also operable to read digital video disc-ROMs (DVD-ROMs) compatible with the game console 1100. TM The BD-ROM reader 1440 is further operable to read BD-ROMs compatible with the game console 1400, as well as regular pre-recorded and recordable Blu-ray discs.

[0185] The game console 1400 is operable to supply audio and video generated or decoded via the reality synthesizer graphics unit 1430 to a display and sound output device 1442 having a display screen 1444 and one or more loudspeakers 1446, such as a monitor or television, via an audio connector 1450 and a video connector 1452. Wireless link port 1118 supplies audio and video to the display device of HMD 1405. In various implementations, audio connector 1450 includes conventional analog and digital outputs, while video connector 1452 differently includes component video, S-video, composite video, and one or more High Definition Multimedia Interface (HDMI) outputs. Therefore, video output can be in formats such as Phase-Inverted (PAL) or National Television System Committee (NTSC), or in 2220p, 1080i, or 1080p high-definition formats. Audio processing, such as generation and decoding, is performed by unit processor 1408. The operating system of game console 1400 is supported. 5.1 surround sound Cinema surround sound (DTS) and from The disc decodes 7.1 surround sound. Display and audio output device 1442 is display device 1 ( Figure 1 Examples of ).

[0186] In some embodiments, the camera, such as camera 1412, includes a single charge-coupled device (CCD), an LED indicator, and hardware-based real-time data compression and encoding means, such that compressed video data is transmitted in a suitable format, such as an in-image-based Moving Picture Experts Group (MPEG) standard, for decoding by the game console 1400. The LED indicator of camera 1412 is arranged to illuminate in response to appropriate control data from the game console 1400, for example, to indicate unfavorable lighting conditions. Some embodiments of camera 1412 are connected to the game console 1400 via USB, Bluetooth, or Wi-Fi communication ports. Various embodiments of the camera include one or more associated microphones and are also capable of transmitting audio data. In several embodiments of the camera, the CCD has a resolution suitable for high-definition video capture. In use, the images captured by the camera are incorporated into the game or decoded as game control input. In another embodiment, the camera is an infrared camera suitable for detecting infrared light.

[0187] In various implementations, in order to enable successful data communication between one of the communication ports of the game console 1400 and a peripheral device (such as, for example, a camera or remote control), appropriate software, such as a device driver, is provided.

[0188] In some implementations, the aforementioned system devices, including game console 1400, HHC, and HMD 1405, enable HMD 1405 to display and capture video of an interactive session of the game. The system devices initiate an interactive session of the game, defining the interactivity between User 1 and other users with the game. The system devices further determine the initial positioning and orientation of the HHC and / or HMD 1405 operated by User 1. Game console 1400 determines the current state of the game based on the interactivity between User 1 and the game. The system devices track the positioning and orientation of the HHC and / or HMD 1405 during the interactive session between User 1 and the game. The system devices generate a viewer video stream of the interactive session based on the current state of the game and the tracked positioning and orientation of the HHC and / or HMD 1405. In some implementations, the HHC renders the viewer video stream on its display screen. In various implementations, HMD 1405 renders the viewer video stream on its display screen.

[0189] refer to Figure 15 The diagram shows a component of HMD 1502. HMD 1502 is an example of HMD1405. Figure 14Examples of HMD1502 include a processor 1500 for executing program instructions. A memory device 1502 is provided for storage purposes. Examples of memory device 1502 include volatile memory, non-volatile memory, or a combination thereof. A display device 1504 is included, which provides any user 1 ( Figure 1 A visual interface for viewing, such as displaying image frames generated from saved data, etc. A battery 1506 is provided as a power source for the HMD 1502. The motion detection module 1508 includes any of various motion-sensitive hardware such as a magnetometer 1510, an accelerometer 1512, and a gyroscope 1514.

[0190] An accelerometer is a device used to measure acceleration and the reaction force induced by gravity. Single-axis and multi-axis models can be used to detect the magnitude and direction of acceleration in different directions. Accelerometers are used to sense tilt, vibration, and shock. In one embodiment, three accelerometers 1512 are used to provide the direction of gravity, which gives an absolute reference to two angles, such as world space pitch and world space roll.

[0191] Magnetometers measure the strength and direction of the magnetic field near HMD 1502. In some embodiments, three magnetometers 1510 are used within HMD 1502 to ensure an absolute reference for the yaw angle in world space. In various embodiments, the magnetometers are designed to span the Earth's magnetic field of ±80 microtesla. Magnetometers are affected by metals and provide yaw measurements that vary monotonically with actual yaw. In some embodiments, the magnetic field is distorted due to metals in the real-world environment, resulting in distortion of the yaw measurement results. In various embodiments, this distortion is calibrated using information from other sensors such as gyroscope 1514, camera 1516, etc. In one embodiment, accelerometer 1512 is used in conjunction with magnetometer 1510 to obtain the tilt and azimuth angles of HMD 1502.

[0192] A gyroscope is a device used to measure or maintain orientation based on the principle of angular momentum. In one embodiment, instead of gyroscope 1514, three gyroscopes provide information about movement across corresponding axes (x, y, and z) based on inertial sensing. Gyroscopes help detect rapid rotation. However, in some embodiments, the gyroscopes drift over time in the absence of an absolute reference. This triggers periodic gyroscope resets, which can be accomplished using other available information such as positioning / orientation determination, based on visual tracking of the object, accelerometers, magnetometers, etc.

[0193] A camera 1516 is provided to capture images and image streams of the real-world environment (e.g., a room, a vehicle, a natural environment, etc.) surrounding any one of users 1 to 3. In various embodiments, the HMD 1502 includes more than one camera, including: a rear-facing camera, for example, facing away from user 1 when user 1 is viewing the display of the HMD 1502, and a front-facing camera, for example, pointing towards user 1 when user 1 is viewing the display of the HMD 1502. Additionally, in several embodiments, the HMD 1502 includes a depth camera 1518 to sense depth information of objects in the real-world environment.

[0194] HMD 1502 includes a speaker 1520 for providing audio output. Additionally, in some embodiments, a microphone 1522 is included to capture audio from the real-world environment, including sounds from the surrounding environment and speech produced by user 1. HMD 1502 includes a haptic feedback module 1524, such as a vibration device, for providing haptic feedback to user 1. In one embodiment, the haptic feedback module 1524 is capable of causing movement and / or vibration of HMD 1502 to provide haptic feedback to user 1.

[0195] An LED 1526 is provided as a visual indicator of the status of the HMD 1502. For example, the LED may indicate battery level, power on, etc. A card reader 1528 is provided to enable the HMD 1502 to read information from and write information to a memory card. A USB interface 1530 is included as an example of an interface for connecting to peripheral devices or to other devices (such as other portable devices, computers, etc.). In various embodiments of the HMD 1502, any of a variety of interfaces may be included to achieve greater connectivity for the HMD 1502.

[0196] Including Wi-Fi TM Module 1532 enables internet connectivity via wireless networking technology. Additionally, HMD1502 includes Bluetooth. TM Module 1534 enables wireless connectivity to other devices. In some embodiments, a communication link 1536 is also included for connecting to other devices. In one embodiment, the communication link 1536 utilizes infrared transmission for wireless communication. In other embodiments, the communication link 1536 utilizes any of a variety of wireless or wired transmission protocols to communicate with other devices.

[0197] Includes input button / sensor 1538 to input to user 1 ( Figure 1It provides an input interface. This includes various types of input interfaces such as buttons, touchpads, joysticks, trackballs, etc. In various embodiments, the HMD 1502 includes an ultrasonic communication module 1540 to facilitate communication with other devices via ultrasonic technology.

[0198] The device includes a biosensor 1542 to enable the detection of physiological data from user 1. In one embodiment, the biosensor 1542 includes one or more dry electrodes for detecting bioelectrical signals of user 1 through user 1's skin.

[0199] The aforementioned components of HMD 1502 have been described as exemplary components that may be included in HMD 1502. In various embodiments, HMD 1502 may or may not include some of the aforementioned components.

[0200] Figure 16 This illustrates an implementation scheme for an Information Service Provider (INSP) architecture. INSP 1302 delivers numerous information services to geographically dispersed users 1 connected via computer network 1606 (e.g., LAN, WAN, or a combination thereof). Computer network 1602 ( Figure 11A This is an example of computer network 1606. An example of a WAN includes the Internet, and an example of a LAN includes an intranet. User 1 operates client device 1620-1, another user 2 operates another client device 1620-2, and yet another user 3 operates yet another client device 1620-3. Client device 1620-1 is client device 602 ( Figure 6 Examples of ).

[0201] In some implementations, each client device 1620-1, 1620-2, and 1620-3 includes a central processing unit (CPU), a display, and an input / output (I / O) interface. Examples of each client device 1620-1, 1620-2, and 1620-3 include personal computers (PCs), mobile phones, netbooks, tablets, gaming systems, personal digital assistants (PDAs), game consoles 1400, and display devices, HMDs 1502, etc. Figure 15 This includes game consoles 1400 and HMD1502, desktop computers, laptops, and smart TVs. In some implementations, INSP 1602 identifies the type of client device and adjusts the communication method accordingly.

[0202] In some implementations, an INSP delivers one type of service, such as updating stock prices; or multiple types of services, such as broadcast media, news, sports, games, etc. Furthermore, the services provided by each INSP are dynamic, meaning services can be added or removed at any time. Therefore, an INSP providing a specific type of service to a particular individual can change over time. For example, when client device 1620-1 is in user 1's hometown, it is served by an INSP located near user 1620-1, and when user 1 travels to different cities, it is served by different INSPs. The hometown INSP will forward the requested information and data to the new INSP, causing the information to "follow" user 1620-1 to the new city, thus bringing the data closer to user 1620-1 and making it easier to access. In various implementations, a master-server relationship is established between a master INSP that manages information for user 1620-1 and a server INSP, which directly interfaces with user 1620-1 under the control of the master INSP. In some implementations, when client device 1620-1 moves around the world, data is transmitted from one ISP to another so that the INSP serving client device 1620-1, located in a better position, is the INSP that delivers these services.

[0203] INSP 1602 includes an Application Service Provider (ASP) 1608 that provides computer-based services to customers via a computer network 1606. Software provided using the ASP model is sometimes also referred to as software-on-demand or Software as a Service (SaaS). A simple form of providing access to computer-based services (e.g., customer relationship management, etc.) is through the use of standard protocols such as Hypertext Transfer Protocol (HTTP). The application software resides on the vendor's servers and is accessed by each client device 1620-1, 1620-2, and 1620-3 via a web browser using Hypertext Markup Language (HTML), etc., through dedicated client software and / or other remote interfaces such as thin clients provided by the vendor.

[0204] Services delivered over vast geographical areas often utilize cloud computing. Cloud computing is a computing approach in which dynamically scalable and often virtualized resources are provided as a service via computer networks 1606. Users 1 through 3 do not need to be experts in the technical infrastructure supporting their “cloud.” In some implementations, cloud computing is divided into different services, such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common business applications online, accessible from a web browser, while software and data are stored on servers. The term cloud is used as a metaphor for computer networks 1606 based on how they are depicted in a computer network diagram, for example, using servers, storage devices, and logic, and is an abstraction of the complex infrastructure hidden within them.

[0205] Additionally, INSP 1602 includes a Game Processing Provider (GPP) 1610, sometimes referred to herein as a Game Processing Server, which client devices 1620-1, 1620-2, and 1620-3 use to play single-player and multiplayer video games. Most video games played over computer network 1606 operate via a connection to a game server. Typically, games use a dedicated server application that collects data from client devices 1620-1, 1620-2, and 1620-3 and distributes that data to other clients operated by other users. This is more efficient and effective than a peer-to-peer arrangement, but uses a separate server to host the server application. In some implementations, GPP 1610 establishes communication between client devices 1620-1, 1620-2, and 1620-3, allowing them to exchange information without further reliance on a centralized GPP 1610.

[0206] Dedicated GPPs are servers that operate independently of the client. These servers typically run on dedicated hardware located in a data center, providing greater bandwidth and dedicated processing power. For most PC-based multiplayer games, dedicated servers are a method of hosting game servers. Large-scale multiplayer online games run on dedicated servers, which are often hosted by the software company that owns the game's title, allowing them to control and update the content.

[0207] The broadcast processing server (BPS) 1612, sometimes referred to herein as a broadcast processing provider, distributes audio or video signals to an audience. Broadcasting to a very small audience is sometimes called narrowcasting. The final stage of broadcast distribution is how the signal reaches the client devices 1620-1, 1620-2, and 1620-3, and in some embodiments, the signal is distributed wirelessly to antennas and receivers, just like a radio or television station, or distributed via cable or wired broadcast through said station. In various embodiments, the computer network 1606 also brings radio or television signals to the client devices 1620-1, 1620-2, and 1620-3, particularly by allowing multicasting that shares signals and bandwidth. Historically, in several embodiments, broadcasting was defined by geographical regions such as national broadcasting, local broadcasting, etc. However, with the widespread availability of high-speed internet, broadcasting is no longer geographically limited because content can reach virtually any country in the world.

[0208] Storage service provider (SSP) 1614 provides computer storage space and related management services. SSP 1614 also provides periodic backups and archiving. By providing storage as a service, client devices 1620-1, 1620-2, and 1620-3 utilize more storage compared to when storage is not used as a service. Another key advantage is that SSP 1614 includes backup services, and client devices 1620-1, 1620-2, and 1620-3 will not lose data in the event of hard drive failure. Additionally, in some implementations, multiple SSPs have all or part of the data received from client devices 1620-1, 1620-2, and 1620-3, thereby allowing client devices 1620-1, 1620-2, and 1620-3 to access data efficiently, independent of the location of client devices 1620-1, 1620-2, and 1620-3 or the type of client. For example, when user 1 is on the move, user 1 accesses personal files via a home computer and via a mobile phone.

[0209] Communication provider 1616 provides connectivity to client devices 1620-1, 1620-2, and 1620-3. One type of communication provider 1616 is an Internet Service Provider (ISP), which provides access to computer network 1606. The ISP uses data transmission technologies suitable for delivering Internet Protocol datagrams (such as dial-up, digital subscriber line (DSL), cable modems, fiber optic, wireless, or dedicated high-speed interconnects) to connect client devices 1620-1, 1620-2, and 1620-3. In some implementations, communication provider 1616 also provides messaging services such as email, instant messaging, and short message service (SMS) text. Another type of communication provider is a Network Service Provider (NSP), which sells bandwidth or network access by providing direct backbone access to computer network 1606. Examples of network service providers include telecommunications companies, data carriers, wireless communication providers, Internet Service Providers, cable television operators providing high-speed Internet access, etc.

[0210] Data switch 1618 interconnects several modules within INSP 1302 and connects these modules to client devices 1620-1, 1620-2, and 1620-3 via computer network 1606. In various implementations, data switch 1618 covers a small area where all modules of INSP 1602 are closely located, or a larger geographical area when the different modules are geographically dispersed. For example, data switch 1602 may include Fast Gigabit Ethernet within a data center rack, or an intercontinental virtual LAN.

[0211] In some implementations, wireless technology can be used to facilitate communication between the server system and client devices 1620-1 to 1620-3. Such technologies may include, for example, 5G wireless communication technology. 5G is the fifth generation of cellular network technology. A 5G network is a digital cellular network in which the service area covered by the provider is divided into small geographical areas called cells. Analog signals representing voice and images are digitized in the telephone, converted into a bit stream by an analog-to-digital converter, and transmitted. All 5G wireless devices in a cell communicate via radio waves with the local antenna array and low-power automatic transceivers (transmitters and receivers) in the cell through frequency channels allocated by transceivers from a frequency pool reused from other cells. The local antennas are connected to the telephone network and the Internet via high-bandwidth fiber optic or wireless backhaul connections. As in other cellular networks, mobile devices moving from one cell to another are automatically routed to the new cell. It should be understood that 5G networks are merely an example type of communication network, and embodiments of the present invention may utilize earlier generations of wireless or wired communication, as well as newer generations of wired or wireless technologies after 5G.

[0212] It should be noted that in various implementation schemes, one or more features of some implementation schemes described herein are combined with one or more features of one or more other implementation schemes described herein.

[0213] The embodiments described in this disclosure can be practiced using various computer system configurations, including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, and the like. In one implementation, the embodiments described in this disclosure are practiced in a distributed computing environment in which tasks are performed via remote processing devices based on wired or wireless network links.

[0214] In light of the above embodiments, it should be understood that, in one implementation, the embodiments described in this disclosure employ various computer-implemented operations involving data stored in a computer system. These operations are those that require physical manipulation of physical quantities. Any operation described herein that forms part of the embodiments described in this disclosure is a useful machine operation. Some embodiments described in this disclosure also relate to devices or apparatuses for performing these operations. The apparatus is specifically constructed for the desired purpose, or the apparatus is a general-purpose computer selectively activated or configured by a computer program stored in a computer. Specifically, in one embodiment, it may be more convenient to use various general-purpose machines with computer programs written in accordance with the teachings of this document, or to construct more specialized apparatus to perform the desired operations.

[0215] In one implementation, some embodiments described in this disclosure are embodied as computer-readable code on a computer-readable medium. A computer-readable medium is any data storage device that stores data subsequently read by a computer system. Examples of computer-readable media include hard disk drives, network attached storage (NAS), ROM, RAM, compact disc ROM (CD-ROM), recordable CD (CD-R), rewritable CD (CD-RW), magnetic tape, optical data storage devices, non-optical data storage devices, and the like. For example, a computer-readable medium includes computer-readable tangible media distributed across network-coupled computer systems, such that computer-readable code is stored and executed in a distributed manner.

[0216] Furthermore, although some of the above implementations describe the gaming environment, in some implementations other environments, such as video conferencing environments, are used instead of gaming.

[0217] Although the method operations are described in a specific order, it should be understood that other housekeeping operations may be performed between operations, or operations may be adjusted so that they occur at slightly different times, or they may be distributed in a system that allows processing operations to occur at various intervals associated with the processing, as long as the processing of the superimposed operations is performed in the desired manner.

[0218] Although the foregoing embodiments described in this disclosure have been described in slightly more detail for clarity of understanding, it will be apparent that certain changes and modifications may be practiced within the scope of the appended claims. Therefore, these embodiments are to be considered illustrative rather than restrictive, and are not limited to the details given herein, but may be modified within the scope and equivalents of the appended claims.

Claims

1. A method for processing a self-directed interface, comprising: Identify gameplay events during gameplay by a user, wherein the gameplay event is marked as when the user's skill drops below a skill threshold while the user is playing the game; Generate a record of the time window for the aforementioned gameplay events; Processing the recorded game telemetry for the gameplay event, wherein the game telemetry is used to identify the progress of interactive actions prior to the gameplay event within the time window; and Overlay content is generated in the self-guided interface, wherein the overlay content is applied to one or more image frames of the recording when viewed via the self-guided interface, and the overlay content is displayed in the one or more image frames during playback of the recording. The overlay content provides a prompt for raising the user's skill to a level above the skill threshold of the marked gameplay event.

2. The method of claim 1, wherein the time window includes a time period before the gameplay event, the time of the gameplay event, and a time period after the gameplay event.

3. The method of claim 1, wherein the time window changes with the gameplay event and other gameplay events that occur prior to the gameplay event.

4. The method of claim 1, wherein the game telemetry is processed to identify the reason why the user's skill has fallen below the threshold.

5. The method of claim 4, wherein the game telemetry includes one or more game states, the one or more game states being processed to identify virtual objects not in the record within the time window.

6. The method of claim 1, wherein the progress of the interactive action includes the movement of virtual objects not in the record within the time window.

7. The method of claim 6, wherein the overlay content includes the virtual object.

8. The method of claim 6, wherein the overlay content includes a frame placed around the virtual object.

9. The method of claim 1, further comprising: Generate timeline scrub bars; Receive the selection of segments from the timeline scrubber bar; as well as The self-guided interface and the overlay content are modified based on the selection of the aforementioned fragments.

10. The method of claim 1, further comprising: Generate another self-guided interface for display together with the said self-guided interface, wherein the other self-guided interface is associated with another user's gameplay and has the same type as the gameplay event.

11. The method of claim 1, further comprising: Generate another self-guided interface for display together with the said self-guided interface, wherein the other self-guided interface is associated with a different game session of the user's gameplay and has the same type as the gameplay event.

12. The method of claim 1, further comprising: Train an artificial intelligence model to determine whether to initiate a guidance session where the overlay content is overlaid on the self-guided interface.

13. The method of claim 1, wherein the self-guided interface does not include one or more virtual objects displayed in the virtual scene during the gameplay event.

14. The method of claim 1, wherein the self-guided interface includes one or more virtual objects that are not displayed in the virtual scene during the gameplay event.

15. A server system for processing a self-guided interface, comprising: Processor, the processor being configured to: Identify gameplay events during gameplay by a user, wherein the gameplay event is marked as when the user's skill drops below a skill threshold while the user is playing the game; Generate a record of the time window for the aforementioned gameplay events; Processing the recorded game telemetry for the gameplay event, wherein the game telemetry is used to identify the progress of interactive actions prior to the gameplay event within the time window; and Overlay content is generated in the self-guided interface, wherein the overlay content is applied to one or more image frames of the recording when viewed via the self-guided interface, and the overlay content is displayed in the one or more image frames during playback of the recording. The overlay content provides a prompt for raising the user's skill to above the skill threshold of the marked gameplay event; as well as A memory device coupled to the processor for storing the records of the gameplay events.

16. The server system of claim 15, wherein the processor is configured to: generate another self-guided interface for display together with the self-guided interface, wherein the other self-guided interface is associated with and related to gameplay events of another user.

17. The server system of claim 15, wherein the processor is configured to: generate another self-guided interface for display together with the self-guided interface, wherein the other self-guided interface is associated with different game sessions of the user's gameplay and with events related to the gameplay.

18. The server system of claim 15, wherein the time window changes with the gameplay event and other gameplay events that occur prior to the gameplay event.

19. A system for processing a self-directed interface, comprising: A client device configured for use by a user to facilitate the generation of gameplay events while playing a game; A server, coupled to the client device via a computer network, wherein the server is configured to: Identify the gameplay event, wherein the gameplay event is marked as when the user's skill drops below a skill threshold while the user is playing the game; Generate a record of the time window for the aforementioned gameplay events; Processing the recorded game telemetry for the gameplay event, wherein the game telemetry is used to identify the progress of interactive actions prior to the gameplay event within the time window; and Overlay content is generated in the self-guided interface, wherein the overlay content is applied to one or more image frames of the recording when viewed via the self-guided interface, and the overlay content is displayed in the one or more image frames during playback of the recording. The overlay content provides a prompt for raising the user's skill to above the skill threshold of the marked gameplay event.

20. The system of claim 19, wherein the server system is configured to: generate another self-guided interface for display together with the self-guided interface, wherein the other self-guided interface is associated with and related to gameplay events of another user.

Images