Player or spectator emotion annotations for generating electronic game clips

Abstract

A system includes a game recorder, an input processor, a trained machine learning model, and a storage device, wherein the game recorder records a first session of an electronic game, the first session including game state data generated by an electronic game processor processing player input data; the input processor identifies a location within the electronic game at which interest of one or more players or spectators exceeds a predetermined threshold; the trained machine learning model determines an emotion of the one or more players or spectators at the location based on real-time game feedback; and the storage device associates an annotation of the emotion with the game state data at the location within the electronic game.

Description

Technical Field

[0001] This invention generally relates to video games. More specifically, this invention relates to annotating player or viewer emotions for the purpose of generating video game clips. Background Technology

[0002] In recent years, computer gaming has become a multi-billion dollar industry. However, the competition for user time has also become more intense than ever before. Some games require more time than a user has available in any given session. As a result, when users feel they don't have enough time to play a lengthy video game, they may avoid playing certain games or even turn to non-gaming activities such as watching television. The ability to generate playable video game segments or minigames would be a significant advancement over traditional methods.

[0003] Unfortunately, creating video game clips presents numerous challenges, one of the most significant being identifying the parts of a video game that appeal to different players. For example, some players might want to enjoy the humorous parts, while others might prefer the most exciting ones. Manually identifying and generating video game clips is not only time-consuming, but it may not always yield optimal results for the diverse range of players. Summary of the Invention

[0004] Embodiments of the present invention include systems and methods for annotating player and / or spectator emotions to generate video game segments. According to one aspect, a computer-implemented method includes recording a first session of a video game, the first session including game state data generated by a video game processor processing player input data. The computer-implemented method further includes identifying a location within the video game where the interest of one or more players or spectators exceeds a predetermined threshold. The computer-implemented method further includes determining the emotion of one or more players or spectators at the location based on real-time game feedback using a trained machine learning model. Furthermore, the computer-implemented method includes associating annotations of the emotions with the game state data at the location within the video game.

[0005] In one example, identifying the location within a video game includes identifying peaks of interest from one or more players or viewers.

[0006] In another example, real-time game feedback includes audio recorded by a microphone associated with the client device used to participate in the first session.

[0007] In yet another example, real-time game feedback includes video recorded by a camera associated with the client device used to participate in the first session.

[0008] In yet another example, real-time game feedback includes motion recorded by motion sensors associated with the client device used to participate in the first session.

[0009] As a further example, real-time game feedback includes text recorded by an input device associated with the client device used to participate in the first session.

[0010] As an additional example, real-time game feedback includes vital sign data recorded by a vital sign monitor associated with the client device used to participate in the first session.

[0011] In one example, the method also includes segmenting the video game into playable segments that include the locations associated with the annotations.

[0012] In another example, segmenting a video game involves generating multiple playable segments of varying lengths.

[0013] In yet another example, segmenting a video game involves determining the boundaries of playable segments within the game's play context and generating playable segments with those boundaries determined.

[0014] In yet another example, the boundary determination is performed by a trained machine learning model or another machine learning model based on one or more of the annotations or real-time game feedback.

[0015] As a further example, the method also includes obtaining one or more player preferences and identifying playable segments among the plurality of playable segments that satisfy one or more preferences.

[0016] As an additional example, the one or more preferences include a preference sentiment for playable segments, and identifying the playable segments includes identifying the playable segments based on the sentiment of associated annotations.

[0017] In one example, the one or more preferences include preferred game time for playable segments, and identifying the playable segments includes identifying playable segments whose game time is within a threshold amount of the preferred game time.

[0018] In another example, obtaining one or more preferences involves determining at least one implicit preference of the player.

[0019] In yet another example, at least one implicit preference includes a preference for game time, and the preference for game time is determined from at least one of player calendar data and player history data.

[0020] In yet another example, identifying playable segments involves using a trained machine learning model or a second machine learning model to identify playable segments based on perceptual matching with one or more preferences.

[0021] In a further example, the method also includes receiving feedback from the player regarding whether the playable segment satisfies one or more of the preferences, and updating a trained machine learning model or a second machine learning model based on the feedback regarding the playable segment.

[0022] In an additional example, the method also includes receiving feedback from a player in a second session of the video game regarding whether the annotation correctly describes the player's mood at the location, and updating the trained machine learning model based on the feedback.

[0023] In yet another example, the method further includes generating a timeline of annotations for a first session that satisfies a set of conditions provided by the player, receiving the player's selection of a first annotation from the timeline, and performing at least one of the following operations: starting a second session of the video game at the location in the video game associated with the annotation, or displaying a recorded screen output of the first session of the video game at the location in the video game associated with the annotation.

[0024] According to another aspect, a system includes a game recorder that records a first session of a video game, the first session including game state data generated by a video game processor processing player input data. The system also includes an input processor that identifies locations within the video game where the interest of one or more players or spectators exceeds a predetermined threshold. The system further includes a trained machine learning model that determines the emotion of one or more players or spectators at the location based on real-time game feedback. The system further includes a storage device that associates annotations of the emotion with the game state data at the location within the video game.

[0025] In one example, the input processor identifies locations within a video game by determining the peak of interest of one or more players or viewers.

[0026] In another example, real-time game feedback includes audio, and the system also includes a microphone for recording audio associated with a client device used to participate in the first session.

[0027] In yet another example, real-time game feedback includes video, and the system also includes a camera that records video and is associated with a client device used to participate in the first session.

[0028] In yet another example, real-time game feedback includes motion, and the system also includes motion sensors associated with a client device used to participate in the first session to record motion.

[0029] In an additional example, real-time game feedback includes text, and the system also includes an input device for receiving text associated with a client device used to participate in the first session.

[0030] In another example, real-time game feedback includes vital sign data, and the system also includes a vital sign monitor associated with a client device used to participate in the first session that records vital sign data.

[0031] As another example, the system also includes a game segment generator that segments the video game into playable segments containing locations associated with the annotations.

[0032] As yet another example, the game fragment generator segments a video game into multiple playable segments of varying lengths.

[0033] As yet another example, the game fragment generator determines the boundaries of playable fragments within the context of game behavior in a video game and generates playable fragments with the determined boundaries.

[0034] In one example, the game fragment generator uses or includes an artificial intelligence (AI) engine, which includes a trained machine learning model or another machine learning model that determines boundaries based on one or more of the annotations or real-time game feedback.

[0035] In another example, the system also includes a preference analyzer that acquires one or more of a player's preferences and identifies playable segments from a plurality of playable segments that satisfy the one or more of those preferences.

[0036] In yet another example, one or more preferences include a preference sentiment for a playable segment, and the preference analyzer identifies the playable segment by identifying the playable segment based on the sentiment of the associated annotation.

[0037] In yet another example, one or more preferences include a preferred amount of game time for playable segments, and the preference analyzer identifies the playable segments by identifying playable segments whose game time falls within a threshold amount of the preferred game time.

[0038] As an additional example, the preference analyzer identifies one or more preferences by determining at least one implicit preference of the player.

[0039] As another example, the at least one implicit preference includes a preference for game time, and the preference for game time is determined by a preference analyzer from at least one of player calendar data and player history data.

[0040] As another example, the preference analyzer includes or uses an artificial intelligence (AI) engine, which includes a trained machine learning model or a second machine learning model to identify playable segments based on perceptual matching with one or more preferences.

[0041] As yet another example, the AI ​​engine receives feedback from players about whether playable segments satisfy one or more of their preferences, and updates the trained machine learning model or a second machine learning model based on the feedback about playable segments.

[0042] As yet another example, the AI ​​engine receives feedback from players in a second session of a video game regarding whether the annotations correctly describe the player's emotions at the stated location, and updates the trained machine learning model based on that feedback.

[0043] In an additional example, the system also includes a game launcher that generates a timeline of annotations for a first session that satisfies a set of conditions provided by the player, receives the player's selection of a first annotation from the timeline, and performs at least one of the following operations: launches a second session of the video game at the location in the video game associated with the annotation; or displays a recorded screen output of the first session of the video game at the location in the video game associated with the annotation. Attached Figure Description

[0044] Figure 1A The image shows users playing cloud-based video games.

[0045] Figure 1B This shows the gaming behavior of multiple users in multiple locations participating in a cloud-based video game.

[0046] Figure 2 The cloud gaming system is shown.

[0047] Figure 3 It showcases the accumulated game titles across various generations of game consoles.

[0048] Figure 4A This illustrates the hierarchical organization of the various parts of a video game.

[0049] Figure 4B The interface shows how to select a portion of the game's timeline to generate game clips or mini-games.

[0050] Figure 5A This demonstrates a system for generating game snippet code for playable mini-games.

[0051] Figure 5B This shows the modifications made to the virtual space to generate game fragments.

[0052] Figure 5C The modifications made to the scene graph to generate game segments are shown.

[0053] Figure 6A This demonstrates the process of annotating recorded game behavior.

[0054] Figure 6B This shows the user interface for annotating recorded game behavior.

[0055] Figure 7A This demonstrates a system that automatically generates annotations, including player and / or audience emotions.

[0056] Figure 7B Various annotations showing the player's and / or viewer's emotions at the peak of interest within a video game.

[0057] Figure 7C This shows an interface displaying the timeline of annotations for the video game.

[0058] Figure 7D This displays an interface for receiving feedback related to annotations.

[0059] Figure 7E This illustrates a system for automatically generating video game clips.

[0060] Figure 7F This shows several automatically generated video game clips of varying lengths.

[0061] Figure 7G This illustrates the process of identifying playable game segments that satisfy explicit user preferences.

[0062] Figure 7H This demonstrates a system that automatically recommends one or more playable game segments based on explicit and / or implicit user preferences.

[0063] Figure 8 A flowchart illustrating methods for annotating and segmenting video games is shown.

[0064] Figure 9 A block diagram of an exemplary electronic entertainment system is shown. Detailed Implementation

[0065] The following embodiments describe methods and systems for creating and sharing video game annotations for one or more video games or video game segments.

[0066] However, those skilled in the art will understand that the invention can be practiced without some or all of these specific details. In other instances, well-known processing operations have not been described in detail to avoid unnecessarily obscuring the invention.

[0067] Figure 1AThe illustration depicts a user playing a cloud-based video game. As shown, user U1 is playing a cloud-based video game displayed on monitor 100. Cloud-based video games are those that primarily execute on a remote server. In one embodiment, the server may include a standalone server or a server executing in a virtual machine data center, where many servers may be virtualized to provide the requested processing.

[0068] In the illustrated embodiment, cloud gaming server 104 executes a video game rendered on display 100. Client 101, located at the user's location, receives and processes input and communicates this input to cloud gaming server 104, and also receives video and audio data from cloud gaming server 104. Client 101 and cloud gaming server 104 communicate via network 102 (e.g., the Internet). In other embodiments, the client can be any device (whether portable or wireless) as long as it can communicate with the network and provide access to the display to render the game and support user input to drive interaction. In one embodiment, the client is a thin client. However, in other embodiments, the client can be a general-purpose computer, a dedicated computer, a game console, a personal computer, a laptop, a tablet, a mobile computing device, a portable gaming device, a mobile phone, a set-top box, a streaming media interface / device, a smart TV or networked display, a virtual reality (VR) or augmented reality (AR) system, or any other computing device capable of being configured to implement the client functionality defined herein. In one embodiment, the cloud gaming server is configured to detect the type of client device the user is using and provide a cloud gaming experience appropriate for the user's client device. For example, image settings, audio settings, and other types of settings can be optimized for the user's client device.

[0069] In various embodiments, the degree of processing performed by the client in terms of input and output processing may vary. However, in some cases, the game state is primarily maintained and executed on the cloud gaming server 104, while the client primarily functions to receive and convey user input and to receive video / audio data for rendering. Client 101 may be a standalone device connected to display 100 and providing video data for rendering on display 100. In other embodiments, the client may be integrated into display 100. In one embodiment, display 100 is a networked display that provides a platform operating system for applications or "apps" utilizing the display's network connectivity. In such embodiments, the client may be defined by an application running on the platform provided by the display's operating system.

[0070] Figure 1BThis diagram illustrates the gaming behavior of multiple users in multiple locations participating in a cloud-based video game. User U1 is displayed in the first location, interacting with a video game rendered on display 100. Users U2 and U3 are displayed in the second location, interacting with a video game rendered on display 106. User U4 is displayed in the third location, participating in a video game rendered on display 108. Users U5, U6, and U7 are displayed in the fourth location, interacting with a video game rendered on display 110.

[0071] At each of the first, second, third, and fourth locations, at least one computing device is provided for processing input from each user and rendering a cloud-based video game on their respective display. It should be understood that the computing device may be integrated into the display or may be a standalone device, such as a personal computer, set-top box, game console, VR headset, or any other type of device with at least one processor and memory for processing and storing data. The computing device may execute or define a client, as described above. These computing devices are networked and communicate with the cloud gaming server 104 via network 102. In some cases, a single computing device at a location may perform processing for multiple users. In other cases, each user at a location may have a dedicated computing device for performing processing, which may also be combined with other computing devices at that location.

[0072] Cloud gaming server 104 executes various video games being played by the user, defines the game state of a given video game over time, and sends video data (including image and audio data) to a computing device at a specific location. The computing device at the given location processes input from the user playing the video game and transmits the input data to the cloud gaming server, which in turn processes the input data to influence the game state of the video game. It should be understood that cloud-based gaming facilitates multiplayer gaming from players located in different locations by executing video games on a remote server accessible to all players over a network. In this way, the execution of the video game does not depend on the hardware or network connectivity of any single player, although these will affect the user experience for a given player.

[0073] Figure 2A cloud gaming system is illustrated. As shown, a user 200 operates a controller 202 to provide input for a cloud-based video game. The controller 202 may include various input devices such as buttons, joysticks, touchpads, trackballs, VR headsets and / or VR controllers, as well as motion-sensing hardware (e.g., accelerometers, magnetometers, and gyroscopes). In one embodiment, the controller 202 may include a light-emitting object that can be tracked to determine the position of the controller 202. The controller 202 may wirelessly communicate with a thin gaming client 204. The client 204 communicates with a cloud gaming service 210 via a network 208. The client 204 processes data from the controller 202 to generate input data that is transmitted to the video game executed by the cloud gaming service 210. Additionally, the client 204 receives video data from the cloud gaming service 210 for rendering on a display 206. In one embodiment, the client 204 may process the received video data to provide a video stream in a format compatible with the display 206. In one embodiment, the client 204 may include a camera for tracking the controller device or objects located on the controller device. As mentioned earlier, the object can be luminescent to further facilitate tracking based on analysis of image frames captured by the camera.

[0074] Cloud gaming service 210 includes resources for providing an environment for executing video games. Broadly speaking, resources can include various types of computer server hardware, including processors, storage devices, and network devices, which can be used to facilitate the execution of video game applications. In the illustrated embodiment, the video game library 212 contains various game titles. Each game title defines executable code and associated data and resource libraries, all used to instantiate the video game. Host 214 can be a single computing device that defines a platform for instantiating virtual machine 216. In another embodiment, host 214 itself can be a virtualization resource platform. In other words, host 214 can run on one or more server computing devices, handling the allocation and use of resources defined by these server computing devices, while presenting a unified platform that can instantiate virtual machine 216.

[0075] Each virtual machine 216 defines a resource environment that can support an operating system, and a video game application 218 can run thereon. In one embodiment, the virtual machine can be configured to emulate the hardware resource environment of a game console, wherein the operating system associated with the game console runs on the virtual machine to support games developed for that game console. In another embodiment, the operating system can be configured to emulate the native operating system environment of the game console, although the underlying virtual machine may or may not be configured to emulate the hardware of the game console. In yet another embodiment, an emulator application runs on top of the virtual machine's operating system, and the emulator is configured to emulate the native operating system environment of the game console to support video games designed for that game console. It should be understood that various current and legacy game consoles can be emulated in a cloud-based gaming system. This allows users to access games on different game consoles through the cloud gaming system.

[0076] When user 200 requests to play a specific video game, the video game is retrieved from library 212. If no compatible virtual machine is already instantiated or available, a new compatible virtual machine is instantiated on host 214. In some cases, if a game has not yet been instantiated or is unavailable, a new game may be instantiated on a dedicated host for performance and security reasons. The retrieved video game is then executed as application 218 on an available or newly instantiated virtual machine 216. In one embodiment, this may require determining the appropriate platform for the video game (e.g., which game console or operating system the game needs to run on) and assigning the video game to an appropriate virtual machine for execution, such as a virtual machine with an emulator application capable of handling the execution of the video game. The executing video game communicates with game client 204 to provide an interactive gaming experience for user 200. More specifically, the executing video game application 218 receives input data from client 204 via network 208. Application 218 processes this input data to update the game state of the executing application. As the game state changes, application 218 outputs video data to client 204 for rendering on display 206. Additionally, application 218 can output feedback data to client 204, which is used to provide additional feedback mechanisms to the user. For example, the user's controller 202 may include a haptic vibration feedback mechanism that can be activated based on the feedback data output by the video game application.

[0077] In one embodiment, the cloud gaming system is configured to detect the type of client device associated with a user, and the type of controller available when the user provides input to a cloud-based video game. For example, in one embodiment, when a user logs into the cloud gaming system, they may be presented with an option to specify the type of client device they are using to access the cloud gaming system. In one embodiment, a range of client device options is presented, from which the user can select the option corresponding to their client device. Alternatively, the user may be presented with an option to specify the type of controller they will use to play the video game. In one embodiment, a range of controller options may be presented to the user, from which the user can select to specify the controller type corresponding to their controller hardware. In other embodiments, the cloud gaming system may be configured to automatically detect the client device type and / or the controller device type.

[0078] For example, upon login, the client device can send information identifying itself and the connected controller device to the cloud gaming server (e.g., in response to a request from the cloud gaming server). Based on this information, the cloud gaming server can determine appropriate game output and input parameter configurations to provide a gaming experience optimized for the user's client device and controller device. In one embodiment, a lookup table is used to determine the game configuration and input parameter configuration based on the detected client device and the detected controller device.

[0079] It should be understood that a given video game may be developed for a specific platform and a specific associated controller device. However, when such a game is provided through the cloud gaming system presented herein, the user may be accessing the video game using different controller devices. For example, a game may be developed for a game console and its associated controllers, while the user may be accessing a cloud-based version of the game from a personal computer using a keyboard and mouse, or through a VR headset and associated VR controllers. In this case, input parameter configuration can define a mapping from inputs that can be generated by the user's available controller device to inputs acceptable to the video game execution. In some cases, the service may use local hardware instead of running the game on cloud gaming server 104, or may use local hardware in addition to running the game on cloud gaming server 104. In some cases, cloud gaming server 104 may coordinate among multiple local hardware devices running the same game session code.

[0080] In another example, users can access a cloud gaming system via tablet computing devices, touchscreen smartphones, or other touchscreen-driven devices. In this case, the client device and controller device are integrated into the same device, and input is provided through detected touchscreen inputs / gestures. For such devices, input parameter configuration can define specific touchscreen inputs corresponding to game inputs in the video game. For example, buttons, directional keys, or other types of input elements can be displayed or overlaid during video game operation to indicate where the user can touch on the touchscreen to generate game input. Gestures such as swipes in a specific direction or specific touch actions can also be detected as game inputs. In one embodiment, a tutorial can be provided to the user to indicate how to provide input for game actions via the touchscreen, such as before starting game actions in the video game, to familiarize the user with operating controls on the touchscreen.

[0081] In some embodiments, the client device acts as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection to transmit input from the controller device to the client device. The client device then processes this input and transmits the input data to the cloud gaming server via a network (e.g., accessed through a local network device such as a router). However, in other embodiments, the controller itself may also be a networked device capable of transmitting input directly to the cloud gaming server via a network without first transmitting such input through the client device. For example, the controller may connect to a local network device (e.g., the aforementioned router) to send and receive data from the cloud gaming server. Therefore, while the client device may still need to receive video output from the cloud-based video game and render it on a local display, input latency can be reduced by allowing the controller to send input directly to the cloud gaming server via the network, bypassing the client device.

[0082] In one embodiment, the networked controller and client device can be configured to send certain types of input directly from the controller to the cloud gaming server, and other types of input via the client device. For example, inputs that can be detected without relying on any additional hardware or processing besides the controller itself can be sent directly from the controller to the cloud gaming server via the network, bypassing the client device. Such inputs may include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), etc. However, inputs utilizing additional hardware or requiring processing by the client device can be sent by the client device to the cloud gaming server. These may include captured video or audio from the game environment, which the client device may process before sending to the cloud gaming server. Additionally, inputs from motion detection hardware in the controller can be processed by the client device in conjunction with captured video to detect the controller's position and movement, which are then relayed by the client device to the cloud gaming server. It should be understood that, according to various embodiments, the controller device may also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server. In some cases, data from the input device can be processed by a local computing device, which can then provide the result of that processing to replace or supplement all or part of the original data. Transmitting the processing results can significantly reduce the bandwidth required for data transmission. For example, eye-tracking sensor data can be processed to determine its processing outcome, such as focus on the display, the object being viewed, blinks, eye opening, eye movement rate, and / or identified emotions. As another example, data from a video camera can be analyzed to recognize user gestures. Recognized gestures can be transmitted to the cloud, while access to the raw data from the video camera can be restricted to the local computing device. In some cases, transmitting the processing results of raw data from user input devices, rather than transmitting the raw data itself, can better protect user privacy.

[0083] Figure 3This illustration shows the accumulation of game titles across different generations of game consoles. In the video game industry, video games are developed for a specific video game console. Over time, a game title library for a particular game console is accumulated. For example, in the illustration, a first-generation console 320 has a collection of games 328 developed for it. A second-generation console 322 is associated with a collection of games 330 developed for it. A third-generation console 324 is also shown, with a collection of games 332 developed for it. In other embodiments, a collection of games 334 may be specifically developed as a cloud-based game to be used with a client 326. Furthermore, other types of games, such as internet games, can be developed and collected for distribution on cloud gaming systems, as described herein. It is understood that game titles from different generations of game consoles can be collected and merged in a cloud gaming library 312. As shown, library 312 contains a first-generation console library 336, which contains game titles developed for the first-generation console 320. In a similar manner, library 312 also contains a second-generation console library 338 and a third-generation console library 340, which respectively contain video games developed for the second-generation and third-generation consoles. Library 312 may also include games developed for client 326, as well as other types of games such as internet games. It is evident that a vast collection of games from various generations of video game consoles can be aggregated and served via a cloud gaming library. As mentioned above, each of these games can run on a virtual machine that simulates the operating system environment associated with the given game console for which the game was developed. In this way, users accessing the cloud-based gaming system can easily access and play games from multiple different consoles, as well as games from other related sources (such as internet games) and games specifically developed for the cloud-based gaming system. Some generations of consoles may be capable of running games originally created for other generations of consoles or gaming systems. Some games may be designed to be playable on multiple generations of consoles or multiple gaming systems. Some games that a user does not own may be available for temporary play, such as through subscriptions, demos, or limited access to game snippets.

[0084] Figure 4AThis illustrates the hierarchical organization of various parts of a video game. For example, a video game can be organized into chapters / sections 400. In the illustrated embodiment, these might include a setup section, an introduction section, individual level sections, and an ending section. A given level can be further divided into scenes. As shown, level 3 is broken down into various scenes 402. In some cases, game levels can be hierarchical, containing other game areas. For example, a game level might contain a continent, which might contain towns, which might contain magic shops, and each of those nested game levels might contain scenes. As a user completes a given scene, a timeline of gameplay behavior for that scene can be recorded, including recorded video of the user's gameplay behavior as well as recorded input data and game states (e.g., variables) of the executed game application. In the illustrated embodiment, the gameplay timeline 404 represents the user's gameplay behavior in scene 2 of game level 3. In some cases, the playable game experience may vary across different game areas, levels, times, or other game segments (such as puzzles, combat challenges, or crafting). The playable gameplay experience varies depending on the player's current state during specific parts of the game (e.g., selected character class, guild membership, previous interactions with NPCs, equipped items, character inventory, or learned skills). In some cases, game segments can be identified based on the player's preferred activities and the game fragments and states that provide those activities.

[0085] In one embodiment, a user can select a portion of their recorded gameplay to generate a mini-game or game segment. For example, in the illustrated embodiment, the gameplay timeline 404 has a start time T0 and an end time T... z We chose to start from time T. x Time T y A segment of the game behavior timeline is used to generate mini-games. In some cases, the game provides players with options for activities or areas to explore. In this case, a portion of scene 402 and / or the game behavior timeline 404 can be identified and used to create one or more playable game segments that can provide players with game activities and / or areas they might enjoy. In other embodiments, such as regarding Figure 7D The described method can automatically generate game segments based on past gaming behavior of individual users using machine learning, and provide them to the user according to the specific user's explicit or implicit preferences in terms of time availability, skills, and interests.

[0086] Figure 4BAn interface is shown for allowing a user to select a portion of a game action timeline to generate mini-games or game segments. In the illustrated embodiment, interface 412 appears on a touchscreen of device 413, such as a tablet computing device. However, in other embodiments, interface 412 may be presented on a television, VR headset, or other display device. Interface 412 includes an optional game action timeline 414. Game action timeline 414 is represented as a strip of film with adjustable markers 416 and 418. Marker 416 specifies the start point along the selected game action timeline 414, while marker 418 specifies the end point along the selected game action timeline. Additionally, marker 419 may be placed within the portion of game action timeline 414 defined by start marker 416 and end marker 418. For ease of use and to provide the user with a visual understanding of their selected game action portion, a start frame 420 corresponding to the point on the game action timeline where marker 416 is located may be displayed. Start frame 420 is an image of a recorded game action video corresponding to the time at which marker 416 is placed. Similarly, representative frame 422 is an image of the recorded game behavior video at the time corresponding to the position marked 419. In a similar manner, end frame 424 is an image of the recorded game behavior video at the time corresponding to the position marked 418. Representative frame 422 can be used as a representative image of a game segment created based on a selected portion of the game behavior timeline. While a touchscreen interface has been provided and described, in other embodiments, various other types of input can be utilized to select the start and end points defining the game behavior portions to generate game segments. For example, input can be provided via a game controller, keyboard, VR controller, gesture input, voice input, and other types of input devices and mechanisms, thereby allowing selection of a portion of the game behavior and representative image frames from the recorded game behavior video.

[0087] In some embodiments, the selection marker cannot be continuously adjusted along the game action timeline, but is instead configured to attach to pre-defined time points on the game action timeline. For example, a preset time point can be defined to correspond to a specific event occurring within the game action timeline. In a given game action timeline, the specific event to be assigned to a preset time point can be generated based on analysis of user game behavior, which will depend on the specific architecture of the video game's gameplay. In one embodiment, assigning a preset time point can be based on the geographical location of a character in the virtual world defined by the video game. For example, a preset time point can be assigned to a specific time when a character moves from one geographical scene to another (e.g., from one scene location to another, from one city to another, enters a building, enters a room in a building, enters a vehicle, enters a different type of environment, etc.), or undergoes any meaningful geographical transition. In another embodiment, preset time points can be assigned based on the development and progression of the user's controlled character or entity within the video game. For example, preset time points can be assigned when a user-controlled character or entity completes a task, acquires a skill, obtains an item, levels up, completes a part of the video game, or performs or achieves a significant event within the game. In some cases, triggering a certain event in the game (such as the appearance of a boss monster, a player triggering a trap, or an earthquake) may activate that playable game segment.

[0088] After a user selects a portion of a game action to generate a game segment, playable mini-games can be created based on the selected action, as shown in the reference. Figure 5A As further elaborated upon. More specifically, game segments allow a user or another user to later replay or play that substantially identical portion of the video game, possibly under substantially the same conditions and parameters. In this sense, a game segment is not merely a replay of the original user's gameplay; it is itself a playable part of the video game. Thus, subsequent users do not need to play through the game linearly to enjoy a substantially similar gaming experience to the original user. In some cases, subsequent users may experience similar gameplay to previous users in a non-linear manner; the user can replay or play game segments from any preceding or subsequent user.

[0089] Figure 5AA system for generating game fragment code is illustrated. The terms "game fragment" and "minigame" are used interchangeably here, defining a standalone playable portion of a video game generated from existing recorded game actions. In the illustrated embodiment, user game action 500 conceptually represents a user interacting with the full version of the video game. This user could be a human player who plays the video game sequentially from the beginning, or, in some embodiments, an AI player.

[0090] The main game code 502 is executed to define the complete version of the video game. As the video game is played, it generates several game behavior outputs, including video data, game state data, and user input data. This data can be recorded by the game behavior recorder 503 to define the user's recorded game behavior. In the illustrated embodiment, the image stream of screen output 504 conceptually represents the video data output generated by the video game. Game state data 506 and user input data 508 are also shown. Game state data 506 contains data for defining the game state at each moment during the execution of the video game during game behavior. In some cases, game state data allows for a finer granularity in starting game behavior for a game segment compared to a preset starting point, allowing the selection of any frame recorded to begin playback. The same game state present in the recorded game behavior can be used to start the game behavior, or the game behavior can be modified when generating the game segment. In some cases, game state data from locations other than the starting point of the game segment can be included in the game segment to make the game behavior of the provided game segment more consistent with the game behavior in the recording. For example, if a player opens a treasure chest during a recorded selection segment, and the items contained in that chest are randomly determined by the game at the time of opening, then those randomly determined items can be included in the game state and applied to the game segment. In other words, if the player opens the same treasure chest while running a game segment, the items inside the chest that were selected when the player opened it will be the same as the items selected in the recorded game action.

[0091] User input data is data generated by capturing actions initiated by the user while interacting with the video game. This can be provided, for example, by triggering input devices on the controller, detecting sensor data (such as motion sensors), or collecting audio input. In some cases, user input data may include data from interactions between the user and things outside the video game, such as interactions with other games or preferences set by the user on a game system's user interface.

[0092] Game state data can contain values ​​of arbitrary variables that define the execution state of the video game. For example, game state data may contain position variables to indicate the X, Y, and Z coordinates of objects such as characters and vehicles in the virtual space of the video game. Camera angle variables indicate the direction of the virtual camera or virtual viewpoint in the video game. In one embodiment, the camera angle is defined by a measured azimuth component relative to an azimuth reference (e.g., along the horizontal plane) and a measured tilt component relative to an tilt reference (e.g., relative to the vertical direction). Action variables indicate the initiation and duration of various actions within the video game. It should be understood that a given action in the video game is context-specific. As an example, actions may include initiating a specific operation, using a skill, triggering a modification mechanism that modifies an existing action (such as increasing intensity or frequency), or any other type of action or behavior that a user can trigger through input during the video game. Weapon variables indicate the activation state of weapons in the video game. Health variables indicate, for example, the user's character's health level in the video game. Button variables indicate the state of buttons on a controller device, such as whether a button is pressed or released. In the illustrated embodiment, the joystick state variable indicates the amount of movement of the joystick relative to a neutral position. The game state variables described above are merely illustrative examples. Those skilled in the art should know that many other types of game state variables can be tracked over time.

[0093] As previously described, the user interface can graphically display a user's recorded game behavior, allowing the user to select a portion of their game behavior to generate a game segment. In the illustrated embodiment, the user has delineated a segment from their recorded game behavior, representing game segment 510. This selected segment of recorded game behavior is used by a game segment generator to generate game segment code 522, which defines a limited game based on the selected portion of the user's game behavior. The game segment generator 512 includes a game state analyzer 514, which analyzes the game state of game segment 510. A game breakpoint processor 516 determines appropriate breakpoints to define the start and end of the game segment based on the analysis of the game state of the recorded segment. Breakpoints can be defined based on geography, time, task or objective completion status, scene boundaries (physical or temporal), or other aspects of the video game, thereby allowing the game behavior of the video game to be segmented to generate game segments. The following brief description of some exemplary embodiments is used to highlight certain possibilities for breakpoint determination.

[0094] For example, some video games require controlling a character to move from one geographic scene or location to another. Selected portions of user gameplay can be determined to originate from gameplay within a specific scene. In such embodiments, the boundaries of that specific scene can define geographic breakpoints for game segments, selecting only that scene and excluding other scenes, including adjacent or neighboring scenes, as well as scenes that are not adjacent, not neighboring, or have a low or no connection to that specific scene. It should be understood that game segment 510 may contain gameplay from multiple scenes; in this case, the game breakpoint processor 516 can be configured to define breakpoints based on the boundaries of the multiple scenes utilized by the recorded gameplay segment. In some cases, the selected game segment 510 is used to create a playable gameplay segment that limits gameplay to the portion of the game that has occurred within the selected game segment. In some cases, the playable portion of a playable gameplay segment created based on the selected game segment 510 may include portions of the game that have occurred outside of those portions within the selected game segment 510. In one example, a playable gameplay segment can serve as a starting point for gameplay, allowing access to the complete game. In another example, a game fragment allows the execution of already completed game segments and other related game content, such as game content that would have occurred in the game state of that fragment, adjacent game locations, the remainder of any game level included in the fragment, or time-limited content. In some cases, multiple selected game fragments 510 can be used to generate a single playable game fragment, such as creating a game fragment that allows access to game actions performed in multiple game fragments 510.

[0095] It should be noted that scenarios can be based on data that is essentially geographical, temporal, or game state data. That is, scenarios can not only be limited to geographical areas within the virtual space defined by the video game, but can also be configured as specific time periods or temporal nodes existing within the larger context of the video game. Such scenarios can possess game aspects determined by game state data, such as whether the player is a guild member, whether they have previously performed a specific action, or the items currently equipped by the player. Such scenarios can set tasks or objectives for the player to complete. Therefore, game breakpoints can be defined based on the temporal sequence or other temporal aspects defined by the video game.

[0096] Furthermore, a given scene may have associated objects or features that appear as part of the scene during gameplay. These objects or features can be analyzed to define more breakpoints based on their contents. For example, objects in a scene may be taken from a subset of a resource library; in this case, the game breakpoint processor 516 can define that subset of the resource library for the game segment to exclude other objects in the resource library that are not used in the scene related to the recorded game segment. Objects and features may be dynamic elements in a given scene with mechanisms to define changes in response to events occurring in the video game. For example, an object may have a damage modeling module to determine and adjust its appearance when the object is damaged (e.g., when hit by a weapon). Features may be vehicles available in the scene with logic defining their appearance, operation, and response to user input during gameplay. Such logic or damage modeling can further define game breakpoints used to generate game segments.

[0097] In video games, defining a selected portion of the game or its various aspects can serve as the basis for defining game breakpoints. The examples described herein are merely illustrative and not restrictive. It should be understood that in other embodiments, other aspects of the video game can also serve as the basis for defining breakpoints to generate game segments.

[0098] In one embodiment, the video game can be organized into multiple scenarios, which are typically completed linearly, requiring the completion of each scenario before attempting subsequent ones. Each scenario may contain several objectives or goals, some necessary for completion and others optional. Objectives may include: moving from a starting position to a preset endpoint within the scenario, surviving for a preset time, eliminating a preset number of enemies, obtaining a certain score, defeating a specific enemy, solving one or more puzzles, and / or any other activity that can define in-game objectives. A scenario may have multiple preset completion points, i.e., points where, once a user reaches a completion point, they can return to that point if they are unable to continue playing for any reason (e.g., the user quits the game, the user's character dies or runs out of health, the user's vehicle crashes, etc.). At preset completion points, the video game may be configured to automatically save the user's progress or provide the user with the option to save their progress.

[0099] In one embodiment, the game breakpoint processor 516 is configured to define game breakpoints at preset completion points. In one embodiment, this can be accomplished by finding the nearest completion points to the selected start and end points of a user-recorded game action segment and using these nearest completion points to define game breakpoints for the game segment. In another embodiment, a start breakpoint is defined using the nearest completion point before the selected start point of the recorded game action segment, and an end breakpoint is defined using the nearest completion point after the selected end point of the recorded game action segment to create a game segment. In yet another embodiment, if a completion point is within a preset radius (e.g., before or after) of the start or end point of a user-recorded game action segment, that completion point is used to define the corresponding start or end breakpoint for the game segment. If no completion point exists within the preset radius, a game breakpoint that more closely matches the start or end point selected by the user for the recorded segment is defined. In other embodiments, the preset radii of the start and end points may be different to determine whether to use existing completion points to define game breakpoints.

[0100] As described above, the game breakpoint processor 516 determines appropriate breakpoints applicable to various aspects of the video game based on the analysis of recorded game behavior segments. The breakpoints defined by processor 516 delineate the scope of game segments generated based on the recorded game behavior segments. In one embodiment, an overlay processor 517 is provided to generate overlay content that can improve the user experience when playing game segments generated by game segment generator 512. For example, in one embodiment, the overlay processor 517 defines game segment pre-processing data, which may define videos, game behaviors, or additional information that can be provided as an introduction to the game segment before the actual game behavior begins. An example of game segment pre-processing data is an introductory video, which can provide contextual background to the user initiating the game behavior that becomes the segment. In another embodiment, game segment pre-processing data may define guiding game behaviors for the game segment, providing the user with an opportunity to learn skills that may be useful or necessary for the game behavior of the game segment. In yet another embodiment, game segment pre-processing data may define a series of one or more information screens or images that provide the user with information related to the game segment. This type of information may include controller configuration, plot background information, objectives or goals, maps, or any other type of information related to the game segment that may be beneficial to the user or enhance the user's experience while playing the game segment.

[0101] The overlay processor 517 can also be configured to define post-game segment data. In some embodiments, the post-game segment data can define a video or image to be displayed after completing gameplay actions within a game segment. For example, a congratulatory video can be played after a user completes a game segment. This video can be customized based on the user's gameplay actions within the game segment, such as displaying information or visuals based on the user's gameplay actions. In one embodiment, the post-game segment data can define a playback mechanism to play a recorded portion of the user's gameplay actions after they have completed them. In another embodiment, the post-game segment data can be configured to display statistics related to gameplay actions within the game segment and can indicate a comparison of those actions with those of other users or the original creator of the game segment. In other embodiments, the post-game segment data can define additional interactive elements presented to the user after completing a game segment. These elements may include options to purchase the video game on which the game segment is based (partial or full purchase), options to redirect to additional sources of information related to that video game, etc.

[0102] In some embodiments, the overlay processor 517 may be configured to define elements overlaid within the game segment. These elements can be customized by the user playing the game segment, such as customization of characters, objects, attributes, annotations, and other types of customization options. In some embodiments, the overlay processor 517 may be configured to define simplified elements for the game segment to reduce the complexity of the game segment code and the amount of resources required to execute the game segment. As an example, many video games contain artificial intelligence (AI) entities, such as characters, vehicles, enemies, etc. In a full video game, such AI entities may be controlled by an AI model that determines the AI ​​entity's reactions and behaviors based on events occurring in the video game. However, in the context of a limited-scope game segment, it may be acceptable to directly define the behavior of the AI ​​entity through hard-coded definitions or simplified deductions, rather than fully modeling the behavior of the AI ​​entity as in a full video game. In some cases, the game segment can be modified before the player starts playing the game. For example, players may be allowed to change their character's equipment before starting the game. Alternatively, the game state of the segment can be adjusted by AI to optimize the game behavior during subsequent gameplay, making it more consistent with the player's gaming preferences. Another example could allow players to modify their character's appearance before playing a game segment. Players could also modify other parts of their game state, such as the options they chose during their last interaction with an NPC, or how they chose to spend attribute points when leveling up. In some cases, game states or portions of game states from one or more other game segments or recorded game segments could be used to modify the game state within a game segment before starting the game; for example, replacing a character in one game state with a character in another game state. All game states could be replaced before starting a game segment.

[0103] For example, if a given AI character moves in a specific way according to its AI model within a recorded segment of gameplay in a full video game, and this movement is unlikely to change throughout the segment, defining an approximation of the AI ​​character's movement for that segment might be more efficient. Such an approximation doesn't require including the complete AI model as part of the game segment's code, yet still provides the user playing the segment with an experience fundamentally similar to the original user gameplay from which the segment originated. The resource savings achieved by approximating the behavior of AI entities are even more significant when multiple AI entities interact within a user's recorded gameplay segment. This is because each AI entity's AI model may depend on the outputs of other AI entities' AI models. However, when gameplay has been recorded, the behavior of each of these AI entities is known and can therefore be reproduced in the game segment through simplification mechanisms, such as directly encoding their control variables and approximating their behavior.

[0104] Continue to refer to Figure 5A The game setup state processor 518 is configured to define the initial state of a game segment. Based on the operations of the game state analyzer 514, the game breakpoint processor 516, and the game setup state processor 518, the code assembler manager 520 assembles the code segments to define the game segment code 522. When the game segment code 522 is executed, user game behavior 524 provides input to define the execution state of the game segment code, which generates game behavior output, including video data and feedback data for rendering the game segment to the user. The video data may include a game segment pre-overlay video 523, a game segment video 525 (i.e., the video generated by the game behavior of the game segment), and a game segment post-overlay video 526.

[0105] It should be understood that, in one embodiment, the game fragment code 522 is completely self-contained, containing all the code segments required to execute the game fragment. However, in other embodiments, the game fragment code 522 may contain references or pointers to existing code segments in the main game code of the complete video game. Furthermore, the game fragment code 522 may reference or use existing resources in the resource library of the main game code of the complete video game. However, in other embodiments, a new resource library may be generated for the game fragment code.

[0106] In one embodiment, the game state analyzer 514 may be configured to analyze game state data 506 of a user's recorded game behavior. Based on the analysis of the user's recorded game behavior, multiple regions of interest (ROIs) within the user's recorded game behavior may be defined and presented to the user as selectable segments for generating game segments. For example, a segment of the user's recorded game behavior may be defined as a game behavior region characterized by an elevated activity level of a specific game state variable. It should be understood that the activity level of a given game state variable may be based on various factors, such as intensity, trigger frequency, number of replays, etc. In some embodiments, the analysis of game state variables may include searching for regions in the game behavior where the activity levels of two or more different game state variables are correlated in a predetermined manner, for example, where the two or more variables simultaneously exhibit elevated activity levels. High activity levels may be determined based on a predetermined threshold. In some embodiments, the frequency or intensity of real-time (synchronous) feedback from the user (e.g., voice feedback) may be used to identify selectable game behavior regions, which may be performed by the user or by machine learning.

[0107] In various embodiments, regions of interest (ROIs) of recorded user gameplay can be automatically determined based on threshold detection of one or more of the following: one or more user inputs, user input rate, input frequency, repetition of input types, occurrence of input patterns, combined inputs (e.g., key combinations), motion vectors, pressure applied to the controller, and / or implicit feedback, such as user arousal based on acquired user image or audio data. Other types of feedback from the user can be employed, such as time spent by the player on various aspects of the game (including different game areas or different play types, such as crafting and hunting monsters), user evaluations (which can be refined to various aspects of their gameplay or gameplay recordings), analyses comparing the player's activity in a particular area with the player's preferred activities, or AI-generated judgments of the player's liking for specific gameplay experiences or aspects of the game. Other types of feedback can be inferred from subsequent users who have played the game segment, including their own feedback on similarities and differences (as described above). Statistical calculations can be performed on the range of differences to derive a user interest score.

[0108] Figure 5BThis illustrates modifications to the virtual space made to generate game segments of a video game. Map 530 represents a scene or geographical area of ​​the video game. The map shown is a two-dimensional representation that depicts the three-dimensional virtual space that can be traversed and experienced during actual gameplay. As shown, map 530 shows area 532 and multiple paths 534, 536, 538, and 540. In a recorded gameplay sequence of the video game, user character 542 moves from area 532 to path 540. Based on this recorded movement and other analyses of gameplay, it can be determined that additional paths 534, 536, and 538 are not necessary for generating game segments. These paths, compared to path 540, may represent incorrect choices, or lead to areas unrelated to the game segment, or diminish the ability of the game segment player to traverse the path and experience gameplay similar to the original user. Furthermore, if the areas led to by paths 534, 536, and 538 are not supported in the game segment, including such paths may confuse the player or at least result in a poor user experience. Therefore, in the modified map 550, paths 534, 536, and 538 are set to be unavailable in the game segment's gameplay, while path 540 and area 532 remain unchanged. Thus, when a user plays a game segment containing terrain defined by map 550, paths 534, 536, and 538 will be impassable in the virtual space they experience. This makes it more likely that the user will follow path 540, just like the original user, and experience similar gameplay.

[0109] It should be understood that the virtual space portion defined by a game segment or mini-game can be defined by boundaries determined based on recorded user gameplay behavior. These boundaries define sub-regions within a larger virtual space and include a subset of available features within that larger virtual space. In some embodiments, virtual space boundaries can be determined by identifying a location within the virtual space defined by user gameplay behavior, and then identifying a predetermined boundary associated with the virtual space, closest to that location, and set to surround that location. For example, user gameplay behavior can define a path traversed by the user's video game character. This path can be analyzed, and based on its location within the virtual space, a set of predetermined boundaries can be selected to define the area surrounding the path. In some embodiments, the predetermined boundaries can be defined by specific features inherently defining different parts of the virtual space, such as doors, windows, walls, rooms, corridors, fences, roads, intersections, passageways, etc.

[0110] Figure 5CModifications to the scene diagram are shown for generating game segments. Scene diagram 560 conceptually illustrates the organization of the various scenes A through G of a video game. It should be understood that the scenes described herein can be geographically and / or temporally dimensional, each representing a playable portion of the video game, such as stages, levels, chapters, locations, or any other organizational unit within the video game that players can use to progress from one scene to another. In scene diagram 560, multiple nodes representing scenes A through G are shown. As illustrated, players can progress from scene A to scene B, and from scene B to either scene D or scene E. Players can also progress from scene A to scene C, and from scene C to either scene F or scene G. Scene diagram 560 is an exemplary illustration of the scene organization for a complete video game. However, not all available scenes are necessary for creating game segments. Therefore, as an example, scene diagram 562 illustrates the scene organization used for game segments. As shown in the figure, scene diagram 562 includes scenes A, B, C, and F, but does not include the remaining scenes included in scene diagram 560 of the complete video game. Thus, the user can progress from scene A to either scene B or scene C, and from scene C to scene F. However, the other scenes in scene diagram 560 of the complete video game are not available in the gameplay of the game segment. As previously described, the system according to embodiments of the present invention can be configured to limit the scenes included when generating game segments. In this way, game segments do not include scenes irrelevant to the limited context and intended purpose of their gameplay.

[0111] Figure 6A The process of annotating a recorded game session to generate annotation 602, which can be presented at selected points in subsequent game behavior sessions, is illustrated. Annotation 602 can be created by a player playing a specific game or game segment (e.g., game code 603), other players, or an automated process. In some cases, the annotation can be provided by a viewer watching the game behavior, or based on behavioral analysis of the viewer, which can be performed by artificial intelligence (AI).

[0112] As shown in the figure, the player's game behavior in the first game session 604B is recorded by the game behavior recorder 503, such as... Figure 5A The relevant description states that the recorded game behavior includes screen output 504 and data transmission via the game processor (MPS). Figure 9(As shown in the image) Game state data 506 generated by processing user input data. Game state data 506 includes a set of variables related to game behavior, such as, but not limited to: the current chapter, scene, level, time position, or spatial position within the video game; one or more attributes of a character within the video game, including game statistics, achievements obtained, and / or puzzles solved; and the position and status of various non-player characters (NPCs) in the game. In some cases, game state data can be determined by analyzing the game's rendered output, such as detecting health levels displayed in a rendered video, identifying enemies in combat by analyzing rendered video footage, or detecting low health alert sounds in rendered audio. Such analysis can be performed by artificial intelligence (AI).

[0113] Note 602 may include video, audio, graphics, text, highlights, and / or metadata, which may be provided by the creator of Note 602 (i.e., its "author"), taken from screen output 504 and / or game state data 506, provided by the game developer, and / or generated by an automated process. The metadata may contain portions of the game state data 506, which can be used to launch the game at different locations later, as will be described in more detail below. The metadata may also include screen coordinates for highlighting and / or locating the video, text, and graphics of Note 602 on the display screen, and / or highlighting Note 602. In some cases, game state data may include information received from the game, such as the current score, location on the map, or player health, in addition to rendered output of game behavior. In some cases, game state data may originate from analysis of the game's rendered output (which may include artificial intelligence (AI) analysis) to determine information such as player health, enemy health, ammunition, character equipment, or enemies the player is fighting.

[0114] Annotation 602 may be linked to or associated with a specific location, place, or segment of the video game. In some embodiments, this location is defined by a specific game state 606, which is a subset or instance of the constantly changing game state data 506 during a recorded game action session. As will be described in more detail below, once annotation 602 is linked to a point within the video game, annotation 602 may be presented (e.g., displayed, played, executed) in response to one or more triggering conditions when game code 603 is run in a subsequent game session 604B. For example, video or text annotation 602 may be overlaid on subsequent game session 604B, and / or the screen output 504 of subsequent game session 604B may be wholly or partially replaced with annotation 602. In some embodiments, this can be achieved by combining Figure 5A The superimposed processor 517 described herein may be implemented using a module that is part of the game code 603 or the game operating system, or a module that works in conjunction with it.

[0115] Triggering conditions may include: a second player reaching the same point in game code 603 in a subsequent game session 604B, such that at least a portion of the game state 606 in the subsequent game session 604B matches at least a portion of the game state 606 in the first game session 604A linked to annotation 602. For example, if annotation 602 is associated with a game state 606 where the level variable is set to level 3, then triggering conditions may include the level variable switching to level 3 in the subsequent game session 604B.

[0116] The triggering condition may also include: the second player choosing to view annotation 602 after receiving a notification that annotation 602 is available. In other words, the presentation of annotation 602 may not be automatic. Instead, the second player may receive a notification that annotation 602 is available during a subsequent game session 604B. This notification may be audio (e.g., sound), visual (e.g., a displayed icon), tactile (e.g., vibration in a game controller or VR headset), etc. The second player may indicate that they wish to present annotation 602 by activating a designated control on the game controller or by selecting it from the on-screen menu. Of course, the triggering condition can also be more complex, involving any number of variables or states within the game state data 506, and may include combinational logic.

[0117] Note 602 may be presented only to its author, or to other players playing the video game, or to specific players under selected conditions. As an example, a subsequent player may need to subscribe to the author's note 602 for it to be displayed. Alternatively, the author may need to share note 602 (or a link to it) with subsequent players, and / or otherwise enable them to unlock note 602 for it to be presented in a subsequent game session 604B.

[0118] In one embodiment, annotation 602 may be created during the author's first game session 604. For example, as... Figure 6B As shown, the author can press the share button 608 on the game controller 610, which temporarily pauses the video game and displays a menu (not shown) from which the player can select an option to create annotation 602 (in other embodiments, the first game session 604A may be recorded by another player or an automated process). In response to this operation, an interface 612 can be presented to the author on a display device 614, which may be a television, tablet, smartphone, VR headset, etc. In some cases, annotation 602 may be created by the game system during the author's gameplay, an operation that may be performed in response to setting specific configurations before the game session. In some cases, annotation 602 may be generated based on the author's gameplay and supplemented by the author after the gameplay, for example, by adding commentary to the gameplay or editing the gameplay to remove less interesting parts and / or highlight more interesting parts.

[0119] Interface 612 may include a selectable game behavior timeline 616, which represents the recorded game behavior, such as... Figure 5A The relevant sections are described and can be graphically presented in the form of a film strip on interface 612. The game action timeline 616 can be associated with at least one adjustable marker 618. The adjustable marker 618 can initially be displayed at the end of the game action timeline 616 by default, representing the last frame of the recorded screen output 504 in the recorded game action. The frame at the point indicated by the adjustable marker 618 can be displayed as a representative frame 620 at another location on interface 612 to aid in browsing the game action timeline 616 and selecting points for inserting annotations 602. The author can move the adjustable marker 618 along the game action timeline 616 to mark specific locations within the recorded game action, which have the function of displaying the corresponding representative frame 620.

[0120] Before or after the author selects a location for annotation 602 within the game, the author may select and / or provide audio, video, text, graphics, highlighting, and / or metadata for annotation 602. In simple cases, the author provides text for annotation 602 via the game controller 610 using a displayed virtual keyboard (not shown). For example, the author may type "Find treasure here." In some cases, audio, video, text, graphics, highlighting, and / or metadata may be derived from recorded gameplay.

[0121] After providing or selecting the content and location of annotation 602, the author can create annotation 602 by pressing the share button 608 again or by selecting it from the screen menu (not shown). Thereafter, the created annotation 602 is linked to or associated with a point in the video game represented by the adjustable marker 618 relative to the game action timeline 616 (and the corresponding game state 606).

[0122] As used herein, the term "link" does not mean inserting any reference into game code 603 (although this is possible in some embodiments). Rather, an indication of a point within game code 603, and / or at least a portion of the game state 606 at the insertion point of annotation 602, may be stored in or associated with annotation 602. In one embodiment, the created annotation 602 containing the reference or game state 606 is stored in association with game code 603. Figure 3 In the game library 312 shown, the overlay processor 517 is made able to access the annotation during a subsequent game session 604B and render it as part of the screen output 504 in response to a triggering condition.

[0123] refer to Figure 7AAnnotation 602 may be generated by an automated process, i.e., using machine learning (ML). Such annotation 602 may include any of the types of annotation 602 described above. Another type of annotation 602 may indicate the emotions (e.g., feelings, attitudes, sensations) of one or more players playing the video game and / or one or more viewers watching the gameplay of the video game. Annotations 602 that convey the emotions of players and / or viewers can be used, for example, to create playable game segments or mini-games, as described in more detail below.

[0124] In some embodiments, the artificial intelligence (AI) engine 702 may receive a variety of inputs, including but not limited to recorded game 704 (e.g., the output of game recorder 503) and real-time game feedback 706 from one or more players and / or viewers of the recorded game 704. The AI ​​engine 702 may include or have access to a trained ML model 703, such as, but not limited to, a large language model (LLM), a bidirectional transformer, a zero / few-shot learner, or a deep neural network (DNN). The AI ​​engine 702 may use the ML model 703 to predict the emotions of one or more players or viewers at different points in the video game, where annotations 602 recording the emotions may be inserted. In some cases, emotions may be determined based on sensor feedback from one or more viewers, for example, by analyzing data from one or more microphones and / or cameras. In some cases, emotions may be determined by user input, such as when a user enters an emoji or text comment in response to game actions. In some cases, emotions may be segmented based on viewer demographic attributes, such as geographic location, or the viewer's experience or history with the game being watched. Emotion determination in this way can be used to train machine learning models that can judge emotions from gameplay behavior without relying on player feedback or observation. In some cases, emotion determination can be based on sensors focused on a group of viewers, such as cameras and / or microphones used to observe a group of people watching esports events.

[0125] In some embodiments, not every expression of player / viewer emotion generates annotation 602. Instead, annotation 602 can be inserted at points in the recorded game 704 where the level of interest reflected by real-time game feedback 706 exceeds a predetermined threshold (i.e., an interest "peak"). For example, simultaneously referencing Figure 7BMultiple interest peaks and corresponding emotion annotations 602 are shown at different points in the recorded game 704. Annotations 602 are associated with different game states 606, which are themselves part of the game state data 506 of the recorded game 704. In several embodiments, some or all of the game states 606 may be stored or associated with annotations 602 to allow the video game to be launched at the point associated with the annotation 602 in a manner similar to a saved game. In one embodiment, annotation 602 may be associated with a point in the game earlier than an interest peak. For example, if annotation 602 is used to launch the game or display a saved screen output 504, it may be expected that the game or display begins before an interest peak, whether that peak is the moment of a laugh or the moment when the player and / or viewer is startled by the appearance of a monster. The time offset of annotation 602 relative to the interest peak may be preset or predicted by an ML model 703, which may then receive feedback, as described below.

[0126] Levels of interest can correspond to levels of activity. For example, for viewers, the number and / or frequency of comments can serve as an indicator of level of interest. For players, levels of interest can correspond to the frequency of inputs, repetition of input types, occurrence of input patterns, combination inputs (e.g., key combinations), motion vectors, pressure applied to the controller, and / or similar factors. For both players and viewers, real-time game feedback 706 can include (but is not limited to) audio feedback, visual feedback, motion feedback, text feedback, and / or vital sign feedback, all of which indicate different levels of interest at different points in the video game. In some cases, real-time game feedback can be based on user interaction with game actions, such as pausing or replaying parts of the game. In other cases, real-time game feedback can be based on user interaction with buffered or recorded game actions, such as watching a delayed or recorded session of game action.

[0127] Audio feedback can be captured by microphone 707A associated with a game console, game controller, or VR headset, and can include verbal expressions of excitement and / or various emotions (such as happiness, sadness, fear, disgust, and anger) experienced while playing or watching video games. Similarly, visual feedback can include facial expressions (such as smiling), eye tracking (as an indicator of attention or engagement), etc., which can be captured by camera 707B associated with a game console, game controller, or VR headset.

[0128] Motion feedback can be obtained from motion sensors 707C (such as one or more accelerometers, magnetometers, cameras, and / or gyroscopes) associated with a game console, game controller, or VR headset. Text feedback can be any form of commentary, including (but not limited to) chat, email, direct messages, ratings, etc., and can be received via user input devices 707D (e.g., a game controller combined with a displayed virtual keyboard, or a physical keyboard, mouse, touchscreen, etc.). Vital signs feedback can be any vital signs (such as heart rate or body temperature) obtained by a vital signs monitor 707E (e.g., a smartwatch, smart ring, or appropriately equipped game controller) that monitors the player or spectator's vital signs during gameplay.

[0129] Data from input devices 707A-E can be aggregated by input processor 708, which may include one or more microprocessors, I / O controllers, network controllers, software modules, and / or the like. Input processor 708 may perform processing on various levels of the data from input devices 707A-E before passing the processed data along with some, all, or no of the original data from input devices 707A-E. For example, input processor 708 may only pass the original data that exceeds certain threshold levels in terms of quantity, frequency, intensity, and / or similar metrics. In one embodiment, input processor 708 may identify locations within a video game where one or more players or viewers show interest exceeding a predetermined threshold. In some cases, this predetermined threshold may be determined by analysis of game behavior using artificial intelligence (AI) and may include other factors such as player interaction and / or viewer interaction. The AI-analyzed game behavior may include game behavior across multiple players to determine thresholds for the game or parts thereof, and / or may include game behavior specific to a particular player to create thresholds tailored to that player.

[0130] The ML model 703 may have been previously trained using pre-defined or automatically generated annotations 602, and instructions on whether these annotations are considered correct by humans (or other AI). For example, if the ML model 703 determines that a particular point in a video game is "sad," a human can confirm or deny the determination, and this confirmation or denial is fed back to the ML model 703 so that it can predict emotions more accurately in the future.

[0131] While the ML model 703 may be pre-trained in many embodiments, it can be further refined in response to explicit or implicit annotation feedback 712 provided by the player or audience for a specific annotation 602. Figure 7CAs shown, in response to user selection, a timeline 714 or list of annotations 602 for a specific game or game segment can be displayed. Annotations 602 can be filtered by user-specified criteria (e.g., funny, sad, hint) so that only the desired annotations 602 are displayed. Furthermore, the displayed annotations 602 can indicate associated emotions, such as increased heart rate, sadness, or fun.

[0132] In some embodiments, a user may select one of annotations 602 to launch the game at the location associated with annotation 602 (e.g., using the associated game state 606) and start playing the game from that point. Alternatively or additionally, the user may choose to view the screen output 504 saved with the recorded game 704 at the point associated with annotation 602.

[0133] Subsequently, the user can provide explicit and / or implicit annotation feedback 712 for the emotion associated with annotation 602. Explicit annotation feedback 712 can be in the form of evaluation, such as graphical evaluation, numerical evaluation, descriptive evaluation, and / or comparative evaluation. For example, such as... Figure 7D As shown, users can provide explicit feedback on whether the annotated parts of the video game are as "fun" as predicted in note 602.

[0134] Implicit annotation feedback 712 can include any form of real-time game feedback 706, such as audio feedback, visual feedback, motion feedback, text feedback, and / or vital sign feedback. In some cases, implicit feedback 712 may contradict explicit feedback 712. For example, a user may be certain a joke is funny, but is recorded saying the opposite. In this case, the AI ​​engine 702 can ignore contradictory feedback, prioritize one type of feedback (e.g., implicit feedback eliminates explicit feedback), and / or determine weighted feedback by assigning different weights to different types of feedback.

[0135] Refer again Figure 7A The annotation feedback 712 is then used to update the ML model 703 of the AI ​​engine 702 so that the ML model 703 will generate annotations 602 that are more likely to be considered as correctly predicting emotions in the future. For example, if the annotation feedback 712 confirms the emotion reflected in annotation 602, the logic (e.g., neurons, nodes, weights) used by the ML model 703 to generate annotation 602 will be strengthened, making it more likely to generate similar annotations 602 for similar inputs in the future. Conversely, if the annotation feedback 712 contradicts the emotion reflected in annotation 602, the logic used by the ML model 703 to generate annotation 602 will be weakened, reducing the likelihood that it will generate that annotation 602 for similar inputs in the future.

[0136] like Figure 7EAs shown, annotation 602, reflecting the emotions of players and / or viewers, can be used to automatically generate playable game segments as a substitute for manually created game segments. Figure 4B A replacement or supplement to ). Artificial Intelligence (AI) Engine 702 (which may be similar to) Figure 7A The AI ​​engine 702 can receive various inputs, including but not limited to recorded game data 704, real-time game feedback 706, game statistics 709, and one or more annotations 602 indicating the player's and / or viewer's emotions at interest peaks in the video game (i.e., points where interest exceeds a predetermined threshold).

[0137] AI engine 702 may include or have access to a trained ML model 703, such as a large language model (LLM), a bidirectional transformer, a zero / few-shot learner, or a deep neural network (DNN). AI engine 702 may utilize ML model 703 to predict one or more game segments 710 in a recorded game 704 that will result in higher completion rates, more positive feedback, higher activity levels, and / or more positive ratings for the user or a specific user. The ML model 703 may have been previously trained using predetermined or automatically generated segments from the recorded game 704, along with indications regarding whether these segments were completed by the user, generated positive feedback, led to increased activity levels, and / or received high user ratings.

[0138] Recorded game data (704) may contain game behavior recorded by one or more human players, such as in combination with... Figure 4A As described above. In some embodiments, at least a portion of the recorded game 704 may have been completed by an AI player, who evaluates game behavior outputs and generates user input in the same manner as a human player.

[0139] Real-time game feedback 706 may be captured simultaneously while one or more human users are playing a recorded game (704). For example, see reference... Figure 7A The real-time game feedback 706 discussed may include, but is not limited to, audible feedback, visual feedback, motion feedback, text feedback and / or vital sign feedback, all of which can indicate the user's excitement, interest, liking and / or engagement with the video game when receiving the real-time game feedback 706.

[0140] Game statistics 709 may include, but are not limited to, information about the structure of the recorded game 704 (such as...). Figure 4A As shown), this includes level and / or scene structure, activity level of each game chapter (i.e., which chapters of the video game are played the most or generate the most input data), the point where the user stopped playing recorded game 704, the point where the user completed a game chapter in less than (or more than) the average time, user feedback or evaluation of recorded game 704 or its chapters, etc.

[0141] Based on various inputs, the ML model 703 of the AI ​​engine 702 may output game segments 710 that are predicted to achieve higher completion rates, increased positive feedback, greater activity, and / or more positive evaluations for users or specific users (e.g., time ranges and / or instructions for levels, scenes, and parts thereof).

[0142] like Figure 7F As shown, game segments 710 can have different lengths (e.g., estimated game time) to accommodate varying time budgets available to different users. For example, game segments 710 can cover the entire scene that takes an average of 30 minutes to complete, or a subset of scenes that take only 10 minutes to complete. In some embodiments, game segments may be generated both for the entire scene and for a subset of the scene to appeal to users with different time budgets.

[0143] In some embodiments, each game segment 710 may be associated with a corresponding annotation 602 of player / viewer emotion, as those annotations 602 correspond to peaks of interest and may be predicted to be attractive to a large number of players. However, this is not necessary in every implementation, and the game segment 710 does not necessarily need to be centered on the point of annotation 602 as shown in the figure. In some embodiments, the game segment 710 may be associated with multiple annotations 602. In any case, the game segment 710 can still be associated with the corresponding annotation 602, because the game state 606 may be completely consistent with the original point in the full video game.

[0144] The game clip 710 output by AI Engine 702 can be based on Figure 5A The technology is used to generate game fragment code that users can play as mini-games. For example, a game state analyzer analyzes the game state of the game fragment, and a game breakpoint handler determines appropriate breakpoints based on this analysis to define the start and end of the game fragment. A code assembler, based on the operations of the game state analyzer, the game breakpoint handler, and the game state settings handler, assembles the code segments to define the game fragment code. The game fragment code is completely self-contained, including all the code segments required to execute game fragment 710.

[0145] The generated game fragment 710 can be stored in the electronic game library 312, such as Figure 3 As shown. Game segment 710 is then available for users (including those who may not have enough time to play the full game but are still interested in playing the standalone game segments that include specific challenges, puzzles, etc.) to play.

[0146] After the user plays game segment 710, the user can provide explicit and / or implicit game segment feedback 713. Explicit game segment feedback 713 can be presented in an evaluative form, such as graphical evaluation, numerical evaluation, descriptive evaluation, and / or comparative evaluation. Implicit game segment feedback 713 may include whether the user has completed game segment 710 (in some embodiments, incomplete completion may be considered negative feedback) and / or a combination thereof. Figure 7A The real-time game feedback 706 may include audio feedback, visual feedback, motion feedback, text feedback, and / or vital sign feedback. Sometimes, some game segment feedback 713 may be positive (e.g., the user completes the mini-game), while other game segment feedback 713 may be negative (e.g., there is a verbal cue in the audio feedback indicating that the user dislikes the game segment 710). In some embodiments, negative implicit game segment feedback 713 may be preferred because it is more accurate or relevant. Alternatively, different weights may be assigned to various types of positive and negative feedback to determine the overall game segment feedback 713 for the user. In some cases, the user may rate different aspects of the same game segment 710, resulting in some aspects receiving positive ratings and others negative ratings, such as rating puzzle elements as highly engaging but combat elements as too difficult.

[0147] like Figure 7E As shown, the game fragment feedback 713 is then used to update the ML model 703 of the AI ​​engine 702, so that the ML model 703 can generate game fragments 710 with higher completion rates, more positive feedback, higher activity, and / or more positive evaluations in subsequent iterations. For example, if the game fragment feedback 713 is positive, the logic (e.g., neurons, nodes, weights) used by the ML model 703 to generate the game fragment 710 will be strengthened, making it more likely to generate similar game fragments 710 for similar inputs in the future. Conversely, if the game fragment feedback 713 is negative, the logic used by the ML model 703 to generate the game fragment 710 will be weakened, reducing the likelihood that it will generate the game fragment 710 for similar inputs in the future.

[0148] like Figure 7G As shown, note 602 allows players to search for game fragments 710 that are associated with one or more desired emotions. For example, the game library 312 may include the ability to use... Figure 7E The process shown generates multiple game segments 710. The player can run a game launcher 718, which is an application configured to help the player identify games that match their interests and available time. In this example, the player can specify one or more explicit preferences 720, such as "I want to laugh or cry for 5 minutes."

[0149] The game launcher 718 can search the game library 312 for game segments 710 that satisfy one or more explicit preferences 720. If one or more matches are found, they can be displayed in the interface 722 provided by the game launcher 718. In some embodiments, the game launcher 718 can be configured to list game segments 710 that match or substantially match one or more explicit preferences 720 within a tolerance range. For example, even if the user indicates that they expect a 5-minute experience, a tolerance factor (e.g., + / - 2 minutes, which can be specified by the user or determined based on the player's historical selections) may allow 6-minute game segments to be displayed, but not 9-minute game segments 710. The interface 722 may list the estimated game duration and / or the mood associated with the corresponding annotation 602.

[0150] After the user has played the selected game segment 710, the game launcher 718 can provide another interface 724 to allow the user to provide explicit game feedback 712 in the form of an evaluation (e.g., excellent, average, needs improvement). Alternatively, as described above, the game feedback 712 can be implicitly derived from the real-time game feedback 706 and / or whether the player has completed the selected game segment 710 and to what extent.

[0151] In other embodiments, the user does not need to provide explicit preferences 720 to the game launcher 718. (See reference...) Figure 7H AI Engine 702 (can be used with) Figure 7A (Similar to AI engine 702) can recommend at least one game segment 710 to a user based on its known implicit preferences about the user, which is expected to be completed and receive positive feedback and / or positive ratings.

[0152] In one embodiment, AI engine 702 receives user request 726, which may be generated by a user logging into a video game console and / or launching a game launcher 718. However, user request 726 may contain some explicit preferences for the video game session, while other preferences may be implicit. For example, user request 726 may include a specific amount of playtime available to the user, such as 10 minutes, 30 minutes, or one hour. Similarly, user request 726 may include a specific game genre (e.g., role-playing, platforming, action, adventure, first-person shooter, sports, simulation, fighting) and / or a specific game that the user wishes to play within their available time. In some embodiments, request 726 may include a user's specific interest in the game session, such as killing time, improving skills, seeking challenges, progressing through levels, or timed races (e.g., speedruns).

[0153] As an alternative or supplement, some or all of the above content can be inferred from the user information 728 available to the AI ​​engine 702. For example, user information 728 may include the user's schedule (e.g., from a calendar application), historical data related to the user's gaming habits (e.g., the user typically plays for one hour at the same time each day), etc. User information 728 may also include historical data related to the game types and / or themes of the user's preferred games, games the user has recently played, and / or specific interests related to the user's gaming sessions. In some embodiments, user information 728 may also include information such as the user's skill level, games played, achievements obtained, and previous feedback from the user on games and game segments.

[0154] In some embodiments, explicit and implicit preferences can be aggregated and analyzed by a preference analyzer 729, which may include one or more microprocessors, I / O controllers, network controllers, software models, etc. The preference analyzer 729 may be a standalone module or part of an AI engine 702. In some embodiments, the preference analyzer 729 can determine a user's explicit and / or implicit preferences without using the AI ​​engine 702.

[0155] In some cases, a user's calendar application can reserve time slots for game sessions, which can be retrieved and used to determine available time without user input. In other cases, multiple users' and / or their friends' calendar applications can reserve time slots for team game sessions. As an alternative or supplement, AI engine 702 can utilize user historical behavior to determine the expected available time and / or goals for a game session. For example, if a user typically engages in 60-minute game sessions after get off work, AI engine 702 can utilize an ML model 703 trained on the user's previous behavior to determine the available time for that game session.

[0156] AI engine 702 can select previously created game segments 710 using ML model 703 based on perceived matching with explicit and / or implicit preferences embodied in user request 726 and user information 728, and / or (using a combination of...) Figure 7E The aforementioned technology creates new game segments 710. In some embodiments, multiple recommended game segments 710 may be provided to the user, such as game segments 710 of different durations, themes, goals, and works that match the user's explicit or implicit interests (including available time). Subsequently, based on... Figure 5A The technology uses recommended game fragments 710 to generate game fragment code that users can play as mini-games.

[0157] After a user plays a mini-game, the user can provide explicit and / or implicit game fragment feedback 713. Explicit game fragment feedback 713 can be in the form of an evaluation (e.g., a numerical rating or a star rating) and / or an indication of whether the user has completed the mini-game. In some embodiments, not completing the mini-game can be considered negative feedback. Implicit game fragment feedback 713 can be combined with... Figure 7A The real-time game feedback 706 is similar to or the same as that provided, such as audio feedback, visual feedback, motion feedback, text feedback, and / or vital sign feedback.

[0158] The game segment feedback 713 is then used to update the ML model 703 of the AI ​​engine 702, enabling the AI ​​engine 702 to subsequently recommend game segments 710 with higher completion rates, more positive feedback, higher activity, and / or more positive evaluations. In other words, if the game segment feedback 713 is positive, the logic (e.g., neurons, nodes, weights) used by the ML model 703 to recommend that game segment 710 will be strengthened, making it more likely to recommend similar game segments 710 for similar inputs in the future. Conversely, if the game segment feedback 713 is negative, the logic used by the ML model 703 to recommend that game segment 710 will be weakened, reducing the likelihood of recommending that game segment 710 for similar inputs in the future.

[0159] Figure 8 A flowchart of method 800 for annotating a video game using indications of player and / or viewer emotions and generating playable segments of the video game based on the annotations, the playable segments of which can then be selected by a user according to explicit or implicit preferences. According to some examples, method 800 includes: recording 802 a first session of the video game, the first session including game state data generated by a video game processor processing player input data. As previously mentioned, the first session may be played and recorded by the author of the annotation, another user, or an automated process.

[0160] Method 800 continues: 804 Identifying a location within the video game where one or more players or viewers show interest exceeding a predetermined threshold. This location may correspond to an interest “peak,” which, in one embodiment, can be determined by activity level and / or real-time game feedback. Examples of real-time game feedback may include audio feedback, visual feedback, motion feedback, text feedback, and / or vital sign feedback. Identifying locations within the video game may be based on data from player gameplay, which may be real-time or recorded. Identifying locations within the video game may also be based on viewer interaction with gameplay, which may be real-time or recorded. In some cases, players may indicate locations within the game while playing the game or replaying a recording of their gameplay to indicate locations within the video game they are interested in.

[0161] Method 800 continues: a trained machine learning model determines the emotion of one or more players or spectators at the location based on real-time game feedback; then, the annotation of the emotion is associated with the game state data at that location within the video game.

[0162] In one embodiment, the method 800 continues by segmenting the video game 810 into playable segments containing locations associated with annotations. This segmentation can be performed by a trained machine learning model or another machine learning model based on one or more of the annotations or real-time game feedback.

[0163] The method 800 continues to execute: obtaining one or more preferences of 812 players regarding the game session. These preferences may be explicit or implicit. In some embodiments, implicit preferences may be determined by a trained machine learning model or another machine learning model, based on, for example, player history data, player calendar data, real-time game feedback, and / or the like.

[0164] The method 800 continues by identifying 814 playable segments that satisfy one or more of these preferences. This identification can be performed by a trained machine learning model or another machine learning model, based on, for example, the player's explicit or implicit preferences, player history data, real-time game feedback, and / or the like. The machine learning model discussed herein can be updated in response to feedback on annotations, such as whether the mood of the annotation is consistent with the mood of the game or a game segment, and / or whether the playable segment satisfies the user's explicit or implicit preferences.

[0165] Figure 9 A block diagram of an exemplary electronic entertainment system 900 is shown, which can... Figure 2 This is used within the context of the cloud gaming system shown. For example... Figure 9 The illustrated electronic entertainment system 900 includes a main memory 902, a central processing unit (CPU) 904, a graphics processor 906, an input / output (I / O) processor 908, a controller input interface 910, a hard disk drive or other storage component 912 (which may be removable), a communication network interface 914, a virtual reality interface 916, a sound engine 918, and an optical disc / media control 920. Each of these components is connected via one or more system buses 922.

[0166] like Figure 9 The illustrated electronic entertainment system 900 may be a video game console. The electronic entertainment system 900 may also be implemented as a general-purpose computer, set-top box, handheld gaming device, tablet computing device, or mobile computing device / telephone. Depending on its specific form factor, purpose, or design, the electronic entertainment system may include some or all of the disclosed components.

[0167] Main memory 902 stores instructions and data for execution by CPU 904. When the electronic entertainment system 900 is running, main memory 902 can store executable code. Figure 9 The main memory 902 can communicate with the CPU 904 via a dedicated bus. The main memory 902 can provide pre-stored programs, as well as programs transferred from the hard disk drive / storage component 912 via the I / O processor 908, from a DVD or other optical disc (not shown) via the optical disc / media controller 920, or downloaded via the communication network interface 914.

[0168] Figure 9 The graphics processor 906 (or graphics card) in the CPU 904 executes graphics instructions received from the CPU 904 to generate an image to be displayed on a display device (not shown). Figure 9 The GPU 906 can transform objects in three-dimensional coordinates to two-dimensional coordinates and vice versa. The GPU 906 can use ray tracing to simulate and track individual rays produced by light sources, thus aiding in the rendering of light and shadow in game scenes. The GPU 906 can achieve 4K-8K resolutions and up to 120 FPS at a 120Hz refresh rate, thanks to its fast startup and loading times. The way the GPU 906 renders or processes images may differ for a specific display device.

[0169] Figure 9 The I / O processor 908 in the middle can also allow content to be exchanged on wireless or other communication networks (e.g., IEEE 802.x (including Wi-Fi and Ethernet), 9G, 4G, LTE and 3G mobile networks, as well as Bluetooth and short-range personal area networks). Figure 9 The I / O processor 908 primarily controls the data exchange between the various devices of the electronic entertainment system 900 (including CPU 904, graphics processor 906, controller interface 910, hard disk drive / storage component 912, communication network interface 914, virtual reality interface 916, sound engine 918, and optical disc / media control 920).

[0170] Figure 9In the electronic entertainment system 900, the user issues commands to the CPU 904 via a controller device communicatively coupled to the controller interface 910. Various controllers can be used to receive commands, including handheld and sensor-based controllers (e.g., controllers for capturing and interpreting eye-tracking, voice-based, and gesture-based commands). The controller receives user commands or input and provides them to the controller interface 910 and then to the CPU 904 for interpretation and execution. These commands can also be used by the CPU 904 to control other components of the electronic entertainment system 900. For example, the user can instruct the CPU 904 to store certain game information on a hard disk drive / storage component 912 or other non-transitory computer-readable storage media. The user can also instruct a character in the game to perform a specific action, which is rendered in conjunction with audio parsed by the graphics processor 906 and the sound engine 918.

[0171] The hard disk drive / storage component 912 may include a removable or non-removable non-volatile storage medium. This medium may be portable and may contain, for example, a digital video optical disc (DVD), a Blu-ray disc, or a USB-coupled memory, for inputting and outputting data and code to and from the main memory 902. Software implementing the embodiments of the present invention can be stored on this medium and input to the main memory via the drive / storage component 912. Software stored on the hard disk drive can also be managed by the optical disc / media controller 920 and / or the communication network interface 914, etc.

[0172] The communication network interface 914 allows communication via various communication networks, including local private networks and / or wider wide area networks such as the Internet. The Internet is a large network of interconnected computers and servers that allows the transmission and exchange of Internet Protocol (IP) data between users connected through a network service provider. Examples of network service providers include the public switched telephone network, wired or fiber optic services, digital subscriber line (DSL) or broadband services, and satellite services. The communication network interface allows the exchange of communication and content between various remote devices, including other electronic entertainment systems associated with other users, cloud-based databases, services and servers, and content hosting systems that can provide or facilitate gameplay and related content.

[0173] The Virtual Reality Interface 916 allows for the processing and rendering of virtual reality, augmented reality, and mixed reality data. It includes display devices capable of creating partially or fully immersive virtual environments. The Virtual Reality Interface 916, in conjunction with audio and haptic feedback processed by the sound engine 918, enables interactive presentation and gaze-based rendering of immersive visual fields.

[0174] The sound engine 918 executes instructions to generate sound signals, which are then output to audio devices such as television speakers, controller speakers, stand-alone speakers, headphones, or other headphone speakers. Different sound sets can be generated for different sound output devices, which may include spatial audio or 3D audio effects.

[0175] The optical disc / media controller 920, which may be implemented using a disk drive or an optical disc drive, is used to store, manage, and control data and instructions for use by the CPU 904. The optical disc / media controller 920 may include system software (operating system) for implementing embodiments of the present invention, which facilitates loading the software into the main memory 902.

[0176] The systems and methods described herein can be implemented using hardware, software, firmware, or a combination thereof. In some examples, the systems described herein can be implemented using a non-transitory computer-readable medium storing computer-executable instructions that, when executed by one or more processors of a computer, cause the computer to perform operations. Computer-readable media suitable for implementing the control systems described herein include non-transitory computer-readable media, such as disk storage devices, chip storage devices, programmable logic devices, random access memory (RAM), read-only memory (ROM), optical read / write memory, cache, magnetic read / write memory, flash memory, and application-specific integrated circuits (ASICs). Furthermore, the computer-readable media for implementing the control systems described herein can be located on a single device or computing platform, or distributed across multiple devices or computing platforms.

[0177] The detailed description of the present technology above is provided for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the technology to the precise forms disclosed. Many modifications and variations are possible based on the foregoing teachings. The selected embodiments are intended to best illustrate the principles of the present technology and its practical application, enabling those skilled in the art to utilize the technology in various embodiments and in combination with various modifications suitable for the intended particular use. The scope of the present technology is intended to be defined by the claims.

Claims

1. A computer-implemented method, comprising: Record the first session of the video game, the first session including game state data generated by the video game processor processing player input data; Identify locations within the video game where one or more players or viewers show interest exceeding a predetermined threshold. The emotions of one or more players or spectators at the location are determined by a trained machine learning model based on real-time game feedback. as well as The annotation of the emotion is associated with the game state data at the location within the video game.

2. The computer-implemented method according to claim 1, wherein, Identifying locations within the video game includes identifying peaks of interest for one or more players or viewers.

3. The computer-implemented method according to claim 1, wherein, The real-time game feedback includes audio recorded by a microphone associated with the client device used to participate in the first session.

4. The computer-implemented method according to claim 1, wherein, The real-time game feedback includes video recorded by a camera associated with the client device used to participate in the first session.

5. The computer-implemented method according to claim 1, wherein, The real-time game feedback includes motion recorded by motion sensors associated with the client device used to participate in the first session.

6. The computer-implemented method according to claim 1, wherein, The real-time game feedback includes text recorded by an input device associated with the client device used to participate in the first session.

7. The computer-implemented method according to claim 1, wherein, The real-time game feedback includes vital sign data recorded by a vital sign monitor associated with the client device used to participate in the first session.

8. The computer-implemented method according to claim 1, further comprising: The video game is segmented into playable segments that include the locations associated with the annotations.

9. The computer-implemented method according to claim 8, wherein, The video game segmentation includes generating multiple playable segments of different lengths.

10. The computer-implemented method according to claim 8, wherein, The video game is segmented as follows: Determine the boundaries of the playable segments within the context of gameplay behavior in the video game; and Generate the playable segment with defined boundaries.

11. The computer-implemented method according to claim 10, wherein, The boundary is determined by the trained machine learning model or another machine learning model based on one or more of the annotations or the real-time game feedback.

12. The computer-implemented method according to claim 9, further comprising: To obtain one or more of a player's preferences; as well as Identify playable segments from the plurality of playable segments that satisfy one or more of the preferences.

13. The computer-implemented method according to claim 12, wherein, The one or more preferences include a preference sentiment for the playable segment, and wherein identifying the playable segment includes identifying the playable segment based on the sentiment of associated annotations.

14. The computer-implemented method according to claim 12, wherein, The one or more preferences include a preferred game time for the playable segment, and wherein identifying the playable segment includes identifying playable segments whose game time is within a threshold amount of the preferred game time.

15. The computer-implemented method according to claim 12, wherein, Obtaining one or more preferences includes determining at least one implicit preference of the player.

16. The computer-implemented method according to claim 15, wherein, The at least one implicit preference includes a preference for game time, wherein the preference for game time is determined from at least one of player calendar data and player history data.

17. The computer-implemented method according to claim 12, wherein, Identifying the playable fragments includes using the trained machine learning model or a second machine learning model to identify the playable fragments based on perceptual matching with one or more of the preferences.

18. The computer-implemented method according to claim 17, further comprising: Receive feedback from the player regarding whether the playable segment satisfies one or more of their preferences; as well as Based on feedback regarding the playable segments, the trained machine learning model or the second machine learning model is updated.

19. The computer-implemented method according to claim 1, further comprising: Receive feedback from the player in the second session of the video game regarding whether the annotation accurately describes the player's emotions at the given location; as well as The trained machine learning model is updated based on the feedback.

20. The computer-implemented method according to claim 1, further comprising: Generate a timeline of annotations for the first session that satisfies a set of conditions provided by the player; Receive the player's selection of the first annotation from the timeline; as well as Perform at least one of the following: Initiate a second session of the video game at the location associated with the annotation in the video game; or The screen output of the recording of the first session of the video game is displayed at the location in the video game associated with the annotation.

21. A system comprising: A game recorder that records the first session of a video game, the first session including game state data generated by the video game processor processing player input data; An input processor identifies a location within the video game where the interest of one or more players or viewers exceeds a predetermined threshold. A trained machine learning model that determines the mood of one or more players or viewers at the location based on real-time game feedback; And a storage device that associates the annotations of the emotions with game state data at the location within the video game.

22. The system according to claim 21, wherein, The input processor identifies locations within the video game by determining the peak interest of one or more players or viewers.

23. The system according to claim 21, wherein, The real-time game feedback includes audio, and the system also includes a microphone associated with a client device used to participate in the first session for recording the audio.

24. The system according to claim 21, wherein, The real-time game feedback includes video, and the system also includes a camera associated with a client device used to participate in the first session to record the video.

25. The system according to claim 21, wherein, The real-time game feedback includes motion, and the system also includes motion sensors associated with a client device used to participate in the first session to record the motion.

26. The system according to claim 21, wherein, The real-time game feedback includes text, and the system also includes an input device associated with a client device used to participate in the first session to receive the text.

27. The system according to claim 21, wherein, The real-time game feedback includes vital sign data, and the system also includes a vital sign monitor associated with the client device used to participate in the first session, which records the vital sign data.

28. The system of claim 21, further comprising: A game segment generator that segments the video game into playable segments containing the locations associated with the annotations.

29. The system according to claim 28, wherein, The game segment generator divides the video game into multiple playable segments of varying lengths.

30. The system according to claim 28, wherein, The game fragment generator determines the boundaries of the playable fragments within the context of gameplay in the video game and generates the playable fragments with the determined boundaries.

31. The system according to claim 30, wherein, The game fragment generator uses or includes an artificial intelligence (AI) engine, which includes the trained machine learning model or another machine learning model to determine the boundaries based on one or more of the annotations or the real-time game feedback.

32. The system of claim 29 further includes a preference analyzer, the preference analyzer acquiring one or more preferences of the player and identifying playable segments among the plurality of playable segments that satisfy the one or more preferences.

33. The system according to claim 32, wherein, The one or more preferences include a preference sentiment for the playable segment, and wherein the preference analyzer identifies the playable segment by recognizing the sentiment based on associated annotations.

34. The system according to claim 32, wherein, The one or more preferences include a preferred amount of game time for the playable segment, and wherein the preference analyzer identifies the playable segment by identifying playable segments whose game time is within a threshold amount of the preferred game time.

35. The system according to claim 32, wherein, The preference analyzer identifies one or more preferences by determining at least one implicit preference of the player.

36. The system according to claim 35, wherein, The at least one implicit preference includes a preference for game time, wherein the preference for game time is determined by the preference analyzer from at least one of player calendar data and player history data.

37. The system according to claim 32, wherein, The preference analyzer includes or uses an artificial intelligence (AI) engine containing the trained machine learning model or the second machine learning model, which identifies the playable segments based on perceptual matching with one or more preferences.

38. The system according to claim 37, wherein, The AI ​​engine receives feedback from the player regarding whether the playable segment satisfies one or more of the player's preferences, and updates the trained machine learning model or the second machine learning model based on the feedback regarding the playable segment.

39. The system according to claim 31, wherein, The AI ​​engine receives feedback from players in a second session of the video game regarding whether the annotation correctly describes the player's emotions at the location, and updates the trained machine learning model based on the feedback.

40. The system of claim 31, further comprising a game launcher, wherein the game launcher: Generate a timeline of annotations for the first session that satisfies a set of conditions provided by the player; Receive the player's selection of the first annotation from the timeline; and Perform at least one of the following: Initiate a second session of the video game at the location associated with the annotation in the video game; or The screen output of the recording of the first session of the video game is displayed at the location in the video game associated with the annotation.

Images