In-game dynamic camera angle adjustment

An AI-driven system dynamically adjusts camera angles in video games based on player interactions, addressing the limitations of fixed developer-set angles by optimizing views for enhanced engagement and realism.

JP7577213B2Active Publication Date: 2024-11-01SONY INTERACTIVE ENTERTAINMENT LLC
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
JP2023532734
Authority / Receiving Office
JP · JP
Patent Type
Patents
Current Assignee / Owner
Priority Date
2020-11-30
Filing Date
2021-10-05
Publication Date
2024-11-01
Estimated Expiration
2041-10-05

AI Technical Summary

Technical Problem

Existing video games rely on predetermined camera angles set by developers, which may not optimally adapt to dynamic player interactions, leading to delayed or suboptimal viewing experiences.

Method used

Implement an artificial intelligence (AI) model trained on datasets of video sequences with tagged metadata to dynamically adjust camera angles during gameplay, using snapshots to identify optimal angles based on player interactions and actions.

Benefits of technology

Provides seamless and interactive camera angle adjustments that enhance the gaming experience by offering views tailored to the current gameplay actions, improving engagement and realism.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 0007577213000001
    Figure 0007577213000001
  • Figure 0007577213000002
    Figure 0007577213000002
  • Figure 0007577213000003
    Figure 0007577213000003
Patent Text Reader

Abstract

Methods and systems are provided for dynamically adjusting camera angles in a video game. One example method includes receiving a dataset of a plurality of video sequences associated with tagging data in metadata that identifies characteristics of actions occurring in each of the plurality of video sequences. The method includes generating an artificial intelligence (AI) model from the dataset. The method includes accessing the AI ​​model during gameplay of the video game. The accessing includes providing snapshots of video sequences generated during gameplay of the video game. The method includes dynamically adjusting a game camera angle during gameplay. The adjusting is responsive to adjustment instructions derived from the AI ​​model.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to methods and systems for dynamically changing camera angles to a video game scene based on learned camera angles that are optimal for an interactive scene being presented to a user in a video game. [Background technology]

[0002] The video game industry has undergone many changes over the years. As computing power has increased, video game developers have also developed game software that takes advantage of this increased computing power. To do so, video game developers code games that incorporate advanced computational and mathematical techniques to create highly detailed and engaging gaming experiences.

[0003] Examples of game platforms include Sony PlayStation®, Sony PlayStation2® (PS2), Sony PlayStation3® (PS3), Sony PlayStation4® (PS4), and Sony PlayStation5® (PS5), each of which is sold in the form of a game console. As is well known, game consoles are designed to connect to a display (typically a television) and allow user interaction via a handheld controller. Game consoles are designed to be provided with dedicated processing hardware, including a CPU, a graphics synthesizer for handling intensive graphic calculations, a vector unit for performing geometric transformations, and other glue hardware, firmware, and software. In addition, for playing locally with the game console, the game console is designed to be provided with an optical disc reader for receiving a game disc. Online gaming is also possible, allowing users to play interactively against or with other users over the Internet. As the complexity of games continues to attract players, game and hardware manufacturers continue to innovate to enable greater interactivity and computer programs.

[0004] Although games continue to see significant improvements in graphics, speed, and realism, the game experience still depends on the camera view into a particular game scene programmed by the game developer. For example, most camera angles are pre-determined or set based on developer coding set during game programming, depending on the interactivity and gamer's positioning within the scene. Unfortunately, when a player tries to experience a game scene in a more dynamic way, in some cases the camera angle lags behind or is not optimally positioned for the interactivity occurring within the game.

[0005] It is in these situations that embodiments of the present disclosure arise. Summary of the Invention

[0006] Embodiments of the present disclosure include devices, methods, and systems related to dynamic selection and generation of camera angles for scenes generated while a game is interactive, based on artificial intelligence-based camera angle selection.

[0007] In one embodiment, a method is provided for dynamically adjusting a camera angle in a video game. The method includes receiving a dataset of a plurality of video sequences associated with tagging data in metadata that identifies characteristics of actions occurring in each of the plurality of video sequences. In one example, the metadata includes a camera angle used in each of the plurality of video sequences. In another example, the camera angle is determined based on an analysis of images in the video sequences. The method includes generating an artificial intelligence (AI) model from the dataset. The method includes accessing the AI ​​model during gameplay of the video game. The accessing includes providing snapshots of video sequences generated during gameplay of the video game. The method includes dynamically adjusting a game camera angle during gameplay. The adjusting is responsive to adjustment instructions derived from the AI ​​model.

[0008] In one embodiment, adjusting the game camera angle modifies or replaces the developer camera angle of the game.

[0009] In one embodiment, generating the AI ​​model includes extracting video feature data from each of the plurality of video sequences and extracting metadata feature data associated with each of the plurality of video sequences. This embodiment includes classifying the video feature data and the metadata feature data for processing by the AI ​​model. The AI ​​model is configured to process the video feature data and the metadata feature data to label characteristics of the actions occurring in each of the plurality of video sequences. The generation of the AI ​​model occurs during a training process.

[0010] In one embodiment, the AI ​​model is configured to process the video sequence snapshots generated during gameplay. Processing the video sequence snapshots includes examining image content of one or more video sequences produced during gameplay to produce video feature data and examining metadata associated with the examined image content to produce metadata feature data. The video feature data and metadata feature data are classified and used by the AI ​​model to identify the adjustment instructions for adjusting game camera angles to present video game scenes during gameplay.

[0011] In one embodiment, snapshots of the video are generated periodically during game play to trigger processing of the adjustment instructions to adjust game camera angles to present video game scenes during game play.

[0012] In one embodiment, the video snapshots are generated in response to metadata generated during game play that indicates that an ongoing action scene should include an adjustment to the game camera angle.

[0013] In one embodiment, the video snapshots are generated in response to metadata generated during game play that indicates that an upcoming action scene should include an adjustment to the game camera angle.

[0014] In one embodiment, the tagging data is obtained from a database of user-generated data (UGD), the UGD including frame sequences that include the camera angles marked as preferred, the marking as preferred being used to select a number of video sequences to use as training data for the AI ​​model.

[0015] In one embodiment, the tagging data is automatically generated based on detected viewing popularity for a frame sequence containing the camera angle, and the viewing popularity is used to select a number of video sequences to use as training data for the AI ​​model.

[0016] In one embodiment, the video sequences in the dataset are sourced from one or more of movies, video game recordings, live sporting event recordings, social media videos, or recordings including one or more action scenes, and the video sequences are used to train the AI ​​model, which is used during gameplay to adjust game camera angles in the video game during gameplay.

[0017] In one embodiment, the plurality of video sequences are used to train the AI ​​model, and the AI ​​model is used during gameplay to adjust game camera angles within the video game during gameplay.

[0018] In one embodiment, the game camera angles provide an alternative view of the video game scene than that programmed by the video game developer.

[0019] In one embodiment, a system is provided for dynamically adjusting camera angles in a video game. The system includes a game server for executing one or more games for streaming to a client device. The system includes an artificial intelligence (AI) camera angle processor for examining snapshots of video sequences produced during gameplay of the video game. In examining the snapshots, an AI model is used to identify game camera angles to use in creating a gameplay scene. The game camera angles used to create a gameplay scene provide an alternative view to the scene than a view programmed by a developer of the video game. The system includes a streaming server configured to compress video frames of the produced scene including the camera angles having the alternative view. The streaming server is configured to transmit the compressed video frames to the client for display on a display viewable during the gameplay.

[0020] In one embodiment, a survey of snapshots using an AI model to identify game camera angles is processed periodically while game play is ongoing.

[0021] In one embodiment, a review of snapshots using an AI model to identify game camera angles is processed during periods of game activity above a predefined threshold.

[0022] In one embodiment, the examination of snapshots using the AI ​​model to identify game camera angles is not processed during periods when game activity does not exceed a predefined threshold.

[0023] In one embodiment, the snapshots are created during periods when game activity exceeds a predefined threshold.

[0024] In one embodiment, snapshots are not created during periods when game activity does not exceed a predefined threshold.

[0025] In one embodiment, when a game camera angle is selected to change from that programmed by the developer, the scene into the video game is generated by the game engine to render the video frames that are streamed to the client.

[0026] For the purposes of providing examples of the disclosed methods and systems, various embodiments are described below. Other aspects and advantages of the present disclosure will become apparent from the following detailed description, taken in conjunction with the accompanying drawings, illustrating by way of example the principles of the present disclosure. [Brief description of the drawings]

[0027] The present disclosure may be better understood by reference to the following description taken in conjunction with the accompanying drawings.

[0028] [Figure 1] 1 shows a graphical illustration of multiple video streams of various different types of video, according to one embodiment.

[0029] [Diagram 2] 1 illustrates an example dataset for identifying video sequences tagged with different types of videos, according to one embodiment.

[0030] [Diagram 3] 1 illustrates an example of an AI model training engine that may be utilized to process video and metadata in training a camera angle AI model, according to one embodiment.

[0031] [Figure 4] 1 illustrates an embodiment in which one or more data centers are utilized to run one or more video games and utilize camera angle AI models, according to one embodiment.

[0032] [Figure 5A] 1 illustrates an example of one of the data centers, according to one embodiment.

[0033] [Figure 5B] 1 illustrates an example of switching between different camera views, according to one embodiment.

[0034] [Figure 6] According to one embodiment, a general embodiment of a method that may be utilized to dynamically select camera angles during gameplay based on learning optimal camera angles for the particular characteristics of the scene being presented in an interactive game is shown.

[0035] [Figure 7] FIG. 1 is a block diagram of a gaming system according to various embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0036] The following embodiments of the present disclosure provide devices, methods, and systems for dynamically adjusting camera angle views into scenes generated for video games and using models to select optimal camera angles. However, it will be apparent to one of ordinary skill in the art that the present disclosure may be practiced without some or all of the specific details described herein. In other instances, well known process operations have not been described in detail in order to not unnecessarily obscure the present disclosure.

[0037] FIG. 1 shows a graphical illustration of multiple video streams of various different types of videos. For example, video 1 may be an action movie, video 2 may be a video game recording, and video game N may be a video of a recorded sporting event. These different types of videos may be selected for training purposes in generating a dataset that includes video sequences that highlight scenes where a particular camera angle is utilized to be optimized. As an example, in video 1, several different video sequences 104 may be identified from the video where different types of action are occurring and corresponding camera angles are determined to be optimized or better positioned compared to other camera angles.

[0038] For example, in video 1, if an action movie has a scene in which a race car crosses a finish line, and a camera angle shows the front bumper crossing the finish line, that camera angle will be preferred over a camera angle showing the race car from behind (e.g., a forward-facing, floating camera view). Specifically, a camera angle showing the front bumper of the race car crossing the finish line will be more interesting and provide more relevant information about the context and characteristics of the action occurring in the video at that particular time.

[0039] Continuing with this example, video sequence 104 (VS1) may represent a number of video frames selected to present action occurring within an action movie, including a video frame showing a race car crossing a finish line, with a camera angle focused on the side of the race car to show the bumper crossing the finish line. Similarly, within the same action movie, video 1, other video sequences, e.g., VS2, VS3, VSN, may be selected to represent video scenes in which camera angles are positioned and selected to optimize interactive views of the action occurring within the video. It should be understood that a video may have more or less video sequences identified as relevant or having optimized camera angles.

[0040] Similarly, if video N represents a recording of a sporting event, a particular video frame may represent an action occurring within the video for which the camera angle is optimized. For example, if the video is a soccer game and a corner kick is occurring within the video, the camera angle may be optimized to show the kicker's view. That is, a camera angle is considered optimal if it is presented as a view into the soccer game scene and has a view from the player taking the kick. As an example, if the camera angle is taken from above (e.g., showing a top view of a soccer field), the camera angle may not be able to show how the soccer ball travels on its trajectory towards the net.

[0041] The selection of each video sequence within the various videos may be performed in various process operations. One process operation may include receiving a selection from a viewer of the video. This selection may occur while the viewer is viewing the video and providing input to tag the video when the camera angle is viewed as optimal or optimized for the content / action being viewed. In one embodiment, the tag may include providing information regarding the content being viewed within the video sequence. For example, the viewer may identify that the action is a corner kick in a soccer game, which provides a scene identifier (ID). In one embodiment, the tag may be applied for a period of time. For example, the tag may be applied for a time slice of the video being viewed, such as from time 1:20 to time 1:23 (corresponding to 3 minutes).

[0042] During these three minutes, approximately 10,800 frames are tagged at 60 frames per second. In some cases, tagging may take less or more time (corresponding to fewer or more tags for a frame). In other embodiments, the tag may be associated with a predefined number of frames. For example, if a viewer tags a video to identify a video sequence that has a camera angle optimized for a particular action scene, the tagging may associate a set number of frames before and after the tag. The number of frames is programmable and may vary depending on the type of video content. For example, a sports video may require more frames (e.g., 400 frames) to be captured for the video sequence, while a particular video game scene may require fewer frames (e.g., 200 frames).

[0043] In still other embodiments, tags may be associated with a shorter number of frames, for example, 5-20 video frames. In some cases, analysis of a tagged video sequence of frames may be limited to one or more frames of the sequence. For example, if analysis of a single frame identifies an action occurring in the frame and a camera angle, additional frames need not be processed. In some embodiments, more frames are processed until a camera angle is identified for the particular action sequence being tagged.

[0044] In another embodiment, the video sequence selection process may be performed automatically using data obtained from a database with user-generated data (UGD). As an example, certain videos may be viewed by many people on social media sites and may be tagged by viewers. Viewer tags may include thumbs up, comments, edits, emojis, etc. This UGD information may be processed to infer that certain time frames within a video are important or popular. In one embodiment, information processing from UGD may be used to automatically tag portions of a video for analysis of the video sequence.

[0045] In one embodiment, the resulting video sequences may be further analyzed to determine the camera angles utilized and the content depicted in those particular images. As with the embodiment in which the user specifically tags videos with optimized camera angles, the automatic tagging process may also examine a different number of video frames. Depending on the type of video, more or fewer frames may be analyzed to determine the camera angles utilized.

[0046] FIG. 2 illustrates an example dataset 200 that identifies video sequences tagged with different types of videos, according to one embodiment. As described above, video sequences (VS) may be tagged utilizing a manual user tagging process or an automatic tagging process using UGD. The dataset 200 illustrates that the video sequences may be defined by various video types 122. For example, the video types 122 may include action movies, racing games, soccer games, interactive games, games of chance, live gaming events, etc. The tagged video sequences may be associated with a scene identifier (ID).

[0047] The scene identifiers may be descriptive terms provided by a person tagging the video. In another embodiment, the scene identifiers may be descriptive terms generated by a computer based on metadata captured from the source video. In another embodiment, the scene identifiers may be generated using machine learning based on activities identified in the video frames. The dataset 200 further includes identified camera angles 126. The camera angles 126 are identified with a descriptive string, for example, overhead, site-left, kicker view, or other descriptive string. In another embodiment, the camera angles 126 may be described as three-dimensional coordinates that capture the scene provided in the video. The three-dimensional coordinates may identify a position (e.g., X, Y, Z) and an orientation (e.g., roll, pitch, yaw). These three-dimensional coordinates identify the camera angle associated with the tagged scene identifier.

[0048] In one embodiment, the three-dimensional coordinates are generated by a processor that approximates the three-dimensional coordinates based on the viewing angle and field of view provided to the video scene. In other embodiments, the three-dimensional coordinates may be provided by a user tagging the video. In some embodiments, the three-dimensional coordinates may be provided as a combination of computer generation and manual input to define the camera angle 126. The dataset 200 may also include frame identifiers (IDs) 128. The frame identifiers identify the number of frames associated with each video sequence of a particular video. For example, for an action movie, the video sequence 104 may be identified as VS1, video 1. The video sequence VS1 may identify a particular time frame within the video, for example, frames between times 1:20-1:23, or a specifically identified numeric frame within the video.

[0049] In the dataset 200, the video type 122, scene identifier 124, camera angle 126, and frame ID 128 may be part of or may define the metadata 204 in the dataset 200. Also shown as part of the dataset 200 are video sequences 129 from a video 206. These video sequences may be separate files that represent portions of the tagged video. In another embodiment, the video 206 may be an index to an actual video that is stored in a separate file. Once the dataset 200 is created, it may be utilized as part of a training algorithm for a camera angle artificial intelligence (AI) model 350 that will be used to dynamically select camera angles during gameplay.

[0050] FIG. 3 illustrates an example of an AI model training engine 220 utilized to process videos 206 and metadata 204 in training a camera angle AI model 350, according to one embodiment. As illustrated, an input video 102 is processed to identify video sequences, as described with reference to FIGS. 1 and 2. A video sequence selection process 302 is utilized to identify video sequences within different videos. As described above, the video sequence selection process 302 may be performed manually by a viewer of the video tagging the video, or may be performed programmatically by a computer that identifies activities, actions, scenes, user feedback, popularity, interests, or other important features for auto-tagging one or more video frames within the input video 102. As described above, tagging the input video results in the identification of video frames of related video sequences, which are then associated with metadata 204.

[0051] Metadata 204 is shown to include video type 122, scene identifier 124, camera angle 126, and frame identifier 128. In some embodiments, more or less of this type of information may be included in metadata 204. For example, some metadata may be obtained directly from the video frames or from metadata embedded in the video frames associated with the video. For example, video type may be a field in a larger metadata file associated with the video from which such data may be captured. Depending on the type of input video provided, other data may also be captured from the source video. As shown, video 206 and metadata 204 are associated with video sequences 304a, 304b, . . . 304n. These captured video sequences are then processed by AI model training engine 222 to extract feature data usable by camera angle AI model 350. As shown, the video sequences are analyzed by video feature extractor 305.

[0052] Again, the input video 102 is selected to include different types of interactive content. The interactive content may be selected from any type of video, including videos recorded from video gameplay, movies, short films, video clips, multimedia videos, social media videos, etc. The input video 102 is processed to identify video sequences 104 that represent interactive content of a particular scene and associated camera angle. As described above, a video sequence selection process 302 may be utilized to select various video sequences 304a, 304b, ... 304n. The video sequences 304a, 304b, ... 304n are generally represented by the video sequences 104 of FIG. 1 and represent one or more video sequences selected from the different types of videos 206 and their associated metadata 204.

[0053] As noted above, the metadata 204 may include one or more of the different types of metadata content indicated in the metadata 204. In some embodiments, the metadata 204 may include other types of metadata information than those identified in the metadata 204. Generally speaking, the metadata 204 is data that describes or identifies information about the video 206. In one embodiment, the AI ​​model training engine 220 receives the video 206 as a video sequence, which is then processed by the video feature extractor 305. In one embodiment, the video feature extractor 305 examines the video content within a particular video frame or image to identify identifying characteristics such as objects, colors, characters, actions, changes in motion, changes in direction, object movement, and the like.

[0054] These extracted features may be examined by pixel data analysis and other learning algorithms to identify and characterize the pixel data, and also associate the pixel data with objects found in the images associated with the video frames. Once the video feature extractor 305 extracts the video feature data, the video feature data is processed by a video feature classifier or classifiers in operation 306. The classification of the identified features is used to identify and track features and their actions during relative changes in the examined video frames. The classified features are then processed by the camera angle AI model 350.

[0055] The camera angle AI model 350 processes the classified features to determine or approximate the camera angle utilized in each of the video frames or groups of video frames. Similarly, the metadata feature extractor 308 reads the metadata 204 to identify metadata feature data. The metadata classifier 310 then classifies those identified metadata features to be processed by the camera angle AI model 350. In some embodiments, the camera angle is identified in the metadata 204. In other embodiments, the camera angle is determined based on the view provided in the classified video features processed by the camera angle AI model 350.

[0056] In other embodiments, camera angle AI model 350 approximates the camera angles utilized in one or more video frames and associates those camera angles with the type of action occurring in the video frames. Over time, camera angle AI model 350 continually learns which types of camera angles best reflect the action occurring in the video frames. Over time, the more training occurs, the better camera angle AI model 350 becomes at identifying or predicting the optimal camera angles to utilize for a particular action scene found in a particular video sequence.

[0057] In the example shown in FIG. 3, once a video game 330 is selected for play, a user may be allowed to play the game at operation 340. As an example, the video game may be hosted by a video streaming service and a user may access the video streaming service to play the game in streaming mode. The video streaming service may utilize one or more servers to handle the execution, streaming, and associated compression of the video game. In one embodiment, one or more servers of the streaming service may also process the AI ​​model training engine 220 or access the camera angle AI model 350. In one embodiment, a camera angle may be accessed during game play 340 if it has been trained in the model 350. During game play 340, one or more video sequences are passed to the camera angle AI model 350.

[0058] In one embodiment, the video sequences are video frames that are the result of a user's interactive gameplay. These video sequences may be snapshots taken continuously, periodically, or programmatically as certain actions occur in the video game being played. While the video game is being played, the camera angle AI model 350 may be able to identify a particular camera angle that is more suited to the type of game activity occurring in the gameplay 340. Correspondingly, the dynamic camera angle adjustment 360 may process the output from the camera angle AI model 350. The dynamic camera angle adjustment 360 then utilizes the information received from the camera angle AI model 350 to provide input to the video game 330.

[0059] The input is in the form of adjustment instructions to adjust the game camera angle being provided to the user while playing the game in gameplay 340. For example, during gameplay 340, if the user is attacking a boss during a boss fight, camera angle AI model 350 may determine that the best or preferred camera angle is a side view focusing on the combat weapon. Based on this determination, the video game will dynamically change its utilized camera angle during execution of the video game by one or more servers such that frames provided to the game player during gameplay 340 convey a side view camera angle. This side view camera angle is different from the camera angle that would have been displayed to the user during gameplay; that is, different from the camera angle programmatically set by the video game developer. This dynamic adjustment of the camera angle utilizing camera angle AI model 350 thus enables the player to view the video game scenes from the camera angle that is determined to be the optimal camera angle for the type of interactivity occurring within the game.

[0060] 4 illustrates an embodiment in which one or more data centers 400 are utilized to execute one or more video games and utilize the camera angle AI model 350, according to one embodiment. An AI camera angle processor 404 is shown, which may be accessed by a game server 4024 executing the video game 330. As shown, the game server 402 may be one or more game servers implemented in a particular data center and is executing the video game 330. For purposes of illustration, a game engine 408 is used to execute the video game 330, and the game engine may be integrated with the camera angle logic 410. The camera angle logic 410 may be integrated with the game engine 408 or may be provided as an add-on when executed by the game server 402.

[0061] It should be appreciated that the camera angle logic 410 may also be executed on a local client, such as a game console or personal computer. In this embodiment, a snapshot extractor 412 is utilized in conjunction with the game engine 408, which may be executed by the game server 402 while the video game 330 is running. In one embodiment, the snapshot extractor 412 is a process that selects video sequences from the currently running game. The video sequences 416 derived from the snapshots 414 may be generated periodically during game play. In another embodiment, the snapshots 414 may be generated over specific points in the game that indicate high activity or when specific game actions are occurring.

[0062] For example, if there are slow activity scenes during the game, such as a character walking down a street, or a character driving a car, or a character standing waiting for the next action adventure, or a user deciding on the next move, snapshots 414 are generated less frequently or are withheld until an action scene occurs. In some embodiments, snapshots 414 may be generated just before an action scene is about to occur, based on processing logic determined from the game engine, or based on learning (or identifying) the type of action that will occur in a particular scene. Generally speaking, snapshots should be generated frequently enough that camera angle AI model 350 can examine these video sequences to determine whether the camera angle should be switched or changed to a different camera angle.

[0063] As described above, camera angle AI model 350 may determine that a different camera angle should be used for a particular type of sequence or characteristic of action occurring within the video game, after which dynamic camera angle adjustment 360 may generate adjustment instructions. The adjustment instructions are sent to camera angle logic 410, which is used by game engine 408 to generate new camera angles and views into the generated scenes of the running video game. In the illustrated example, a user 456 is playing video game 330 in streaming mode utilizing a controller 458, and a video scene is being displayed on display 454.

[0064] In this embodiment, the video scene displayed on the display 454 may be updated with respect to the camera angle utilized for the video game scene based on the game camera angle changes provided by execution of the camera angle logic 410. Again, the game camera angle changes are performed dynamically based on what is happening in the game play 452 and a prediction of what a better camera angle would be that will be processed by the camera angle AI model 350.

[0065] FIG. 5A illustrates an example of one of the data centers 400, according to one embodiment. In this example, the data center may include one or more databases for storing video games 330a-330n. These games are accessible to one or more users in a streaming format over the Internet 450. It should be understood that the video games 330 may be full video games, mini-games, shared slices of games, or interactive programs. The data center 400 may also include one or more game servers 402a-402n that are entrusted with running one or more video games depending on load requirements.

[0066] Additionally, the data center may also include a streaming server 502 that works in conjunction with the game server 402 to provide streaming services to various clients 506 of the user 456. In one embodiment, the streaming server 502 may perform encoding / compression and packetization of the video stream for efficient presentation to the client device 506. As shown, the data center may also execute one or more AI camera angle processors 404. The AI ​​angle processors 404 may be utilized by the one or more game servers 402 to dynamically adjust the game camera view provided to the user 456. As shown in FIG. 5A, the user 456 is playing a video game using a controller 458. The user 456 is watching the game on the display 454 while playing a boss fight scene of the video game. Initially, the AI ​​camera angle processor 404 determines that a side view of the fight scene is optimal at time 0.

[0067] At another time t1, the AI ​​camera angle processor 404 will determine that a first person perspective will provide the most interesting or dynamic view into the scene based on the current action occurring within the video game and the characteristics of the action. Then, at another time tn, the AI ​​camera angle processor 404 will determine that a top view is optimal for showing the activity occurring between the user and the boss during the boss fight. This illustration shows that the dynamic change of camera angle occurs seamlessly and is used to intelligently provide the best scene for the video game recommended by the AI ​​camera angle processor 404. To the user 456, the viewing of the video game will be seamless and it will not be obvious that the camera angle has been dynamically changed from what was originally programmed by the game designer. However, the view into the video game activity will be provided in a more interactive and context-specific manner based on the optimal view learned for the type of scene displayed or encountered during video game play.

[0068] 5B illustrates an example of switching between different camera views. As shown, at time t0, a side view is selected for the game camera view. At time t1, the game camera view is selected to a first person view (i.e., a view seen by the character player), and at time tn, the game camera view switches to a top down game camera view. This dynamic switching of camera angles for the video game occurs seamlessly, and the selection is made based on the AI ​​camera angle processor 404 having learned during training what the optimal type of camera angle is for a particular type of scene occurring within the video game.

[0069] As noted above, the selection of optimal camera angles may be based not only on research of past game videos, but also on research from action movies, live sporting events, video clips, social media clips, social media posts, user submitted videos, annotated videos, animated videos, etc. Thus, training of the camera angle AI model 350 benefits from different types of learned camera angles that are optimal for different types of content that a video game user may encounter when interacting within interactive content.

[0070] In one embodiment, the video game is executed locally on a game console, a personal computer, or executed on a server. In some cases, the video game is executed by one or more servers in a data center. When the video game is executed, some instances of the video game may be simulations of the video game. For example, the video game may be executed by an environment or server that generates a simulation of the video game. A simulation, in some embodiments, is an instance of the video game. In other embodiments, the simulation may be generated by an emulator. In either case, when the video game is represented as a simulation, the simulation may be executed to render interactive content that may be interactively streamed, executed, and / or controlled by user input.

[0071] 6 illustrates a general embodiment of a method that may be utilized to dynamically select camera angles during gameplay based on learning optimal camera angles for particular characteristics of a scene presented in an interactive game, according to one embodiment. At operation 602, a data set of multiple video sequences is received that identifies characteristics of action occurring within a video game. Metadata associated with these video sequences may also be processed, and may include the camera angles used in each of the multiple video sequences.

[0072] In other embodiments, the metadata does not include the camera angle used, and the camera angle can be determined using machine learning when examining the action occurring in the video sequence and the view provided for the scene. In one embodiment, the dataset includes a number of video sequences selected from among many types of videos captured in the dataset. The transmitted data may include videos from video game recordings, movies, multimedia content, social media videos, etc. The video sequences that are part of the dataset are sequences that show the optimal camera angles for viewing a particular type of interactive content, or camera angles that have been determined to be the best or most popular camera angles.

[0073] As noted above, the camera angles may be manually selected by viewers of the video who tag particular scenes as containing the optimal or best camera angles based on subjective selection. In another embodiment, the camera angles may be automatically selected using a program that identifies the most popular camera angles for a particular type of scene or interactivity.

[0074] In operation 604, an artificial intelligence (AI) model is generated from the dataset based on a training process. The training process uses as input data from a dataset that includes video sequences and associated camera angles that have been determined to be optimal for the type of content and characteristics of the content. Once the AI ​​model is trained, the final model can be dynamically utilized during gameplay to help automatically select different camera angles that are optimal for the particular type of content that a gamer is experiencing.

[0075] In operation 606, the AI ​​model may be accessed by a server or process during gameplay of the video game. The AI ​​model processes snapshots of a video sequence generated during gameplay of the video game for analysis. The snapshots include one or more frames that are analyzed to determine interactivity, content characteristics, scene interactivity, scene characteristics, and other features that may be extracted, identified, classified, and utilized to identify interactive content characteristics. During gameplay, in operation 608, the game camera angle may be dynamically adjusted. This dynamic adjustment may be in response to adjustment instructions derived from the AI ​​model.

[0076] In one embodiment, dynamic adjustments may be made periodically during gameplay, during specific scenes of high activity or interactivity, or programmatically based on specific scenes within the video game that require more accurate camera view and camera angle selection. To the user, the adjustments to the camera angle are seamless and unnoticeable, except to accommodate the improved camera angle for the video gameplay scene. The adjustments to the camera angle correspond to the introduction of a different camera angle into the video game than was programmed by the game developer.

[0077] 7 illustrates components of an exemplary device 700 that can be used to implement aspects of various embodiments of the present disclosure. The block diagram illustrates device 700, which can incorporate or be a personal computer, video game console, personal digital assistant, server, or other digital device suitable for implementing embodiments of the present disclosure. Device 700 includes a central processing unit (CPU) 702 for executing software applications and optionally an operating system. CPU 702 may be comprised of one or more homogeneous or heterogeneous processing cores.

[0078] For example, CPU 702 is one or more general-purpose microprocessors having one or more processing cores. Further embodiments can be implemented using one or more CPUs having a microprocessor architecture that is particularly adapted for highly parallel and computationally intensive applications, such as processing operations of interpreting queries, identifying contextually relevant resources, and immediately implementing and rendering the contextually relevant resources within a video game. Device 700 may be localized to a player playing a game segment (e.g., a game console), or remote from the player (e.g., a back-end server processor), or one of many servers using virtualization in a game cloud system for remote streaming of game play to clients.

[0079] Memory 704 stores applications and data used by CPU 702. Storage 706 provides non-volatile storage and other computer-readable media for applications and data and may include fixed disk drives, removable disk drives, flash memory devices, and CD-ROM, DVD-ROM, Blu-ray, HD-DVD, or other optical storage devices, as well as signal transmission and storage media. User input devices 708 communicate user input from one or more users to device 700, and examples of user input devices 708 may include a keyboard, a mouse, a joystick, a touchpad, a touch screen, a still or video recorder / camera, a tracking device that recognizes gestures, and / or a microphone.

[0080] The network interface 714 enables the device 700 to communicate with other computer systems over an electronic communications network, which may include wired or wireless communications over a local area network or a wide area network such as the Internet. The audio processor 712 is adapted to generate analog or digital audio output from instructions and / or data provided by the CPU 702, memory 704, and / or storage 706. The components of the device 700, including the CPU 702, memory 704, data storage 706, user input device 708, network interface 710, and audio processor 712, are connected via one or more data buses 722.

[0081] A graphics subsystem 720 is further coupled to the data bus 722 and the components of the device 700. The graphics subsystem 720 includes a graphics processing unit (GPU) 716 and a graphics memory 718. The graphics memory 718 includes a display memory (e.g., a frame buffer) used to store pixel data for each pixel of an output image. The graphics memory 718 may be integrated in the same device as the GPU 708, may be coupled as a separate device from the GPU 716, and / or may be incorporated within the memory 704. The pixel data may be provided to the graphics memory 718 directly from the CPU 702. Alternatively, the CPU 702 provides data and / or instructions defining a desired output image to the GPU 716, from which the GPU 716 generates pixel data for one or more output images. The data and / or instructions defining the desired output image may be stored in the memory 704 and / or the graphics memory 718. In an embodiment, GPU 716 includes 3D rendering capabilities for generating pixel data for output images from instructions and data defining the geometry, lighting, shading, texturing, motion, and / or camera parameters of a scene. GPU 716 may further include one or more programmable execution units capable of executing shader programs.

[0082] Graphics subsystem 714 periodically outputs pixel data of an image from graphics memory 718 for display on display device 710. Display device 710 may be any device capable of displaying visual information in response to signals from device 700, including CRT, LCD, plasma, and OLED displays. Device 700 may provide analog or digital signals to display device 710, for example.

[0083] It should be noted that access services distributed over a wide geography, such as providing access to games in the current embodiment, often use cloud computing. Cloud computing is a computing paradigm in which dynamically scalable, often virtualized resources are provided as a service over the Internet. Users do not need to be experts in the technical infrastructure of the "cloud" that supports them. Cloud computing can be categorized into different services such as Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). Cloud computing services often provide common applications, such as video games, online, accessed through a web browser, but the software and data are stored on servers in the cloud. The term cloud is used as a metaphor for the Internet based on how the Internet is depicted in computer network diagrams, and is an abstract concept that hides a complex infrastructure.

[0084] A game server may be used in some embodiments to perform the operation of a persistent information platform for video game players. Most video games played over the Internet operate through a connection to a game server. Typically, the game uses a dedicated server application that collects data from the player and distributes the collected data to other players. In other embodiments, the video game may be executed by a distributed game engine. In these embodiments, the distributed game engine may be executed on multiple processing entities (PEs), such that each PE executes a functional segment of the given game engine on which the video game is executed. Each processing entity is viewed by the game engine as simply a computational node.

[0085] A game engine typically performs a functionally diverse set of operations to execute a video game application together with additional services that are experienced by a user. For example, the game engine implements game logic and performs game calculations, physics, geometry transformations, rendering, lighting, shading, audio, and additional in-game or game-related services. The additional services may include, for example, messaging, social utilities, audio communication, gameplay playback functions, help functions, etc. The game engine may run on an operating system virtualized by a hypervisor on a particular server, while in other embodiments the game engine itself may be distributed across multiple processing entities, each of which may reside on a different server unit in a data center.

[0086] According to this embodiment, for execution, each processing entity may be a server unit, a virtual machine, or a container, depending on the needs of each game engine segment. For example, if a game engine segment is responsible for camera transformation, since it will be performing a lot of relatively simple mathematical operations (e.g., matrix transformations), that particular game engine segment may be provisioned with a virtual machine associated with a graphics processing unit (GPU). Other game engine segments, requiring fewer but more complex operations, may be provisioned with processing entities associated with one or more higher-powered central processing units (CPUs).

[0087] By distributing the game engine, the game engine has elastic computational characteristics that are not bound by the capabilities of a physical server unit. Instead, the game engine is provisioned with more or fewer computation nodes as needed to meet the demands of the video game. From the perspective of the video game and the video game player, a game engine that is distributed across multiple computation nodes is indistinguishable from a non-distributed game engine running on a single processing entity, since a game engine manager or supervisor distributes the workload and seamlessly integrates the results to provide the video game output component to the end user.

[0088] Users access remote services through a client device that includes at least a CPU, a display, and I / O. The client device may be a PC, a cell phone, a netbook, a PDA, etc. In one embodiment, a network running on the game server recognizes the type of device used by the client and adjusts the communication method employed. In another case, the client device accesses the application on the game server over the Internet using a standard communication method such as HTML.

[0089] It should be understood that a given video game or game application may be developed for a particular platform and a particular associated controller device. However, when making such games available through a game cloud system as presented herein, users may access the video game with different controller devices. For example, a game may be developed for a game console and its associated controller, but a user may access a cloud-based version of the game from a personal computer utilizing a keyboard and mouse. In such a scenario, the input parameter configuration may define a mapping from inputs that can be generated by the user's available controller device (in this case, a keyboard and mouse) to inputs that are acceptable for execution of the video game.

[0090] In another example, a user may access the cloud gaming system via a tablet computing device, a touchscreen smartphone, or other touchscreen driven device. In this case, the client device and the controller device are integrated into the same device, with input provided as detected touchscreen inputs / gestures. For such devices, the input parameter configuration may define specific touchscreen inputs that correspond to game inputs for the video game. For example, buttons, directional pads, or other types of input elements may be displayed or overlaid during the execution of the video game to indicate locations on the touchscreen that the user may touch to generate game inputs. Gestures such as swipes in specific orientations, or specific touch motions may also be detected as game inputs. In one embodiment, to familiarize the user with control operations on the touchscreen, a tutorial may be provided to the user showing how to input gameplay via the touchscreen, for example, before beginning gameplay of the video game.

[0091] In some embodiments, the client device serves as a connection point for the controller device. That is, the controller device communicates with the client device via a wireless or wired connection and transmits inputs from the controller device to the client device. The client device may then process these inputs and then transmit the input data to a cloud gaming server over a network (e.g., a network accessed through a local network device such as a router). However, in other embodiments, the controller itself may be a networked device that has the ability to communicate inputs directly to the cloud gaming server over a network, without the need for such inputs to be communicated through the client device first. For example, the controller may connect to a local network device (such as the router mentioned above) and transmit data to and receive data from the cloud gaming server. Thus, while the client device may still need to receive video output from a cloud-based video game and render it on a local display, it is possible to reduce input latency by allowing the controller to transmit inputs directly over the network to the cloud gaming server, bypassing the client device.

[0092] In one embodiment, the networked controller and client devices can be configured to transmit certain types of inputs directly from the controller to the cloud gaming server and other types of inputs via the client device. For example, inputs whose detection does not depend on any additional hardware or processing away from the controller itself can be transmitted directly from the controller to the cloud gaming server over the network, bypassing the client device. Such inputs can include button inputs, joystick inputs, embedded motion detection inputs (e.g., accelerometers, magnetometers, gyroscopes), and the like. However, inputs that utilize additional hardware or require processing by the client device can be transmitted by the client device to the cloud gaming server. These can include video or audio captured from the game environment that can be processed by the client device prior to transmission to the cloud gaming server. In addition, inputs from the controller's motion detection hardware can be processed by the client device in conjunction with the captured video to detect the position and movement of the controller, which is then communicated by the client device to the cloud gaming server. It should be understood that the controller device according to various embodiments can also receive data (e.g., feedback data) from the client device or directly from the cloud gaming server.

[0093] It should be understood that the various embodiments defined herein may be combined or assembled into specific implementations using various features disclosed herein. Thus, the examples provided are only some of the possible examples and are not intended to limit the various embodiments that may be combined to define many more embodiments. In some examples, an embodiment may include fewer elements without departing from the spirit of the disclosed or equivalent embodiments.

[0094] Embodiments of the present disclosure may be practiced with a variety of computer system configurations including handheld devices, microprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Embodiments of the present disclosure may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a wire-based or wireless network.

[0095] Although the method operations have been described in a particular order, it should be understood that other housekeeping operations may be performed between operations, or operations may be coordinated to occur at slightly different times, or operations may be distributed within the system to allow processing operations to occur at various processing-related intervals, so long as the processing of telemetry data and game state data to generate modified game state is performed in the desired manner.

[0096] One or more embodiments may also be fabricated as computer readable code on a computer readable medium. A computer readable medium is any data storage device that can store data and can then be read by a computer system. Examples of computer readable media include hard drives, network attached storage (NAS), read only memory, random access memory, CD-ROM, CD-R, CD-RW, magnetic tape, and other optical and non-optical data storage devices. The computer readable medium may include computer readable tangible media distributed across networked computer systems such that the computer readable code is stored and executed in a distributed fashion.

[0097] Although the foregoing embodiments have been described in some detail for clarity of understanding, it will be apparent that certain changes and modifications can be practiced within the scope of the appended claims. Thus, the present embodiments should be considered as illustrative and not restrictive, and the present embodiments should not be limited to the details set forth herein, but may be modified within the scope of the appended claims and their equivalents.

Claims

1. 1. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences, the metadata including the camera angle used in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, Generating the AI ​​model includes: extracting video feature data from each of the plurality of video sequences; extracting metadata feature data associated with each of the plurality of video sequences; classifying the video feature data and the metadata feature data for processing by the AI ​​model, the AI ​​model being configured to process the video feature data and the metadata feature data to label characteristics of the actions occurring in each of the plurality of video sequences; Including, A method wherein generating the AI ​​model occurs during a training process.

2. The method of claim 1 , wherein said adjusting of said game camera angle modifies a developer camera angle of said game.

3. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences, the metadata including the camera angle used in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, The AI ​​model is configured to process snapshots of the video sequence generated during play of the game, the processing of the snapshots of the video sequence comprising: examining image content of one or more video sequences produced during play of the game to produce video characteristic data; examining metadata associated with the examined image content to produce metadata characteristic data; Including, The method, wherein the video feature data and the metadata feature data are classified and used by the AI ​​model to identify the adjustment instructions for adjusting the game camera angle to present the video game scene during the gameplay.

4. 2. The method of claim 1 , wherein snapshots of the video are generated periodically during the gameplay to trigger processing of the adjustment instructions to adjust the game camera angle to present the video game scene during the gameplay.

5. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences, the metadata including the camera angle used in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, A method, wherein the video snapshots are generated in response to metadata produced during game play that indicates that an ongoing action scene should include an adjustment to the game camera angle.

6. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences, the metadata including the camera angle used in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, A method wherein the video snapshots are generated in response to metadata produced during game play that indicates that an upcoming action scene should include an adjustment to the game camera angle.

7. 2. The method of claim 1, wherein the tagging data is obtained from a database of user-generated data (UGD), the UGD including frame sequences that include the camera angles marked as preferred, and the marking as preferred is used to select the plurality of video sequences to use as training data for the AI ​​model.

8. 2. The method of claim 1 , wherein the tagging data is generated automatically based on detected viewing popularity for frame sequences that include the camera angles, and the viewing popularity is used to select the plurality of video sequences to use as training data for the AI ​​model.

9. the video sequences in the dataset are sourced from one or more of a plurality of movies, a plurality of video game recordings, a recording of a live sporting event, or a recording including one or more action scenes, and the plurality of video sequences are used to train the AI ​​model; The method of claim 1 , wherein the AI ​​model is used during gameplay to adjust the game camera angle within the video game during gameplay.

10. The plurality of video sequences is used to train the AI ​​model; The method of claim 1 , wherein the AI ​​model is used during gameplay to adjust the game camera angle within the video game during gameplay.

11. The method of claim 10 , wherein the game camera angle provides an alternative view of the video game scene than a view programmed by a developer of the video game.

12. 1. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, Generating the AI ​​model includes: extracting video feature data from each of the plurality of video sequences; extracting metadata feature data associated with each of the plurality of video sequences; classifying the video feature data and the metadata feature data for processing by the AI ​​model, the AI ​​model being configured to process the video feature data and the metadata feature data to label characteristics of the actions occurring in each of the plurality of video sequences; Including, A method wherein generating the AI ​​model occurs during a training process.

13. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, The AI ​​model is configured to process snapshots of the video sequence generated during play of the game, the processing of the snapshots of the video sequence comprising: examining image content of one or more video sequences produced during play of the game to produce video characteristic data; examining metadata associated with the examined image content to produce metadata characteristic data; Including, The method, wherein the video feature data and the metadata feature data are classified and used by the AI ​​model to identify the adjustment instructions for adjusting the game camera angle to present the video game scene during the gameplay.

14. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, A method, wherein the video snapshots are generated in response to metadata produced during game play that indicates that an ongoing action scene should include an adjustment to the game camera angle.

15. A method for dynamically adjusting a camera angle in a video game, comprising: receiving a data set of a plurality of video sequences associated with tagging data in metadata identifying characteristics of action occurring in each of the plurality of video sequences; generating an artificial intelligence (AI) model from the dataset; and accessing the AI ​​model during gameplay of the video game, said accessing comprising providing snapshots of video sequences generated during the gameplay of the video game; dynamically adjusting a game camera angle during gameplay, said adjusting being responsive to adjustment instructions derived from the AI ​​model; and Including, A method wherein the video snapshots are generated in response to metadata produced during game play that indicates that an upcoming action scene should include an adjustment to the game camera angle.

Citation Information

Patent Citations

  • Information processor, information processing system, information processing program and information processing method

    JP2015013103A

  • Methods and systems for computer video game streaming, highlight, and replay

    US20170157512A1

  • Spectating Virtual (VR) Environments Associated With VR User Interactivity

    US20170269685A1